Configuration of maxRedirects support

[bushev]

Which package is the feature request for? If unsure which one to select, leave blank

@crawlee/cheerio (CheerioCrawler)

Feature

It would be created if the CheerioCrawler will support option maxRedirects=N for the configuration of maximum allowed redirects.

Motivation

I faced the issue when I needed more than 10 redirects. (12 in my particular case).

ERROR CheerioCrawler: Request failed and reached maximum retries. MaxRedirectsError: Redirected 10 times. Aborting.

Ideal solution or implementation, and any additional constraints

Option maxRedirects=N for CheerioCrawler

Alternative solutions or implementations

No response

Other context

No response

[B4nan]

FYI this redirect limit is coming from got (or better say got-scraping), it's not a crawlee limit. There already is a maxRedirects option in got, I believe you can set that via preNavigationHooks, where you get the options object (that gets passed to the got library) in the last parameter of the hook.