PlaywrightCrawler - Possible to pause the crawl in case of 403 blocking

[yossi-s]

Hey, I have a question related to the PlaywrightCrawler.
Is it possible to pause the crawl in case I started to get 403 due to rate limit violation? wait for 2 min and back to crawl again?

Also the url's that was not succeed to crawled due to this blocking will back to the queue?
Thanks!

[harshmaur]

You can first define sessionPoolOptions in your crawler like this

then you can handle the 403 as a middleware in your routes.

[yossi-s]

Not sure I got it.
Do you have an example for that?

[yossi-s]

Is it possible if you can paste here an example how to create a middleware that catch the blocked requests, pause the crawler for some amount of time (lets say 2 min) and resume to work after that.