Crawl failed request after Crawling is finished

[yossi-s]

Hey,

My crawl logic is like that, I am using Playwright crawler to crawl a websites.
In the config I set the maxRequestRetries to 2.

When the crawl is finished there are some url's that wasn't crawled due to blocking or network issue, so lets say I was able to crawl 190 url's from 200 so I have 10 requests that failed.

There is an option to mark those requests as no handled and try to run them again after the crawl finished?

something like that:

const crawler = new PlaywrightCrawler({ maxRequestRetries:2 ...});
await crawler.run([startingUrl]);

if(failedUrlsSet.size() > 0){
//mark each request from failedUrlSet as not handled
//rerun the crawl again only with those failedUrls
}