Crawler crashing due to a RequestQueue somehow being deleted and not found?

[dbilyk]

Hello! I have a crawler which needs to be invoked and cleanly completed multiple times. I'm using {purgeOnStart:true} so I would expect the crawler to start everything 'fresh'. And it works for 20-50 times that the crawler is called, but eventually starts throwing this error: AutoscaledPool: isTaskReadyFunction failed. Is there a way to either A) prevent the crawler from deleting any requestQueues it's using or B) to prevent it from referencing some older request queue or C) catch this error and ignore this missing queue?

PlaywrightCrawler:AutoscaledPool: isTaskReadyFunction failed
Error: ENOENT: no such file or directory, open '/Users/admin/CodeProjects/IGScraper/storage/request_queues/default/dDL0pGj5X9jotg9.json'

[B4nan]

Are you on latest version? Namely v3.3.3 could help with this issue.

Few notes:

• purgeOnStart happens once per app context, so basically only once when you start the node.js app, it is not about purging on each crawler.run.
• only since v3.3.2 it is supported to call crawler.run multiple times (feat: allow running single crawler instance multiple times #1844), and the run method itself now clears the default RQ and resets the crawler internal state (stored in a KVS item), this is configurable CrawlerRunOptions.purgeRequestQueue (the internal state is always reset)
• since v3.3.2 you can also set the RQ to use in-memory storage only, so there won't be any JSON files at all (feat: support alternate storage clients when opening storages #1901)
• maybe you could also use keepAlive option, that way you call run only once, and the crawler will wait for new requests even if the queue gets emptied

[dbilyk]

Got it, thanks for the advice, I'm not on the latest version so I'll give that a try!