cannot crawl the same URL twice

[creative-ae]

Which package is this bug report for? If unsure which one to select, leave blank

@crawlee/puppeteer (PuppeteerCrawler)

Issue description

the crawler actually crawl the URL 1 time only,
when i try to crawl the same url again, i get the cached data.

Code sample

import { Dataset, PuppeteerCrawler } from "crawlee";
const crawl = async (url) => {
    const crawler = new PuppeteerCrawler({
        maxRequestsPerCrawl: 50,
        async requestHandler({ request, page, enqueueLinks, log }) {
            const title = await page.title();
            await Dataset.pushData({ title });
        },
        failedRequestHandler({ request, log }) {
            log.error(`Request ${request.url} failed too many times.`);
        },
    });
    await crawler.addRequests([url]);
    await crawler.run();
    console.log('Crawler finished.');
}
// the first time the functionis is called, the puppeteer crawler actually crawl and get the data from the url
/*
INFO  PuppeteerCrawler: Final request statistics: {"requestsFinished":1,"requestsFailed":0,"retryHistogram":[1],"requestAvgFailedDurationMillis":null,"requestAvgFinishedDurationMillis":7108,"requestsFinishedPerMinute":8,"requestsFailedPerMinute":0,"requestTotalDurationMillis":7108,"requestsTotal":1,"crawlerRuntimeMillis":7303}
*/
crawl('https://example.com')

/*
if the function is called again with the same URL, the crawler just respond with all URLS has been completed 
INFO  PuppeteerCrawler: Final request statistics: {"requestsFinished":0,"requestsFailed":0,"retryHistogram":[],"requestAvgFailedDurationMillis":null,"requestAvgFinishedDurationMillis":null,"requestsFinishedPerMinute":0,"requestsFailedPerMinute":0,"requestTotalDurationMillis":0,"requestsTotal":0,"crawlerRuntimeMillis":130}
*/
crawl('https://example.com')

Package version

3.5.4

Node.js version

18

Operating system

mac

Apify platform

• Tick me if you encountered this issue on the Apify platform

I have tested this on the next release

No response

Other context

No response

[adrianhorning08]

+1 for this. I think it's stored in memory and the memory needs to be purged, but I can't figure out how to do that :(

[adrianhorning08]

Found the answer: #2026

[creative-ae]

worked perfectly. Thanks

[B4nan]

There are several ways to go about this:

• have different RQ instances (as in not able to call two different crawler with same url #2026)
• enqueue with a random uniqueKey (as that is what is used for deduplication, and defaults to the URL basically)
• disable storage persistence and run things with different config instances (as in https://crawlee.dev/docs/deployment/aws-cheerio)

[creative-ae]

having different requestQueue instances (as in #2026) worked well.
disable storage persistence did not do the job.

[B4nan]

disable storage persistence did not do the job.

You need to specify the config instances to all the places that use storages, mainly the crawler options, but same applies for working with the RQ, KVS and datasets. The guide I linked is well tested and definitely works on its own - but we had to do some fixes to make it work correctly, so be sure to test things on latest crawlee version. If you provide some code, I'll be happy to look into why it does not work. Note that with that approach, it is crucial to have new config instance for each crawler instance.

To add a bit more context - if you don't specify the config, the default (global) one is used. Similarly, if you dont provide a RQ explicitly, the default one is used - so the very same instance for multiple crawler, as we cache this internally. And to make things work with a single RQ, the approach with uniqueKey is needed.