crawl incorrect with given requestQueue

[zshnb]

Hello I'm using crawlee to crawl web pages. The scene in question is I try to crawl same url multi times, I have tried add uniqueKey like await crawler.run([{url: config.url, uniqueKey: uuid}]); and

await enqueueLinks({
  globs: typeof config.match === "string" ? [config.match] : config.match,
  transformRequestFunction: (request) => {
    request.uniqueKey = `${request.url}:${uuid}`;
    return request;
  }
});

but it doesn't work. Then I try use requestQueue

const requestQueue = await RequestQueue.open(uuid)
await enqueueLinks({
  globs: typeof config.match === "string" ? [config.match] : config.match,
  transformRequestFunction: (request) => {
    request.uniqueKey = `${request.url}:${uuid}`;
    return request;
  },
  requestQueue
});

with maxPageCrawl: 5 config, I got output

INFO  PlaywrightCrawler: Starting the crawler.
INFO  PlaywrightCrawler: Crawling: Page 1 / 5 - URL: https://www.prisma.io/docs...
INFO  PlaywrightCrawler: All requests from the queue have been processed, the crawler will shut down.
INFO  PlaywrightCrawler: Final request statistics: {"requestsFinished":1,"requestsFailed":0,"retryHistogram":[1],"requestAvgFailedDurationMillis":null,"requestAvgFinishedDurationMillis":32848,"requestsFinishedPerMinute":2,"requestsFailedPerMinute":0,"requestTotalDurationMillis":32848,"requestsTotal":1,"crawlerRuntimeMillis":33028}
INFO  PlaywrightCrawler: Finished! Total 1 requests: 1 succeeded, 0 failed. {"terminal":true}
Found 1 files to combine...
data/6c9d4f5a-5629-49fa-b299-ca27de45391f-1.json
Wrote 1 items to data/6c9d4f5a-5629-49fa-b299-ca27de45391f-1.json

what's the problem about my code? and how can I implement this scene?
Thanks

[B4nan]

uniqueKey works fine, your code looks like you are reusing a single UUID generated upfront, you need to generate a new one for each request - so this needs to happen inside the transformRequestFunction. Also depends on what you are after, if it's about respecting URL fragments (the thing after #), there is an option for that too, that would be better than adding random strings to unique key.

const requestQueue = await RequestQueue.open(uuid)

This is a wrong take, you are creating a named queue (with a random name), and those are not removed automatically on start. You just generate data that you will need to clean up manually. Using the default queue is fine here, your problem is deduplication on the request level, so the uniqueKey approach is the one to take.

[zshnb]

sorry I add uniqueKey in startUrl, but seems still doesn't work, I committed code, you can check my repo.

[B4nan]

I guess there is still some global state shared between both, probably the crawler statistics.

You should really go with the approach I showed above, having two configs (or better say having new config for each invocation/crawler instance), that's the proper way to run things multiple time, not this.

[zshnb]

yes, config is work.
But why uniqueKey in startUrl still not work? is it a bug in crawlee?

[B4nan]

The problem is not unique key, its the crawler state, which is shared too - that's where we keep the number of processed results.

[zshnb]

ok got it, thanks for your answer.