scrape page count is exceed maxRequestsPerCrawl too much

[zshnb]

Which package is this bug report for? If unsure which one to select, leave blank

@crawlee/playwright (PlaywrightCrawler)

Issue description

the actual scrape page count is more exceed than expect, even 2x(13:5)

Code sample

sample code in guide

Package version

@builder.io/gpt-crawler@1.2.0 /Users/zheng/Workbench/github/gpt-crawler └── crawlee@3.7.0 -> ./node_modules/.pnpm/crawlee@3.7.0_playwright@1.40.1/node_modules/crawlee

Node.js version

v18

Operating system

MacOS

Apify platform

• Tick me if you encountered this issue on the Apify platform

I have tested this on the next release

No response

Other context

No response

[B4nan]

This happens with a higher initial concurrency and is expected behavior, we never ignore requests that were started (in fact it's usually not possible to cancel them in the first place), as you can see yourself in the logs (the message starting with earlier... is about this).

This fact is described in several places in the docs, e.g. https://crawlee.dev/docs/introduction/adding-urls#limit-your-crawls-with-maxrequestspercrawl

sample code in guide

This is not very helpful, reproductions need to be complete and specific.

[zshnb]

you mean even in log display actual count is more than config, crawlee can't cancel exceed requests which are already in request queue?

[B4nan]

I don't understand what you are trying to say. The problem is not about requests in queue, it's about requests that already started processing, the request handler for them was already started, that's what cannot be canceled.

The problem is parallelism, if you would limit the concurrency to 1, this could never happen, but if you allow parallel processing (which is the default), you can get to a state like this, when several requests started processing and only after that you reach the limit (as the limit is about what requests finished processing).