Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add waitForAllRequestsToBeAdded option to enqueueLinks #2318

Open
barjin opened this issue Feb 2, 2024 · 0 comments
Open

Add waitForAllRequestsToBeAdded option to enqueueLinks #2318

barjin opened this issue Feb 2, 2024 · 0 comments
Assignees
@vladfrangu
Labels
feature Issues that represent new features or improvements to existing features. t-c&c Team covering store and finance matters. t-console Issues with this label are in the ownership of the console team. t-tooling Issues with this label are in the ownership of the tooling team.

Comments

@barjin
Copy link
Contributor

barjin commented Feb 2, 2024

Which package is the feature request for? If unsure which one to select, leave blank

@crawlee/core

Feature

As discussed in this WCC PR, the enqueueLinks method (and the contexted version too) is missing the waitForAllRequestsToBeAdded option, which means that the method always resolves after enqueuing 1000 URLs, with no opt-in for waiting until the end.

cc @vladfrangu as discussed :)

Motivation

Some (high-concurrency?) use cases could profit from this.

Ideal solution or implementation, and any additional constraints

Should be quite straightforward to implement (just pass the new option to the RQ).

Alternative solutions or implementations

No response

Other context

No response
@barjin barjin added the feature Issues that represent new features or improvements to existing features. label Feb 2, 2024
@mtrunkat mtrunkat added t-tooling Issues with this label are in the ownership of the tooling team. t-console Issues with this label are in the ownership of the console team. t-c&c Team covering store and finance matters. labels Apr 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vladfrangu vladfrangu

Labels
feature Issues that represent new features or improvements to existing features. t-c&c Team covering store and finance matters. t-console Issues with this label are in the ownership of the console team. t-tooling Issues with this label are in the ownership of the tooling team.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mtrunkat @vladfrangu @barjin