Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add "exclude" property to enqueueLinksByClickingElements like "enqueueLinks" #2298

Open
AraCoders opened this issue Jan 22, 2024 · 0 comments
Labels
feature Issues that represent new features or improvements to existing features. t-tooling Issues with this label are in the ownership of the tooling team.

Comments

@AraCoders
Copy link

Which package is the feature request for? If unsure which one to select, leave blank

None

Feature

PlaywrightCrawler enqueueLinks has two properties: "regexps" and "exclude". however only "regexps" is present for "enqueueLinksByClickingElements".

Motivation

consistency between enqueueLinks and enqueueLinksByClickingElements. because i have a scope to crawl using "regexps" but frequently i need to filter some urls (add them to a blacklist). so for enqueueLinks it's easy. but for enqueueLinksByClickingElements i had to provide 2 regexp: one for the normal scope of the crawler, the other is a negative lookbhind regex to filter some of the urls, however i think it's still not working as expected, because some urls get filtered from the first regex, but still make it to enqueued requests because of the second negative lookbehind regex.

Ideal solution or implementation, and any additional constraints

add the property exclude to "enqueueLinksByClickingElements". and also make it clear in the docs wether the list of regex supplied to "regexps" property should work in a "and" or "or" relationship. same thing for relationship between "regexps" and "exclude" when they are both supplied.

Alternative solutions or implementations

No response

Other context

No response

@AraCoders AraCoders added the feature Issues that represent new features or improvements to existing features. label Jan 22, 2024
@B4nan B4nan added the t-tooling Issues with this label are in the ownership of the tooling team. label Jan 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature Issues that represent new features or improvements to existing features. t-tooling Issues with this label are in the ownership of the tooling team.
Projects
None yet
Development

No branches or pull requests

2 participants