Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide an addon for Broad Crawls #6331

Open
kmike opened this issue May 3, 2024 · 2 comments
Open

Provide an addon for Broad Crawls #6331

kmike opened this issue May 3, 2024 · 2 comments
Labels

Comments

@kmike
Copy link
Member

kmike commented May 3, 2024

There are common practices for broad crawls, explained here: https://docs.scrapy.org/en/latest/topics/broad-crawls.html. It involves modifying many settings. It seems we can provide a Scrapy addon to simplify all of that.

It can start as a third-party package, but I think Scrapy itself may also benefit from such addon.

@kmike kmike added the discuss label May 3, 2024
@wRAR
Copy link
Member

wRAR commented May 10, 2024

Are all of those settings safe to enable for all broad crawls?

@kmike
Copy link
Member Author

kmike commented May 11, 2024

Are all of those settings safe to enable for all broad crawls?

I'm not 100% sure. It seems we should review them. Also, we should ensure that the users can change the values we set in the addon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants