-
Notifications
You must be signed in to change notification settings - Fork 10.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make brotlipy a hard Scrapy dependency #4698
Comments
Related: #4267 |
How about doing this in 2.7? |
Related: https://www.lapierrebikes.com/page-data/de-de/haendler/page-data.json sends |
It seems we can do https://github.com/python-hyper/brotlicffi#using-brotlicffi-in-projects for PyPy support, and for Windows I think https://docs.scrapy.org/en/latest/intro/install.html#windows already covers everything needed, although it would be great to have a Windows user confirm so. |
@kmike Any thoughts on this? If we are going ahead, we could mark this as a good first issue. |
It seems brotli is popular and maintained (e.g. there are wheels uploaded recently), even if last release was in 2020. That's probably fine to add it as a dependency. On the other hand, it's still a possible source of install errors in various environments, as it's a binary package. So, no strong opinion on that. |
google/brotli#944 suggests there will be a new release soon. But so far we decided not to make it a dep as it's a binary module. |
When we added support to Scrapy (#2535), we did so without making brotlipy a hard dependency, seemingly because there seem to be some potential for installation issues.
However, given that most web browsers support Brotli (
br
) compression out of the box, I think we should consider revisiting that. It would also slightly simplify our code.Related to #4697
The text was updated successfully, but these errors were encountered: