Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a AuthenticatedRoundRobinProxyHTTP method to authenticate proxies #578

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

wowinter13
Copy link
Contributor

This PR addresses the issue #343

Hey!
Right now colly doesn't have an ability to authenticate proxies out of the box. So I added this functionality:)
Also I decided not to modify proxy_switcher.go because it would break an example without having correct credentials.
(Also my deepest gratitude to @andystroz)

@wowinter13
Copy link
Contributor Author

@asciimoo Also we could rewrite RoundRobinProxySwitcher and make credentials to be nullable. Therefore we would be able to delete AuthenticatedRoundRobinProxySwitcher. (but imho for now that is better to have both methods)

@WGH-
Copy link
Collaborator

WGH- commented Jan 22, 2021 via email

@wowinter13
Copy link
Contributor Author

Hey @WGH- !
Most crawlers have the functionality to authenticate proxies. At least, Scrapy.
The situation is a little bit inconvenient when you have a built-in method to rotate proxies and don't have any methods to do basic auth for these proxies.

Also if you check the issue #343, you will see some demand for this feature.

@wowinter13 wowinter13 changed the title Add a AuthenticatedRoundRobinProxyHTTP method to authenticate proxiees Add a AuthenticatedRoundRobinProxyHTTP method to authenticate proxies Jan 22, 2021
@wowinter13
Copy link
Contributor Author

@asciimoo ?

@samirsss
Copy link

Any thoughts on this approach - i'd like this to be included as part of Gocolly too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants