Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pyspider设置了隧道代理,在调试时生效,实际爬取时不生效 #975

Open
Acloud11 opened this issue Sep 16, 2021 · 0 comments

Comments

@Acloud11
Copy link

Acloud11 commented Sep 16, 2021

crawl_config = { 'proxy': 't13177827711279:24qefecw@tps158.kdlapi.com:15818', #'proxy':'t13038971957781:fqtmsm8i@tps187.kdlapi.com:15818' }
我配置配置了快代理的隧道代理。在调试时代理能够正常工作。
但是在爬虫run起来之后,代理却没有生效。
我判断代理没有生效的原因是:
1.快代理网站对我购买的隧道代理的监控中,代理使用次数没有增加
图中零星的代理使用,是在调试过程,尝试爬取页面时的代理正确生效。
2.在Active Tasks任务的详情内容中,fatch中,无代理的配置。
fetch { "save": { "key": "肯尼亚航空" } }

我尝试使用下述配置启动,同样无效

pyspider --phantomjs-proxy 用户名:密码@tps158.kdlapi.com:15818
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant