Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Scrapy Selenium #13

Open
KatBotkowska opened this issue May 25, 2020 · 0 comments
Open

Scrapy Selenium #13

KatBotkowska opened this issue May 25, 2020 · 0 comments

Comments

@KatBotkowska
Copy link

In my scrapy app, I use scrapy -selenium with geckodriver. When I try use command 'scrapy-deploy', I get an error:
file "/tmp/scrapy_job_it-1590437526-tezy3xue.egg/scrapy_job_it/spiders/bdcrawler.py", line 3, in \nModuleNotFoundError: No module named 'scrapy_selenium'\n"}
What can I do to deploy app with selenium?
All track for error is here:
Server response (200):
{"node_name": "cbb42d40-da02-4d65-bb04-ff436c5aacc1", "status": "error", "message": "/app/.heroku/python/lib/python3.6/site-packages/scrapy/utils/project.py:94: ScrapyDeprecationWarning: Use of environment variables prefixed with SCRAPY_ to override settings is deprecated. The following environment variables are currently defined: EGG_VERSION\n ScrapyDeprecationWarning\nTraceback (most recent call last):\n File "/app/.heroku/python/lib/python3.6/runpy.py", line 193, in _run_module_as_main\n "main", mod_spec)\n File "/app/.heroku/python/lib/python3.6/runpy.py", line 85, in _run_code\n exec(code, run_globals)\n File "/app/.heroku/python/lib/python3.6/site-packages/scrapyd/runner.py", line 40, in \n main()\n File "/app/.heroku/python/lib/python3.6/site-packages/scrapyd/runner.py", line 37, in main\n execute()\n File "/app/.heroku/python/lib/python3.6/site-packages/scrapy/cmdline.py", line 142, in execute\n cmd.crawler_process = CrawlerProcess(settings)\n File "/app/.heroku/python/lib/python3.6/site-packages/scrapy/crawler.py", line 280, in init\n super(CrawlerProcess, self).init(settings)\n File "/app/.heroku/python/lib/python3.6/site-packages/scrapy/crawler.py", line 152, in init\n self.spider_loader = self._get_spider_loader(settings)\n File "/app/.heroku/python/lib/python3.6/site-packages/scrapy/crawler.py", line 146, in _get_spider_loader\n return loader_cls.from_settings(settings.frozencopy())\n File "/app/.heroku/python/lib/python3.6/site-packages/scrapy/spiderloader.py", line 60, in from_settings\n return cls(settings)\n File "/app/.heroku/python/lib/python3.6/site-packages/scrapy/spiderloader.py", line 24, in init\n self._load_all_spiders()\n File "/app/.heroku/python/lib/python3.6/site-packages/scrapy/spiderloader.py", line 46, in _load_all_spiders\n for module in walk_modules(name):\n File "/app/.heroku/python/lib/python3.6/site-packages/scrapy/utils/misc.py", line 77, in walk_modules\n submod = import_module(fullpath)\n File "/app/.heroku/python/lib/python3.6/importlib/init.py", line 126, in import_module\n return _bootstrap._gcd_import(name[level:], package, level)\n File "", line 994, in _gcd_import\n File "", line 971, in _find_and_load\n File "", line 955, in _find_and_load_unlocked\n File "", line 656, in _load_unlocked\n File "", line 626, in _load_backward_compatible\n File "/tmp/scrapy_job_it-1590437526-tezy3xue.egg/scrapy_job_it/spiders/bdcrawler.py", line 3, in \nModuleNotFoundError: No module named 'scrapy_selenium'\n"}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant