CRAWLY

Scrapy Scripts for scraping data

Run Commands

• Clone Project and go to crawly/

  $ git clone git@github.com:natalieconan/crawly.git
  $ cd crawly
  

• (Optional) To install pipenv with Homebrew:

  $ brew install pipenv
  

• Activate Python Virtual Env using Pipenv and install packages

  $ pipenv shell
  $ pipenv install
  

• Finally, run spider for crawling

  $ scrapy crawl ${spider_name}
  

  In this case spider_name = carlist, so run this command to start crawling:

  $ scrapy crawl carlist