Skip to content

Web Scraping Craigslist's Engineering Jobs in NY with Scrapy

Notifications You must be signed in to change notification settings

GoTrained/Scrapy-Craigslist

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

5 Commits
 
 
 
 
 
 
 
 

Repository files navigation

Scrapy-Craigslist

Scrapy tutorial to build a Craigslist crawler that scrapes engineering jobs in New York.

Scrapy Tutorial: Craigslist

Check this blog post for the full Scrapy tutorial:

http://python.gotrained.com/scrapy-tutorial-web-scraping-craigslist

Usage

In Terminal or CMD, navigate to the main Scrapy project folder, and run one of the spiders:

  • Scraping Craigslist's Engineering Job Titles from One Page

scrapy crawl titles -o job-titles.csv

  • Scraping Craigslist's Engineering Job Titles, Addresses, and URLs from One Page

scrapy crawl jobsone -o job-one-page.csv

  • Scraping Craigslist's Engineering Job Titles, Addresses, and URLs from All Pages

scrapy crawl jobsall -o job-all-pages.csv

  • Scraping Craigslist's Engineering Job Details from Each Job's Web Page

scrapy crawl jobscontent -o job-all-pages-content.csv

Scrapy Online Course

Check this Scrapy tutorial to learn much more: