Skip to content

Scrapy spider which gets prices form a web catalouge and saves them in Excel.

Notifications You must be signed in to change notification settings

macrusso/price_scraper

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

7 Commits
 
 
 
 
 
 

Repository files navigation

Siemens Mall Price Scrapper

Small stand alone crawler script. Made to make my life easier, it goes through part numbers in a db and then goes to that parts pages and gets its prices.

How to use

Download both files. Start with db_seed.py, fill in part numbers you want to get prices for and then run the file. Then in prices.py insert your Siemens Mall credentials where USER_LOGIN and USER_PASSWORD are.

Check locations of your Selenium Webdiver and SQLite files and change the code if needed.

Spider automatically logs in with provided credentials then iterates through the db. Two prices are taken, one is a list price for the item and second is a price with your company discount. If the webpage for the given part is non existent or there is no prices for some reason, script inserts error message to the table.

Below you can find example table with two prices scrapped and one wrong part number and error notification.

Part number Discounted Price List Price
6ES7215-1HF40-0XB0 753.45 984.34
6ES7414-3XM07-0AB1 error error
6ES7221-1BF32-0XB0 72.94 96.90

Technologies used

  • Scrapy
  • Selenium
  • SQLite3