Skip to content

sh-alireza/General-Crawler

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

10 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Data Crawler

This is a simple data crawler that crawls data from a website and stores it in a database. Then you can download your data in your local storage. For now it only supports crawling data from houzz, amazon(only some products), google-images, youtube.

Installation

Go to the root of the project and run the following commands:

pip install -r requirements.txt
pip install -e .

Usage

crawler

Contributing

Pull requests are welcome. You can add your own website crawler and downloader to help the project grow.