This repository is designed for data scraping, data mining and data visualization in house price, job interviewing and SNS data mining. We collect data with scrapy and requests, data pre-processing and machine learning with scikit-learn and pandas. Data storing with MongoDB.
There are several modules belongs to different application.
Python version >= 3.5
requests
scrapy
pandas
scikit-learn
pymongo
TensorFlow >= 1.6
PyTorch >= 0.3.1
-
Installation
sudo pip3 install -r requirements.txt -
Start MongoDB Service
sudo service mongod start
- This repository can be only used for research and non-commercial applications.