Skip to content

Latest commit

 

History

History
72 lines (37 loc) · 1.4 KB

README.md

File metadata and controls

72 lines (37 loc) · 1.4 KB
Raw

Crawley

This project allows you to run crawlers and save the results in various formats.

Installing and Running

1. Clone Repository

2. Install with Poetry

Inside the Crawley directory, do:

  • poetry install

To install the enviroment via poetry.

Then, do:

  • poetry shell

To activate the virtual enviroment.

This will setup the enviroment to run the cli.

3. Running Crawley

Available Crawlers:

  • VultrCrawler
  • HostgatorCrawler

Arguments:

  • print
  • save_json
  • save_csv

Inside Crawley, you will run the command:

-python -m crawley.cli (Crawler) (argument)

To print the required information crawled from the website:

  • python -m crawley.cli (Crawler) print"

This should print the required information crawled from the website.

To save file in .json, execute:

  • python -m crawley.cli (Crawler) (argument) --filename (output).json

The .json file will be saved in the current directory

  • You can choose the name of your file by switching in (output).

To save a file in .csv, execute:

  • python -m crawley.cli (Crawler) (argument) --filename (output).csv

The .csv file will be saved in the current directory

  • You can choose the name of your file by switching in (output).

Final Note

That's the end of it. I hope this application can live up to the standards of good python Implementation.