Skip to content

zytedata/sctools

Repository files navigation

Analyze your jobs on Scrapy Cloud.

The tool is hosted on Github Pages: https://zytedata.github.io/sctools/

Table of Contents

Screenshots

Display multiple jobs stats as a table

Job Stats Table

Sort jobs by some metric

Sorting

Filter jobs by spider args

Filtering

Customize which stats to show

Customize

Select which scrapy stats to show

Stats

Reorder the columns by drag and drop

Drag and drop to reorder the columns:

Reorder

Visualizing the jobs stats

Number of items crawled over time:

Crawled Items Chart

Development Guide

Install pnpm

curl -fsSL https://get.pnpm.io/install.sh | sh -

Install clojure

Follow this document.

Launch the dev server

pnpm watch

It could take a while running for the first time.

In another terminal:

pnpm watch-css

Now visit http://127.0.0.1:3345 in a web browser.

License

Copyright © 2021 Lucy Wang

Distributed under the the Apache License, Version 2.0.