Crawlers for extracting measurements from the web for Scroll datasets

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

The All in One Framework to build Awesome Scrapers.

🚀 OFFICIAL STARTER TEMPLATE FOR BOTASAURUS SCRAPING FRAMEWORK 🤖

This is a Twitter Scraper which uses Selenium for scraping tweets. It is capable of scraping tweets from home, user profile, hashtag, query or search, and advanced searches.

An asyncronous web crawling library for Python

Unveiling the Hidden Layers of the Web – A Comprehensive Web Reconnaissance Tool

Open Source Search Engine with built-in web/document crawler and an indexing method.

An easy-to-use library for the SpeedyShot Capture service.

An advanced recommender system for U.S. museums, using English-language text analytics on TripAdvisor reviews to enhance the visitor experience.

Another curated list of Python frameworks

REST-ly perform an HTTP GET and POST request with rotating proxy and user agents

🌐 Network Information Toolkit: Your all-in-one Python solution for network analysis. Explore IP addresses, DNS records, SSL certificates, and BGP data with ease. Stay efficient and secure with features like port scanning, whois lookup, and web crawling. Uncover valuable insights effortlessly. 🛠️🔍

JAW: A Graph-based Security Analysis Framework for Client-side JavaScript

A utility to crawl specified domains and download .zip files

Appache Airflow DAGs for e-commerce pricing collection.

Crawls download urls of albums from freehardmusic.com website

This project is a set of Python scripts designed to crawl and extract data from the Credly platform, focusing on skills, organizations, and badges. The scripts allow users to perform searches using command-line arguments, predefined search terms, or skills listed in a JSON file. The collected data is then saved to JSON files for further analysis an

Crawler is a Python package that crawls web pages and converts their content into Markdown format, making it easy to create documentation, notes, or other text-based representations. It features domain restrictions, flexible output options, and graph visualization.

Library for Rapid (Web) Crawler and Scraper Development