Skip to content

0.15.0 Release

Latest
Compare
Choose a tag to compare
@oltarasenko oltarasenko released this 11 Apr 08:16
· 22 commits to master since this release

What's Changed

  1. Created a simple management Web UI. Try it on localhost:4001
  2. Added the possibility of creating spiders with the help of the YML format. Read more here: https://github.com/elixir-crawly/crawly/blob/master/documentation/spiders_in_yml.md
  3. Added the possibility to run Crawly (and your scraping projects) without Elixir. Read more here: https://github.com/elixir-crawly/crawly/blob/master/documentation/standalone_crawly.md
  4. Added generators for Crawly spiders and configuration files to reduce boilerplate
  5. Improved UniqueRequest middleware so that it can store hashes instead of complete URLs (special thanks to @serpent213)
  6. Added SameDomainFilter middleware, my favorite, which will probably deprecate the need to rely on the base_url in the future. Again thanks to @serpent213!