How to run a spider?

[dogweather]

I saw the docs that show starting a REPL:

iex -S mix run -e "Crawly.Engine.start_spider(Spider)"

And that works for my spider when testing it in development. But how should we run it from the command line as a normal CLI app that runs and exits when done?

[oltarasenko]

I think that the most simple way to achieve it is to add the following line to your configuration:
on_spider_closed_callback: fn _, _, _ -> System.halt(1) end,

[dogweather]

Thanks, that's cool. I'll give it a try.

[dogweather]

Yep, this works great. I just needed to add a setting that'd close the spider: it appears that the spider doesn't close when it's processed all its requests. I guess it stays running and open like a server for handling more. Anyhow I added:

closespider_timeout: 1

...to close the spider when it has processed zero requests for one minute. A little indirect, but it does work.