Granular crawl

[blinjrm]

Spider seems great for broad crawls, but is it possible to do more granular crawls ?

I really like the enqueueLinks API in Crawlee, as it allows to selectively add links to the queue based on regex, css selector, and more.
Can a similar behavior be implemented with Spider ?

2 use cases I'm trying to implement:

• crawl www.some_website.com/fr, but not www.some_website.com/en or any url that does not start with www.some_website.com/fr/*
• on an e-commerce catalog webpage, crawl all product pages but ignore all other links

[j-mendez]

Yes use website.queue along with subscriptions to add links mid crawl.

[j-mendez]

Thanks for your reply.

Looking at the code and examples I'm still struggling to understand subscriptions though. How is it different from "regular" crawling ?

Using the queue I can add links mid-crawl, but how do i stop the crawler from just crawling every link it sees in the first place ?

This example could help https://github.com/spider-rs/spider/blob/main/examples/queue.rs. The blacklisting needs to be done with the configuration or website.with_blacklist. It allows a regex if the regex feature flag is enabled.

[blinjrm]

Yes I saw that example, but does it have to be combined with a subscription? How is a subscription different from a normal crawl?

[j-mendez]

Yes I saw that example, but does it have to be combined with a subscription? How is a subscription different from a normal crawl?

Yes, it kinda needs to be combined with a subscription or a thread. The subscriptions let you get access to the page object concurrently so you do not wait till the end of the crawl.

[blinjrm]

The subscriptions let you get access to the page object concurrently so you do not wait till the end of the crawl

Ok things make a lot more sense now, thanks!

[LiamK]

I have also been struggling with this using the Python wrapper. Is is possible to implement the queue example given above with Python? I don't understand rust, nor whether the proposed answer involving the queue might work with Python.

I ended up getting a Page, getting its list of links, filtering the list with a regex, and then getting the individual pages, but that seems convoluted.

Re: the blacklisting, the regex doesn't accept negative look-aheads. From the rust documentation:

The regex syntax supported by this crate is similar to other regex engines, but it lacks several features that are not known how to implement efficiently. This includes, but is not limited to, look-around and backreferences.

What I would like to do is:

1. start with sub-page, e.g. https://domain.com/events/
2. blacklist urls that don't contain '/event/'

Or possibly in a single positive regex:

1. Get 'https://domain.com/event(s)?/

Without that, I would have to know all the other possible urls on the site and blacklist them. That's impractical and error-prone.

It seems more intuitive to say what I do want, than what I don't. But I can imagine use cases for both approaches.