Is there a way to remove link from descending into?

[kulak]

I preserve article crawled and don't need to read it again on the next run of the application.

Is there a way to have a callback to tell not to add link to the crawl list?

[j-mendez]

@kulak yes. You can use the blacklist configuration to add exact paths to be ignored.

Once this PR gets released 06chaynes/http-cache#62 we can use a Cache Option with the “cache” feature flag to handle this by default.