Presales question: scrape and summarize

[fulldecent]

Which package is the feature request for? If unsure which one to select, leave blank

None

Feature

I would like to give you a URL and then you scrape it using what technique is required (different for Facebook vs. static vs. JS SPA sites) and then you return it to me in Markdown or some other usable format. Possibly summarized using LLM.

Where do I put in my credit card number to buy this ^^ service?

Motivation

.

Ideal solution or implementation, and any additional constraints

.

Alternative solutions or implementations

No response

Other context

No response

[B4nan]

Hi there, we currently don't offer a dynamic solution that would automatically pick the best technique (crawler and proxy type), but it's something we are currently researching, and it is on our roadmap.

If you are up for gathering data for LLMs, you might want to take a look at Website Content Crawler, which was designed for this purpose.

https://apify.com/apify/website-content-crawler

Once we have that dynamic crawling support, the website content crawler will be updated to use it, now you need to pick the setup yourself (the defaults should work most of the time).