docs/guides/result-storage

[giscus[bot]]

docs/guides/result-storage

Where are you going to store all of that juicy scraped data?!

https://crawlee.dev/docs/guides/result-storage

[gablabelle]

Hello, is there a way to save the data as a single big file? This would save us from having to read the many json files from the disk and having to write a new big one after.

[janbuchar]

Absolutely! Check out https://crawlee.dev/docs/examples/export-entire-dataset

[gablabelle]

Thank you @janbuchar, this is exactly what I was looking for.

[aminebeh]

Hello, how can you setup a dataset and pass it to a crawler instance (to use it in requestHandler) ?

I want to create a dataset instance configure it using a configuration object and pass it to the crawler to be used as the "default" one.

Here's a sample code for illustration:

// Setup configurations
const datasetConfig = new Configuration({
  purgeOnStart: false,
});

// Create a dataset
const dataset = await Dataset.open("products", {
  config: datasetConfig,
});
// Initiate and run crawler
new CheerioCrawler({
  async requestHandler({ $, request, pushData }) {
    const title = $('title').text();
    console.log(`The title of "${request.url}" is: ${title}.`);
    pushData({
      title,
      url: request.url,
      succeeded: true,
    })
  }
  dataset: dataset
}).run('https://example.com/products')

Thanks in advance !

[janbuchar]

You can just push the data directly to your dataset - await dataset.pushData({ ... })

[franciscohanna92]

Hi! Is it possible to store results in a database? Should I just make the DB call where I otherwise would call dataset.pushData?

[janbuchar]

Hi, yes, that's one option. Another one is, if you're on the Apify platform, to use an integration specific to the database server that you'd like to use.