Using crawlee in an existing project

[SteveGrixti]

Hey Everyone,

I followed the crawlee tutorial and was very happy as after getting everything to work it was just what I was looking for to test out some stuff on my existing solution.

However I have run into an issue when trying to implement crawlee on my existing api.

Basically, I already have an api set up which was created with nodeJS and my router is already being used as the below examples:

const router = express.Router();

router.get("/api/test/:query", (req, res) => {
   //logic goes in here
});

Now as you can see I am handling my router using express.

I created an endpoint which calls a js file containing the crawlee implementation as below:

const crawlee = require("crawlee");
const playwrightextra = require("playwright-extra");
const stealth = require("puppeteer-extra-plugin-stealth");

async function spec(url) {
  let results = [];

  const crawler = new crawlee.PlaywrightCrawler({
      launchContext: {
        launcher: playwrightextra.chromium,
        launchOptions: {
          headless: true,
        },
      },
      sessionPoolOptions: {
        blockedStatusCodes: [],
      },
      browserPoolOptions: {
        fingerprintOptions: {
          fingerprintGeneratorOptions: {
            browsers: [crawlee.BrowserName.firefox],
            devices: [crawlee.DeviceCategory.desktop],
            operatingSystems: [crawlee.OperatingSystemsName.windows],
          },
        },
      },
      requestHandler: async ({ page, request, enqueueLinks }) => {
          // logic goes here
      },
  });

  await crawler.run([url]).then((x) => {
    return results;
  });
}

module.exports = spec;

However when I call the endpoint which calls the spec method, seems that this causes my api to restart.

Below is the output I am getting:

[nodemon] 2.0.22
[nodemon] to restart at any time, enter `rs`
[nodemon] watching path(s): *.*
[nodemon] watching extensions: js,mjs,json
[nodemon] starting `node app.js`
Listening on PORT: 8080
(node:6528) ExperimentalWarning: Import assertions are not a stable feature of the JavaScript language. Avoid relying on their current behavior and syntax as those might change in a future version of Node.js.
(Use `node --trace-warnings ...` to show where the warning was created)
(node:6528) ExperimentalWarning: Importing JSON modules is an experimental feature and might change at any time
[nodemon] restarting due to changes...
[nodemon] restarting due to changes...
INFO  PlaywrightCrawler: Starting the crawler.
[nodemon] starting `node app.js`
Listening on PORT: 8080
(node:12068) ExperimentalWarning: Import assertions are not a stable feature of the JavaScript language. Avoid relying on their current behavior and syntax as those might change in a future version of Node.js.
(Use `node --trace-warnings ...` to show where the warning was created)
(node:12068) ExperimentalWarning: Importing JSON modules is an experimental feature and might change at any time
[nodemon] restarting due to changes...
[nodemon] starting `node app.js`
Listening on PORT: 8080
(node:15328) ExperimentalWarning: Import assertions are not a stable feature of the JavaScript language. Avoid relying on their current behavior and syntax as those might change in a future version of Node.js.
(Use `node --trace-warnings ...` to show where the warning was created)
(node:15328) ExperimentalWarning: Importing JSON modules is an experimental feature and might change at any time
[nodemon] restarting due to changes...
INFO  PlaywrightCrawler: Starting the crawler.
[nodemon] restarting due to changes...
[nodemon] starting `node app.js`
Listening on PORT: 8080
(node:2648) ExperimentalWarning: Import assertions are not a stable feature of the JavaScript language. Avoid relying on their current behavior and syntax as those might change in a future version of Node.js.
(Use `node --trace-warnings ...` to show where the warning was created)
(node:2648) ExperimentalWarning: Importing JSON modules is an experimental feature and might change at any time

Has anybody encountered this issue before and how would I go about solving it?

Thanks :)

[B4nan]

Crawlee will write its state into to storage directory (request queue as well as the dataset and kvs records), and you are probably watching that directory for changes.

Try doing this:

https://crawlee.dev/docs/deployment/aws-cheerio

[SteveGrixti]

Ok thanks will try it out 😁

[SteveGrixti]

This worked perfectly thank you 💪