Getting "ow" errors when I click start

[pguardiario]

Does anyone know what causes these? There is no ow in my script so it's something in Crawlee.

2023-10-11T03:55:23.049Z ACTOR: Pulling Docker image of build lXdAHIe554fRDGSHm from repository.
2023-10-11T03:55:25.886Z ACTOR: Creating Docker container.
2023-10-11T03:55:26.569Z ACTOR: Starting Docker container.
2023-10-11T03:55:32.366Z INFO System info {"apifyVersion":"3.1.12","apifyClientVersion":"2.8.0","crawleeVersion":"3.5.7","osType":"Linux","nodeVersion":"v18.18.0"}
2023-10-11T03:55:32.942Z /usr/src/app/node_modules/ow/dist/index.js:33
2023-10-11T03:55:32.944Z (0, test_1.default)(value, () => (0, infer_label_1.inferLabel)(stackFrames), labelOrPredicate);
2023-10-11T03:55:32.945Z ^
2023-10-11T03:55:32.946Z
2023-10-11T03:55:32.948Z ArgumentError: (array requests) Any predicate failed with the following errors:
2023-10-11T03:55:32.949Z - Expected values to be of type string but received type Object
2023-10-11T03:55:32.950Z - Expected property url to be of type string but received type Object in object values
2023-10-11T03:55:32.952Z - Expected property requestsFromUrl to be of type string but received type undefined in object values
2023-10-11T03:55:32.953Z at ow (/usr/src/app/node_modules/ow/dist/index.js:33:28)
2023-10-11T03:55:32.954Z at RequestQueue.addRequestsBatched (/usr/src/app/node_modules/@crawlee/core/storages/request_provider.js:304:26)
2023-10-11T03:55:32.955Z at RequestQueue.addRequestsBatched (/usr/src/app/node_modules/@crawlee/core/storages/request_queue.js:266:22)
2023-10-11T03:55:32.957Z at CheerioCrawler.addRequests (/usr/src/app/node_modules/@crawlee/basic/internals/basic-crawler.js:606:29)
2023-10-11T03:55:32.958Z at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
2023-10-11T03:55:32.959Z validationErrors: Map(1) {
2023-10-11T03:55:32.960Z 'requests' => Set(1) {
2023-10-11T03:55:32.961Z '(array requests) Any predicate failed with the following errors:\n' +
2023-10-11T03:55:32.962Z ' - Expected values to be of type string but received type Object\n' +
2023-10-11T03:55:32.963Z ' - Expected property url to be of type string but received type Object in object values\n' +
2023-10-11T03:55:32.965Z ' - Expected property requestsFromUrl to be of type string but received type undefined in object values'
2023-10-11T03:55:32.966Z }
2023-10-11T03:55:32.967Z }
2023-10-11T03:55:32.968Z }
2023-10-11T03:55:32.970Z
2023-10-11T03:55:32.971Z Node.js v18.18.0

My script:

import { Actor } from 'apify';
import { CheerioCrawler, Dataset } from 'crawlee';
import router from "./router.js"
await Actor.init();

const {
    startUrls = ['https://www.zillow.com/ca'],
    maxRequestsPerCrawl = 100,
} = await Actor.getInput() ?? {}

const config = {}

const crawler = new CheerioCrawler({
    ...config,
    maxRequestsPerCrawl,
    requestHandler: () => {} // router
});

for(let url of startUrls){
  crawler.addRequests([{
    url,
    label: "SEARCH"
  }])
}

await crawler.run()

await Actor.exit()

TIA

[barjin]

Hi @pguardiario,
the problem is caused by the format the Apify platform parses the start URLs in. The Actor.getInput() call returns an object with the following properties:

{
 "startUrls": [
   { "url": "https://google.com/..." },
   { "url": "https://yahoo.com/..." },
 ]
}

I.e. each start url is a string "wrapped" in an object with the url key.

In your code, you are treating the start urls as strings, which causes a problem in the for loop with crawler.addRequests(), as you are passing the entire object in the url field, which Crawlee expects to be a string.

Let us know in case of any other problems. Thanks!

[pguardiario]

@barjin sorry I didn't quite follow that, are you saying startUrls should be strings or objects?

[barjin]

@pguardiario startUrls coming from the Apify platform are objects with "url" (and possibly) other keys.

Consider the following example:

import { Actor } from 'apify';
import { CheerioCrawler, Dataset } from 'crawlee';
await Actor.init();

const {
    startUrls = [
        {
            url: 'https://www.zillow.com/ca'
        },
        {
            url: 'https://www.google.com'
        },
        {
            url: 'https://www.example.com'
        },
    ],
    maxRequestsPerCrawl = 100,
} = await Actor.getInput() ?? {}

const config = {}

const crawler = new CheerioCrawler({
    ...config,
    maxRequestsPerCrawl,
    requestHandler: () => {} // router
});

await crawler.addRequests(startUrls.map(({ url }) => ({ url, label: 'SEARCH' })));
await crawler.run();
await Actor.exit()

Aside from the strings/objects issue, there was another problem in your code - the call to addRequests is async and needs to be awaited - plus you don't even need the for loop, as the addRequests call accepts an array (you can check out how I'm using map to add the correct label to all the requests that are being enqueued).

[pguardiario]

So yes startUrls need to be objects you're saying? It's strange that this works fine locally.
Anyway thanks for the help, I will try it that way

[barjin]

Great, let me know whether this has fixed your issue - and if it has, feel free to mark one of the comments as an answer. Thanks!

[pguardiario]

I don't think I can do that in discussions, can I? I don't see anywhere to do that.

[barjin]

Ah, my mistake, it's a maintainer-only feature. Sorry and thank you for the feedback! :)

[pguardiario]

Also just as a comment I would consider changing the property name to startRequests because startUrls invites confusion.