Incorrect Request Timeout in Error Message

[teammakdi]

Which package is this bug report for? If unsure which one to select, leave blank

@crawlee/cheerio (CheerioCrawler)

Issue description

We have been given a URL that takes more than 10 seconds to load, so the expected error message is 'request timed out after 10 seconds'. However, in the log, we can see it is saying 'request timed out after 55 seconds'.

Could someone please check this?"

Code sample

const { CheerioCrawler } = require('crawlee')

const crawler = new CheerioCrawler({
    requestHandlerTimeoutSecs: 15,
    keepAlive: true,
    requestHandler: async ({}) => { 

    },
    preNavigationHooks: [
        async (crawlingContext, gotOptions) => {
            gotOptions.timeout = {
                request: 10000
            }
        }
    ],
    errorHandler: ({ request }, err) => {
        console.log(`Error handler: ${request?.url}`, err)
    },
    failedRequestHandler: async ({ request, response }, err) => {
        console.log(`Failed Request Handler: ${request?.url}`, error)
    }
})

crawler.run(['http://localhost:8080/internal'])

Package version

3.8.2

Node.js version

20.11.1

Operating system

No response

Apify platform

• Tick me if you encountered this issue on the Apify platform

I have tested this on the next release

No response

Other context

No response

[B4nan]

That is not coming from got, it's not the timeout you are setting in the got options (10s), it's the request handler timeout (most probably), which is computed from several things, namely the requestHandlerTimeoutSecs you are setting to 15, but the final number is combined with other timeouts like the navigationTimeoutSecs as that is happening inside the handler too.

This is the part where its computed:

https://github.com/apify/crawlee/blob/master/packages/http-crawler/src/internals/http-crawler.ts#L366

The BASIC_CRAWLER_TIMEOUT_BUFFER_SECS is 10s, navigationTimeoutSecs defaults to 30s and then there is your 15s requestHandlerTimeoutSecs, which when you sum up, you get the 55s you see in the error message.

[teammakdi]

The request is timing out just after 10 seconds, and the log was showing incorrect information, which is why I am confused here.

[B4nan]

What is the full stack trace?

[teammakdi]

Error handler: http://localhost:8080/internal Error: request timed out after 55 seconds.
at CheerioCrawler._handleRequestTimeout (/node_modules/@crawlee/http/internals/http-crawler.js:588:15)
at CheerioCrawler._requestFunction (/node_modules/@crawlee/http/internals/http-crawler.js:444:22)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async wrap (/node_modules/@apify/timeout/index.js:52:21)

[B4nan]

Yes, so it's coming from the request handler as I expected, and that has the 55s timeout.