Handling Errors from failedRequestHandler in Single URL Flow with PlaywrightCrawler

[wojtekKrol]

Hello Crawlee maintainers and community,

I am working on a project where I use PlaywrightCrawler from the Crawlee library to handle HTML content for individual URLs. Unlike the typical use case of crawling multiple pages, my flow involves handling one URL at a time and executing different scenarios based on the content of each URL. For non-HTML content such as DOCX or XLSX files, I use different tools, so my use of PlaywrightCrawler is exclusively for HTML content processing.

Issue

The challenge arises when the failedRequestHandler gets invoked after several unsuccessful attempts to access a page. In such cases, I need a way to propagate this failure to the main execution context so I can handle it in a centralized manner. However, errors in the failedRequestHandler are currently not thrown back to the main function, preventing me from applying my error handling logic.

Expected Behavior

What I expect is a mechanism by which failedRequestHandler can throw an error back to the main function, allowing me to either retry the operation, halt execution, or execute other custom logic.

Current Behavior

As of now, errors in the failedRequestHandler are logged but do not interrupt or propagate back to the main loop, which keeps running independently of the error.

Code Snippet

Here is the relevant part of the main function that orchestrates the crawling process:

// ... previous code
export const main = async (dbs) => {
  // ... existing main function logic
  while (true) {
    // ... more code
    try {
      // ...
      const result = await checkUrl(checkUrlParams);
      // ...
    } catch (error) {
      // I want to handle errors from failedRequestHandler here
    }
    // ...
  }
};
workerpool.worker({ main });
// ... more code

The PlaywrightCrawleePageResolver class is set up as follows, including the failedRequestHandler:

import { BasicCrawler, PlaywrightCrawler } from 'crawlee';
import { Page } from 'playwright';
import { CONFIG } from '../config.js';

// ... other code

export class PlaywrightCrawleePageResolver extends CrawleePromiseResolver<
  PlaywrightCrawler,
  Page
> {
  constructor() {
    super(
      (resolveUrl) =>
        new PlaywrightCrawler({
          // ... other configurations
          failedRequestHandler({ request, log }, error) {
            log.info(`Request to: ${request.url} failed...`);
            // Here, I want to propagate this error to the main function's context
            // throw new CriticalCrawlError('Critical error occurred', { cause: error });
          },
          // ... rest of the configuration
        })
    );
  }
}

Specific Flow

I do not follow a broad crawling flow as outlined in the Crawlee documentation; my application processes URLs one at a time, and based on the content, different scenarios are handled. Therefore, I am looking for a solution that fits this particular use case.

Request

Is there an existing pattern or a recommended way to propagate errors from failedRequestHandler up to the main execution context in Crawlee? If such a mechanism does not exist, could you please suggest a workaround or consider this as a feature request?

Handling errors effectively in my specified flow is crucial, and I believe having this feature would significantly improve Crawlee's flexibility for various use cases.

Thank you for your time and for the excellent work on the library.

[metalwarrior665]

What you can do is to re-add it to the queue (with a different uniqueKey) to be processed again with a fresh set of retries. A request can error inside requestHandler but also before. You can also manage the errors before it reaches all retries with errorHandler. This all should give enough flexibility for any problem.

[wojtekKrol]

@metalwarrior665

Actually I do not want to re-add anything, I'm fine with unsuccessful attempt, my problem is that I don't know how to propagate this error to upper scope.

My execution stack is:

main() {
   checkUrl(...args) {
     //some operations
      anotherFunction(...args){
          //some conditions
                 yetAnotherFunction(...args){
                      const playwrightCrawleePageResolver = new PlaywrightCrawleePageResolver()
                      
                      playwrightCrawleePageResolver.resolve([url]) // [1]
                      const test = 'another Line Of Code That Will Never Execute If
                                   `failedRequestHandler` or `errorHandler` executes' // [2]
                      ...
                      ...
                     }
               }
        }
}

//And that's how `resolve` function looks like

  async resolve(urls: string[]) {
    this.client.addRequests(urls)

    const promises = urls.map(
      (url) =>
        new Promise<R | null>((resolve) => {

          this.urlToHandler[url] = (result) => {
              resolve(result)
          }
        })
    )

    return Promise.all(promises)
      .then((results) => {
        return results.filter((result): result is Awaited<R> => result !== null)
      })
      .catch((error) => {
          console.error('An error occurred:', error)
        throw error
      })
  }                   

[1] this resolve is method of CrawleePageResolver that is extended by PlaywrightCrawleePageResolver class and once there are 3 unsuccessful attempts code after that line is never executed, flow goes to failedHandlerRequest / errorHandler and it hangs out there (and never goes back)

[2] This line of code is never executed in case of execution of failedRequestHandler or errorHandler

And thats why i want somehow to let main function know that there was an error in failedRequestHandler or errorHandlerand break the execution for current url

[metalwarrior665]

I'm sorry but I'm a bit lost in here. I think you got lost a bit in the asynchronous flow and might need to re-think your flow. Keep in mind all the handlers are executed asynchronously by Crawlee's execution engine (Autoscaled Pool).

[wojtekKrol]

@metalwarrior665

But even if i do something like this it's not working as expected - throwing error so I could catch it

const main = async () => {
  while (true) {
    const playwrightCrawleePageResolver = new PlaywrightCrawleePageResolver()
    try {
      const url = 'https://osadnik.com' // let's assume that this page refuse to open 3 times

      await playwrightCrawleePageResolver.resolve([url])
    } catch (error: any) {
      // I want to propagate error here
    }
  }
}

import { BasicCrawler, PlaywrightCrawler } from 'crawlee'
import { Page } from 'playwright'
import { CONFIG } from '../config.js'

export class CrawleePromiseResolver<
  T extends BasicCrawler | PlaywrightCrawler,
  R
> {
  private client: T
  private urlToHandler: Record<
    string,
    (result: R | null, error?: any) => void
  > = {}

  constructor(builder: (resolveUrl: (url: string, result: R) => void) => T) {
    this.client = builder((url, result) => {
      if (!this.urlToHandler[url]) {
        return
      }

      this.urlToHandler[url](result)

      delete this.urlToHandler[url]
    })

    this.client.run()
  }

  async resolve(urls: string[]) {
    this.client.addRequests(urls)

    const promises = urls.map(
      (url) =>
        new Promise<R | null>((resolve) => {
          this.urlToHandler[url] = (result) => {
              resolve(result)
          }
        })
    )

    return Promise.all(promises)
      .then((results) => {
        return results.filter((result): result is Awaited<R> => result !== null)
      })
      .catch((error) => {
          console.error('An error occurred:', error)
        throw error
      })
  }
}

export class PlaywrightCrawleePageResolver extends CrawleePromiseResolver<
  PlaywrightCrawler,
  Page
> {
  constructor() {
    super(
      (resolveUrl) =>
        new PlaywrightCrawler({
          failedRequestHandler({ request, log }, error) {
            log.info(`Request to: ${request.url} failed...`) // this is the last line that executes, it never goes back to tha main() function even if I throw error/return anything
            // from here I want to propagate error to catch it in main() funciton
          },
          async requestHandler({ page, request, log }) {
            log.info(`Processing ${request.url} ...`)
            resolveUrl(request.url, page)
          },
        })
    )
  }
}

Is this case possible to achieve in your API?

Because in logs I only get this:

WARN  PlaywrightCrawler: Reclaiming failed request back to the list or queue. page.goto: net::ERR_CERT_COMMON_NAME_INVALID at https://osadnik.com/
=========================== logs ===========================
navigating to "https://osadnik.com/", waiting until "load"
============================================================
 {"id":"ydnavj1OebVBzxx","url":"https://osadnik.com","retryCount":3}
ERROR PlaywrightCrawler: Request failed and reached maximum retries. page.goto: net::ERR_CERT_COMMON_NAME_INVALID at https://osadnik.com/
=========================== logs ===========================
navigating to "https://osadnik.com/", waiting until "load"
============================================================
    at gotoExtended (C:\Users\pla481wojkro\WBO\openweb-crawler\node_modules\@crawlee\playwright\internals\utils\playwright-utils.js:154:17)
    at PlaywrightCrawler._navigationHandler (C:\Users\pla481wojkro\WBO\openweb-crawler\node_modules\@crawlee\playwright\internals\playwright-crawler.js:112:52)
    at PlaywrightCrawler._handleNavigation (C:\Users\pla481wojkro\WBO\openweb-crawler\node_modules\@crawlee\browser\internals\browser-crawler.js:299:51)
    at async PlaywrightCrawler._runRequestHandler (C:\Users\pla481wojkro\WBO\openweb-crawler\node_modules\@crawlee\browser\internals\browser-crawler.js:236:13)
    at async PlaywrightCrawler._runRequestHandler (C:\Users\pla481wojkro\WBO\openweb-crawler\node_modules\@crawlee\playwright\internals\playwright-crawler.js:109:9)
    at async wrap (C:\Users\pla481wojkro\WBO\openweb-crawler\node_modules\@apify\timeout\index.js:52:21) {"id":"ydnavj1OebVBzxx","url":"https://osadnik.com","method":"GET","uniqueKey":"https://osadnik.com"}
  

PS

I also tried to re-add like you suggest but it do not help me as it never hit requestHandler once again

 new PlaywrightCrawler({
          launchContext: { launchOptions: { headless: true } },
          requestHandlerTimeoutSecs: CONFIG.SOURCE_REQUEST_TIMEOUT / 1000, // 60 seconds
          // Limit to 10 requests per one crawl
          maxRequestsPerCrawl: 10,
          maxRequestRetries: 1,
          keepAlive: true,
          async failedRequestHandler({ request, log, crawler }) {
            log.info(`Request to: ${request.url} failed...`)
            log.info('Re-add request to queue')
            const uniqueKey = `Re-added_${request.url}_${Math.random()}`

            await crawler.addRequests([{ url: request.url, uniqueKey }])
          },
          async requestHandler({ page, request }) {
            const isRequestReAdded = request.uniqueKey.startsWith('Re-added')

            if(isRequestReAdded){
              throw new Error('Request re-added')
            }
            resolveUrl(request.url, page)
          },
        })

[wojtekKrol]

I manage to solve this issue by adding reject

 const promises = urls.map(
      (url) =>
        new Promise<R | null>((resolve, reject) => {
          this.urlToHandler[url] = { resolve, reject } 
        })
    )

and properly handling it in other methods