Implement Automatic Memory Management in Playwright for Enhanced Stability in Web Crawling Operations

[wojtekKrol]

Which package is the feature request for? If unsure which one to select, leave blank

@crawlee/playwright (PlaywrightCrawler)

Feature

Dear Development Team,

I am encountering a significant challenge with our web crawling application that utilizes Playwright. The core issue pertains to excessive memory consumption leading to system instability.

The application's primary function is to perform web crawling in headless mode. It successfully processes a considerable number of websites (approximately 1000-1500) before encountering performance degradation. Post this threshold, the application experiences a drastic slowdown, ultimately culminating in a freeze. This has necessitated a complete system reboot, as the application becomes unresponsive.

Detailed Workflow:

1. Initiate the crawling process by opening a website in headless mode.
2. Execute the designated business logic and crawling operations.
3. Conclude each session by closing the website. I believe this also includes terminating the browser process, as the application uses Crawlee.
4. This process repeats for each website.

A critical observation is the memory usage reported by Playwright. Logs indicate that memory consumption reaches 400%, translating to roughly 800MB. This is a clear indication of resource overutilization, leading to the aforementioned issues.

The objective of this request is to explore potential strategies to optimize memory usage without restarting the entire application. Any insights or suggestions on memory management techniques, code optimizations, or best practices in this context would be immensely beneficial.

Thank you for your attention to this matter.

Best regards

Motivation

The motivation for requesting this feature is to enhance the stability and efficiency of the application by optimizing memory usage, thereby preventing crashes and ensuring smoother, more reliable web crawling operations.

Ideal solution or implementation, and any additional constraints

Ideal Solution:

The ideal solution involves integrating a feature in Playwright, possibly through its constructor, that allows for automatic memory management. This could be implemented as a flag or an object within the Playwright constructor, which enables the clearing of cache or freeing up memory at specified intervals or after processing a certain number of URLs.

Implementation:

Automatic Memory Clearing Option: Introduce a parameter like autoMemoryManagement in the Playwright constructor. This parameter could accept an object defining the conditions for triggering memory cleanup, such as time intervals or the number of URLs processed.

Example:

new PlaywrightCrawler({
    autoMemoryManagement: {
        urlThreshold: 500, // Clear memory after every 500 URLs
        timeInterval: 3600 // or every hour (value in seconds)
    }
});

Alternative solutions or implementations

1. Dynamic Resource Allocation: The feature should dynamically monitor memory usage and adjust its behavior based on current system resource availability, ensuring optimal performance without manual intervention.

2. Customizable Settings: Offer customization for different levels of memory cleanup, enabling users to balance between performance and memory usage based on their specific requirements.

Other context

No response

[B4nan]

I am not sure what you are proposing here (you just said you want us to add a magic option to deal with memory leaks if I read things right), but I can say one thing - we at Apify use playwright on a daily basis, for scrapes that can take days, sometimes even weeks. So if you have issues with memory, it's probably a memory leak in your code, not something to fix at our end.

We'll be happy to help with a complete reproduction of your problem, but don't expect us to add some magical toggles like this. There is no "playwright url cache" we just need to clear.

[wojtekKrol]

Thank you for your response. I understand from your experience at Apify that long-duration crawls are feasible without significant memory issues. This leads me to believe that the memory leak I'm experiencing with my Playwright-based application might be due to something specific in my implementation.

I've been using Playwright, the Microsoft package for testing, and I've noticed that other developers also encounter similar memory leak issues. However, if your team can run crawls for weeks without such issues, I'm inclined to think the problem lies in my code. I'm struggling to pinpoint where this memory leak could be occurring, particularly in the parts of my code related to crawling. Here's a snippet:

public pageProcess: PageProcessType = async ({
  url,
  pageProcessingCallback,
}) => {
  const urlWithUniqueKey = {
    url,
    uniqueKey: `${url}_${Math.random()}`,
  }

  await this.crawler.addRequests([urlWithUniqueKey])

  const promise = new Promise<Set<string>>((resolve, reject) => {
    this.urlToResolver[url] = {
      pageProcessingCallback,
      resolve,
      reject,
    }
  })

  const result = await promise

  return result
}

private requestHandler: PlaywrightRequestHandler = async ({
  page,
  request: { url },
  log,
}) => {
  log.info(`Request to: ${url} ...`)
  try {
    await playwrightUtils.infiniteScroll(page, {
      scrollDownAndUp: true, // some pages require scrolling up and down to load new content
      timeoutSecs: 0.1, // scroll max for N seconds
    })
    const result = await this.processPageRequest({ url, page })

    this.resolvePageProcessing({ url, result })

    await page.close()
    await browserController.close()
    await browserController.kill()
  } catch (error) {
    if (error instanceof Error) {
      this.rejectPageProcessing(url, error)
    }
  } finally {
    delete this.urlToResolver[url]
  }
}

Additionally, here is how I've set up my Playwright instance:

 private createCrawler = (): PlaywrightCrawler => {
    return new PlaywrightCrawler({
      autoscaledPoolOptions: {
        autoscaleIntervalSecs: 5, // Interval in seconds to check and adjust pool size for optimal concurrency
        loggingIntervalSecs: null, // Disables periodic logging of the pool's state; set a value to enable
        maxConcurrency: CONFIG.SOURCE_MAX_CONCURRENCY, // Maximum number of requests processed concurrently
        minConcurrency: CONFIG.SOURCE_MIN_CONCURRENCY, // Minimum number of requests to keep processing concurrently
      },
      maxRequestsPerMinute: 250,
      browserPoolOptions: {
        operationTimeoutSecs: 5, // Maximum time in seconds for a single page operation (like goto, waitForSelector)
        retireBrowserAfterPageCount: 5, // Number of pages after which the browser will be retired (closed and replaced)
        maxOpenPagesPerBrowser: 1, // Maximum number of pages (tabs) to open concurrently in a single browser
        closeInactiveBrowserAfterSecs: 5, // Time in seconds after which an inactive browser will be closed
      },
      launchContext: {
        launchOptions: {
          chromiumSandbox: false, // Disables Chromium's sandbox for compatibility with Docker environments
          headless: CONFIG.SOURCE_HEADLESS, // Launches browser in headless mode (without GUI)
        },
      },
      requestHandlerTimeoutSecs: 300, // Timeout in seconds for processing a single request (15 seconds)
      maxRequestRetries: 1, // Number of retries for a request if it fails
      keepAlive: true, // Keeps the crawler alive even if all requests are handled; useful for long-running crawls
      retryOnBlocked: false, // Automatically retries a request if it is identified as blocked (e.g., by bot detection)
      requestHandler: this.requestHandler.bind(this), // Function to handle each request
      failedRequestHandler: this.failedRequestHandler.bind(this), // Function to handle each failed request
    })
  }

Given this information, could you offer any insights or suggestions on what might be going wrong or how I can handle it better? Any hints or tips on memory management and optimization in this context would be highly appreciated.

Thank you for your assistance.

my packages:

  "crawlee": "3.7.2",
  "playwright": "1.41.0",

[B4nan]

Have you tried checking where the memory actually grows?

e.g. https://nodejs.org/en/guides/diagnostics/memory/using-heap-snapshot