Option to exclude certain resource types

[mbledkowski]

Which package is the feature request for? If unsure which one to select, leave blank

@crawlee/playwright (PlaywrightCrawler)

Feature

Hi,
I would like to have an option to exclude certain resource types from being loaded in browser. E.g. "stylesheet", "media", "texttrack", "eventsource", "websocket", "manifest", etc.

Motivation

It would help with the speed of scraping, and lower the internet usage.

Ideal solution or implementation, and any additional constraints

In PlaywrightCrawler object constructor.

Alternative solutions or implementations

No response

Other context

No response

[yellott]

Do you mean something like this https://crawlee.dev/api/playwright-crawler/namespace/playwrightUtils#blockRequests?

[LeMoussel]

Or Skipping navigations for certain requests.
Or you can do like this:

import {
  PlaywrightCrawler,  // https://crawlee.dev/docs/examples/playwright-crawler
  log
} from 'crawlee';


// https://playwright.dev/docs/api/class-request#request-resource-type
const RESOURCE_EXCLUSTIONS = ['image', 'stylesheet', 'media', 'font', 'other'];
/* More ressource type
For a new site I try to block everything in BLOCKED_IMG_CSS_JS.
In case the site does not work (pages not rendered properly) I try the BLOCKED_IMG_CSS.
If it is still not work - BLOCKED_IMG and than - no block at all.

const BLOCKED_IMG = ['image', 'imageset', 'object', 'object_subrequest', 'ping', 'web_manifest', 'xslt',  'media', 'font', 'other', 'beacon', 'csp_report', 'speculative', 'sub_frame', 'xbl', 'xml_dtd', 'texttrack', 'fetch', 'eventsource', 'manifest'];
const BLOCKED_IMG_CSS = ['stylesheet', 'image', 'imageset', 'object', 'object_subrequest', 'ping', 'web_manifest', 'xslt',  'media', 'font', 'other', 'beacon', 'csp_report', 'speculative', 'sub_frame', 'xbl', 'xml_dtd', 'texttrack', 'fetch', 'eventsource', 'manifest'];
const BLOCKED_IMG_CSS_JS = ['websocket', 'xhr', 'xmlhttprequest', 'script', 'stylesheet', 'image', 'imageset', 'object', 'object_subrequest', 'ping', 'web_manifest', 'xslt',  'media', 'font', 'other', 'beacon', 'csp_report', 'speculative', 'sub_frame', 'xbl', 'xml_dtd', 'texttrack', 'fetch', 'eventsource', 'manifest'];
*/
const crawler = new PlaywrightCrawler({
  headless: true,
  preNavigationHooks: [
    async ({ page, request }) => {
      await page.route('**/*', (route) => {
        if (RESOURCE_EXCLUSTIONS.includes(route.request().resourceType())) route.abort()
        else {
          log.info(`Request: ${route.request().url()} to resource type: ${route.request().resourceType()}`);
          route.continue()
        }
      });
    },
  ],
  async requestHandler({ request, page, log }) {
    log.info(`Title of ${page.url()} is '${await page.title()}'`);
  },
});

await crawler.run(["https://amazon.com"]);

[mbledkowski]

Thank you very much. This is the answer I needed