Programatically steer smart mode detection

[blinjrm]

When crawling with with smart mode detection, is there a way to tell the crawler when to use JS rendering / HTTP only?

For example when crawling an e-commerce website, JS rendering may be needed to navigate catalog pages, but HTTP only can suffice to extract data from product pages, which represent >90% of pages.

[j-mendez]

When crawling with with smart mode detection, is there a way to tell the crawler when to use JS rendering / HTTP only?

For example when crawling an e-commerce website, JS rendering may be needed to navigate catalog pages, but HTTP only can suffice to extract data from product pages, which represent >90% of pages.

Right now smart mode is only automatic and handles it in that manner. It only loads the JS when it is needed to render markup ( if it has a certain dom mutation or JS framework being set ). It's hard to have this handled programmatically since theres no way to control the input prior.

If smart mode needs more flexibility use chrome flag with the crawl_raw to perform http crawling by default. Using subscriptions you can add logic to determine when to use the chrome page.

[blinjrm]

Using subscriptions you can add logic to determine when to use the chrome page

What is the best way to tell the crawler to use JS here?
I see a couple a solution reading the code, like Page::new() (with chome flag) or directly getting the response from fetch_page_html_chrome_base(), but both require the page argument. Creating a page from scratch duplicates a lot of work done at Website initialization, is there a better way to handle this?

[j-mendez]

The 7th param is called page_set if set to true pass in the raw html instead of a url for the first param source and the html will be re-used.

[blinjrm]

Isn't it the opposite though? I have the url but not the (js-rendered) html.
It looks like everything needed to get the chrome page is "hidden" in the shared object in crawl_concurrent, is there a way to reuse this logic from another function?

let shared = Arc::new((
    client.to_owned(),
    selectors,
    self.channel.clone(),
    self.configuration.external_domains_caseless.clone(),
    self.channel_guard.clone(),
    browser,
    self.configuration.clone(),
));

// lots happens to configure the new_page

let mut page = Page::new(
    &target_url,
    &shared.0,
    &new_page,
    &shared.6.wait_for,
    &shared.6.screenshot,
    false,
    &shared.6.openai_config,
)
.await;