HTTP/2 requests through non-http2 aware proxies

[corford]

If useHttp2: true is passed to requestAsBrowser() (or as a request option to CheerioCrawler); how are the resulting HTTP/2 requests proxied if a non HTTP/2 aware proxy (e.g. squid) has been specified with proxyUrl ?

Asking because Got itself, or more specifically http2-wrapper, seems to require an intermediary native HTTP/2 speaking proxy (see: https://github.com/szmarczak/http2-wrapper/#proxy-support) i.e. it has no support for directly proxying HTTP/2 requests through a standard CONNECT tunnel.

Related: would be awesome to have a little more detail in general in the docs around how CheerioCrawler and requestAsBrowser handle HTTP/2 given it's now the dominant traffic protocol (https://radar.cloudflare.com/) and a lot of anti-bot vendors are using support (or lack thereof) as a strong signal to block non-browser based scrapers.

[mnmkng]

To be honest, we don't know at this moment. But it's one of our top priorities for Q1 2021 to figure this out and update the SDK where needed.

[corford]

Ok, great to know! FYI (in case it helps), I created an issue earlier this year asking about Got's (well, http2-wrapper's) HTTP/2 proxy support: szmarczak/http2-wrapper#40 As a result of that, some limited support & examples were merged in to the v1.0.0-beta.5.2 release.

I've just checked back now and it seems a lot more proxy related commits have gone in to master since then (see: https://github.com/szmarczak/http2-wrapper/commits/master/README.md), and from a quick glance, it looks possible to pass a "classic" proxy agent to Got for http2-wrapper to then use (https://github.com/szmarczak/http2-wrapper/blob/master/examples/proxies/h2-over-h1.js). It's been a while since I looked into all of this though, so maybe it's not as simple as it appears.

[szmarczak]

Just a heads up - since 2.0.0 http2-wrapper now supports proxies out of box. Apify uses got-scraping under the hood which automatically detects the proxy protocol, the destination protocol + has some additional features in order to mimic the browsers.