Http crawler does not return response in gzip format

[teammakdi]

Which package is this bug report for? If unsure which one to select, leave blank

@crawlee/http (HttpCrawler)

Issue description

The HttpCrawler always returns response body in string format, even though accept-encoding is set to gzip

Code sample

import {
   log,
   LogLevel,
   HttpCrawler
} from 'crawlee';

log.setLevel(LogLevel.ERROR);

const crawler = new HttpCrawler({
   minConcurrency: 2,
   maxConcurrency: 2,
   maxRequestRetries: 1,
   requestHandler: async ({
      request,
      response,
      enqueueLinks,
      body,
      contentType
   }) => {
      console.log("====REQUEST HEADERS====")
      console.log(JSON.stringify(response?.request?.options?.headers))
      console.log("====REQUEST HEADERS====")
      console.log("====RESPONSE HEADERS====")
      console.log(JSON.stringify(response?.headers))
      console.log("====RESPONSE HEADERS====")
      console.log("Content Type: ", contentType)
      // console.log("Body: ", body.toString())
   },
   failedRequestHandler({
      request
   }) {
      log.debug(`Request ${request.url} failed twice.`);
   },
   preNavigationHooks: [
      async (crawlingContext, gotOptions) => {
         if (!gotOptions.headers) {
            gotOptions.headers = {}
         }
         gotOptions.headers['Accept-Encoding'] = 'gzip, deflate, br, zstd'
         gotOptions.headers['Accept-Language'] = 'en-US,en;q=0.9'
      }
   ]
});

await crawler.run([
   'https://github.com/apify/crawlee/issues',
]);

Package version

3.8.1

Node.js version

20.11.1

Operating system

Mac OS

Apify platform

• Tick me if you encountered this issue on the Apify platform

I have tested this on the next release

No response

Other context

No response

[B4nan]

I believe this works as expected. You won't get the raw gzip encoded data, but they are encoded during the transfer, that's how the header should be working afaik.

[teammakdi]

Any way to get the compressed data @B4nan ?

[B4nan]

Why do you want this? What else would you do with it other than decode it?

[teammakdi]

@B4nan we are planning to put into another database in compressed fashion, hence do not want to waste CPU cycles twice, i.e. from decompressing and compressing again.