Istio is Truncating (cut off ) Json Response Body

[hassanbsee2071]

we have multi container pod.
php-fpm_exporter
nginx

Without istio, every thing is working. But when we enable istio and execute curl request, response body becomes truncated and gives partial response.
I enable debug logs on nginx container.

2024/04/24 10:30:50 [warn] 29#29: *111 upstream sent more data than specified in "Content-Length" header while reading upstream, client: 127.0.0.6, server: _, request: "GET /flight-api/flight/resource/codes/RUH,JED,DMM,F3,XY HTTP/1.1", upstream: "fastcgi://127.0.0.1:9002", host: "hub-api", referrer: "https://hub-staging.almosafer.com/unify/sales/add/flights/6628d7e4c5a35b209259ebf8?l=5d8a2e4d949764f32e021d8d"

Without istio nginx logs does not throw this warning.

My nginx file:

    upstream api-backend {
        least_conn;
        server 127.0.0.1:9002 max_fails=15 fail_timeout=45s weight=2;
        keepalive 8;
    }
    server {
            error_log /var/log/nginx/error.log debug;
            listen 80 default_server;
            listen [::]:80 default_server;
            server_name _;
            # Set nginx to serve files from the shared volume!
            set $static_root /var/www/hub-api/public;
            root $static_root;
            index index.php index.html index.htm index.nginx-debian.html;
    
            include /etc/lb-config/site.d/headers_api.conf;
            include /etc/lb-config/site.d/error_page.conf;
    
            include /etc/lb-config/site.d/extra_protect.conf;
            # static content
            include /etc/lb-config/site.d/static_location.conf;
    
            # The minimum length of the response to be compress
            gzip_min_length  256;
    
            # Disable PHP execution in var and media
            location /var { location ~ \.php$ {return 403;} }
    
            ## These locations are protected
            location ~ /(app|includes|pkginfo|var|composer.json|errors/local.xml)/ {
                deny all;
            }
    
            ## Bots trap from nginx.conf
    
            ## Main @location
            location / {
                index index.php index.html index.htm;
                try_files $uri $uri/ @rewrite;
               
            }
    
            location @rewrite {
                rewrite / /index.php?$args;
            }
    
            set $fastcgi_https "";
    
            if ($http_x_forwarded_proto = "https") {
                set $fastcgi_https "on";
            }
    
            include /etc/lb-config/site.d/lb_opt.conf;
    
            ## Execute PHP scripts
            location ~ \.php$ {
                include /etc/lb-config/site.d/fastcgi_params.conf;
    
                fastcgi_pass   api-backend;
                fastcgi_param  HTTPS       $fastcgi_https;
                fastcgi_param  SCRIPT_FILENAME $document_root/$fastcgi_script_name;
            }
       proxy_hide_header Content-Length;
 
     }

Partial response looks like this:

{
    "airline": [
        {
            "code": "ET",
            "label": {
                

I tried below in nginx conf but it does not allow it.

proxy_ignore_headers Content-Length;

nginx: [emerg] invalid value "Content-Length"

Also tried below filters but none of them seems working

---
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: increase-buffer-limits
  namespace: istio-system
spec:

  configPatches:
    - applyTo: NETWORK_FILTER
      match:
        context: SIDECAR_INBOUND
        listener:
          filterChain:
            filter:
              name: "envoy.http_connection_manager"
      patch:
        operation: MERGE
        value:
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
            per_connection_buffer_limit_bytes: 524288000  # Increase buffer size 
---
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: max-request-headers
  namespace: istio-system
spec:
  configPatches:
  - applyTo: NETWORK_FILTER # http connection manager is a filter in Envoy
    match:
      context: ANY
      listener:
        filterChain:
          filter:
            name: "envoy.filters.network.http_connection_manager"
    patch:
      operation: MERGE
      value:
        typed_config:
          "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager"
          max_request_headers_kb: 100
---

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: application-gzip
  namespace: istio-system
spec:
  configPatches:
    - applyTo: HTTP_FILTER
      match:
        context: ANY
        listener:
          filterChain:
            filter:
              name: envoy.http_connection_manager
              subFilter:
                name: envoy.router
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.filters.http.compressor
          typed_config:
            '@type': type.googleapis.com/envoy.extensions.filters.http.compressor.v3.Compressor
            compressor_library:
              name: text_optimized
              typed_config:
                '@type': type.googleapis.com/envoy.extensions.compression.gzip.compressor.v3.Gzip
            remove_accept_encoding_header: true

Any help is really appreciated.

[hassanbsee2071]

I got the issue and I fixed by below filter.

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: add-encoding-header
  namespace: hub
spec:
  workloadSelector:
    labels:
      app: hub-api
  configPatches:
    - applyTo: HTTP_FILTER
      match:
        context: SIDECAR_OUTBOUND
        listener:
          portNumber: 80
          filterChain:
            filter:
              name: "envoy.filters.network.http_connection_manager"
              subFilter:
                name: "envoy.filters.http.router"
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.lua
          typed_config:
            "@type": "type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua"
            inlineCode: |
              function envoy_on_response(response_handle)
                local response_headers = response_handle:headers()
                response_handle:logWarn("Removing content-length and adding transfer-encoding chukcked header for hub-api:")
                response_headers:remove("content-length")
                response_headers:add("transefer-encoding", "chuncked")
              end