Istio is Truncating (cut off ) Json Response Body when curl is executed

[hassanbsee2071]

we have multi container pod.
php-fpm
nginx

Without istio, things are working and curl gives full json response body. But when we enable istio and execute curl request, response body becomes truncated and gives partial response.

First, I used nginx:1.25.2 and enable debug logs on this container, I saw a log when response is truncated.

upstream sent more data than specified in "content-length" header while reading upstream

this error is gone when response is full and not truncated. Some of the curl request gives may be 4/100 gives full response.

Then I changed the nginx image to nginx:1.15.2.

Still gives me the partial response but with log.

2024/04/24 19:44:58 [info] 7#7: *594 client 127.0.0.6 closed keepalive connection

In case of full response this log is not present.

nginx configuration:

    upstream api-backend {
        least_conn;
        server 127.0.0.1:9002 max_fails=15 fail_timeout=45s weight=2;
        keepalive 80;
    }
    server {
            error_log /var/log/nginx/error.log debug;
            listen 80 default_server;
            listen [::]:80 default_server;
            server_name _;

            keepalive_timeout 600s;
            client_body_timeout 600s;
            client_header_timeout 600s;
            
            # Set nginx to serve files from the shared volume!
            set $static_root /var/www/hub-api/public;
            root $static_root;
            index index.php index.html index.htm index.nginx-debian.html;
    
            include /etc/lb-config/site.d/headers_api.conf;
            include /etc/lb-config/site.d/error_page.conf;
    
            include /etc/lb-config/site.d/extra_protect.conf;
            # static content
            include /etc/lb-config/site.d/static_location.conf;
    
            # The minimum length of the response to be compress
            gzip_min_length  2056;
    
            # Disable PHP execution in var and media
            location /var { location ~ \.php$ {return 403;} }
    
            ## These locations are protected
            location ~ /(app|includes|pkginfo|var|composer.json|errors/local.xml)/ {
                deny all;
            }
    
            ## Bots trap from nginx.conf
    
            ## Main @location
            location / {
                index index.php index.html index.htm;
                try_files $uri $uri/ @rewrite;
            }
    
            location @rewrite {
                rewrite / /index.php?$args;
            }
    
            set $fastcgi_https "";
    
            if ($http_x_forwarded_proto = "https") {
                set $fastcgi_https "on";
            }
    
            # include /etc/lb-config/site.d/lb_opt.conf;
            
            proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
            proxy_next_upstream_timeout 600s;
            proxy_read_timeout 600s;
            proxy_connect_timeout 600s;
            ###
            fastcgi_keep_conn on;
            fastcgi_next_upstream error timeout http_500  http_503;
            fastcgi_next_upstream_timeout 5s;
            fastcgi_connect_timeout 10;
            fastcgi_ignore_client_abort on;
            #fastcgi_send_timeout 180;
            #fastcgi_read_timeout 180;

            # Adjust buffering settings
            fastcgi_buffering off;  # Disable buffering

            fastcgi_buffer_size 64k;  # Adjust buffer size as needed
            fastcgi_buffers 4 64k;    # Adjust number of buffers


            ## Execute PHP scripts
            location ~ \.php$ {
                include /etc/lb-config/site.d/fastcgi_params.conf;
    
                fastcgi_pass   api-backend;
                fastcgi_param  HTTPS       $fastcgi_https;
                fastcgi_param  SCRIPT_FILENAME $document_root/$fastcgi_script_name;
            }
            
    
     }

Partial Response:

{
    "airline": [
        {
            "code": "ET",
            "label": {
                "en": "Ethiopian Airlines",
                "ar":"الخطوط الجو\

Full response:

{
    "airline": [
        {
            "code": "ET",
            "label": {
                "en": "Ethiopian Airlines",
                "ar": "الخطوط الجوية الأثيوبية"
            }
        }
    ],
    "airport": []
}

With istio enabled sometimes I get the full response but usually response body is truncated.
we do not have istio gateway, no virtual service and destination, we just enabled the sidecar injection.

istio version:

client version: 1.18.2
control plane version: 1.18.2
data plane version: 1.18.2 (559 proxies)

EKS version:

Client Version: v1.28.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.7-eks-b9c9ed7

[hassanbsee2071]

I got the issue and I fixed by below filter.

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: add-encoding-header
  namespace: hub
spec:
  workloadSelector:
    labels:
      app: hub-api
  configPatches:
    - applyTo: HTTP_FILTER
      match:
        context: SIDECAR_OUTBOUND
        listener:
          portNumber: 80
          filterChain:
            filter:
              name: "envoy.filters.network.http_connection_manager"
              subFilter:
                name: "envoy.filters.http.router"
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.lua
          typed_config:
            "@type": "type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua"
            inlineCode: |
              function envoy_on_response(response_handle)
                local response_headers = response_handle:headers()
                response_handle:logWarn("Removing content-length and adding transfer-encoding chukcked header for hub-api:")
                response_headers:remove("content-length")
                response_headers:add("transefer-encoding", "chuncked")
              end

[zirain]

1.18 is EOL for a while, please try with latest version, it maybe fixed in envoy.