Cookie not applied in inter-namespace communication

[Raven6681]

Hello,

I am trying to apply cookies on a service and my setup is the following:

---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: myservice-sticky-vs
  namespace: myservice
spec:
  hosts:
    - my.gateway.fqdn
    - myservice.namespace.svc.cluster.local
  gateways:
    - istio-ingress-internal/gateway-internal
    - mesh
  http:
  - match:
    - uri:
        prefix: /path
    route:
    - destination:
        host: myservice.namespace.svc.cluster.local
        port:
          number: 8080
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: myservice-sticky-dr
  namespace: myservice
spec:
  host: myservice.namespace.svc.cluster.local
  trafficPolicy:
    loadBalancer:
      consistentHash:
        httpCookie:
          name: istio
          ttl: 0s
    tls:
      mode: ISTIO_MUTUAL

EKS cluster version: 1.27
Istio version: 1.21.1

The Gateway sits behind an AWS ALB (gateway deployment installed with a NodePort service and the ALB as ingress).
If I hit the Gateway FQDN with curl, I can see the cookie (real path and fqdn have been substituted), < set-cookie: istio="a9bbe686634a6ce8"; HttpOnly:

curl -v https://my.gateway.fqdn/path
*   Trying 10.102.5.74:443...
* Connected to my.gateway.fqdn (10.102.5.74) port 443 (#0)
* ALPN: offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=my.gateway.fqdn
*  start date: Jan 26 00:00:00 2024 GMT
*  expire date: Feb 23 23:59:59 2025 GMT
*  subjectAltName: host "my.gateway.fqdn" matched cert's "my.gateway.fqdn"
*  issuer: C=US; O=Amazon; CN=Amazon RSA 2048 M02
*  SSL certificate verify ok.
* using HTTP/2
* h2 [:method: GET]
* h2 [:scheme: https]
* h2 [:authority: my.gateway.fqdn]
* h2 [:path: /path]
* h2 [user-agent: curl/8.1.2]
* h2 [accept: */*]
* Using Stream ID: 1 (easy handle 0x7fb1ca80c800)
> GET /path HTTP/2
> Host: my.gateway.fqdn
> User-Agent: curl/8.1.2
> Accept: */*
>
< HTTP/2 200
< date: Fri, 12 Apr 2024 09:58:35 GMT
< content-type: application/json
< x-content-type-options: nosniff
< x-xss-protection: 1; mode=block
< cache-control: no-cache, no-store, max-age=0, must-revalidate
< pragma: no-cache
< expires: 0
< x-frame-options: DENY
< content-security-policy: script-src 'self'
< referrer-policy: no-referrer
< x-envoy-upstream-service-time: 5
< server: istio-envoy
< set-cookie: istio="a9bbe686634a6ce8"; HttpOnly
<
{
    ...
    some_json_output
}

but when I am using curl from a pod within the cluster, hitting the service, I see no cookie:

curl -v http://myservice.namespace.svc.cluster.local:8080/path
*   Trying 10.102.3.57:8080...
* Connected to myservice.namespace.svc.cluster.local (10.102.3.57) port 8080 (#0)
> GET /path HTTP/1.1
> Host: myservice.namespace.svc.cluster.local:8080
> User-Agent: curl/7.88.1
> Accept: */*
>
< HTTP/1.1 200 OK
< x-content-type-options: nosniff
< x-xss-protection: 1; mode=block
< cache-control: no-cache, no-store, max-age=0, must-revalidate
< pragma: no-cache
< expires: 0
< x-frame-options: DENY
< content-security-policy: script-src 'self'
< referrer-policy: no-referrer
< content-type: application/json
< date: Fri, 12 Apr 2024 10:03:12 GMT
< x-envoy-upstream-service-time: 2
< server: envoy
< transfer-encoding: chunked
<
{
    ...
    some_json_output
}

I have enabled debug level in logs and I can see the cookie in the ingressgateway pod logs:

2024-04-12T09:18:46.800592Z	debug	envoy http external/envoy/source/common/http/conn_manager_impl.cc:1869	[Tags: "ConnectionId":"5628","StreamId":"1461258412630245050"] encoding headers via codec (end_stream=false):
':status', '200'
'x-content-type-options', 'nosniff'
'x-xss-protection', '1; mode=block'
'cache-control', 'no-cache, no-store, max-age=0, must-revalidate'
'pragma', 'no-cache'
'expires', '0'
'x-frame-options', 'DENY'
'content-security-policy', 'script-src 'self''
'referrer-policy', 'no-referrer'
'content-type', 'application/json'
'date', 'Fri, 12 Apr 2024 09:18:46 GMT'
'x-envoy-upstream-service-time', '3'
'server', 'istio-envoy'
'set-cookie', 'istio="a9bbe686634a6ce8"; HttpOnly'

but not in the istio-proxy logs of the pod that originates the curl request:

2024-04-12T09:19:42.290888Z	debug	envoy http external/envoy/source/common/http/conn_manager_impl.cc:1869	[Tags: "ConnectionId":"998","StreamId":"8952761916989281991"] encoding headers via codec (end_stream=false):
':status', '200'
'x-content-type-options', 'nosniff'
'x-xss-protection', '1; mode=block'
'cache-control', 'no-cache, no-store, max-age=0, must-revalidate'
'pragma', 'no-cache'
'expires', '0'
'x-frame-options', 'DENY'
'content-security-policy', 'script-src 'self''
'referrer-policy', 'no-referrer'
'content-type', 'application/json'
'date', 'Fri, 12 Apr 2024 09:19:42 GMT'
'x-envoy-upstream-service-time', '2'
'server', 'envoy'

I would greatly appreciate any feedback on why this is not working.