istio grpc vs grpc-web service selection issue

[achovelon]

Is this the right place to submit this?

• This is not a security vulnerability or a crashing bug
• This is not a question about how to use Istio

Bug Description

Hello,

I am using istio to expose grpc and grpc-web protocols on the same application.
On my kubernetes namespace, I have 1 deployment which is my grpc server, and 2 services : one for grpc and another for grpc-web with grpc filters. On my service spec, I specify the protocol (grpc or grpc-web), and I use 2 separate DNS: grpc.test.com and grpc-web.test.com. For this issue I will only talk about the grpc-web DNS : grpc-web.test.com which is connected to grpc-web service.
Now when I deploy my chart with these 3 resources, web app grpc-web.test.com only works if the first service created is the grpc-web - the one with grpc web filters. If not, all the grpc-web traffic is routed over the grpc service (which is the first one created in this case) and it fails because it has no filters.

I used this documentation for explicit protocol selection with service name (grpc/grpc-web) :
explicit-protocol-selection

Steps to reproduce

To reproduce the problem it's simple, if you have a grpc server and 2 services grpc + grpc-web, you can delete and recreate the grpc-web service : even after recreation the traffic will continue to be routed on the grpc service (without filter).

logs

This is logs from istio gateway:

when not working (grpc service created before grpc-web)

istio logs

2024-04-25T16:00:51+02:00 [2024-04-25T14:00:50.468Z] "OPTIONS /back.testgrpc.SomethingController/RaiseGrpcException HTTP/1.1" 200 - cors_response - "-" 0 0 0 - "82.210.2.233,192.168.8.2,10.250.0.48" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36" "51f1a083-5441-4f76-81ee-7fba85aa2c99" "grpc-web.test.com" "-" outbound|80||grpc-web.test.svc.cluster.local - 10.42.123.188:8080 82.210.2.233:0 - - traceID=- grpc_status=-

2024-04-25T16:00:51+02:00 [2024-04-25T14:00:50.491Z] "POST /back.testgrpc.SomethingController/RaiseGrpcException HTTP/1.1" 503 URX via_upstream - "-" 5 85 40 39 "82.210.2.233,192.168.8.2,10.42.125.65" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36" "866f789f-cb9a-477b-8501-13484acf32a3" "grpc-web.test.com" "10.42.190.208:8000" outbound|80||grpc-web.test.svc.cluster.local 10.42.123.188:40754 10.42.123.188:8080 82.210.2.233:0 - - traceID=17f7714648ff1c47f1f43b932f8ba696 grpc_status=-

application logs

2024-04-25T16:00:06+02:00 D0425 14:00:06.708340622      32 server.cc:1637]                       Failed call creation: UNKNOWN:Missing :te header {grpc_status:2}
2024-04-25T16:00:06+02:00 D0425 14:00:06.730257865      32 server.cc:1637]                       Failed call creation: UNKNOWN:Missing :te header {grpc_status:2}
2024-04-25T16:00:09+02:00 E0425 14:00:09.380900898      32 hpack_parser.cc:999]                  Error parsing 'content-type' metadata: invalid value
2024-04-25T16:00:09+02:00 D0425 14:00:09.381074793      32 server.cc:1637]                       Failed call creation: UNKNOWN:Missing :te header {grpc_status:2}
2024-04-25T16:00:09+02:00 D0425 14:00:09.403796601      32 server.cc:1637]                       Failed call creation: UNKNOWN:Missing :te header {grpc_status:2}
2024-04-25T16:00:09+02:00 D0425 14:00:09.421239629      32 server.cc:1637]                       Failed call creation: UNKNOWN:Missing :te header {grpc_status:2}

when working fine (grpc-web service created before grpc)

istio logs

2024-04-25T16:04:33+02:00 [2024-04-25T14:04:33.580Z] "OPTIONS /back.testgrpc.SomethingController/Create HTTP/1.1" 200 - cors_response - "-" 0 0 0 - "82.210.2.233,192.168.8.2,10.42.193.65" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36" "c199d647-6f4a-4b03-a5cf-0d286ba5054a" "grpc-web.test.com" "-" outbound|80||grpc-web.test.svc.cluster.local - 10.42.123.188:8080 82.210.2.233:0 - - traceID=- grpc_status=-

2024-04-25T16:04:33+02:00 [2024-04-25T14:04:33.603Z] "POST /back.testgrpc.SomethingController/Create HTTP/1.1" 200 - via_upstream - "-" 22 95 22 21 "82.210.2.233,192.168.8.2,10.42.120.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36" "e4b107a7-a5f9-4652-a495-f592a34dfd53" "grpc-web.test.com" "10.42.190.208:8000" outbound|80||grpc-web.test.svc.cluster.local 10.42.123.188:59008 10.42.123.188:8080 82.210.2.233:0 - - traceID=4a1c881580873d07485659d5182199a1 grpc_status=-

application logs

2024-04-25T16:04:32+02:00 level=INFO name=django_grpc.request line=/opt/code/.venv/lib/python3.12/site-packages/django_grpc/services/servicer_proxy.py:329 message="OK : Something/Create" service_name="Something" action="Create" time=2024-04-25T14:04:32.562000 levelno=20 usermanagement_uuid="" user_roles="User,Developer,User" client_id="" funcName=log_response trace_id=20b86c68c6f640ca68b6241c4099d35c span_id=c6fec09e727e1d41 resource.service.name=back status_code=0 service_name="Something" action="Create" usermanagement_uuid="" user_roles="User,Developer,User"

It seems that istio does not distinguish between grpc and grpc-web. Is this a bug or something I misunderstood ?

Version

$ istioctl version
client version: 1.20.4
control plane version: 1.20.4
data plane version: 1.20.4 (92 proxies)
$ kubectl version
Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.27.4+rke2r1

Additional Information

service-grpc-web.yaml

kind: Service
metadata:
  name: service-grpc-web
  namespace: test
spec:
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: grpc-web
    port: 80
    protocol: TCP
    targetPort: 8000
  selector:
    app: server-grpc
  sessionAffinity: None
  type: ClusterIP

virtualservice-grpc-web.yaml

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: grpc-web-api
  namespace: test
spec:
  gateways:
  - grpc-web-api
  hosts:
  - grpc-web.test.com
  http:
  - corsPolicy:
      allowCredentials: true
      allowHeaders:
      - grpc-timeout
      - content-type
      - keep-alive
      - user-agent
      - cache-control
      - content-type
      - content-transfer-encoding
      - custom-header-1
      - x-accept-content-transfer-encoding
      - x-accept-response-streaming
      - x-user-agent
      - x-grpc-web
      - filters
      - pagination
      - headers
      - traceparent
      - depth
      - filter
      - disable_pagination
      - project
      - project_id
      allowMethods:
      - POST
      - GET
      - OPTIONS
      - PUT
      - DELETE
      allowOrigin:
      - '*'
      exposeHeaders:
      - custom-header-1
      - grpc-status
      - grpc-message
      - filters
      - pagination
      - headers
      - project_id
      maxAge: 1728s
    match:
    - uri:
        prefix: /
    route:
    - destination:
        host: service-grpc-web
        port:
          number: 80

service-grpc.yaml

apiVersion: v1
kind: Service
metadata:
  name: service-grpc
  namespace: test
spec:
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: grpc
    port: 80
    protocol: TCP
    targetPort: 8000
  selector:
    app: server-grpc
  sessionAffinity: None
  type: ClusterIP

virtualservice-grpc.yaml

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: grpc-api
  namespace: test
spec:
  gateways:
  - grpc-api
  hosts:
  - grpc.test.com
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: service-grpc
        port:
          number: 80