How to set Envoy --drain-time-s more than 1h

[s-dwinter]

Istio sets the maximum duration to 1 hour, but I want to set it to 100 hours.

istio/pkg/config/validation/validation.go

Line 71 in c5463a0

drainTimeMax = time.Hour

I would like to know how to bypass the validation and set the duration to 100h.

[ramaraochavali]

What is the use case you are trying to solve?

[s-dwinter]

Thank you reply! I'm testing a Blue/Green deployment for TCP Proxy using the manifests as follows.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: tcp-bg-test
spec:
  gateways:
    - istio-ingressgateway
  hosts:
    - example.com
  tcp:
    - match:
        - port: 21935
      route:
        - destination:
            host: example.default.svc.cluster.local
            port:
              number: 11935
            subset: blue
          weight: 100
        - destination:
            host: example.default.svc.cluster.local
            port:
              number: 11935
            subset: green
          weight: 0
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: example
spec:
  host: example.default.svc.cluster.local
  subsets:
    - labels:
        version: blue
      name: blue
    - labels:
        version: green
      name: green
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN

When I change the weight to switch between Blue and Green, The existing TCP connections, currently linked to Blue, are terminated. This behavior was unexpected as I hoped that updating the weight would not affect existing connections.
Upon further investigation, I found out that the --drain-time-s command line option in Envoy allows for setting the duration before existing connections are terminated. However, Istio has set the maximum limit to 1h.

[ramaraochavali]

AFAIK, there is no draining for TCP proxy (if you are using Envoy TCP proxy). And how do you expect it to work if you increase it to 100h - would n't 1 hour draining not sufficient if at all the draining works for other protocols like http/redis?

[s-dwinter]

I believe it's almost certain that modifying the weight will trigger a drain. Here are the reasons:

• According to Envoy's documentation, modifying the weight results in an LDS update, which leads to a drain.
  • https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/operations/draining

  • Draining occurs at the following times:

    • ...
    • Individual listeners are being modified or removed via LDS.

• When measuring the time until disconnection actually occurs, connections were severed in about 45 seconds.
  • This coincides with the default value of Istio's drain-time-s.

And how do you expect it to work if you increase it to 100h - would n't 1 hour draining not sufficient if at all the draining works for other protocols like http/redis?

I agree that 1 hour should be sufficient for HTTP and other standard protocols. However, the protocol I intend to use is RTMP, which is crucial for receiving video streams from end-users.
Disconnection of existing TCP connections will stop the video distribution to end-users, necessitating a longer disconnection time to prevent this.

I believe there is no upper limit to the --drain-time-s setting in Envoy. What could be the reason for Istio to set a limit? I would be most pleased if this limit were removed.

[ramaraochavali]

if RTMP is configured as TCP in Envoy, TCP does not have a draining behaviour. so it just waits for 100h blindly. Is that what you are looking for? I think there is no specific reason to apply that 1h limit in Istio. We can relax if that helps?

[s-dwinter]

if RTMP is configured as TCP in Envoy, TCP does not have a draining behaviour. so it just waits for 100h blindly. Is that what you are looking for?

Yes. I expect Envoy to maintain existing TCP connections without interruption for up to 100h.

I think there is no specific reason to apply that 1h limit in Istio. We can relax if that helps?

If there is no specific reason for that 1h limit, I would appreciate it if it could be removed or relaxed.

[ramaraochavali]

#50508

[s-dwinter]

Thank you very much!