Istio Graceful Shutdown #44457
Replies: 2 comments
-
I am not 100% if I follow if you want the endpoints to keep working, or to stop working. If you want it to keep working: it is your application container shutting down, you need to make it catch the SIGTERM and not exit immediately. Its not Istio exiting early, but your app If you want it to stop working: set terminationDrainDuration=1ms or something, and envoy will exit immediately when terminating. Probably you don't want this. |
Beta Was this translation helpful? Give feedback.
-
You can configure Istio to exclude the /metrics endpoint from the graceful termination period using a VirtualService. This ensures requests to /metrics bypass Istio's sidecar proxy, allowing access during pod termination. Here's a sample YAML:
|
Beta Was this translation helpful? Give feedback.
-
I am trying to understand if there is a way to avoid that while some pods (which have the injected sidecar) are in the Terminating phase, if Istio in the meantime is able to intercept any requests toward the service and avoid that those got propagated.
I mean, this is what I am observing:
My application is exposing some metrics on the
/metrics
endpoint (Prometheus integration).Basicly while killing the pod, and while the pod is in the Terminating phase (graceful shutdown intercept from the application), I still see some request towards the endpoint, that's why I see :
Apr 19 17:48:12 service-7589696bd7-5lv2d istio-proxy error failed scraping application metrics: error scraping http://localhost:3260/metrics: Get "http://localhost:3260/metrics": dial tcp [::1]:3260: connect: connection refused
I would expect that this should not happen, but I did not get why it is, so I assume that the sidecar did not realize that the other container is dying.
I would expect that those kind of request will be then discarded since the
/endpoint
is not valid (service dying).Is there any way for that?
Imagine the following scenario:
while running I performed into one of those on the istio-proxy sidecar:
kubectl exec -it example-c4944957-hfb4h bash -c istio-proxy
and then
curl -i -X GET http://localhost:3260/metrics
endpoint and it works correctly.On another tab I deleted the previous pod
example-c4944957-hfb4h
From this,
example-c4944957-hfb4h
, on the istio-proxy sidecar still active I try again calling the previous endpoint and I get :to notice that the application is still running, making a graceful shutdown and the
/metrics
endpoint should answer to such requests.I would expect something different since I have:
Basicly while the main app is dying I would like to mantain
/metrics
endpoint enabled.Maybe is there a way to exclude from the Graceful Termination period (I expect it is stopping any new requests) any specific endpoint I would still manage while the main app is dying.
Beta Was this translation helpful? Give feedback.
All reactions