Istio-proxy pods consuming high cpu for few hundreds of requests

[krishna-aw]

istio version: 1.15.7
EKS: 1.26

The istio-proxy pods are consuming lots of CPU for few hundreds(~400) of requests. We have a HPA setup so that if CPU utilization reaches 80%, they scale up. Due to this several replicas(~30) are created. How do I troubleshoot to find the source of high CPU usage?

Here is the number of requests graph:

The CPU usage graph

Number of Pods scaled up by HPA during this time period

[zirain]

can you check the number of stats by running istioctl x es <podname>.<namespace> -oprom|wc -l