envoyMetrics

In this project, We will demo how to expose the http response time in [Prometheus] format through Envoy.

This demo has three components: App, Envoy and statsd_exporter. These three components will deployed in three Docker containers.

How to deploy this demo in Kubernetes can be found here.

Overview of the deployment of the demo

App

This App will simulate a http service, which will have a latency of specified milliseconds for each http request.

Start this App in Docker container:

cd components/app
sh run.sh

If the app works, it can be accessed via http://localhost:8080/workload.php/?value=100.

Envoy

Envoy will work as a http proxy for the App: All http requests for the App will go through Envoy, and Envoy will monitor the http response time for the requests.

Before start the Envoy, provide a correct configuration file for Envoy image:

cd components/envoy
vim ./conf/envoy.json  ## modify this file according to your host IP address
sh run.sh

If everything works fine, the app can be accessed via http://localhost:9090/workload.php/?value=100.

In addition, some metrics (not including the response time) can be accessed via http://localhost:8001/stats.

Statsd_exporter

The timer metris of Envoy are only forwarded to statsd server (in this demo, it is statsd_exporter). Two functions of statsd_exporter in this demo:

• Receive all the metrics (including the response time);
• Convert the metrics from statsd format into Prometheus format.

Start statsd_exporter in Docker container:

cd components/statsd_exporter
sh run.sh

If everything works fine, the metrics can be accessed via http://localhost:9102/metrics.

Make sure envoy_cluster related metrics can be found in the results of http://localhost:9102/metrics. If not, check the two settings in Envoy configuration file:

• address of statsd_exporter
• address of App

Replace the two IP addresses to a non-loopback address.

Sample metrics from statsd_exporter

...
envoy_cluster_manager_total_clusters 1
# HELP envoy_cluster_video_service_external_upstream_rq_200 Metric autogenerated by statsd_exporter.
# TYPE envoy_cluster_video_service_external_upstream_rq_200 counter
envoy_cluster_video_service_external_upstream_rq_200 1672
# HELP envoy_cluster_video_service_external_upstream_rq_2xx Metric autogenerated by statsd_exporter.
# TYPE envoy_cluster_video_service_external_upstream_rq_2xx counter
envoy_cluster_video_service_external_upstream_rq_2xx 1672
# HELP envoy_cluster_video_service_external_upstream_rq_time Metric autogenerated by statsd_exporter.
# TYPE envoy_cluster_video_service_external_upstream_rq_time summary
envoy_cluster_video_service_external_upstream_rq_time{quantile="0.5"} 104
envoy_cluster_video_service_external_upstream_rq_time{quantile="0.9"} 106
envoy_cluster_video_service_external_upstream_rq_time{quantile="0.99"} 109
envoy_cluster_video_service_external_upstream_rq_time_sum 173175
envoy_cluster_video_service_external_upstream_rq_time_count 1673
...