-
Notifications
You must be signed in to change notification settings - Fork 327
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Otel Metrics Not Received #10070
Comments
@stephen-galea-weavr could you share more info about your Grafana agent setup? I've just tested it, with the basic setup from Grafana docs, and it works for me |
Hi, sure this is my setup. This is working fine for all other microservices etc... in the cluster.
|
Hi. I tried to reproduce the issue, and on the first attempt, I had a problem communicating with the agent.
If I understand correctly, the host was resolved to the ClusterIP of a service. Unfortunately, the default service doesn't expose ports for gRPC and HTTP. After adding these ports to the definition of the alloy service, everything started to work. Could you check your service definition and ensure the necessary ports are included? If that is a different error could you check the logs of your sidecar? K8s alloy's service configuration ports:
- name: grpc
port: 4317
protocol: TCP
targetPort: 4317
- name: http
port: 4318
protocol: TCP
targetPort: 4318 alloy config I've used(empty HTTP and gRPC points to provided in an example config) otelcol.receiver.otlp "default" {
grpc {}
http {}
output {
metrics = [otelcol.exporter.logging.default.input]
}
}
otelcol.exporter.logging "default" {
verbosity = "detailed"
sampling_initial = 1
sampling_thereafter = 1
} Policy: apiVersion: kuma.io/v1alpha1
kind: MeshMetric
metadata:
name: otel-metrics
namespace: kuma-system
labels:
kuma.io/mesh: default
spec:
targetRef:
kind: Mesh
default:
backends:
- type: OpenTelemetry
openTelemetry:
endpoint: alloy.obs.svc.cluster.local:4317 |
What happened?
I am using Grafana Agent as an otel collector and is working for other otel sources. However when I set a MeshMetric based on (https://kuma.io/docs/2.7.x/guides/otel-metrics/) I am not receiving any metrics.
This is my config:
I can also confirm that the sidecar is being configured to send metrics
The text was updated successfully, but these errors were encountered: