send custom headers in tracing backend requests

[jmazzitelli]

fixes: #7266

[jmazzitelli]

I think the only thing left to do is figure out how to get custom headers sent via the gRPC client that is created here:

https://github.com/kiali/kiali/blob/v1.84.0/tracing/client.go#L150-L158

conn, err := grpc.Dial(address, opts...)
if err == nil {
	cc := model.NewQueryServiceClient(conn)
	client, err = jaeger.NewGRPCJaegerClient(cc)
	if err != nil {
		return nil, err
	}
	log.Infof("Create %s GRPC client %s", cfgTracing.Provider, address)
	return &Client{httpTracingClient: httpTracingClient, grpcClient: client, ctx: ctx}, nil
...

Need to investigate the APIs used here because it isn't clear to me if you can set client-side headers to be sent over the underlying HTTP/2 transport. (UPDATE: latest commits do this. So this is done).

[jmazzitelli]

1. With this PR checked out, build the server: make build build-ui test
2. Start a cluster with Kiali (using what you built in step 1) w/Tracing enabled: hack/run-integration-tests.sh -so true
3. Add a custom header to the Tracing config: kubectl edit cm kiali -n istio-system

external_services:
  tracing:
    url: http://tracing.istio-system:16685/jaeger
    custom_headers:
      X-MazzWasHere_Foo: mazz_WasHere

4. Restart the Kiali pod to pick up the change: kubectl delete pod -n istio-system -l app.kubernetes.io/name=kiali
5. Confirm the header was added to the client:

$ kubectl logs -n istio-system -l app.kubernetes.io/name=kiali --tail=-1 | grep "custom headers"
2024-05-17T14:27:30Z TRC Adding [1] custom headers to Tracing client

6. At this point it would be nice to actually see the header in the HTTP request, but you need to use kubeshark to do it. If you are so inclined, install kubeshark and look at the traffic going from kiali to the tracing pod and you'll see the header.

[jmazzitelli]

Using Tempo without gRPC (hack/setup-kind-in-ci.sh -t), I can see the headers using kubeshark:

[jmazzitelli]

latest commit has it working with grpc:

[jmazzitelli]

This is all ready. I was able to confirm the header is passed with using gRPC and non-gRPC.

[jmazzitelli]

@josunect I added you as a reviewer to take a look at the code since you are most familiar with it. The person who asked for this feature tested it in his cluster and it works for him, and I tested it (as you see above) and it works for me, too. I just want to make sure the code doesn't break something else that I cannot see.

[jshaughn]

I didn't run it because the community originator of the RFE already tested it successfully. Code looks good for me.

[josunect]

I haven't tested it yet, I've just added one comment after looking at the code. Except for that, it looks good!

[josunect]

I can't add the comment in the specific line, but I think the ctx should also need to be added in the Tempo grpc client? This would be in line 195:
clientConn, _ := grpc.Dial(grpcAddress, dialOps...)

I think that could be replaced with something like:

clientConn, _ := grpc.DialContext(ctx, grpcAddress, dialOps...)

[jmazzitelli]

I made this change - see commit abb5e7a

[jmazzitelli]

The failure is the known flake unrelated to this PR

[jshaughn]

Re-approving, +1 to Josune's suggested change.