-
Notifications
You must be signed in to change notification settings - Fork 5.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Extend kube_inventory plugin to support Custom Resource metrics #14917
Comments
Hi, Assume I don't know much about this plugin and how it is used ;)
Looking briefly at the plugin it has a list of collectors, where it goes through and pulls metrics from pods, nodes, secrets, etc. Are you wanting to extend this list to allow the user to collect from some other resource? What resource is missing the list that isn't already captured? Thanks |
Hey @powersj, |
As I asked above, can you give me an example?
I am not a K8s expert, so I don't know this :) Can you explain what an example would be please? |
Ohh ok @powersj, So can we have a feature similar to this which would allow us to monitor custom resources as well as any other resources which are not there in the default list of resources monitored by Telegraf. |
Thanks for the pointers and additional background. I spent some time looking to see if or how the kubernetes/client-go can talk to CRDs to collect information about them. I saw some talk about talking directly to them, but others talk about building custom REST clients. I looked through this blog post where the user had to define their own REST client + pass the custom resource and a RedHat post on CRDs. With my limited understanding, it doesn't seem practical to create a generic CRD client in Telegraf. Given the need to pass a CRD's API this certainly could vary from CRD to CRD. It may make more sense for a user of Telegraf to create their own client that pulls down the data from whatever CRD API they need and return metrics via the Thoughts? |
Yes, i also went through some of the links and using the I was thinking, the user can provide a list of CRDs as part of config that needs to be monitored, and we can have a function like The struct should look something like this -
this struct would be of type array and gets populated based on the CRDs provided in the config. Does this sound feasible ? |
What does the return value of the the REST call look like? Given you could have many different CRDs that I assume could produce different data, how would telegraf know how to transform that result into metrics? |
I think we would want scrape pretty generic metrics which should be common across all CRDs. These metrics could be like - Maybe adding support for custom metrics specific to a CRD could be a thing for future but this is a good starting point. |
I asked for an example of the response, do you have one to share?
It was my understanding that it was up the creator of the CRD as to what this endpoint produces. Is that correct? If so how would we pick "generic metrics"? |
So CRD basically allows us to define a custom resource which has set of specifications, we are not worried about that. K8s CRD and Custom resource doc -> https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/ |
Are you planning to put up a PR? Otherwise I need your help to provide some answers to my questions. I have asked multiple times now to understand what we would need to do for development. If you are not going to provide one, then let's close this. The link you provided tells me nothing that I can see about getting metrics from a CRD. It is all about creating one. |
I have literally provided you all the information that you have asked for. I don't understand what else do you need ? I am not sure what else do you need ? |
We are using the k8s client library as it can help manage authentication and calls for us. The existing client in use does not seem to have a way to talk directly to CRDs or get information about them. Have I misunderstood that? You have made the leap to just calling API endpoints, but ignores the entire set of authentication and other existing tooling that exists with using the k8s client. While there is some use of a generic http client for kubelets, I'd like to avoid extending that usage. I have asked twice from you to get an example response from that API. I want to see the full API response so we can get an idea of the metrics that Telegraf would create. |
AFAIU, the existing client should be able to query and fetch the custom resource. The client implements the RESTCLIENT interface and that can be reused. Since it is the same client, i don't think we should face any auth issues. (this is my assumption). I am assuming we can store the data in an
Let me look further into it, will get back and see if it is possible or not. |
Nice find with the dynamic client. What do you consider to be the next steps for your branch? Happy to see a PR! |
@powersj can you assign the issue to me ? |
So far it does. What would be the remaining work? I assume something around parsing the metrics? |
@powersj yes, we need to add more metrics, currently i have only tested it for collecting the created time metric. |
@Shubhama19 - were you able to get something in a state where you wanted to put up a PR? |
Hello! I am closing this issue due to inactivity. I hope you were able to resolve your problem, if not please try posting this question in our Community Slack or Community Forums or provide additional details in this issue and reqeust that it be re-opened. Thank you! |
Use Case
This feature would allow users to configure telegraf to fetch state metrics for Custom resources. Users should provide a list of CRDs and the gauges that needs to be monitored by telegraf.
This can either be provided as a config to telegraf or as input flags in telegraf deployment yaml.
This is similar to how native
kube-state-metrics
supports the monitoring for custom resource.https://github.com/kubernetes/kube-state-metrics/blob/main/docs/customresourcestate-metrics.md
Expected behavior
Users should be able to monitor Custom resource State metrics.
Actual behavior
Currently there is no way to retrieve metrics for resources other than the ones supported out of the box. This feature would provide extensibility.
Additional info
No response
The text was updated successfully, but these errors were encountered: