New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Explore adding table ids tags to some metrics #4511
Comments
Other than well defined Accumulo system tables (root, metadata,...) it is probably not a good idea to add unbounded tags. Systems that have large numbers of dynamic tables will have issues. |
The following may be relevant if something is done for this issue. https://docs.micrometer.io/micrometer/reference/concepts/meter-provider.html |
Yeah there is big difference there. The monitor metrics only track the tables that currently exist, it will not track any deleted tables. Adding a table id tag for external metrics would cause the metrics system to have data for tables that no longer exists. So that is a huge difference. If anything is done twoards this, would probably need to only add tags for tables that are specified by a user. So could provide a configurable list of tables to tag and only add tags for tables in that list. |
I saw this as well. Additionally, there is a If we are going to do something that will dynamically create Meters, then we need to be sure to remove them (MeterRegistry.remove). Otherwise the process will continue to report the metrics for it's lifetime. |
It may be more useful to tag the metrics with a table name instead of a table id. |
One way to think of it is that each tag will create a unique time-series in some (most?) back ends. So, in addition to the number of metrics that are reported each interval, there can be negative impact to the collection / storage / display systems. |
Would something like tracing work? If we could activate / de-activate tracing on-demand, then it seems that maybe would could collect the needed values for profiling and then turn them off when not being used? No idea what it would look like at this point. |
If tracing provided the level of information needed, then it could be enabled on a subset of the scan servers / tablet servers via settings in accumulo-env.sh. If there are problems with specific queries, those queries could be directed to specific scan servers via the resource group mechanism. |
Yeah, I think for this to be workable would need a user provided list of tables to tag. This could be a metric property with a value of a list of table names or it could be a per table property that is a boolean. So this would allow control over which tables are tagged.
I think that would be out of scope of this issue. This issue could be closed if its not something that seems workable. This issue was about supporting current functionality of the monitor in an external metrics system. Currently in the monitor if a table is clicked on it will show metrics that are per table and per tserver. Also in the tables view in the monitor can see metrics like how many compactions are queued for a table and what the current ingest rate is for a table. |
The code needs to be responsive to changes in the list so that processes don't need to be restarted for metrics to be enabled / disabled. |
For example guidance, Prometheus label guidelines provides the following:,
|
These guidelines aren't general time series database guidelines, right? They are based on the limitations of Prometheus? |
While those are specific to Prometheus - my understanding is that they apply generally across various metric systems. There are similar limitations for things like InfluxDB and it all relates to unique tags (labels) creating a unique time-series. |
Yeah, I get that total cardinality is an issue for most of the TSDBs, but I think limitations are different for each. My point was that we should not take the numbers from Prometheus as a hard limit. |
Is your feature request related to a problem? Please describe.
The monitor currently has a custom metrics system. This custom system in the monitor tracks certain metrics per tablet server per table id. The monitor currently lacks any metrics for scan severs that are similar to the current monitor tablet server metrics. There are metrics being emitted by scan servers that almost have enough information to construct a scan server metrics dashboard in an external metrics system, however the lack of table id information prevents having parity with the tablet server view in the monitor.
In general, if table ids were added to some subset of tablet server and scan server metrics then it could allow an external metrics system to achieve parity with the metrics offered by the monitor. The metrics in the monitor that do this are primarily concerned with table read, write, and compactions.
Describe the solution you'd like
For a subset of metrics related to reading and writing data tag them with table ids. This would like result in a
Map<TableId, Meter>
in the code. For this map would need to be able to add new meters as new table ids are encountered and remove table ids that have not been used in a while. The meters in this map would have the table id tag.Describe alternatives you've considered
Do not try to achieve feature parity in external metrics and custom monitor metrics.
The text was updated successfully, but these errors were encountered: