New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Problem with null values #6198
Comments
Hello, VictoriaMetrics/lib/storage/storage.go Lines 1811 to 1816 in 5e8c087
But as I can see, this behavior hasn't changed since VictoriaMetrics/lib/protoparser/vmimport/parser.go Lines 142 to 143 in 5e8c087
So if you import timeseries with all "null" values like your example above, the whole timeseries will be dropped and they won't be counted in
Did you ingest the same values to v1.87.14, set all the values to "null"? |
Hi Haleygo,
We have vmagent sending the same metrics to VM v1.87.14 and v1.93.14 with absolutely same settings. Only v1.93.14 has the issue with registering new timeserries if they have nulls value and hadn't existed before. |
1.93.14
We have process_exporter which is scraped by vmagent and then metrics sent via one more vmagent to two VMs v1.93.14 and v1.87.14 In v1.87.14 i see metric like NaNs in VMUI and like nulls in export: Look's like related to staleness markers somehow but not sure how to reproduce issue from the scratch. |
Managed to reproduce it with promremotecli - just modified it for sending "staleNaNBits uint64 = 0x7ff0000000000002". |
Related to #5069. In VictoriaMetrics, there are two different NaN, one is called staleNaN [using uint64 = 0x7ff8000000000002] VictoriaMetrics/lib/decimal/decimal.go Lines 407 to 409 in d386a68
the other one is NormalNaN as math.NaN() [using uint64 = 0x7ff8000000000001] vmagent or vmsingle only generate Then when it comes to store, VictoriaMetrics can tell the difference between staleNaN and NormalNaN, and only store staleNaN values. VictoriaMetrics/lib/storage/storage.go Lines 1811 to 1816 in 5e8c087
But in v1.93.14, the "NormalNaN" is dropped as always, the "staleNaN" won't be considered as valid value and be dropped as well, so the series won't be registered successfully. |
We don't need to store time series with only NaN values. We want VM to be fast and stable when it (for some reasons) gets a lot of NaNs. |
This looks like a very narrow case. In your example, you're trying to ingest StaleNaNs, a reserved type of NaN for staleness detection. VM accepts StaleNaNs only if series for this sample was registered before with any value different from StaleNaN. There is no sense in recording/registering series which contains only StaleNaNs. To verify if series contains only StaleNaNs, VM does cache and index lookup, which is counted as cache miss and slowInsert. vmagent will create a stale marker in two cases:
I wasn't able to reproduce vmagent to send stale markers with this error. It is likely something weird is happening to vmagents in your setup. Could you try setting |
Describe the bug
Hi,
VictoriaMetrics v1.93.14 processes samples with null values in way.
When VM receives metrics like
it doesn't insert them (can't see them in export or cardinality explorer in VMUI), doesn't count them in active series but counts them as 'slow inserts' and 'cache misses' (storage/tsid).
This is critical issue because it affects performance of the whole ingesting.
Same metrics are processed correctly in v1.87.14 - they are counted in active series, we can access them and no strange 'slow inserts', 'cache misses' and so on.
To Reproduce
Try to insert null values in clean database (so no metrics exists before) to VM v1.93.14
Version
victoria-metrics-20240419-095826-tags-v1.93.14-0-g345a53d8b0
Logs
No response
Screenshots
No response
Used command-line flags
No response
Additional information
No response
The text was updated successfully, but these errors were encountered: