Incorrect label saved when writing metrics with prometheus remote write after flush memtables #3881
Comments
|
Thanks for the feedback. I tried to reproduce this issue on the same build but failed.
cat <<EOF | base64 --decode | curl -i -XPOST --data-binary @- -H "Content-Encoding: compress" localhost:4000/v1/prometheus/write
yAHICjAKEAoIX19uYW1lX18SBHRlc3QKEwoKdGVzdF9sYWJlbBIFMTBtaW4SBxC4qbqv7DEKqjIAEPjEs7zzkjIAMGRhaWx5EgcQgPvL+um2MgAQu5fE7DE=
EOF
cat <<EOF | base64 --decode | curl -i -XPOST --data-binary @- -H "Content-Encoding: compress" localhost:4000/v1/prometheus/write
O+gKOQoQCghfX25hbWVfXxIEdGVzdAoTCgp0ZXN0X2xhYmVsEgUxMG1pbhIQCZqZmZmZmbk/EPi0x8XzMQ==
EOF
mysql> select *, cast(greptime_timestamp as bigint) as ts from test order by greptime_timestamp desc;
+---------------------+----------------+------------+---------------+
| greptime_timestamp | greptime_value | test_label | ts |
+---------------------+----------------+------------+---------------+
| 2024-05-02 09:45:31 | 0.1 | 10min | 1714643131000 |
| 2024-05-02 04:25:31 | 0 | 10min | 1714623931000 |
| 2024-04-10 15:00:00 | 0 | daily | 1712761200000 |
| 2024-04-10 02:55:31 | 0 | 10min | 1712717731000 |
| 2024-04-02 15:00:00 | 0 | daily | 1712070000000 |
+---------------------+----------------+------------+---------------+
5 rows in set (0.01 sec)
|
|
oh ok, I sorted the timestamps for the first four metrics. You can try with: cat <<EOF | base64 --decode | curl -i -XPOST --data-binary @- -H "Content-Encoding: compress" localhost:4000/v1/prometheus/write
yAHQCjAKEAoIX19uYW1lX18SBHRlc3QKEwoKdGVzdF9sYWJlbBIFZGFpbHkSBxCA+8v66TEKMAqCMgAwMTBtaW4SBxC4qbqv7JIyAABkEWQIu5fEljIANDEwbWluEgcQ+MSzvPMx
EOFcat <<EOF | base64 --decode | curl -i -XPOST --data-binary @- -H "Content-Encoding: compress" localhost:4000/v1/prometheus/write
O+gKOQoQCghfX25hbWVfXxIEdGVzdAoTCgp0ZXN0X2xhYmVsEgUxMG1pbhIQCZqZmZmZmbk/EPi0x8XzMQ==
EOFgot
the config file that I use [wal]
file_size = "256MB"
purge_threshold = "4GB"
purge_interval = "1m"
read_batch_size = 128
sync_write = false
# Mito engine options
[[region_engine]]
[region_engine.mito]
num_workers = 8
worker_channel_size = 128
worker_request_batch_size = 64
manifest_checkpoint_distance = 10
compress_manifest = false
max_background_jobs = 4
auto_flush_interval = "2m"
global_write_buffer_size = "1GB"
global_write_buffer_reject_size = "2GB"
sst_meta_cache_size = "128MB"
vector_cache_size = "512MB"
page_cache_size = "512MB"
sst_write_buffer_size = "8MB"
scan_parallelism = 0
parallel_scan_channel_size = 32
allow_stale_entries = false |
|
We've confirmed the issue and will fix it ASAP. |
|
This bug is related to the internal encoding of GreptimeDB's memtable. In greptimedb/src/mito2/src/memtable/partition_tree/tree.rs Lines 135 to 136 in 314f270 and we store the mapping from sparse key to pk index: While in the inner and we also store the mapping from mem-comparable encoded primary keys in a dictionary: But when a PartitionMemtable gets freezed, for example, flushed onto disk, the inner dictionary may re-order the keys: For some particular primary key combanitions, the sparsely-encoded bytes and the mem-comparable encoded bytes may have two different pk indices (which means the byte order of sparse encoding and mcmp encoding are different), so the last inserted metric has an incorrect label. #3927 fixes this issue by also carrying a sparsely encoded key in the inner dictionary so that when dictionary is reordered, it can updates the outer mapping: @0neSe7en Big thanks for your feedback! |
|
awesome, thanks for your work and explanation |
What type of bug is this?
Incorrect result
What subsystems are affected?
Standalone mode, Storage Engine, Datanode
Minimal reproduce step
starts a clean greptime database
save these four metrics with prometheus remote write
{ "records": { "schema": { "column_schemas": [ { "name": "greptime_timestamp", "data_type": "TimestampMillisecond" }, { "name": "greptime_value", "data_type": "Float64" }, { "name": "test_label", "data_type": "String" } ] }, "rows": [ [ 1712717731000, 0.0, "10min" ], [ 1714623931000, 0.0, "10min" ], [ 1712070000000, 0.0, "daily" ], [ 1712761200000, 0.0, "daily" ] ] } }wait until the first memtable flush finished.
write a new metric with prometheus remote write
Then
select * from test_metric order by greptime_timestamp descWhat did you expect to see?
What did you see instead?
Then it changed back to the expect value
10minafter restart the greptimeWhat operating system did you use?
macOS 14.2.1 (23C71)
What version of GreptimeDB did you use?
commit: 9038e1b version: 0.7.2
Relevant log output and stack trace
The text was updated successfully, but these errors were encountered: