New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
vmagent panic on remoteWrite.streamAggr.dedupInterval #6205
Comments
Thanks for report! |
It only happens when you have multiple remotewrite targets with:
The remotewrite (with deduplicator)
And clear(tss)
While the remotewrite (without deduplicator)
And here's the critical part:
The goroutine (without deduplicator) refer timeseries data with index tsSrc := &src[i] , where the timeseries data might be cleared.
While the goroutine(with deduplicator) refer timeseries data with a copy:
It could be reproduced whenever you have:
Hope this could help |
@alexintech just curious if you change the order - |
It'd be great to build vmagent with race detector: Note, it significantly reduces performance of application and must be used only for testing. |
the most obvious reason is this as mentioned by @jiekun, I've reproduces an issue as well and I've tested these changes |
The same error, but it crashes quicker, just after the start.
I'll check |
seems that it's working! |
…multiple remote write contexts (#6206) When at least one remote write has deduplication configured it cleans up timeseries while they can be in use by another remote write without deduplication #6205 --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com>
…multiple remote write contexts (#6206) When at least one remote write has deduplication configured it cleans up timeseries while they can be in use by another remote write without deduplication #6205 --------- Signed-off-by: hagen1778 <roman@victoriametrics.com> Co-authored-by: hagen1778 <roman@victoriametrics.com> (cherry picked from commit 8797718)
Re-opening issue since #6206 isn't released yet. It will be included into the next release. |
Describe the bug
vmagent crashes periodically when the
-remoteWrite.streamAggr.dedupInterval="0s,120s"
flag set.To Reproduce
vmagent configured with
remoteWrite.streamAggr.dedupInterval
configuration:Version
./vmagent-prod --version
vmagent-20240425-145801-tags-v1.101.0-0-g5334f0c2c
Logs
Screenshots
No response
Used command-line flags
command-line flags
-httpListenAddr=":8429"
-promscrape.config="/etc/vmagent/config_out/vmagent.env.yaml"
-promscrape.streamParse="true"
-remoteWrite.maxDiskUsagePerURL="1073741824"
-remoteWrite.streamAggr.dedupInterval="0s,2m0s"
-remoteWrite.tmpDataPath="/vmagent_pq/vmagent-remotewrite-data"
-remoteWrite.url="secret"
Additional information
No response
The text was updated successfully, but these errors were encountered: