VictoriaMetrics linux-ppc64le build crashes in runtime

[SlavaLatokhin]

Describe the bug

After 3 hours of maximum write load from two vmagent VictoriaMetrics crashes with the error "fatal error: concurrent map read and map write".

To Reproduce

3 hours of write load from two vmagents with maximum CPU usage on a computer with 100 cores with ppc64 architecture. At the end of the third hour, VictoriaMetrics drops

Version

(victoria-metrics-linux-ppc64le-prod)
victoria-metrics-20240404-181724-heads-master-0-g0dda3978a-dirty-31074ca3

Logs

https://pastebin.com/ZZ6kxKf2
https://pastebin.com/1becPs6f#Xtj3DjAg

Screenshots

No response

Used command-line flags

victoria-metrics-linux-ppc64le -httpListenAddr=":8426" -storageDataPath=$base_path/vmstorage-data-60 -enableTCP6 -retentionPeriod=1y
vmagent-linux-ppc64le -enableTCP6 -httpListenAddr=":8436" -remoteWrite.url={http://sp-0:8426/api/v1/write,http://sp-1:8426/api/v1/write} -remoteWrite.tmpDataPath=$base_path/vmstorage-tmp-data-60

Additional information

No response

[f41gh7]

Can you please also share build environment details:
version of GO compiler.

[SlavaLatokhin]

The build was done via docker, most likely the default version

[f41gh7]

@SlavaLatokhin may I ask you to build victoria-metrics with param -race. It enables race detector and must help to identify source of the problem.

make victoria-metrics-race
# produces bin/victoria-metrics-race binary

Please note it has significantly lower performance compared to production builds.

[SlavaLatokhin]

I can't compile, I can't find at all how to configure zstd for a local build under ppc64le. Please tell me how do you build VictoriaMetrics for ppc64le? There is no c library for ppc64le in gozstd or on the Internet

[SlavaLatokhin]

@f41gh7, do you have zstd for ppc64le? Or maybe you can throw in an already compiled version or know the workarounds

[f41gh7]

I can't compile, I can't find at all how to configure zstd for a local build under ppc64le. Please tell me how do you build VictoriaMetrics for ppc64le? There is no c library for ppc64le in gozstd or on the Internet

It's a problem, application cannot be compiled with -race if CGO is disabled for given arch.

I'll take a look at compilation issues. Most probably, we have to add ppc64le to gozstd. And it should unblock CGO compilation.

[f41gh7]

It should be possible to build -race version of VictoriaMetrics for ppc64le with following changes (See example at draft PR)

1. replace gozstd lib at go.mod: add the following line: replace github.com/valyala/gozstd => github.com/f41gh7/gozstd v1.12.1-0.20240514104918-c42dd7d06806
2. execute command go get github.com/valyala/gozstd
3. execute command go mod vendor.

Unfortunately, I dont have any power8 instances to properly test it. And local cross-compilation doesn't work for me.

example PR https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6271/files