db/config.cc: increment components_memory_reclaim_threshold config default #18611
Conversation
…fault Incremented the components_memory_reclaim_threshold config's default value to 0.8 as the previous value was too strict and caused unnecessary eviction in otherwise healthy clusters. Fixes scylladb#18607 Signed-off-by: Lakshmi Narayanan Sreethar <lakshmi.sreethar@scylladb.com>
lkshminarayanan
added
backport/5.2
🟢 CI State: SUCCESS✅ - Build Build Details:
|
There was a problem hiding this comment.
@denesb No, what are you doing. 0.8 effectively disables the entire mechanism.
The system gradually adjusts memtable flushing speed to keep memtable memory usage under 0.5 (and stops accepting writes at the extreme), and doesn't even start flushing until it reaches 0.25 0.15.
Bloom filter memory usage isn't going to reach 0.8 because the system is all but guaranteed to OOM at 0.75 0.85, and that's even without counting all the non-cache, non-memtable memory usage.
If you want to prevent OOM crashes with this feature, the threshold must be way under 0.85 (with enough margin to cover all non-LSA, non-bloom memory usage). If you want to guarantee that, then it must be somewhere under (0.5 - margin).
For example, take the last case of bloom explosion from a few days ago (https://github.com/scylladb/scylla-enterprise/issues/4156). The affected node started choking on bad_allocs at bloom fraction of 0.6, and crashed (due to bad_alloc-induced flush failure) at 0.65.
Edit: I misremembered the defaults — we start flushing at 0.15, not at 0.25. Which means that 0.8 might be enough to prevent some OOMs, but that's cutting it very close.
0.6 is way too strict. In another customer case, the cluster is healthy and steady at 0.6 BF memory usage ratio and forcing it below makes IO and latencies jump and results in escalation. So we have to use the max() of all use-cases as the default and lower on deployments where the default is already problematic. |
Steady at 0.6? Which customer case was that? |
https://github.com/scylladb/scylla-enterprise/issues/4183 |
|
@avikivity - per our discussion on Sunday - please review. |
|
The goal of the feature was to prevent OOM. If we use the max() of all clusters, it will never prevent OOM since different clusters have other non-LSA components (prepared statements, cached queriers, running ops, background writes, repair with its large buffers, group 0 holding all the tablets metadata). We can increase it to 0.2, but not beyond. |
If we make the default too conservative, field is just going to disable it on all clusters and then we also aren't preventing OOMs. |
|
@avikivity - please help move this forward. |
Incremented the components_memory_reclaim_threshold config's default value to 0.8 as the previous value was too strict and caused unnecessary eviction in otherwise healthy clusters.
Fixes #18607