Changing MutableState cache to host-level by default #5894
Conversation
| @@ -1254,12 +1254,12 @@ will wait on workflow lock acquisition. Requires service restart to take effect. | |||
| ) | |||
| EnableHostHistoryCache = NewGlobalBoolSetting( | |||
| "history.enableHostHistoryCache", | |||
| false, | |||
| true, | |||
There was a problem hiding this comment.
Is this enabled in functional tests?
There was a problem hiding this comment.
let me check that.
There was a problem hiding this comment.
It is not enabled in functional tests. But it is not disabled explicitly. So after merging this change, it should be enabled.
I am explicitly setting cli dev server cache size to be 8k in dynamic config yaml file: https://github.com/temporalio/temporal/pull/5894/files#diff-18c8bb8717f6896034d89c0eb07448ffce8fe45aeb752362b798ce18ef22e92f This should limit the cache size in dev server. |
The cli ( I'd also recommend trying to keep those files as short as possible. In local testing we should run as close to the default as possible. |
Right! Default host level cache size is 128K entries. That is very large for local clusters. Previously it was per-shard and cli server only has 1 shard. Do you think if setting this dynamic config key(cache size) in cli a good idea? |
What changed?
We had added a configuration flag to make MutableState cache a host-level cache.
Changing dynamic config to enable this flag by default and adjusting cache size to 128K. We found that this size is appropriate for most workloads we see.
Also setting host-level cache size to be 8K in development dynamic config.
Why?
Host level cache is better for cache utilization and it prevents the problem of a smaller shard-level cache getting full with pinned elements.
How did you test it?
Running the cluster with load.
Potential risks
None
Documentation
Is hotfix candidate?
No