Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enforce object store total memory size constraint during recovery #359

Open
wants to merge 7 commits into
base: main
Choose a base branch
from

Conversation

yrajas
Copy link
Contributor

@yrajas yrajas commented May 5, 2024

Problem
Whenever object store operations are performed, corresponding heap size impact is computed and tracked to ensure that the total memory size is within the user configured limit. However, this constraint is not currently enforced during recovery. This change addresses that.

Fix description

  • Ability to register a deserialization observer, OnDeserializationObserver, that can compute and use the CacheSizeTracker to track heap usage when pages are read in during recovery. LogOperationObserver is added to perform this.
  • During recovery, if total object store size needs to be tracked, the log memory is not fully saturated by reading pages from disk rather done sequentially.
  • If reading in a page causes the memory limit to be breached, earlier pages are evicted EvictPage one at a time until the total memory size is back within the limit.
  • If pages were freed to constrain memory limits, DoPostRecovery updates headAddress to account for the freed pages.

@yrajas yrajas force-pushed the yrajas/objectstorerecovery branch from 66a4cb5 to dd22fd0 Compare May 8, 2024 04:00
@yrajas yrajas force-pushed the yrajas/objectstorerecovery branch from e48ea03 to 71eb38a Compare May 10, 2024 04:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant