Support WAL Compression #12476
Open
Support WAL Compression #12476
+456
−54
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
THUMarkLau
force-pushed
the
wal-compress-formal-branch
branch
from
834e61d
to
70d8400
Compare
THUMarkLau
force-pushed
the
wal-compress-formal-branch
branch
2 times, most recently
from
89b0f35
to
94152b2
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #12476 +/- ##
============================================
+ Coverage 43.37% 43.44% +0.07%
Complexity 69 69
============================================
Files 3164 3165 +1
Lines 196704 196931 +227
Branches 23431 23459 +28
============================================
+ Hits 85316 85557 +241
+ Misses 111388 111374 -14 ☔ View full report in Codecov by Sentry. |
remove metrics in mem table flush task, cache hash code in partial path, use gzip to compress wal batch update metrics
THUMarkLau
force-pushed
the
wal-compress-formal-branch
branch
from
e7250f8
to
069795d
Compare
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The write-ahead logging consumes a significant amount of I/O resources during data writing, sometimes even 2-7 times the amount of TsFile I/O. Currently, the write-ahead logging in the system is uncompressed, this is a main reason for the large volume of WAL. Therefore, compressing the write-ahead logging can save I/O resources and enhance system performance. This PR primarily introduces support for compressing write-ahead logging, featuring the following characteristics:
The basic unit for compression and decompression is a WAL Buffer, which may contain several WAL Entries, with a default size of 16MB. This design achieves a good compression ratio while minimizing modifications to upper-layer applications.
To maintain compatibility with previous versions, we have replaced the original Magic String in the WAL file with a Version String to distinguish between different versions of WAL. If there are subsequent new versions of WAL, we can continue to add new Version Strings.
Multiple compression methods are supported. Currently, the choice of compression method is not open to users, but theoretically, our design supports various compression methods, and we may open this option in the future.
Changes in system resource consumption and system performance caused by this feature will be supplemented in subsequent comments.