Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[AMORO-2540] Report expiring snapshots event for Iceberg and mixed format #2541

Open
wants to merge 14 commits into
base: master
Choose a base branch
from

Conversation

XBaith
Copy link
Contributor

@XBaith XBaith commented Feb 5, 2024

Why are the changes needed?

Close #2540 .

Brief change log

  • Report expire snapshots event to downstream listeners

How was this patch tested?

  • Add some test cases that check the changes thoroughly including negative and positive cases if possible

  • Add screenshots for manual tests if appropriate

  • Run test locally before making a pull request

Documentation

  • Does this pull request introduce a new feature? (no)
  • If yes, how is the feature documented? (not documented)

@github-actions github-actions bot added module:core Core module module:ams-server Ams server module type:build labels Feb 5, 2024
@XBaith XBaith changed the title [AMORO-2540] report expiring event for iceberg and mixed format [AMORO-2540] Report expiring snapshots event for Iceberg and mixed format Feb 5, 2024
Copy link

codecov bot commented Feb 6, 2024

Codecov Report

Attention: Patch coverage is 74.33628% with 58 lines in your changes are missing coverage. Please review.

Project coverage is 32.61%. Comparing base (2808413) to head (f15c295).

Files Patch % Lines
...etease/arctic/server/events/MixedExpireEvents.java 73.68% 16 Missing and 4 partials ⚠️
...s/api/events/mixed/ExpireMixedSnapshotsResult.java 0.00% 7 Missing ⚠️
.../optimizing/maintainer/IcebergTableMaintainer.java 89.70% 4 Missing and 3 partials ⚠️
.../netease/arctic/server/utils/IcebergTableUtil.java 57.14% 6 Missing ⚠️
...netease/arctic/ams/api/events/ExpireOperation.java 0.00% 5 Missing ⚠️
.../ams/api/events/iceberg/ExpireSnapshotsResult.java 0.00% 5 Missing ⚠️
...ms/api/events/iceberg/RemoveOrphanFilesResult.java 0.00% 3 Missing ⚠️
...a/com/netease/arctic/ams/api/events/EventType.java 0.00% 2 Missing ⚠️
...ease/arctic/ams/api/events/IcebergReportEvent.java 0.00% 2 Missing ⚠️
...com/netease/arctic/ams/api/events/ExpireEvent.java 0.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master    #2541      +/-   ##
============================================
+ Coverage     32.44%   32.61%   +0.16%     
- Complexity     4412     4433      +21     
============================================
  Files           590      596       +6     
  Lines         50055    50260     +205     
  Branches       6622     6640      +18     
============================================
+ Hits          16242    16391     +149     
- Misses        32533    32578      +45     
- Partials       1280     1291      +11     
Flag Coverage Δ
core 30.71% <74.33%> (+0.18%) ⬆️
trino 50.93% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Contributor

@zhoujinsong zhoujinsong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@XBaith I left some comments. PTAL.

# Conflicts:
#	ams/api/pom.xml
#	ams/server/src/main/java/com/netease/arctic/server/optimizing/maintainer/IcebergTableMaintainer.java
#	core/src/main/java/com/netease/arctic/utils/TableFileUtil.java
#	pom.xml
@github-actions github-actions bot removed the module:core Core module label Feb 28, 2024
@majin1102
Copy link
Contributor

Hi, thanks for this wonderful feature.

I wonder if you can somehow explain the motivation of this definition?

@XBaith
Copy link
Contributor Author

XBaith commented Mar 28, 2024

I wonder if you can somehow explain the motivation of this definition?

The purpose is to report information about files that have been deleted after the expiration timestamp, such as the path to the file, so that downstream systems can monitor or utilize this information for appropriate operations. But after discussing with @zhoujinsong , I think the reported information should be as concise as possible, and also need to wait for the API to be refactored to handle it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Subtask]: Collect and report expire snapshots event for Mixed and Iceberg format
3 participants