Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

persist: turn on CRDB changefeeds -> s3 for all prod environments #27162

Open
Tracked by #26869
benesch opened this issue May 19, 2024 · 1 comment
Open
Tracked by #26869

persist: turn on CRDB changefeeds -> s3 for all prod environments #27162

benesch opened this issue May 19, 2024 · 1 comment
Assignees
Labels
A-PERSIST Area: persistence of (meta)data

Comments

@benesch
Copy link
Member

benesch commented May 19, 2024

This is an important insurance policy in the case of a data loss disaster.

@benesch benesch changed the title Turn on CRDB changefeeds -> s3 for all prod environments persist: turn on CRDB changefeeds -> s3 for all prod environments May 19, 2024
@benesch benesch self-assigned this May 19, 2024
@benesch benesch added the A-PERSIST Area: persistence of (meta)data label May 19, 2024
@benesch
Copy link
Member Author

benesch commented May 19, 2024

From discussions with @danhhz and @pH14:

  • Paul has an old PR for this: https://github.com/MaterializeInc/cloud/pull/4555
  • One pain point in the past was setting up IAM delegation for this instead of using static access keys; plausibly this was fixed when we set up IAM delegation for backups and restores
  • Another blocker was that SHOW CHANGEFEED JOBS took like literal minutes to complete. I think this is fixed in v23.1 due to some optimizations they made to not scan the whole jobs table to compute SHOW CHANGEFEED JOBS.

tl;dr is that I'll try to find some time this quarter to revive Paul's PR and see if I can get it across the finish line.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-PERSIST Area: persistence of (meta)data
Projects
None yet
Development

No branches or pull requests

1 participant