Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle GCS rate limit on manifest write contention #2272

Open
wjones127 opened this issue Apr 29, 2024 · 0 comments
Open

Handle GCS rate limit on manifest write contention #2272

wjones127 opened this issue Apr 29, 2024 · 0 comments
Labels
enhancement New feature or request rust Rust related tasks

Comments

@wjones127
Copy link
Contributor

We have essentially the same problem as in delta lake, where concurrent writers can hit a rate limit while trying to commit the manifest. We should handle the 429 as another writer beat us, rather than as a request error.

TODO: see if this applies to other object stores (S3, minio, Cloudflare R2, Azure blob)

See: delta-io/delta-rs#2451

@wjones127 wjones127 added enhancement New feature or request rust Rust related tasks labels Apr 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request rust Rust related tasks
Projects
None yet
Development

No branches or pull requests

1 participant