Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: refactor read/write of kafka wal #3809

Draft
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

niebayes
Copy link
Contributor

I hereby agree to the terms of the GreptimeDB CLA.

Refer to a related PR or issue link (optional)

What's changed and what's your intention?

  • Refactor the code by introducing abstractions for Producer and Consumer functionalities.
  • Migrate the buffering logic utilized by rskafka to our codebase to enhance our control over writing to Kafka.
  • Each kafka record now can contain entries from multiple regions.
  • Allow entries from a region to be written to Kafka out of order.
  • Support fetching multiple entries from Kafka within a single Kafka record.
  • Enable buffering when consuming a Kafka topic.

This PR is still in its draft version. I will add necessary tests and refine the codes.

Checklist

  • I have written the necessary rustdoc comments.
  • I have added the necessary unit tests and integration tests.
  • This PR does not require documentation updates.

@niebayes niebayes self-assigned this Apr 26, 2024
@niebayes niebayes marked this pull request as draft April 26, 2024 09:07
@github-actions github-actions bot added the docs-not-required This change does not impact docs. label Apr 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
docs-not-required This change does not impact docs.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant