Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

config/datalake: add options for datalake topics #18515

Draft
wants to merge 1 commit into
base: dev
Choose a base branch
from

Conversation

jcipar
Copy link
Contributor

@jcipar jcipar commented May 15, 2024

This change adds two new configuration options for controlling datalake support as part of the Iceberg project. One option, at the cluster level, can be used to globally enable or disable datalake support. The topic option turns on datalake support for a specific topic. A topic is considered "datalake-enabled" when both options are set.

This change also adds logging in ntp_archiver_service to indicate the state of topics as they are uploaded.

These new config options are experimental, and should not be enabled outside of development experiments.

  • Cluster-level config: experimental_enable_datalake
  • Topic-level config: experimental_datalake_topic

Backports Required

  • none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v24.1.x
  • v23.3.x
  • v23.2.x

Release Notes

  • none

This change adds two new configuration options for controlling datalake
support as part of the Iceberg project. One option, at the cluster
level, can be used to globally enable or disable datalake support. The
topic option turns on datalake support for a specific topic. A topic is
considered "datalake-enabled" when both options are set.

This change also adds logging in ntp_archiver_service to indicate the
state of topics as they are uploaded.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant