Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add resource for UC schemas to DABs #1413

Open
wants to merge 12 commits into
base: main
Choose a base branch
from

Conversation

shreyas-goenka
Copy link
Contributor

@shreyas-goenka shreyas-goenka commented May 2, 2024

Changes

This PR adds support for UC Schemas to DABs. This allows users to define schemas for tables and other assets their pipelines/workflows create as part of the DAB, thus managing the life-cycle in the DAB.

The first version has a couple of intentional limitations:

  1. The owner of the schema will be the deployment user. Changing the owner of the schema is not allowed (yet). Tangentially related, but run_as will also not affect UC resources like a schema or registered model. Lets limit the scope of run_as to the compute identity used instead of ownership.
  2. API fields that are present in the update API but not the create API. For example: enabling predictive optimization is not supported in the create schema API and thus is not available in DABs at the moment.
  3. Force destroy for schemas. Schema deletion/bundle destroy will fail if the schema has data in it. Support for force destroy for schemas can be added as a followup, as a command line flag. The HTTP API does not support force deletes (the SQL API does via the CASCADE key), so we would have to use the volumes / tables / models / functions APIs to manually delete all of them before deleting the schema. This is how terraform does it today.

Note: This PR requires #1414 to be merged.

Tests

Manually and integration test. Manually verified the following work:

  1. Development mode adds a "dev_" prefix.
  2. Modified status is correctly computed in the bundle summary command.
  3. Grants work as expected, for assigning privileges.
  4. Variable interpolation works for the schema ID.

@shreyas-goenka shreyas-goenka reopened this May 2, 2024
@shreyas-goenka shreyas-goenka marked this pull request as ready for review May 7, 2024 11:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant