Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use BigQuery Storage Write API #184

Open
Tang8330 opened this issue Sep 6, 2023 · 2 comments
Open

Use BigQuery Storage Write API #184

Tang8330 opened this issue Sep 6, 2023 · 2 comments
Labels
enhancement New feature or request

Comments

@Tang8330
Copy link
Contributor

Tang8330 commented Sep 6, 2023

https://cloud.google.com/bigquery/docs/write-api

Context https://artie-technologies.slack.com/archives/C04Q7GG6DT9/p1693997817550579
@Tang8330 Tang8330 added the enhancement New feature or request label Sep 6, 2023
@Tang8330
Copy link
Contributor Author

Potentially doable with the dynamic JSON parser: https://github.com/googleapis/google-cloud-go/blob/7a46b5428f239871993d66be2c7c667121f60a6f/bigquery/storage/managedwriter/integration_test.go#L424-L442

@claytongentry
Copy link

FYI, https://cloud.google.com/pubsub/docs/bigquery#change_data_capture

This looks like a way simpler alternative to interfacing with Storage Write directly. Pub/Sub supports a native BigQuery subscription, which uses Storage Write under the hood.

Transfer could simply register those native BigQuery subscriptions to the topics it already registers in Pub/Sub and potentially replace the temp table / merge query workflow altogether.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Tang8330 @claytongentry