Service Account for gcp_bigquery output

[jean-altura]

Hello!

We have a service account (SA) under a certain project (project_A). We want to use this to write data to another project (project_B) under a data set (dataset_B1) using the gcp_bigquery output component. We have granted read and write access to this dataset_B1 dataset for this SA as we use same SA to query data from it using the gcp_bigquery_select input component and it works.

When we tested the output component, we are getting this error:

ERRO Failed to send message to gcp_bigquery: googleapi: Error 403: Access Denied: Project project_B: User does not have bigquery.jobs.create permission in project project_B., accessDenied  @service=benthos label="" path=root.output.switch.output

Could you help us why this error is being triggered? We have jobs from other services that are able to write data in same scenario but we are not sure why it doesn't work with benthos gcp_bigquery output component.

Hoping to hear from you soon.

[mihaitodor]

Hey @jean-altura 👋 I'm not that familiar with IAM in GCP, but I'm guessing that you might've left the create_disposition field set to the default value, which is CREATE_IF_NEEDED. In this case, BigQuery will create the table if it does not already exist, so you might be hitting this. If that's what you want, then I guess you also need to grant the create permission as well.

[Roviluca]

Hey @mihaitodor thanks for your quick answer,
I work with @jean-altura and I've tried to take a look at the code and it seems like the issue can be caused by the fact that, when the projectID is set, both client and the DatasetInProject function call are use that value making them coupled. In GCP it is possible to have a dataset in projectB while running the jobs ( where the client is initialized ) in projectA. Benthos code making that not possible.

I also noticed that datasetID on gcp is in the format project_name:dataset_name while in the benthos code the dataset name is used instead. This part is not very clear even in the go lib documentation so I tried to retrieve one dataset with the bq command to confirm that and the ID returned is project_name:dataset_name format.

I think that one option to solve this could be to require the full dataset ID as an input and use the project id obtained from it in DatasetInProject() function.
What are your thoughts?

[mihaitodor]

Thanks for digging into that @Roviluca! PRs are definitely welcome. The gcp_bigquery components are still marked as beta, but it would be nice to avoid breaking changes if possible. I'm not sure I fully understand what changes should be made, so it would help to look at a diff you have time to dig into it and raise a PR.

[Roviluca]

This is one possible way of doing it: Roviluca@12da668
I will edit other bigquery resources after testing properly and later open a PR.
Please, let me know if you have any suggestion.

[mihaitodor]

Cool, that looks like a good start, thank you! Some of those changes look unnecessary and I think there's some issue with the logic here: Roviluca@12da668#diff-99c5313d09dbae11c94d4cc88ba5f8bea1cccb47e0c7cd98f56e51211d9f1aaaR78-R80 since you're doing gconf.ProjectID = gconf.ProjectID. I'd probably mark that new field as Advanced() too. Once you're ready, please run make docs, create a signed commit (use the -s flag for git commit) and send a PR.

[Roviluca]

Opened this PR