Dataform --dry-run fails with large number of actions

[elanor-sparx]

I have a medium to large setup: 875 actions

What I'd like to do is add --dry-run to our CICD to check executions, but it fails when I try with dataform run --dry-run --timeout 2m:

Dataform encountered an error: request to https://oauth2.googleapis.com/token failed, reason: socket hang up
FetchError: request to https://oauth2.googleapis.com/token failed, reason: socket hang up
    at ClientRequest.<anonymous> (/usr/local/lib/node_modules/@dataform/cli/node_modules/node-fetch/lib/index.js:1505:11)
    at ClientRequest.emit (events.js:315:20)
    at ClientRequest.EventEmitter.emit (domain.js:467:12)
    at TLSSocket.socketOnEnd (_http_client.js:493:9)
    at TLSSocket.emit (events.js:327:22)
    at TLSSocket.EventEmitter.emit (domain.js:467:12)
    at endReadableNT (internal/streams/readable.js:1327:12)
    at processTicksAndRejections (internal/process/task_queues.js:80:21)

When I try a smaller individual action with lots of dependents it works, but a bigger one fails similarly to above - hence my thinking that it has to do with the number of actions and/or a token timeout.

Any thoughts on what's going on here? The above were run using my own GCP credentials, we are going to try a service account next.

[elanor-sparx]

Additional info: my Dataform version is 2.6.7

[Ekrekr]

Thanks for the report!

It being a token timeout seems like a reasonable idea - however it's hard for me to reproduce it without being given a project that can replicate it.

It doesn't seem to be caused by our timeout options:

• The --timeout option only applies to compilation - it should probably be renamed to --compilation-timeout:

  dataform/cli/index.ts

  Line 148 in c55ca6b

  const timeoutOption: INamedOption<yargs.Options> = {

  .
• There is an option in execution run configs that allows a timeout to be specified for each action:

  dataform/cli/api/commands/run.ts

  Line 115 in c55ca6b

  if (!!this.graph.runConfig && !!this.graph.runConfig.timeoutMillis) {

• However, this option is never set, and there isn't an option to configure it currently

  dataform/cli/index.ts

  Line 505 in c55ca6b

  const executionGraph = await build(

  . So execution shouldn't be timing out because of that.