[Bug] dbt clone not redirecting exception to logs

[jp-dbt]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

Any errors that occur while running dbt clone are written to stderr instead of stdout.

Expected Behavior

Any errors that occur while running dbt clone are written to stdout, which is the case for other dbt commands.

Steps To Reproduce

1. Modify a project so that an error occurs. In my case I added a non-existent doc ref for a column in a model so that a parsing error occurs
2. Run dbt parse 1>out.txt 2>err.txt. The out.txt file contains the whole output including the error while err.txt is empty
3. Run dbt clone 1>out.txt 2>err.txt. The out.txt file contains the non-error dbt output and err.txt contains the error and stacktrace

Relevant log output

No response

Environment

- OS: MacOS Sonoma 14.4.1, Debian 10
- Python: 3.11.9
- dbt: 1.7.1

Which database adapter are you using with dbt?

No response

Additional Context

Possibly related: errors during dbt clone are not written to the dbt.log file.

[dbeatty10]

Thanks for reporting this @jp-dbt !

Root cause

It appears that the root cause is dbt clone failing to capture the underlying exception and redirect it to the logs. Rather, the exception is being raised directly to the user via standard error (stderr).

See below for the details of an extremely simple reproducible example.

Reprex

Replace postgres below with the name of your dbt connection profile. The last line is invalid, and it will raise an error for most dbt commands.

dbt_project.yml

name: "my_project"
profile: "postgres"

give_me: an_error

Run these commands:

dbt parse 1>parse-out.txt 2>parse-err.txt
dbt clone 1>clone-out.txt 2>clone-err.txt

Expected output

Here's what the contents of *-out.txt should be for nearly every command:

[0m15:36:22  Running with dbt=1.8.0
[0m15:36:22  Encountered an error:
Runtime Error
  at path []: Additional properties are not allowed ('give_me' was unexpected)

Error encountered in /path/to/dbt/project/dbt_project.yml

But dbt clone has a big Python stacktrace in *-err.txt instead:

Traceback (most recent call last):
  File "/Users/dbeatty/projects/environments/dbt_1.8/bin/dbt", line 8, in <module>
    sys.exit(cli())
...
  File "/Users/dbeatty/projects/environments/dbt_1.8/lib/python3.10/site-packages/dbt/config/project.py", line 390, in create_project
    raise ProjectContractError(e) from e
dbt.exceptions.ProjectContractError: Runtime Error
  at path []: Additional properties are not allowed ('give_me' was unexpected)

Error encountered in /path/to/dbt/project/dbt_project.yml

Checking other dbt commands

Here's the script that I used to test a large range of dbt commands (pretty much everything but dbt init, I think:

check_stdout_and_stderr.sh

#!/bin/bash

# Define an array of dbt commands
commands=(
    "build"
    "clean"
    "clone"
    "compile"
    "debug"
    "deps"
    "list"
    "parse"
    "retry"
    "run"
    "run-operation"
    "seed"
    "show"
    "snapshot"
    "test"
    "docs generate"
    "docs serve"
    "source freshness"
    "source snapshot-freshness"
)

# Loop through each command in the array
for cmd in "${commands[@]}"; do
    echo "Running dbt $cmd..."
    
    # Replace spaces with hyphens for file names
    filename=$(echo "$cmd" | tr ' ' '-')

    # Execute dbt with the current command, redirect stdout and stderr to files
    dbt $cmd 1>${filename}-out.txt 2>${filename}-err.txt
    
    # Use wc to count lines, words, and characters in output and error files
    wc ${filename}-out.txt
    wc ${filename}-err.txt
done

Run these commands:

chmod +x check_stdout_and_stderr.sh
./check_stdout_and_stderr.sh

Output

Running dbt build...
       6      25     267 build-out.txt
       0       0       0 build-err.txt
Running dbt clean...
       6      25     267 clean-out.txt
       0       0       0 clean-err.txt
Running dbt clone...
       1       4      41 clone-out.txt
     600    1584   47342 clone-err.txt
Running dbt compile...
       6      25     267 compile-out.txt
       0       0       0 compile-err.txt
Running dbt debug...
      43     154    1803 debug-out.txt
       0       0       0 debug-err.txt
Running dbt deps...
       6      25     267 deps-out.txt
       0       0       0 deps-err.txt
Running dbt list...
       6      25     267 list-out.txt
       0       0       0 list-err.txt
Running dbt parse...
       6      25     267 parse-out.txt
       0       0       0 parse-err.txt
Running dbt retry...
       6      25     267 retry-out.txt
       0       0       0 retry-err.txt
Running dbt run...
       6      25     267 run-out.txt
       0       0       0 run-err.txt
Running dbt run-operation...
       0       0       0 run-operation-out.txt
       4      15     112 run-operation-err.txt
Running dbt seed...
       6      25     267 seed-out.txt
       0       0       0 seed-err.txt
Running dbt show...
       6      25     267 show-out.txt
       0       0       0 show-err.txt
Running dbt snapshot...
       6      25     267 snapshot-out.txt
       0       0       0 snapshot-err.txt
Running dbt test...
       6      25     267 test-out.txt
       0       0       0 test-err.txt
Running dbt docs generate...
       6      25     267 docs-generate-out.txt
       0       0       0 docs-generate-err.txt
Running dbt docs serve...
       6      25     267 docs-serve-out.txt
       0       0       0 docs-serve-err.txt
Running dbt source freshness...
       6      25     267 source-freshness-out.txt
       0       0       0 source-freshness-err.txt
Running dbt source snapshot-freshness...
       6      25     267 source-snapshot-freshness-out.txt
       0       0       0 source-snapshot-freshness-err.txt