Bug in Docs: Tutorials > Getting Started > Part 3 - Process data - Bug when meltano invoke dbt-postgres:run is run

[nivinsrinivas]

Meltano Version

3.3.1

Python Version

3.11

Bug scope

CLI (options, error messages, logging, etc.)

Operating System

macOS

Description

• I was trying to follow the documentation to the dot, to get started with meltano to build an ETL pipeline.

• And finally landed to this section: https://docs.meltano.com/getting-started/part3#run-the-transformation-process

• But when I ran meltano invoke dbt-postgres:run I was running into an error saying the following

18:10:42 Encountered an error:
Runtime Error
dbt will not clean the following directories outside the project: ['/Users/nivinsrinivas/my_meltano_project/first_one/.meltano/transformers/dbt/target']
error invoking dbt clean error_message=pre invoke step of dbt clean failed returncode=2

• The issue is that in dbt_project.yml, clean-targets is set to ../.meltano/transformers/dbt/target instead of .meltano/transformers/dbt/target

• The fix is to somehow change the clean-targets value in dbt_project.yml file when dbt-postgres is configured

Code

No response

[edgarrmondragon]

Thanks for reporting @nivinsrinivas!

The fix for that should probably go to https://github.com/meltano/dbt-ext/blob/a3f1abf79f6d577a949fc8ccf1d656a717cce292/files_dbt_ext/bundle/transform/dbt_project.yml#L21-L22.

PRs welcome!

[sevkw]

I also encountered the same error.
After reading this post, i simply removed the path under the clean-targets parameter in the dbt_project.yml because I could not identify this path in the .meltano directory.

I removed - ../.meltano/transformers/dbt/target and dbt ran successfully:

(.venv) user@DESKTOP-USER:~/meltano_learn/my-meltano-project/transform$ meltano invoke dbt-postgres:run
2024-02-08T21:36:21.320345Z [info     ] Environment 'dev' is active   
2024-02-08T21:36:21.375558Z [warning  ] `kind: password` is deprecated for setting definitions in favour of `sensitive: true`, and is currently in use by the following settings of utility 'dbt-postgres': 'password'. Please open an issue or pull request to update the plugin definition on Meltano Hub at https://github.com/meltano/hub/blob/main/_data/meltano/utilities/dbt-postgres/dbt-labs.yml.
Extension executing `dbt clean`...
21:36:23  Running with dbt=1.7.7
21:36:23  [WARNING]: Deprecated functionality
The `target-path` config in `dbt_project.yml` has been deprecated, and will no
longer be supported in a future version of dbt-core. If you wish to write dbt
artifacts to a custom directory, please use the --target-path CLI flag or
DBT_TARGET_PATH env var instead.
21:36:24  Checking /home/user/meltano_learn/my-meltano-project/transform/logs/*
21:36:24  Cleaned /home/user/meltano_learn/my-meltano-project/transform/logs/*
21:36:24  Checking /home/user/meltano_learn/my-meltano-project/transform/dbt_packages/*
21:36:24  Cleaned /home/user/meltano_learn/my-meltano-project/transform/dbt_packages/*
21:36:24  Finished cleaning all paths.


Extension executing `dbt deps`...
21:36:26  Running with dbt=1.7.7
21:36:26  [WARNING]: Deprecated functionality
The `target-path` config in `dbt_project.yml` has been deprecated, and will no
longer be supported in a future version of dbt-core. If you wish to write dbt
artifacts to a custom directory, please use the --target-path CLI flag or
DBT_TARGET_PATH env var instead.
21:36:26  Warning: No packages were found in packages.yml
21:36:26  Warning: No packages were found in packages.yml


Extension executing `dbt run`...
21:36:29  Running with dbt=1.7.7
21:36:29  [WARNING]: Deprecated functionality
The `target-path` config in `dbt_project.yml` has been deprecated, and will no
longer be supported in a future version of dbt-core. If you wish to write dbt
artifacts to a custom directory, please use the --target-path CLI flag or
DBT_TARGET_PATH env var instead.
21:36:29  Registered adapter: postgres=1.7.7
21:36:29  Unable to do partial parsing because saved manifest not found. Starting full parse.
21:36:30  Found 1 model, 1 source, 0 exposures, 0 metrics, 401 macros, 0 groups, 0 semantic models
21:36:30
21:36:31  Concurrency: 2 threads (target='dev')
21:36:31
21:36:31  1 of 1 START sql table model analytics.authors ................................. [RUN]
21:36:31  1 of 1 OK created sql table model analytics.authors ............................ [SELECT 2 in 0.15s]
21:36:31
21:36:31  Finished running 1 table model in 0 hours 0 minutes and 0.34 seconds (0.34s).
21:36:31
21:36:31  Completed successfully
21:36:31
21:36:31  Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1

On the side: I also noticed that the target-path configuration should probably be removed (and instead set up as an environment variable) according to the WARNING message.

[edgarrmondragon]

@sevkw see meltano/dbt-ext#56