Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Temporary tables for volume and freshness tests are not properly cleaned on Athena #710

Closed
bebbo203 opened this issue May 10, 2024 · 1 comment

Comments

@bebbo203
Copy link

Describe the bug
After a succesful run of volume and source freshness tests, tables in the form:

data_monitoring_metrics_tmp_<timestamp>

are left in Athena and s3.
Moreover, while not present in Athena, is still possible to find files about other temporary tables in the form:

test_<hex_code>_elementary_volume_anomalies_<model_name>_<other_descs>

in the s3 bucket

To Reproduce
Steps to reproduce the behavior:

  1. Add an Elementary's volume or freshness test to any model or source in your dbt project while using dbt-athena adapter
  2. Run dbt run --select elementary
  3. Run dbt test
  4. Check the tables in athena and in the bucket s3_data_dir

Expected behavior
Temporary tables must be completely dropped after a run (metadata + parquet files in S3).

Environment (please complete the following information):

  • edr Version: 0.14.1
  • dbt package Version: 0.14.0
  • dbt version: 1.7.13
@haritamar
Copy link
Collaborator

Closing this issue to keep discussion in the main Elementary repo

elementary-data/elementary#1514

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants