You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We have been using Elementary for a long time and love the work you do. I do not always look through our daily runs, but sometimes I do and i noticed some heavy models. One of them was elementary.dbt_columns which seems to be an incremental model, it runs with our daily job at 05:00 together with a bunch of other models and tests. This incremental model (according to the logs, see screenshot) merges 0 rows, but takes over 2500seconds (~42 minutes ) to build. Which is kinda insane, so I think something is off here.
I also tried bumping up to elementary 0.14.1 to test out the Alerts functionality for Teams, did a test run for the whole setup locally against our dev environment which holds around 100 models/tests. It also had another similiar model which also took tremendously long amount of time.
I haven't deepdived into the logs in the VM's running these daily jobs. So not sure exactly what is happening here. But when I ran the edr monitor for alerting locally i got this folder called edr_target/ which had a edr.log file that I looked inside. It gave me some indications of some errors, could they be correlated? I'll attach the logfile to this issue.
I just saw that this config is available: "disable_dbt_columns_autoupload". Is that a fix I could use to fix running this incremental model? If so, how will that affect my dashboard/ edr report?
Hi @christopherekfeldt !
Sorry for only responding on this issue, I'm not sure if this is still relevant / was answered in the community.
I'm closing for now, but if still relevant please feel free to open a new issue in the main Elementary repo.
Hi Elementary team.
We have been using Elementary for a long time and love the work you do. I do not always look through our daily runs, but sometimes I do and i noticed some heavy models. One of them was elementary.dbt_columns which seems to be an incremental model, it runs with our daily job at 05:00 together with a bunch of other models and tests. This incremental model (according to the logs, see screenshot) merges 0 rows, but takes over 2500seconds (~42 minutes ) to build. Which is kinda insane, so I think something is off here.
our setup:
elementary 0.13.0
dbt-core 1.7.0
dbt-bigquery 1.7.0
I also tried bumping up to elementary 0.14.1 to test out the Alerts functionality for Teams, did a test run for the whole setup locally against our dev environment which holds around 100 models/tests. It also had another similiar model which also took tremendously long amount of time.
I haven't deepdived into the logs in the VM's running these daily jobs. So not sure exactly what is happening here. But when I ran the edr monitor for alerting locally i got this folder called edr_target/ which had a edr.log file that I looked inside. It gave me some indications of some errors, could they be correlated? I'll attach the logfile to this issue.
edr.log
Thanks in advance.
// Christopher
The text was updated successfully, but these errors were encountered: