feat(pipeline): add an ability to auto truncate

[IlyaFaer]

Closes #1104

[netlify]

✅ Deploy Preview for dlt-hub-docs canceled.

Name	Link
🔨 Latest commit	eada0dd
🔍 Latest deploy log	https://app.netlify.com/sites/dlt-hub-docs/deploys/6644ff6a8d2eb10007ec523e

[sh-rp]

@rudolfix since we have two different concepts of "staging", a staging destination as well as a staging dataset, maybe we should use those terms in config vars more clearly and not just "staging" which could mean both really

[sh-rp]

@IlyaFaer in the ticket it says we want to truncate the staging dataset, not the staging destination. I also think that the setting should probably be part of the LoaderConfiguration and not be passed down from the pipeline. @rudolfix should we always truncate or only if all loadjobs where successfull? If a mergejob fails because of a connection error or something like that I don't think we should truncate the staging tables, because you can retry the load.

[rudolfix]

@sh-rp we should truncate only when we complete the load id successfully. so after the complete method is executed.

[IlyaFaer]

One thing that I didn't manage to solve is how to patch the LoaderConfiguration, to run tests for both cases. I tried mocking, direct replacing the class in the module, but none worked 🤔

[rudolfix]

OK almost there, see review and

1. add info on new config ie to ## Data left behind of running.md
2. we need to enable this behavior for one of the merge tests ie. test_pipeline_upfront_tables_two_loads and make sure that all tables in staging dataset are truncated - this is to test it on all destinations

[IlyaFaer]

@rudolfix, okay, what do we do, if the client doesn't have should_load_data_to_staging_dataset()? Qdrant supports both merge and replace, but doesn't have the method.

[rudolfix]

@rudolfix, okay, what do we do, if the client doesn't have should_load_data_to_staging_dataset()? Qdrant supports both merge and replace, but doesn't have the method.

yeah because qdrand does not use staging dataset for merge!

look at my comments: client must implement WithStagingDataset trait which has should_load_data_to_staging_dataset so just move stuff below instance type check!

[IlyaFaer]

@rudolfix, hm, some tests fail for Dremio. It seems to me, there staging is always used. Does it mean that staging dataset is not used? I don't see credentials for Dremio in the Secret Manager, so I can't test locally what's going on there.

[rudolfix]

@rudolfix, hm, some tests fail for Dremio. It seems to me, there staging is always used. Does it mean that staging dataset is not used? I don't see credentials for Dremio in the Secret Manager, so I can't test locally what's going on there.

we only have local container and there are instructions. take a look in dremio folder in destinations. but I think you won't need it if you fix the pr. see review

[rudolfix]

LGTM!

[rudolfix]

LGTM!