WIP - Glue integration #529
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…run-kc/dbt-data-reliability into ele-47-add-integration-with-s3
Arun-kc
commented
Sep 7, 2023
| {# Glue - truncate and insert (non-atomic) #} | ||
| {% macro glue__replace_table_data(relation, rows) %} | ||
| {% set intermediate_relation = elementary.create_intermediate_relation(relation, rows, temporary=True) %} | ||
| {% do dbt.glue_exec_query(dbt.get_insert_overwrite_sql(intermediate_relation, relation)) %} |
There was a problem hiding this comment.
Ideally should be using run_query, but not able to pass the flag as 'False' for DDL and DML statements which in turn is causing issues at run_query when it tries to change the case of columns into lowercase. So using glue_exec_query for now.
Note: When DDL and DML statements are passed to run_query in dbt it will return none
Arun-kc
commented
Sep 7, 2023
| {% endfor %} | ||
| {% elif insert_rows_method == 'chunk' %} | ||
| {% set rows_chunks = elementary.split_list_to_chunks(rows, chunk_size) %} | ||
| {% for rows_chunk in rows_chunks %} | ||
| {% set insert_rows_query = elementary.get_chunk_insert_query(table_relation, columns, rows_chunk) %} | ||
| {% do elementary.run_query(insert_rows_query) %} | ||
| {% if target.type == 'glue' %} | ||
| {% do dbt.glue_exec_query(insert_rows_query) %} |
There was a problem hiding this comment.
Same reason as #r1318786335
Arun-kc
commented
Sep 7, 2023
| @@ -22,13 +22,21 @@ | |||
| {% set queries_len = insert_rows_queries | length %} | |||
| {% for insert_query in insert_rows_queries %} | |||
| {% do elementary.file_log("[{}/{}] Running insert query.".format(loop.index, queries_len)) %} | |||
| {% do elementary.run_query(insert_query) %} | |||
| {% if target.type == 'glue' %} | |||
| {% do dbt.glue_exec_query(insert_query) %} | |||
There was a problem hiding this comment.
Same reason as #r1318786335
Arun-kc
commented
Sep 7, 2023
| @@ -57,7 +65,7 @@ | |||
| {% do rendered_column_values.append(column_value) %} | |||
| {% else %} | |||
| {% set column_value = elementary.insensitive_get_dict_value(row, column.name) %} | |||
| {% do rendered_column_values.append(elementary.render_value(column_value)) %} | |||
| {% do rendered_column_values.append(elementary.render_value(column_value, column.data_type)) %} | |||
There was a problem hiding this comment.
Passing column data_type in order to cast timestamp columns for glue
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
[ELE-47] Add integration with S3 as a data lake