You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This means the generated assertion SQL does not contain filters for partition columns.
Problem Details:
This is the assertion SQL for the minimal reproduction.
SELECT*FROM (
SELECT
id,
COUNT(1) AS index_row_count
FROM`project.dataform.repro`GROUP BY id
// missing a filter for column "id"
) AS data
WHERE index_row_count >1
The SQL is generated by Adaptor.indexAssertion and it is missing the filter. This is because that BigQueryAdaptor{.indexAssertion || .rowConditionsAssertion} disregards BigQueryOptions during assertions, causing partition settings to be neglected(=assertion query does not contain partition clause.).
BigQueryAdaptor should have implementations for the methods, like below.
exportclassBigQueryAdapterextendsAdapterimplementsIAdapter{// for uniqueKeypublicindexAssertion(dataset: string,indexCols: string[]){// do something to filter partition columnsconstpartitionColumnStatement= ...
return`SELECT *FROM ( SELECT${commaSeparatedColumns}, COUNT(1) AS index_row_count FROM ${dataset} GROUP BY ${commaSeparatedColumns} AND ${partitionColumnStatement} ) AS dataWHERE index_row_count > 1 `}}
The text was updated successfully, but these errors were encountered:
Problem Summary:
I found that assertions for SQLX scripts fail with the following error when
config.bigquery.requirePartitionFilter: true
:Minimal Reproduction:
The error message originates from BigQuery. I found BigQueryAdaptor uses Adaptor.indexAssertion for the assertion, ignoring BigQueryOptions.
This means the generated assertion SQL does not contain filters for partition columns.
Problem Details:
This is the assertion SQL for the minimal reproduction.
The SQL is generated by Adaptor.indexAssertion and it is missing the filter. This is because that
BigQueryAdaptor{.indexAssertion || .rowConditionsAssertion}
disregardsBigQueryOptions
during assertions, causing partition settings to be neglected(=assertion query does not contain partition clause.).BigQueryAdaptor
should have implementations for the methods, like below.The text was updated successfully, but these errors were encountered: