You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hello everyone, today my team incurred in a very strange bug using Iceberg via Athena. I'll descrive the steps we used to reproduce the error below:
1. We create an iceberg table with an "id" column and 321 other columns with random strings - in the example below we use awsrangler to create the table, but the same happens when the table is created using Athena directly.
2. we then run the following query in athena to insert a random value
MERGE INTO my_database.iceberg_limits_322as existing
using (
SELECT'something'as id
) as new onexisting.id=new.id
WHEN NOT MATCHED
THEN INSERT (id) VALUES (new.id)
WHEN MATCHED THEN DELETE
3. which results in the error:
[ErrorCode: INTERNAL_ERROR_QUERY_ENGINE] Amazon Athena experienced an internal error while executing this query. Please contact AWS support for further assistance. You will not be charged for this query. We apologize for the inconvenience.
Notice that the error only occurs when multiple WHEN are used in the MERGE INTO query! - in case one WHEN is used (just to insert or to delete records) everything works fine, and the table can be used normally.
We can replicate this behaviour on multiple AWS accounts and with different tables/databases/s3 locations.
After trying with different number of columns we consistently found that 321 is the maximum limit for the number of columns of the table. Everything works fine below this threshold.
The text was updated successfully, but these errors were encountered:
andreaschiappacasse
changed the title
Running MERGE INTO with more than one condition fails if number of columns is > 321
Running MERGE INTO with more than one WHEN condition fails if the number of columns in the target table is > 321
May 10, 2024
Update: it seems that even a MERGE INTO with a single NOT MACHED THEN INSERT condition fails, given that the table is big enough (in our case 633 columns)
Apache Iceberg version
None
Query engine
Athena (engine v3)
Please describe the bug 馃悶
Hello everyone, today my team incurred in a very strange bug using Iceberg via Athena. I'll descrive the steps we used to reproduce the error below:
1. We create an iceberg table with an "id" column and 321 other columns with random strings - in the example below we use awsrangler to create the table, but the same happens when the table is created using Athena directly.
2. we then run the following query in athena to insert a random value
3. which results in the error:
[ErrorCode: INTERNAL_ERROR_QUERY_ENGINE] Amazon Athena experienced an internal error while executing this query. Please contact AWS support for further assistance. You will not be charged for this query. We apologize for the inconvenience.
Notice that the error only occurs when multiple WHEN are used in the MERGE INTO query! - in case one WHEN is used (just to insert or to delete records) everything works fine, and the table can be used normally.
We can replicate this behaviour on multiple AWS accounts and with different tables/databases/s3 locations.
After trying with different number of columns we consistently found that 321 is the maximum limit for the number of columns of the table. Everything works fine below this threshold.
The text was updated successfully, but these errors were encountered: