Spark: Schema evolution is not reflected on branches

[javrasya]

Apache Iceberg version

1.4.3

Query engine

Spark

Please describe the bug 🐞

We have added to columns in a nested struct field by using Iceberg Java API. I can query and see the new columns when no version is passed down to the query;

select * 
from spark_catalog.my_db.my_table 
where myStructField.newlyAddedField='123' ;

The query above works. But not the one below

ALTER TABLE spark_catalog.my_db.my_table CREATE BRANCH `my_test_branch`;

select * 
from spark_catalog.my_db.my_table.branch_my_test_branch
where myStructField.newlyAddedField='123' ;

I get Error: org.apache.spark.sql.AnalysisException: No such struct field newlyAddedField

[javrasya]

An interesting finding, once there is any update on the main branch after the schema is evolved, this no longer becomes a problem.

I ran the following to trigger a dummy update to test it and the problem went away;

MERGE INTO spark_catalog.spark_catalog.my_db.my_table AS target
USING (SELECT * FROM spark_catalog.my_db.my_table LIMIT 1) AS source
ON target.id = source.id
WHEN MATCHED THEN
    UPDATE SET *;

ALTER TABLE spark_catalog.my_db.my_table CREATE BRANCH `my_test_branch`;

SELECT * 
FROM spark_catalog.my_db.my_table.branch_my_test_branch
WHERE myStructField.newlyAddedField='123' ;