AWS: Fix S3FileIO tests failing on ListObjects for Express buckets

[muddyfish]

Issue

When S3FileIO integration tests were being ran against an S3 Express bucket, they were failing due to a change in Express requiring all ListObjects requests end with a delimiter (/).

Fix

Changed the calls to ListObjects to end in /. This works because the prefix was always proceeded by a / in tests.

I'm avoiding changing s3FileIO.listPrefix here in case the previous functionality is required for external integrations.

Testing

The tests still pass when ran under S3 Standard, and when ran under S3 Express, the ListPrefix test now passes, and bucket cleanup now works as expected.

[singhpk234]

LGTM, thanks for the change @muddyfish !

This makes me wonder if we have tested s3 express with HadoopFileIO (where we require version hint and hence listing) as well as ran iceberg maintenance actions (like clean orphan files) with the same I am assuming these ops will require listing as well and if we have suff data points proving that it works out of the box with these subtle differences and no changes in iceberg lib is required to work like we are doing here to fix the integ tests.

[muddyfish]

I did not try with HadoopFileIO or with Iceberg maintenance operations.

According to this PR #7914, it doesn't seem that delete_orphan_files supports S3FileIO. If/when it does, it still might not work with S3 Express due to it requiring a / at the end of a ListObjects request, whereas S3 Standard does not. We could change s3FileIO.listPrefix to add the / if it's not already present, but that could potentially break usages where a list request is required without a trailing delimiter. (E.g when the list request is for a partial prefix)

With regards to HadoopFileIO, does that refer to this? https://iceberg.apache.org/docs/1.5.0/aws/#hadoop-s3a-filesystem
Is this still recommended for use?

[singhpk234]

According to this PR #7914, it doesn't seem that delete_orphan_files supports S3FileIO. If/when it does, it still might not work with S3 Express due to it requiring a / at the end of a ListObjects request, whereas S3 Standard does not. We could change s3FileIO.listPrefix to add the / if it's not already present, but that could potentially break usages where a list request is required without a trailing delimiter. (E.g when the list request is for a partial prefix)

If we claim, S3 express is supported with iceberg we have to make sure it works with all of the iceberg unless / until we can have a dedicated page to document limitation, my questions are mostly comming from there as i see the integ test being updated but i am not able to figure what does fixing integ test imply, are we claiming S3 express is supported with iceberg ?

With regards to HadoopFileIO, does that refer to this? https://iceberg.apache.org/docs/1.5.0/aws/#hadoop-s3a-filesystem

yes, HadoopFileIO can use any of the FS impl available, the link you shared is when using S3A, we also have EMR-FS using the HadoopFile system. HadoopFileIO default with i think HiveCatalog.

Is this still recommended for use?

It depends, from what i am aware this is not prod recommended, since S3FileIO is the one we would be using.