[HUDI-7772] HoodieTimelineArchiver##getCommitInstantsToArchive need skip limiting archiving of instants #11245
Conversation
…kip limiting archiving of instants
xuzifu666
changed the title
| } catch (UnsupportedOperationException unsupportedOperationException) { | ||
| // If tableMetadata is FileSystemBackedTableMetadata would throw UnsupportedOperationException, should skip it to | ||
| // confirm next operation success | ||
| LOG.warn("tableMetadata is FileSystemBackedTableMetadata and skip limiting archiving of instants."); |
There was a problem hiding this comment.
Why the metadata backend is fs based if metadata is available?
There was a problem hiding this comment.
refer：HoodieTableMetadata#create If the MDT is not initialized would fallback to FileSystemBackedTableMetadata，so need add a guard for it @danny0405 When mdt is not enable, but mdt dir exsist before would return FileSystemBackedTableMetadata
There was a problem hiding this comment.
The isMetadataTableAvailable checks the hoodie.properties for FILES metadata partition instead, it should be empty if the MDT does not initialize successfully.
There was a problem hiding this comment.
The condition is mdt with enable firstly , and make it disable after several insert. Then mdt table run archive would ocurr this sences. At this condition mdt dir is not empty and would occur the error. @danny0405
There was a problem hiding this comment.
The condition is mdt with enable firstly , and make it disable after several insert
Then the isMetadataTableAvailable should be false? Beause once the mdt is disabled, the FILES partition would also be removed from hoodie.properties.
There was a problem hiding this comment.
Emm，but the logic stack is inexplicable，do you think the guard check for UnsupportedOperationException is neccessary? @danny0405
There was a problem hiding this comment.
Not really because it is unexpected. I guess there might be some inconsistency for metadata table configurations.
|
close it first |
Change Logs
refer issue : https://issues.apache.org/jira/browse/HUDI-7772
When user alter table by adding a column then insert new data to the table with set mdt enable, would error out with follow, from the stack we find that FileSystemBackedTableMetadata not support it.
org.apache.hudi.exception.HoodieException: Error limiting instant archival based on metadata table
at org.apache.hudi.client.HoodieTimelineArchiver.getInstantsToArchive(HoodieTimelineArchiver.java:522)
at org.apache.hudi.client.HoodieTimelineArchiver.archiveIfRequired(HoodieTimelineArchiver.java:167)
at org.apache.hudi.client.BaseHoodieTableServiceClient.archive(BaseHoodieTableServiceClient.java:791)
at org.apache.hudi.client.BaseHoodieWriteClient.archive(BaseHoodieWriteClient.java:890)
at org.apache.hudi.client.BaseHoodieWriteClient.autoArchiveOnCommit(BaseHoodieWriteClient.java:619)
at org.apache.hudi.client.BaseHoodieWriteClient.mayBeCleanAndArchive(BaseHoodieWriteClient.java:585)
at org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:248)
at org.apache.hudi.client.SparkRDDWriteClient.commit(SparkRDDWriteClient.java:104)
at org.apache.hudi.HoodieSparkSqlWriter$.commitAndPerformPostOperations(HoodieSparkSqlWriter.scala:1020)
at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:405)
at org.apache.spark.sql.hudi.command.InsertIntoHoodieTableCommand$.run(InsertIntoHoodieTableCommand.scala:108)
at org.apache.spark.sql.hudi.command.InsertIntoHoodieTableCommand.run(InsertIntoHoodieTableCommand.scala:61)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:80)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:78)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:89)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:110)
Caused by: java.lang.UnsupportedOperationException
at org.apache.hudi.metadata.FileSystemBackedTableMetadata.getLatestCompactionTime(FileSystemBackedTableMetadata.java:260)
at org.apache.hudi.client.HoodieTimelineArchiver.getInstantsToArchive(HoodieTimelineArchiver.java:512)
... 67 more
org.apache.hudi.exception.HoodieException: Error limiting instant archival based on metadata table
Impact
low
Risk level (write none, low medium or high below)
low
Documentation Update
Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".
ticket number here and follow the instruction to make
changes to the website.
Contributor's checklist