Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"Iceberg.engine.hive.enabled" Conf is not honouring for HIVE CATALOG #10286

Open
shivjha30 opened this issue May 8, 2024 · 2 comments
Open
Labels
bug Something isn't working

Comments

@shivjha30
Copy link

Apache Iceberg version

1.4.3

Query engine

Hive

Please describe the bug 🐞

In the iceberg documentation it could be seen that
"To enable Hive support globally for an application, set iceberg.engine.hive.enabled=true in its Hadoop configuration."

The value of iceberg.engine.hive.enabled must be true in order to enable hive support. However, it could still use Iceberg's hive support even if it is set as "iceberg.engine.hive.enabled" to false. If i understand correctly, the value "iceberg.engine.hive.enabled" is irrelevant.

Current scenario:

if it is a hive catalog we are setting engine.hive.enabled as true.
In

iceberg/hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java

Line 486 in 01bc864

Even when we set ConfigProperties.ENGINE_HIVE_ENABLED to FALSE in Hadoop configuration, we are passing the ConfigProperties.ENGINE_HIVE_ENABLED as TRUE in the HiveIcebergMetaHook#commitCreateTable if it's of HiveCatalog type.
We are initially evaluating the 'engine.hive.enabled' value in HiveTableOperations#hiveEngineEnabled. In cases where it's a Hive catalog, this value is consistently 'True', rendering the 'iceberg.engine.hive.enabled' as unnecessary.

@shivjha30 shivjha30 added the bug Something isn't working label May 8, 2024
@pvary
Copy link
Contributor

pvary commented May 8, 2024

This is intentional.
If you are using HiveIcebergMetaHook then you are using Hive to create the table. The expectation in this case, that you want to be able read the table with Hive.

See the discussion here: #1495 (comment)

@shivjha30
Copy link
Author

@pvary then, for a cluster level if we set icebeg.engine.hive.enabled as false and we don’t pass the table properties(hive.engine.enabled as false), even then we are creating the table with Iceberg, ideally it shouldn’t be.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants