Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

catalog issue #10324

Open
Shantanu0701 opened this issue May 12, 2024 · 1 comment
Open

catalog issue #10324

Shantanu0701 opened this issue May 12, 2024 · 1 comment

Comments

@Shantanu0701
Copy link

Shantanu0701 commented May 12, 2024

Below code:

import pyspark
from pyspark.sql import SparkSession
conf = (
    pyspark.SparkConf()
       .setAppName('app_name')  # Set the application name for Spark
        # Specify the Iceberg package to be used
       .set('spark.jars.packages', 'org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.5.1')
        # Add Iceberg SQL Extensions to enable Iceberg-specific SQL operations
       .set('spark.sql.extensions', 'org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions')
        # Configure the Iceberg catalog
       .set('spark.sql.catalog.localcat', 'org.apache.iceberg.spark.SparkCatalog')  # Define the catalog implementation
       .set('spark.sql.catalog.localcat.type', 'hadoop')  # Set the catalog type to Hadoop
       .set('spark.sql.catalog.localcat.warehouse', '/home/user/Downloads/warehouse')  # Specify the warehouse directory
)

Create a SparkSession with the previously defined configurations

spark = SparkSession.builder.config(conf=conf).getOrCreate()
print("Spark Running")

Create a table named 'first_table' in the 'localcat.localdb' catalog using Iceberg format

spark.sql("CREATE TABLE localcat.localdb.first_table (id bigint, first_name string, last_name string) USING iceberg")

ISSUE:
So basically my table is under localcat catalog and localdb database.
But if I tried to run a session with a different catalog such as demo_cat it's able to show all databases and table referred to localcat catalog.

@nastra
Copy link
Contributor

nastra commented May 13, 2024

But if I tried to run a session with a different catalog such as demo_cat it's able to show all databases and table referred to localcat catalog.

Do you have example output that describe what you're seeing vs what you're expecting here?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants