You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
import pyspark
from pyspark.sql import SparkSession
conf = (
pyspark.SparkConf()
.setAppName('app_name') # Set the application name for Spark
# Specify the Iceberg package to be used
.set('spark.jars.packages', 'org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.5.1')
# Add Iceberg SQL Extensions to enable Iceberg-specific SQL operations
.set('spark.sql.extensions', 'org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions')
# Configure the Iceberg catalog
.set('spark.sql.catalog.localcat', 'org.apache.iceberg.spark.SparkCatalog') # Define the catalog implementation
.set('spark.sql.catalog.localcat.type', 'hadoop') # Set the catalog type to Hadoop
.set('spark.sql.catalog.localcat.warehouse', '/home/user/Downloads/warehouse') # Specify the warehouse directory
)
Create a SparkSession with the previously defined configurations
Create a table named 'first_table' in the 'localcat.localdb' catalog using Iceberg format
spark.sql("CREATE TABLE localcat.localdb.first_table (id bigint, first_name string, last_name string) USING iceberg")
ISSUE:
So basically my table is under localcat catalog and localdb database.
But if I tried to run a session with a different catalog such as demo_cat it's able to show all databases and table referred to localcat catalog.
The text was updated successfully, but these errors were encountered:
Below code:
Create a SparkSession with the previously defined configurations
Create a table named 'first_table' in the 'localcat.localdb' catalog using Iceberg format
spark.sql("CREATE TABLE localcat.localdb.first_table (id bigint, first_name string, last_name string) USING iceberg")
ISSUE:
So basically my table is under localcat catalog and localdb database.
But if I tried to run a session with a different catalog such as demo_cat it's able to show all databases and table referred to localcat catalog.
The text was updated successfully, but these errors were encountered: