Simplify Reading Partitioned Data with Automatic Glob Expansion

[gabrieldernbach]

Currently, reading partitioned data, especially with Hive, requires specifying a detailed glob-pattern expansion:

SELECT * FROM read_parquet('orders/*/*/*.parquet', hive_partitioning = true)

In comparison, other libraries like pandas allow a simpler approach:

pd.read_parquet('orders')

Proposal

It would be helpful to introduce automatic glob expansion to simplify reading partitioned data. This would enable users to specify the root directory without manually counting the partitioning levels.

Expected Behavior

Allow the following command to automatically expand to include all relevant partitioned files:

SELECT * FROM read_parquet('orders', hive_partitioning = true)

• Ease of Use: Reduces the need to manually define partitioning levels.
• Consistency: Aligns with the behavior of other data processing libraries.
• Error Reduction: Decreases the likelihood of errors in glob pattern specification.

Considerations

Are there potential issues with implementing automatic glob expansion?

Are there edge cases I didn't considered?

Thank you for considering and sharing your insights.