Enable updating Parquet key-value metadata of opened ParquetWriter before closing

[kylebarron]

Describe the enhancement requested

In pyarrow, it would be great to have the ability to update Parquet key-value metadata of an existing ParquetWriter instance before closing the file. For example:

import pyarrow.parquet as pq

with pq.ParquetWriter("output.parquet", schema) as writer:
    writer.write_batch(...)
    writer.add_key_value_metadata({b"hello": "world"})

This is akin to the arrow-rs ArrowWriter::append_key_value_metadata, which lets you mutate an existing writer to add key-value metadata before writing the final metadata.

My use case for this is in writing to GeoParquet efficiently. Part of the GeoParquet metadata (stored as Parquet key-value metadata) includes an optional bounding box of all geometries in the file. But many applications might need a pass over the dataset to infer the bounding box of the data. If this key-value metadata can only be defined when opening the ParquetWriter, then streaming applications (such as converting other file formats to GeoParquet) might need two passes over the data: one to infer the bounding box and construct the schema to pass to the ParquetWriter constructor, and other to actually write the batches.

Component(s)

Python

[mapleFU]

C++ has this API, but this might not integrated to Python SDK. Would you like to support this?

This means exporting this api ( 3bd57e3#diff-0042b97c44521fde7da4ce3b4446cd7bc85bbc25eae63d2b4896ce58241e3b94 ) to Python SDK

[kylebarron]

That would be great. I don't know C++ myself so I can't help out. But I may be able to test a branch if necessary

[mapleFU]

I would finish it in this week