Add return_file_name in load_dataset

[juliendenize]

Proposition to fix #5806.

Added an optional parameter return_file_name in the dataset builder config. When set to True, the function will include the file name corresponding to the sample in the returned output.

There is a difference between arrow-based and folder-based datasets to return the file name:

• for arrow-based: a column is concatenated after the table is cast.
• for folder-based: dataset.info.features has the entry file_name and the original file name is passed to the sample_metadata dictionary.

The difference in behavior might be a concern, also I do not know whether the file_name should return the original file path or the downloaded one for folder-based datasets.

I added some tests for the datasets that already had a test file.

[lhoestq]

Thanks for the change !

Since return in python often refers to what is actually returned by the function (here load_dataset), I think we can use another word for the parameter. Maybe name it with_file_names?

cc @mariosasko in case you have an opinion

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[juliendenize]

Thanks for the change !

Since return in python often refers to what is actually returned by the function (here load_dataset), I think we can use another word for the parameter. Maybe name it with_file_names?

cc @mariosasko in case you have an opinion

I changed the argument name to your suggestion, I agree that it should be less confusing :)

[mariosasko]

Thanks! I've left some comments.

@lhoestq WDYT about returning a data file's name (the last part) instead of the full path? This way we could have the same values in the streaming and the non-streaming mode. (In the non-streaming mode, we would also have to iterate over remote files to not output the files' hash (from the HF cache))

[mariosasko]

Let's check here if the file_name column is already present in the table and raise an error if it is.

datasets does not support accessing columns with integers as pyarrow does, so multiple columns with the same name are not allowed.

[juliendenize]

I added the test for the different datasets, let me know if you want to change the error message.

[mariosasko]

I think we ensure there is some data in the table before yielding it, so this is not needed

[juliendenize]

Here if I remove the condition it fails so I left it.

[mariosasko]

The parquet builder performs a cast if features are defined in the file's metadata. So, you need to specify a feature for with_file_names in the self.info.features dictionary to avoid an error in this scenario.

PS: You can reproduce the error by creating a dataset, saving it with to_parquet("path/to/parquet_file"), and then loading it with load_dataset("parquet", data_files="path/to/parquet_file", with_file_name=True).

[juliendenize]

This particular error is fixed, however I did not test every use cases as I am not familiar enough with the library, if you have more suggestions to improve tests I'll work on it

[juliendenize]

Thanks! I've left some comments.

@lhoestq WDYT about returning a data file's name (the last part) instead of the full path? This way we could have the same values in the streaming and the non-streaming mode. (In the non-streaming mode, we would also have to iterate over remote files to not output the files' hash (from the HF cache))

Concerning the last part of the file name, do you have suggestions on how to do that? Because it can happen that the files are located in different folders with the same name so I am wondering what would be the way to go.