Propagate storage_options and other read parameters when pickling LanceFragment

[wjones127]

No description provided.

[wjones127]

I'm actually not sure we should do this. We found when debugging ray-project/ray#45392 that pickling a list of LanceFragment is terribly inefficient. On unpickling: each fragment will load an independent copy of the LanceDataset from disk, using up a whole lot of IO and memory.

Instead, we should probably encourage users to send just the fragment ids and the dataset across processes. We might consider deprecating pickling LanceFragment in favor of this.