OOM error when training on a 220G Memory machine with 8 V100.

[SefaZeng]

Is your feature request related to a problem? Please describe.
I tried to train a model with my own data about 300G after binarized. And there will be an OOM error for this. Is there any parameter that can be used for loading part of the data in a stream way?

[Quentin-Anthony]

There is not. I recommend you shard your dataset like the Pile (https://the-eye.eu/public/AI/pile/train/) and then unpack and train on one shard at a time.

[SefaZeng]

There is not. I recommend you shard your dataset like the Pile (https://the-eye.eu/public/AI/pile/train/) and then unpack and train on one shard at a time.

Thank you for your reply. Do you mean to change the train data path from

    "train-data-paths": [
        "/data/pile-train_text_document",
        "/data/clue_pretrain_0-train_text_document",
    ],
    "train-data-weights": [40.0, 60.0],

to

    "train-data-paths": [
        "/data/pile-00-train_text_document",
        "/data/pile-01-train_text_document",
        ...
        "/data/clue_pretrain_0-train_text_document",
    ],
    "train-data-weights": [40.0, 40.0, ..., 60.0],

?
I thought the gpt-neox will load all the data in the train-data-paths into the memory one time. I can shard the data into multiple shards but I am not sure how to load one shard at a time. Is there any example?
Thanks for your kindly response again!!! :-)