[QUESTION] How to pre-build the dataset's index ?

[etiennemlb]

How to pre-build the dataset's index ?

I want to avoid using compute node for this task:

> WARNING: could not find index map files, building the indices on rank 0 ...
> elasped time to build and save doc-idx mapping (seconds): 270.614145

[ethanhe42]

you can use --data-cache-path to specify where you want to cache. And precompute it using a single node.

Megatron-LM/megatron/training/arguments.py

Lines 1349 to 1350 in 9de386d

	group.add_argument('--data-cache-path', default=None,
	help='Path to a directory to hold cached index files.')