How to load a pre-trained model using horovod

[ForawardStar]

Environment:

1. Framework: PyTorch 1.4
2. python version: horovod with ppytorch
3. Horovod version: latest version
4. MPI version: 4.0.3
5. CUDA version: cuda10.1
6. NCCL version: 2.5.6
7. Python version: python3.6

Question
I can successfully train my model using horovod in a multi-machine-multi-card fashion, and save the checkpoint models only on worker 0 to prevent other workers from corrupting them. My question is how to load such pre-trained models using horovod in a multi-machine-multi-card fashion, Do I need to load the pre-trained models only on worker 0 ?

[maxhgerlach]

You could restore the checkpoint on rank 0, then broadcast the variables to the other workers, similarly to what you would do after initialization.