Broadcasting local arrays among all nodes

[Snoeprol]

I'm trying to broadcast an array among all the nodes. I found this piece of code which is almost what I'm looking for:
hvd.broadcast_parameters(model.state_dict(), root_rank=0), but not exactly. I on all nodes I have arrays say indices = [1, 2, 3, 4], with different values and lengths for all nodes. I want all the nodes to have access to all these arrays. What would be the best way to do this?
I can make some kind of for loop:

indices{hvd.rank()} = []
# Fill it up
for i in range(hvd.size()):
                hvd.broadcast_parameters(indices{i}, root_rank = i)

What is a more efficient way of doing this? And does my proposed method work?

[maxhgerlach]

on all nodes I have arrays say indices = [1, 2, 3, 4], with different values and lengths for all nodes. I want all the nodes to have access to all these arrays.

I think allgather is the operation you are looking for. :)

• https://horovod.readthedocs.io/en/stable/api.html#horovod.tensorflow.allgather
• https://horovod.readthedocs.io/en/stable/api.html#horovod.torch.allgather
• etc.