RuntimeError: ('Put data input KVStore server failed.', URLError(error(104, 'Connection reset by peer'),))

[xiaogengyaokeyan]

Hi guys, recently I'm using horovod.run.runner.run() python entry for one machine 8gpus trainning,

from horovod.run.runner import run as horovodrun
ret = horovodrun(distributed_train, args=(pt, data, save_path), np=8)[0]

and I run with this RuntimeError:

176509 Sat Jan 29 12:27:02 2022[5]<stderr>:Traceback (most recent call last):
176510 Sat Jan 29 12:27:02 2022[1]<stderr>:  File "/opt/conda/envs/py27_cu101/lib/python2.7/runpy.py", line 174, in _run_module_as_main
176511 Sat Jan 29 12:27:02 2022[5]<stdout>:Rank: 5, train start: 1643430183.24, end: 1643430422.87, cost: 239.625784159
176512 Sat Jan 29 12:27:02 2022[5]<stderr>:  File "/opt/conda/envs/py27_cu101/lib/python2.7/runpy.py", line 174, in _run_module_as_main
176513 Sat Jan 29 12:27:02 2022[1]<stdout>:Rank: 1, train start: 1643430183.24, end: 1643430422.87, cost: 239.626456022
176514 Sat Jan 29 12:27:02 2022[1]<stderr>:    "__main__", fname, loader, pkg_name)
176515 Sat Jan 29 12:27:02 2022[5]<stderr>:    "__main__", fname, loader, pkg_name)
176516 Sat Jan 29 12:27:02 2022[5]<stderr>:  File "/opt/conda/envs/py27_cu101/lib/python2.7/runpy.py", line 72, in _run_code
176517 Sat Jan 29 12:27:02 2022[5]<stderr>:    exec code in run_globals
176518 Sat Jan 29 12:27:02 2022[5]<stderr>:  File "/opt/conda/envs/py27_cu101/lib/python2.7/site-packages/horovod/run/run_task.py", line 38, in <module>
176519 Sat Jan 29 12:27:02 2022[5]<stderr>:    main(driver_addr, run_func_server_port)
176520 Sat Jan 29 12:27:02 2022[5]<stderr>:  File "/opt/conda/envs/py27_cu101/lib/python2.7/site-packages/horovod/run/run_task.py", line 32, in main
176521 Sat Jan 29 12:27:02 2022[5]<stderr>:    put_data_into_kvstore(addr, port, 'runfunc_result', str(rank), pickled_ret_val)
176522 Sat Jan 29 12:27:02 2022[1]<stderr>:  File "/opt/conda/envs/py27_cu101/lib/python2.7/runpy.py", line 72, in _run_code
176523 Sat Jan 29 12:27:02 2022[1]<stderr>:    exec code in run_globals
176524 Sat Jan 29 12:27:02 2022[5]<stderr>:  File "/opt/conda/envs/py27_cu101/lib/python2.7/site-packages/horovod/run/http/http_client.py", line 50, in put_data_into_kvstore
176525 Sat Jan 29 12:27:02 2022[5]<stderr>:    raise RuntimeError("Put data input KVStore server failed.", e)
176526 Sat Jan 29 12:27:02 2022[1]<stderr>:  File "/opt/conda/envs/py27_cu101/lib/python2.7/site-packages/horovod/run/run_task.py", line 38, in <module>
176527 Sat Jan 29 12:27:02 2022[5]<stderr>:RuntimeError: ('Put data input KVStore server failed.', URLError(error(104, 'Connection reset by peer'),))
176528 Sat Jan 29 12:27:02 2022[1]<stderr>:    main(driver_addr, run_func_server_port)
176529 Sat Jan 29 12:27:02 2022[1]<stderr>:  File "/opt/conda/envs/py27_cu101/lib/python2.7/site-packages/horovod/run/run_task.py", line 32, in main
176530 Sat Jan 29 12:27:02 2022[1]<stderr>:    put_data_into_kvstore(addr, port, 'runfunc_result', str(rank), pickled_ret_val)
176531 Sat Jan 29 12:27:02 2022[1]<stderr>:  File "/opt/conda/envs/py27_cu101/lib/python2.7/site-packages/horovod/run/http/http_client.py", line 50, in put_data_into_kvstore
176532 Sat Jan 29 12:27:02 2022[1]<stderr>:    raise RuntimeError("Put data input KVStore server failed.", e)
176533 Sat Jan 29 12:27:02 2022[1]<stderr>:RuntimeError: ('Put data input KVStore server failed.', URLError(error(104, 'Connection reset by peer'),))
176534 Process 1 exit with status code 1.
176535 Process 5 exit with status code 1.
176536 Traceback (most recent call last):
176537   File "job/run.py", line 170, in <module>
176538     main()
176539   File "job/run.py", line 143, in main
176540     chosen_pt = horovodrun(distributed_train, args=(ac_pt, object_id_list, progress_path), np=PPOConf.NP)[0]
176541   File "/opt/conda/envs/py27_cu101/lib/python2.7/site-packages/horovod/run/runner.py", line 815, in run
176542     return _run(hargs)
176543   File "/opt/conda/envs/py27_cu101/lib/python2.7/site-packages/horovod/run/runner.py", line 644, in _run
176544     _launch_job(args, remote_host_names, settings, nics, command)
176545   File "/opt/conda/envs/py27_cu101/lib/python2.7/site-packages/horovod/run/runner.py", line 717, in _launch_job
176546     args.verbose)
176547   File "/opt/conda/envs/py27_cu101/lib/python2.7/site-packages/horovod/run/runner.py", line 694, in run_controller
176548     gloo_run()
176549   File "/opt/conda/envs/py27_cu101/lib/python2.7/site-packages/horovod/run/runner.py", line 706, in gloo_run_fn
176550     gloo_run(settings, remote_host_names, nics, env, driver_ip, command)
176551   File "/opt/conda/envs/py27_cu101/lib/python2.7/site-packages/horovod/run/gloo_run.py", line 312, in gloo_run
176552     launch_gloo(command, exec_command, settings, nics, env, server_ip)
176553   File "/opt/conda/envs/py27_cu101/lib/python2.7/site-packages/horovod/run/gloo_run.py", line 304, in launch_gloo
176554     .format(name=name, code=exit_code))
176555 RuntimeError: Gloo job detected that one or more processes exited with non-zero status, thus causing the job to be terminated. The first process to do so was:
176556 Process name: 1
176557 Exit code: 1

which seems like an http error, however, I run the train 10 times about 4 times it ran into this error, the other times it just ran fine without any error...
btw, when I use shell command:
horovodrun -np 8 python distributed_train.py
I have never ran into this error, if this is caused by the http... how could I avoid this error using python entry