OSError: [Errno 98] Address already in use

[chencjcj]

dlrover version：v0.3.5
megatron version：main
I encountered an error when using flash checkpoint in megatron：

Exception in thread checkpoint-saver:
Traceback (most recent call last):
File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/usr/lib/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.10/dist-packages/dlrover/python/elastic_agent/torch/ckpt_saver.py", line 422, in _saver
saver: AsyncCheckpointSaver = class_def(**class_meta.kwargs)
File "/usr/local/lib/python3.10/dist-packages/dlrover/python/elastic_agent/torch/ckpt_saver.py", line 386, in init
self._event_queue = SharedQueue(name=qname, create=True)
File "/usr/local/lib/python3.10/dist-packages/dlrover/python/common/multi_process.py", line 369, in init
super().init(name, create)
File "/usr/local/lib/python3.10/dist-packages/dlrover/python/common/multi_process.py", line 188, in init
self._init_socket()
File "/usr/local/lib/python3.10/dist-packages/dlrover/python/common/multi_process.py", line 210, in _init_socket
self._server = _create_socket_server(self._socket_file)
File "/usr/local/lib/python3.10/dist-packages/dlrover/python/common/multi_process.py", line 71, in _create_socket_server
server.bind(path)
OSError: [Errno 98] Address already in use
Exception ignored in: <function AsyncCheckpointSaver.del at 0x7efed6fbb490>
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/dlrover/python/elastic_agent/torch/ckpt_saver.py", line 402, in del
[2024-05-10 07:57:02,115] [INFO] [ckpt_saver.py:429:_factory] Start the checkpoint saver factory.
self.close()
File "/usr/local/lib/python3.10/dist-packages/dlrover/python/elastic_agent/torch/ckpt_saver.py", line 494, in close
if not self._event_queue.empty():
AttributeError: 'MegatronCheckpointSaver' object has no attribute '_event_queue'

[workingloong]

Can you retry it with dlrover[torch]==0.3.7. We have fixed some bugs for Megatron-LM after 0.3.5 and the bug may have been fixed.

[chencjcj]

Can you give me a tested version of megatron-lm? When I use megatron-lm-main, I get some errors.

[workingloong]

You can test it with the repo https://github.com/workingloong/Megatron-LM-CKPT forked from Megatron-LM in 2024.02.