[BUG] deepspeed.runtime.zero.utils.ZeRORuntimeException: You are using ZeRO-Offload with a client provided optimizer

[LUMO666]

Run /scripts/run_raft_align.sh in docker and get an error.

deepspeed.runtime.zero.utils.ZeRORuntimeException: You are using ZeRO-Offload with a client provided optimizer (<class 'transformers.optimization.AdamW'

) which in most cases will yield poor performance. Please either use deepspeed.ops.adam.DeepSpeedCPUAdam or set an optimizer in your ds-config (https://www.deepspeed.ai/docs/config-json/#optimizer-parameters). If you really want to use a custom optimizer w. ZeRO-Offload and understand the performance impacts you can also set <"zero_force_ds_cpu_optimizer": false> in your configuration file.

Is it related to mpi4py? I'm doubting whether I have mpi4py installed correctly. Thanks.

[research4pan]

@WeiXiongUST @hendrydong I am wondering if you could take a look? Thanks 🙏

[hendrydong]

Hi, it looks that the configuations of "ZeRO-Offload" is not correct, you may double check the yaml file.

BTW, this might be more related to the configuration of deepspeed.