[BUG] OpenMPI backend doesn't support custom mpi launch args

[flyhighzy]

Describe the bug
In our business, we need to specify a custom mpi port rather than default 22 to communicate, in normal mpi, we can pass -mca plm_rsh_args -p 5000 to specify the port, but deepspeed not work.

I looked into the source code, found OpenMPI multi-node runner doesn't use the launcher_args passed by command --launcher_args

Expected behavior
deepspeed bin accept launcher_args and pass them to openMPI launcher

ds_report output

--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
cpu_adam ............... [NO] ....... [OKAY]
cpu_adagrad ............ [NO] ....... [OKAY]
fused_adam ............. [NO] ....... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
sparse_attn ............ [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... [NO] ....... [NO]
transformer_inference .. [NO] ....... [OKAY]
utils .................. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/usr/local/lib/python3.6/dist-packages/torch']
torch version .................... 1.10.0+cu102
torch cuda version ............... 10.2
nvcc version ..................... 10.2
deepspeed install path ........... ['/usr/local/lib/python3.6/dist-packages/deepspeed']
deepspeed info ................... 0.5.6, unknown, unknown
deepspeed wheel compiled w. ...... torch 0.0, cuda 0.0

System info (please complete the following information):

• OS: Ubuntu 18.04.6 LTS
• GPU count and types ： dynamic number of nodes, V100 or A100
• Interconnects (if applicable) [not clear]
• Python version： 3.6.9

Launcher context
Are you launching your experiment with the deepspeed launcher, MPI, or something else?
launch cmd like:

deepspeed --master_addr ${master_addr} --master_port 1234 --hostfile ${HOST_FILE} \
       --launcher OpenMPI \
       --launcher_args "--allow-run-as-root -mca plm_rsh_args -p 5000" \
       {user_script} {user_script_args}

[mrwyattii]

I think the fix for this is simple. Can you try #1933 and let me know if that works?

git clone https://github.com/microsoft/DeepSpeed.git && cd DeepSpeed
git pull origin pull/1933/head
pip install .

[jeffra]

@flyhighzy, can you confirm if this fixes your issue or not?

[flyhighzy]

maybe it's not that simple.
If I pass launcher args as "-mca plm_rsh_args -p 5000", it will be splited as ["-mca", "plm_rsh_args", "-p", "5000"], but expected is ["-mca", "plm_rsh_args", "-p 5000"]

[mrwyattii]

@flyhighzy I modified the PR to use shlex.split. This should fix the problem you noted with -p 5000 being split into separate strings. One thing to note is that you will need to place a quotes around that flag. So your launch command would be:

deepspeed --master_addr ${master_addr} --master_port 1234 --hostfile ${HOST_FILE} \
       --launcher OpenMPI \
       --launcher_args "--allow-run-as-root -mca plm_rsh_args \"-p 5000\"" \
       {user_script} {user_script_args}

Can you pull the latest chanes in #1933 and let me know if that works? Thanks!

[flyhighzy]

Now it works!
Thanks for the update

[mrwyattii]

Thanks for testing it. We'll get this merged :)