Distributed learning with Async update does not work.

[absalama]

Hi All,

I am running currently distributed learning using the launch.py tool. My plan is to run the distributed learning by using Sync and Async distributed modes. For the Sync PS, the training works fine and I got the number. However, when I switch the kvstore type to "dist_async", the following error shows up:

terminate called after throwing an instance of 'dmlc::Error' what(): [11:58:12] src/kvstore/././kvstore_dist_server.h:294: Check failed: sync_mode_ Updater needs to be set for async mode Stack trace returned 10 entries: [bt] (0) /work/projects/Project00755/envs/mxnet_gpu/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x30cbe2) [0x2aedf8ebdbe2] [bt] (1) /work/projects/Project00755/envs/mxnet_gpu/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x30d1b8) [0x2aedf8ebe1b8] [bt] (2) /work/projects/Project00755/envs/mxnet_gpu/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x2ab1f11) [0x2aedfb662f11] [bt] (3) /work/projects/Project00755/envs/mxnet_gpu/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x2ab6f10) [0x2aedfb667f10] [bt] (4) /work/projects/Project00755/envs/mxnet_gpu/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x2ab74d4) [0x2aedfb6684d4] [bt] (5) /work/projects/Project00755/envs/mxnet_gpu/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x2a97b06) [0x2aedfb648b06] [bt] (6) /work/projects/Project00755/envs/mxnet_gpu/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x2f640d5) [0x2aedfbb150d5] [bt] (7) /shared/apps/gcc/4.9.4/lib64/libstdc++.so.6(+0xc9090) [0x2aee34139090] [bt] (8) /lib64/libpthread.so.0(+0x7e25) [0x2aedc9953e25] [bt] (9) /lib64/libc.so.6(clone+0x6d) [0x2aedca36934d]

Any idea why this error happens?

Thanks :)

[rahul003]

Are you using Gluon? You would have to pass update_on_kvstore=True to trainer if you want to do async updates in Gluon.

[sandeep-krishnamurthy]

@rahul003 - Is this documented in example or on mxnet site?

[absalama]

@rahul003

yes I am using Gluon. The trainer is ImageClassification, would it work if export update_on_kvstore=True?

[rahul003]

@sandeep-krishnamurthy Improved this here #11910

@absalama You need to pass update_on_kvstore=True when creating the Trainer. Refer the test above.

[absalama]

Hi all,

Thank you all guys for you help. However, I am still seeing the issue even after passing "update_on_kvstore=True".
I am using example/gluon/ImageClassification with slurm tracker from dmlc. Please could you reference to the error trace on the top since the error is still generated from the same corner.

Thanks

[rahul003]

Are you sure you set this variable on each of your machines, and can you post a snippet? I just tested the change in above pr and it worked for me.

[absalama]

In trainer.py I changed the default value of update_on_kvstore from None to True in the init method. The imageclassification.py initialises the trainer object so I assume that if the update_on_kvstore is True in the init method then it should be set?

We are using slurm cluster, and all nodes are sharing the same folder where mxnet source resides. So the any change should be seen by all worker.

Do I need to set something extra for slurm configuration?

[rahul003]

What optimizer are you using?

[absalama]

I did not set optimizer so I think the default one is used, below is more detailed log.

INFO:root:Starting new image-classification task:, Namespace(batch_norm=False, batch_size=128, builtin_profiler=0, data_dir='', dataset='dummy', dtype='float32', epochs=120, gpus='0', kvstore='dist_async', log_interval=50, lr=0.01, lr_factor=0.1, lr_steps='30,60,90', mode=None, model='alexnet', momentum=0.9, num_workers=4, prefix='', profile=False, resume='', save_frequency=10, seed=123, start_epoch=0, use_pretrained=False, use_thumbnail=False, wd=0.0005) [17:42:05] src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:107: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable) terminate called after throwing an instance of 'dmlc::Error' what(): [17:42:08] src/kvstore/././kvstore_dist_server.h:294: Check failed: sync_mode_ Updater needs to be set for async mode

The following command I use in my training:

python3 launch.py -n 4 -s 1 -H ${NODES_DIR}/hosts --launcher slurm alex_train_single_GPU_dist_async.sh

and alex_train_single_GPU_dist.sh is as follows:

export USE_CUDA=1
export MXNET_CUDNN_AUTOTUNE_DEFAULT=1
export USE_PROFILER=1
export USE_CUDNN=1
export USE_NVRTC=1
export DMLC_INTERFACE="ib0"
export USE_DIST_KVSTORE=1

python $MXNET_GLUON_ASYNC/image_classification.py --dataset dummy  --kvstore dist_async --gpus=0  --model alexnet  --batch-size 128  --lr 0.01  --mom 0.9 --wd 0.0005