distributed kvstore bug in MXNet

[eric-haibin-lin]

I'm using distributed kvstore with Gluon trainer. I found the two following bugs:

1. Initializing trainer = gluon.Trainer(update_on_kvstore=True) doesn't work. Inspecting trainer._update_on_kvstore shows that the value is still set to False.

2. When distributed kvstore is used, by default gluon.Trainer doesn't work with mx.optimizer.LRScheduler if a worker has more than 1 GPU. To be more specific, the trainer updates once per GPU, the LRScheduler object is shared across GPUs and get a wrong update count.

This means one cannot train imagenet classification using resnet with gluon trainer.

[vrakesh]

@eric-haibin-lin Thank you for reporting the issue,
@mxnet-label-bot [Bug, Gluon]

[ragavvenkatesan]

+1

[sandeep-krishnamurthy]

Working on this. Will update my findings.

[lupesko]

@sandeep-krishnamurthy is this issue fixed with #12786 ?
If so - please close the issue or comment.

[sandeep-krishnamurthy]

• Initializing trainer = gluon.Trainer(update_on_kvstore=True) doesn't work. Inspecting trainer._update_on_kvstore shows that the value is still set to False.

This is fixed.

• When distributed kvstore is used, by default gluon.Trainer doesn't work with mx.optimizer.LRScheduler if a worker has more than 1 GPU. To be more specific, the trainer updates once per GPU, the LRScheduler object is shared across GPUs and get a wrong update count.

This needs to be fixed.

[lebeg]

As mentioned in #12786 the fix for the 1st problem has issues on the v1.3.x release branch.