参数初始化的问题？

[junranhe]

我用了以前在cxxnet上跑的数据测试mxnet（网络结构用了alexnet和inception），发现一个奇怪的现象，只要不用batchnorm，网络都不收敛，而且运行的结果看起来学不到任何东西，简单地在卷积层后加上bn，网络就收敛正常了，一开始怀疑是inplace优化的问题，后来禁止了inplace还是一样，之前我在cxxnet上做过类似的实验，这个数据集都是能很好收敛的，是不是初始化参数权重的地方要特别注意的地方？能不能提供像cxxnet那么精确参数设置？
例如：
momentum = 0.9
wmat:lr = 0.05
wmat:wd = 0.0001
bias:wd = 0.000
bias:lr = 0.1

[tqchen]

You should be able to change the optimizer behavior easily by https://github.com/dmlc/mxnet/blob/master/python/mxnet/optimizer.py#L85

Consider to submit a PR on this :)

[antinucleon]

Thanks for letting us know We figure out a bug in initialization and we are fixing it. To set params for model, you may find this doc to be useful. https://mxnet.readthedocs.org/en/latest/python/model.html

[tqchen]

Was caused by inconsistent scale xavier initialization. Batchnorm makes things scale invariant so it poses a less problem

[antinucleon]

I changed default Initializer, Now it works on many nets without BN. Could you try again?

[junranhe]

更新代码不收敛的现象没了，btw，model.py 里,eval_metric.update() 可不可以挪到 调用epoch_end_callback 前， 这样比较方便我测试的时候传入自己的eval_metric和epochcallback来检查迭代情况，否则容易第一个batch抛除零异常。
还有一个发现一个奇怪的现象，可能跟机器相关，用的都是cudnn-v3，alexnet的计算时间是比cxxnet上快了很多，之前我的780是每秒180张，现在时260张，但inception-bn的速度却跟之前cxxnet的一样，都是34张每秒，没发现明显提高

[antinucleon]

What is your batch size of Inception BN?

On Tue, Sep 29, 2015 at 21:47 junranhe notifications@github.com wrote:

更新代码不收敛的现象没了，btw，model.py 里,eval_metric.update() 可不可以挪到
调用epoch_end_callback 前，
这样比较方便我测试的时候传入自己的eval_metric和epochcallback来检查迭代情况，否则容易第一个batch抛除零异常。

还有一个发现一个奇怪的现象，可能跟机器相关，用的都是cudnn-v3，alexnet的计算时间是比cxxnet上快了很多，之前我的780是每秒180张，现在时260张，但inception-bn的速度却跟之前cxxnet的一样，都是34张每秒，没发现明显提高

—
Reply to this email directly or view it on GitHub
#182 (comment).

[junranhe]

two 780，mxnet is 64batch，so each gpu 32batch， cxxnet is 20 batch each（I modify the cxxnet so it can run 20batch on 3GB memory）

[tqchen]

Normally you cannot expect mxnet to be faster than cxxnet in all cases as cxxnet is already heavily optimized in c++. But you can expect they run at same performance, with mxnet more memory efficient and more flexible.

You proposal seems to be valid, please open a PR on this

Tianqi

[junranhe]

well, when I run the same inception-bn on purine，and it run 50 sample per sec

[tqchen]

hmm, maybe you can try set update_on_kvstore=True http://mxnet.readthedocs.org/en/latest/python/model.html#mxnet.model.FeedForward.fit which will change default behavior. But I do not have tried 780 before so don't know if that would really help :)

[tqchen]

Sorry update_on_kvstore=False

[junranhe]

Oh, I try update_on_kvstore both True and False, it has no effect, and I try to run mxnet on single gpu, and inception-bn is 65 sample per sec... compare with two gpu 68 sample per sec(so 34 sample/sec), It seem just like used one gpu, Is there some lock make it slowdown? (I use dmesg can not find any error message, and I test alexnet is no any problem like inception-bn),