CPU speed problem: MxNet 4 times slow than Caffe

[wangzhangup]

I convert same model and parameter from caffe to MxNet. Everything works fine. But time spend more than about 4 times. Only using CPU
They both use atlas, same machine

[qiaohaijun]

caffe and mxnet all in the cpu model?

[wangzhangup]

@qiaohaijun yes, CPU on same machine

[starimpact]

@tqchen @piiswrong

[winstywang]

Could you help to do some profile like in http://www.network-theory.co.uk/docs/gccintro/gccintro_80.html ?

[uestcbingo]

I encountered the same issue, CPU version of mxnet is 3 times slower than caffe both using MKL. Did you solve the problem? @wangzhangup

[yzhliu]

@wangzhangup @uestcbingo Curious about this issue, mind listing the caffe model configuration?

[hjk41]

Guys, is there any performance profiling tools in mxnet? I am also trying
to figure out the issue on CPU. Currently mxnet does not fully utilize my
CPUs on Windows.

On Tue, Feb 2, 2016 at 12:28 AM, Yizhi Liu notifications@github.com wrote:

@wangzhangup https://github.com/wangzhangup @uestcbingo
https://github.com/uestcbingo Curious about this issue, mind listing
the caffe model configuration?

—
Reply to this email directly or view it on GitHub
#1222 (comment).

HONG Chuntao
System Research Group
Microsoft Research Asia

[piiswrong]

@hjk41 Not that I'm aware of. I have been thinking about adding profiling to the engine for a while. For example add a string tag to each Push and count time for each tag. But @tqchen Thinks this has negative performance implications.

BTW, there is a pretty easy fix for not fully using CPU: change mshadow/mshadow/tensor_cpu-inl.h:143 to a OpenMP parallel for. We have been talking about this for a while but no body has done it yet.

[hjk41]

ok. Back in Minerva v2, we had some kind of lightweight profiling which can
be turned on when needed. I think we can do the same for mxnet.

For the mshadow fix, why isn't this enabled by default?

On Tue, Feb 2, 2016 at 10:06 AM, Eric Junyuan Xie notifications@github.com
wrote:

@hjk41 https://github.com/hjk41 Not that I'm aware of. I have been
thinking about adding profiling to the engine for a while. For example add
a string tag to each Push and count time for each tag. But @tqchen
https://github.com/tqchen Thinks this has negative performance
implications.

BTW, there is a pretty easy fix for not fully using CPU: change
mshadow/mshadow/tensor_cpu-inl.h:143 to a OpenMP parallel for.

—
Reply to this email directly or view it on GitHub
#1222 (comment).

HONG Chuntao
System Research Group
Microsoft Research Asia

[hjk41]

@wangzhangup I am looking into this issue. Is it possible to give me the program that you are using? Both for mxnet and caffe?

[piiswrong]

Because... Its a Pr people have been too lazy to push?
On Feb 1, 2016 6:10 PM, "Chuntao Hong" notifications@github.com wrote:

ok. Back in Minerva v2, we had some kind of lightweight profiling which can
be turned on when needed. I think we can do the same for mxnet.

For the mshadow fix, why isn't this enabled by default?

On Tue, Feb 2, 2016 at 10:06 AM, Eric Junyuan Xie <
notifications@github.com>
wrote:

@hjk41 https://github.com/hjk41 Not that I'm aware of. I have been
thinking about adding profiling to the engine for a while. For example
add
a string tag to each Push and count time for each tag. But @tqchen
https://github.com/tqchen Thinks this has negative performance
implications.

BTW, there is a pretty easy fix for not fully using CPU: change
mshadow/mshadow/tensor_cpu-inl.h:143 to a OpenMP parallel for.

—
Reply to this email directly or view it on GitHub
#1222 (comment).

HONG Chuntao
System Research Group
Microsoft Research Asia

—
Reply to this email directly or view it on GitHub
#1222 (comment).