[call for contribution] Improving CPU performance

[hjk41]

Currently we are still slow on CPU (#1222).

There are several things we can do:

1. Do a good profile on a chosen set of benchmarks to understand the bottlenecks.
   Here are some candidates:
   1. MNIST CNN model: since the MNIST model is very small, performance will suffer if system overhead is high, and it will show us potential bottlenecks in non-CNN operations
   2. Cifar: this is much heavier than MNIST, so it will mostly show us the performance of the underlying library we are using, that is OpenBLAS/MKL + MShadow. I think the configuration (#of threads) of the libraries has quite a lot of impact on overall performance.
2. Integrate libraries like NNPACK (https://github.com/Maratyszcza/NNPACK) and MKLDNN (https://software.intel.com/en-us/articles/deep-neural-network-technical-preview-for-intel-math-kernel-library-intel-mkl).
3. Improve the operators not included in NNPACK and MKLDNN. This would include some code in MShadow and some in mxnet operators.

[winstywang]

Can you first confirm that the bottleneck is in computation? As far as I know, caffe uses the same computational engine as ours, but much faster than ours.

[Darwin2011]

@winstywang @hjk41
I suspect that patch2cols in mxnet is one bottleneck rather than blas library.

[tqchen]

@Darwin2011 can you try to do some change to see if that is really the case? Thanks!

[xmchen1987]

@hjk41 After uncomment OpenMP in mshadow/mshadow/tensor_cpu-inl.h, mxnet is still slower than caffe. The detail performance data on Intel(R) Xeon(R) CPU E5-4657L v2 is shown as follow:

i can follow up the issue:

1. give a performance analysis of mxnet, show which part is slower.
2. evaluate NNPACK and MKL DNN, to see any potential performance gain.
3. integrate NNPACK or MKL DNN into mxnet.

[Maratyszcza]

Unlike MKL, NNPACK is not x86-only: it now includes kernels implemented with Clang vector extensions. The implementation was originally developed for PNaCl port of NNPACK, but can be compiled to any clang-supported architecture; LLVM will automatically lower the vector operations to the SIMD ISA of the target. To build NNPACK with portable SIMD implementation instead of x86-64 assembly, configure with python configure.py --enable-psimd. On Haswell+ targets, however, performance would be about 4 times slower than with x86-64 assembly.

[winstywang]

@xmchen1987 Could you help run a profile on the CPU analysis? Let's figure out the bottleneck.

[xmchen1987]

@hjk41 @winstywang @Darwin2011 I add time measurement function in src/operator/convolution-inl.h, and dump the running time for Alexnet.
From the table below, we can see:

1. convolution layers consume most of running time.
2. time for patch2col/col2patch is also the same with that for gemm. i remeber time for patch2col/col2patch on caffe is very small.
   

now there are two options to fix this problem:

1. fix patch2col/col2patch performance issue
2. use MKL DNN to avoid use patch2col/col2patch.

what do you guys think about?

[hjk41]

@xmchen1987 great finding.
I guess we can use the same code as Caffe on patch2col? That should solve the problem.

Integrating MKL DNN is also a good idea. But I think if we decide to do that, we should do it systematically and replace every operator that MKL DNN has. @xmchen1987 are you interested in making this contribution?

[hjk41]

@ALL I just updated the issue. As some of you suggested, we should do some profiling first. And we should make sure we have comparable performance with Caffe on most of the critical applications. I just listed MNIST-CNN and Cifar as two of the candidate benchmarks. What else can you think of?

[xmchen1987]

@hjk41 sure, i can do that.

Caffe uses memory copy to do patch2col, which is more effciency. no problem to use that method.
i think we can add MKL DNN as another options, as MKL DNN can helps us to achieve better scalibility on cores.

[xmchen1987]

@hjk41 can we add Alexnet, as most of benchmark for DL framework has Alexnet?

[tornadomeet]

@xmchen1987 does MKL DNN only support intel cpu? because many user will use mxnet in the mobile devices using ARM, so it's better using the general implementation.

[hjk41]

Good idea. Alexnet would show us the performance of the CNN implementation.
I guess MKL DNN would really show its benefit in this case.

On Fri, Aug 12, 2016 at 11:19 AM, xmchen notifications@github.com wrote:

@hjk41 https://github.com/hjk41 can we add Alexnet, as most of
benchmark for DL framework has Alexnet?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#2986 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABFI4YgWUAjK4Q39unja8awG1wcBXWHyks5qe-ZYgaJpZM4Jg0fu
.

[hjk41]

@tornadomeet MKL supports only x86 CPUs. I think we should add a compile option like we do for OpenBLAS/MKL. I believe MKL DNN will become the de-facto choice for DNN on x86, just like cuDNN for GPU.