Skip to content
This repository was archived by the owner on Nov 17, 2023. It is now read-only.
This repository was archived by the owner on Nov 17, 2023. It is now read-only.

MXNet 1.5.0 is slower than 1.3.0 when intputs are variant #13928

Description

@wkcn

Description

Hi! I have an experiment about Object Counting, which needs variant inputs.
I write the code with Gluon, and hybridize the model with static_alloc=True
I found there is obvious difference between MXNet 1.5.0 and MXNet 1.3.0, and I checked it on two servers.

I think the method of memory allocation for Gluon may be changed after MXNet 1.3.0.

Thanks!

Update:
When there are dilated Convolutional layers in the model, and the input size is variational, the performance will drop.
I think it may be related to one of the two PRs: #11742 #12722

Environment info (Required)

OS: ubuntu 14.04
GPU: Tesla M40 x 4

Minimum reproducible example

I write a minimum reproducible example without dataset.
Code

  • Performance for test code [a fully convolutional model(vgg16 without FC layer), variant inputs]:
    MXNet 1.5.0: 10 images / sec
    MXNet 1.3.0: 40+ images / sec

The performances are the same when input shape is fixed.

Input shape: (9, 3, 300~512, 300~512) in NCHW order

Package used (Python/R/Scala/Julia):
Python 2.7.12, 3.7.1

MXNet is installed by pip:

# MXNet 1.5.0
pip install mxnet-cu80 --pre
# MXNet 1.3.0
pip install mxnet-cu80==1.3.0

Steps to reproduce

Download the test code.
Run the test code in different version (1.3.0 and 1.5.0) of MXNet.

Performance

I test several versions of MXNet.

version performance
1.4.0b20181207 slow
1.3.1b20181101 slow
1.3.1b20181010 slow
1.3.1b20181004 fast
1.3.1b20181001 fast

Some pre-build versions don't support CUDA9.0, so I cound't test it.
The performance drops during 20181004 to 20181010.

If changing the dilation of dilated conv to 1, the performance will be normal.
It seems the problem occurs in dilated conv.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions