Address already in use during tutorial test

[marcoabreu]

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/PR-10656/7/pipeline/

+ nosetests-3.4 test_tutorials.py --nologcapture

....[19:25:36] src/io/iter_image_recordio_2.cc:170: ImageRecordIOParser2: ./data/caltech.rec, use 4 threads for decoding..

.......[19:27:51] src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:107: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)

.....[19:28:58] src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:107: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)

..[19:32:06] src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:107: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)

[19:37:22] src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:107: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)

....[19:39:25] src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:107: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)

.[19:39:51] src/nnvm/legacy_json_util.cc:209: Loading symbol saved by previous version v0.8.0. Attempting to upgrade...

[19:39:51] src/nnvm/legacy_json_util.cc:217: Symbol successfully upgraded!

.....[19:40:24] src/operator/././../common/utils.h:416: Optimizer with lazy_update = True detected. Be aware that lazy update with row_sparse gradient is different from standard update, and may lead to different empirical results. See https://mxnet.incubator.apache.org/api/python/optimization/optimization.html for more details.

..[19:40:43] src/operator/././../common/utils.h:416: Optimizer with lazy_update = True detected. Be aware that lazy update with row_sparse gradient is different from standard update, and may lead to different empirical results. See https://mxnet.incubator.apache.org/api/python/optimization/optimization.html for more details.

..Traceback (most recent call last):

  File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main

    "__main__", mod_spec)

  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code

    exec(code, run_globals)

  File "/usr/local/lib/python3.5/dist-packages/ipykernel_launcher.py", line 16, in <module>

    app.launch_new_instance()

  File "/usr/local/lib/python3.5/dist-packages/traitlets/config/application.py", line 657, in launch_instance

    app.initialize(argv)

  File "<decorator-gen-123>", line 2, in initialize

  File "/usr/local/lib/python3.5/dist-packages/traitlets/config/application.py", line 87, in catch_config_error

    return method(app, *args, **kwargs)

  File "/usr/local/lib/python3.5/dist-packages/ipykernel/kernelapp.py", line 456, in initialize

    self.init_sockets()

  File "/usr/local/lib/python3.5/dist-packages/ipykernel/kernelapp.py", line 248, in init_sockets

    self.control_port = self._bind_socket(self.control_socket, self.control_port)

  File "/usr/local/lib/python3.5/dist-packages/ipykernel/kernelapp.py", line 180, in _bind_socket

    s.bind("tcp://%s:%i" % (self.ip, port))

  File "zmq/backend/cython/socket.pyx", line 547, in zmq.backend.cython.socket.Socket.bind

  File "zmq/backend/cython/checkrc.pxd", line 25, in zmq.backend.cython.checkrc._check_rc

zmq.error.ZMQError: Address already in use

F..

======================================================================

FAIL: test_tutorials.test_unsupervised_learning_gan

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 198, in runTest

    self.test(*self.arg)

  File "/work/mxnet/tests/tutorials/test_tutorials.py", line 203, in test_unsupervised_learning_gan

    assert _test_tutorial_nb('unsupervised_learning/gan')

AssertionError: 

-------------------- >> begin captured stdout << ---------------------

Kernel died before replying to kernel_info



--------------------- >> end captured stdout << ----------------------



----------------------------------------------------------------------

Ran 35 tests in 1024.216s



FAILED (failures=1)

build.py: 2018-05-31 19:41:38,780 Running of command in container failed (1): nvidia-docker run --rm -t --shm-size=3g -v /home/jenkins_slave/workspace/it-tutorials-py2:/work/mxnet -v /home/jenkins_slave/workspace/it-tutorials-py2/build:/work/build -u 1001:1001 mxnet/build.ubuntu_gpu /work/runtime_functions.sh tutorialtest_ubuntu_python2_gpu

build.py: 2018-05-31 19:41:38,780 You can try to get into the container by using the following command: nvidia-docker run --rm -t --shm-size=3g -v /home/jenkins_slave/workspace/it-tutorials-py2:/work/mxnet -v /home/jenkins_slave/workspace/it-tutorials-py2/build:/work/build -u 1001:1001 -ti --entrypoint /bin/bash mxnet/build.ubuntu_gpu /work/runtime_functions.sh tutorialtest_ubuntu_python2_gpu

into container: False

Traceback (most recent call last):

  File "ci/build.py", line 307, in <module>

    sys.exit(main())

  File "ci/build.py", line 243, in main

    container_run(platform, docker_binary, shared_memory_size, command)

  File "ci/build.py", line 154, in container_run

    raise subprocess.CalledProcessError(ret, cmd)

subprocess.CalledProcessError: Command 'nvidia-docker run --rm -t --shm-size=3g -v /home/jenkins_slave/workspace/it-tutorials-py2:/work/mxnet -v /home/jenkins_slave/workspace/it-tutorials-py2/build:/work/build -u 1001:1001 mxnet/build.ubuntu_gpu /work/runtime_functions.sh tutorialtest_ubuntu_python2_gpu' returned non-zero exit status 1

script returned exit code 1

Possibly related to hardcoding ports.

[marcoabreu]

@ThomasDelteil

[ThomasDelteil]

I tracked this issue for a while, still not sure of why it is happening. My first guess was that the previous notebook run didn't clear the socket in time for the next run.

I added a delay between each notebook run which should have been plenty of time for socket to be released.

I'll do some more digging.

@marcoabreu, how many containers are running per instance concurrently? only one right?

[marcoabreu]

They are running concurrently. We got up to 4 containers in parallel. Why should it matter whether a socket is released or not? First of all, networking is virtualized per container, so there should be no problem on that side. Second, we should not require a certain port but just use a random free one.

[ThomasDelteil]

The ports are already random, but I don't know whether it checks for availability. I think the issue could be that the ports are not released and rarely but sometimes it happens to try to reuse the same random port.

I have also seen a 'linger=1000', which could mean that the port might be still taken 1s after closing, so ugprading the delay between notebook to 1.1 from 0.5 could also solve the issue.

I'll see if I can make the port deterministically different.

[marcoabreu]

I'm not a fan of the delay in between notebooks anyways because it masks a problem. I'd propose that we now remove the delay entirely and track down all the issues coming from that. Otherwise, we're flaky and depend on timing.

[ThomasDelteil]

the linger variable is hard coded in the kernelapp.py, if we don't want to use this technique then we need to assign deterministic ports that are different from each other and update the config file of the python notebook between runs.

[marcoabreu]

I think that's what we should do. In general, this is required to prepare the path for parallel execution.

[access2rohit]

HI Marco,
For reproducing this issue I have to build docs within MXNet since it requires gan.ipynb file. Can you share the setup to reproduce it

[marcoabreu]

@ThomasDelteil would you mind assisting here?

[anirudh2290]

assigned to @reminisce. @access2rohit is working on this.

[ThomasDelteil]

To reproduce the setup I suggest looking at the jenkins function for tutorial tests. I had put a fix in my last PR before removing them from CI, the issue might be gone already. Can someone start a few hundred runs of the tutorial tests and see if it still happen? Note that they take ~25min so that could take a few days. Actually commenting out most tests except three very fast ones might be a better idea since it isn't related to a specific test, and as one simple test runs in 2-3s with the jupyter kernel overhead. To know which one is fast check the tutorials, some do not much like the NDArray ones. My current best guess is that the issue is related to the fact that the ports used by jupiter internal mechanism are chosen randomly and that there is linger=1000 hard-coded in the jupiter code somewhere that keep it being used for 1sec. For every test there is ~1/10000 chance that the same port will be reused (3 ports are picked between 1-100000), which makes it 1/300 because we have 30 tests and 1/150 because we run on python2 and python3. That seems roughly consistent with the number of reports we've had, about once every 150 CI runs. There is no easy way to set the ports to fixed deterministic value. My latest fix added a non-ideal 1.1 sleep between tests. Let's see if that fixed it. The above explanation might be bogus too. I'm on my phone in a plane and can assist more from Friday onwards. Thanks for looking into it @reminisce <https://github.com/reminisce> and @access2rohit <https://github.com/access2rohit>!

…

On Wed, Jun 27, 2018, 20:45 Anirudh Subramanian ***@***.***> wrote: assigned to @reminisce <https://github.com/reminisce> @access2rohit <https://github.com/access2rohit> is working on this. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#11120 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADi001F-ydKxRBX2r1PYR-DJXVfJv14lks5uBBkDgaJpZM4UWPoF> .

[marcoabreu]

Just to be on the safe side: please try what Thomas suggested only locally and don't submit these jobs to CI. This test suite causes resource exhaustion on our CI and running so many in parallel would basically knock off our instances. Thomas Delteil <notifications@github.com> schrieb am Do., 28. Juni 2018, 11:29:

…

To reproduce the setup I suggest looking at the jenkins function for tutorial tests. I had put a fix in my last PR before removing them from CI, the issue might be gone already. Can someone start a few hundred runs of the tutorial tests and see if it still happen? Note that they take ~25min so that could take a few days. Actually commenting out most tests except three very fast ones might be a better idea since it isn't related to a specific test, and as one simple test runs in 2-3s with the jupyter kernel overhead. To know which one is fast check the tutorials, some do not much like the NDArray ones. My current best guess is that the issue is related to the fact that the ports used by jupiter internal mechanism are chosen randomly and that there is linger=1000 hard-coded in the jupiter code somewhere that keep it being used for 1sec. For every test there is ~1/10000 chance that the same port will be reused (3 ports are picked between 1-100000), which makes it 1/300 because we have 30 tests and 1/150 because we run on python2 and python3. That seems roughly consistent with the number of reports we've had, about once every 150 CI runs. There is no easy way to set the ports to fixed deterministic value. My latest fix added a non-ideal 1.1 sleep between tests. Let's see if that fixed it. The above explanation might be bogus too. I'm on my phone in a plane and can assist more from Friday onwards. Thanks for looking into it @reminisce <https://github.com/reminisce> and @access2rohit <https://github.com/access2rohit>! On Wed, Jun 27, 2018, 20:45 Anirudh Subramanian ***@***.***> wrote: > assigned to @reminisce <https://github.com/reminisce> @access2rohit > <https://github.com/access2rohit> is working on this. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > < #11120 (comment) >, > or mute the thread > < https://github.com/notifications/unsubscribe-auth/ADi001F-ydKxRBX2r1PYR-DJXVfJv14lks5uBBkDgaJpZM4UWPoF > > . > — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#11120 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ARxB60DSOdndKcaobWgfxBGzIAMRSv4Dks5uBKH-gaJpZM4UWPoF> .