Flaky test test_gluon.test_export

[KellenSunderland]

Failed build:
http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/master/1148/pipeline
http://jenkins.mxnet-ci.amazon-ml.com/blue/rest/organizations/jenkins/pipelines/incubator-mxnet/branches/master/runs/1148/nodes/822/log/?start=0

Output:

======================================================================
ERROR: test_gluon.test_export
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Anaconda3\envs\py3\lib\site-packages\nose\case.py", line 197, in runTest
    self.test(*self.arg)
  File "C:\jenkins_slave\workspace\ut-python-cpu@3\tests\python\unittest\common.py", line 157, in test_new
    orig_test(*args, **kwargs)
  File "C:\jenkins_slave\workspace\ut-python-cpu@3\tests\python\unittest\test_gluon.py", line 787, in test_export
    prefix='resnet', ctx=ctx, pretrained=True)
  File "C:\jenkins_slave\workspace\ut-python-cpu@3\pkg_vc14_cpu\python\mxnet\gluon\model_zoo\vision\resnet.py", line 406, in resnet18_v1
    return get_resnet(1, 18, **kwargs)
  File "C:\jenkins_slave\workspace\ut-python-cpu@3\pkg_vc14_cpu\python\mxnet\gluon\model_zoo\vision\resnet.py", line 390, in get_resnet
    root=root), ctx=ctx)
  File "C:\jenkins_slave\workspace\ut-python-cpu@3\pkg_vc14_cpu\python\mxnet\gluon\model_zoo\model_store.py", line 114, in get_model_file
    os.remove(zip_file_path)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Windows\\system32\\config\\systemprofile\\.mxnet\\models\\resnet18_v1-a0666292.zip'

-------------------- >> begin captured stdout << ---------------------
Model file is not found. Downloading.
Downloading C:\Windows\system32\config\systemprofile\.mxnet\models\resnet18_v1-a0666292.zip from https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/models/resnet18_v1-a0666292.zip...
--------------------- >> end captured stdout << ----------------------

-------------------- >> begin captured logging << --------------------
requests.packages.urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): apache-mxnet.s3-accelerate.dualstack.amazonaws.com
requests.packages.urllib3.connectionpool: DEBUG: https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com:443 "GET /gluon/models/resnet18_v1-a0666292.zip HTTP/1.1" 200 43452554
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=767602412 to reproduce.
--------------------- >> end captured logging << ---------------------

[szha]

Looks like a CI problem @marcoabreu

[marcoabreu]

To me, it looks like a test problem. It seems like we are hardcoding paths, causing problems if multiple runs are triggered in parallel. The same could happen on a user system if they have multiple processes starting up inside the same container or operating system, so we should consider this as an important bug.

Also, why de we write to system32?

[szha]

I was treating sandboxing as a CI problem, as I could hardly think of any circumstance in which we'd want tests from one Jenkins executor to interact with tests in another executor on the same host.

[szha]

Even if there is such case, I imagine we'd want to manage it explicitly.

[szha]

That particular API uses $HOME/.mxnet as the default path, so the system path likely comes from $HOME. One simple solution to fix such case is to override $HOME and let it point to a managed temporary file path.

[marcoabreu]

On production systems while running a service, we can't necessarily assume that every single request is handled in a sandbox. On Ubuntu we actually run in a separate container for every single run, but this is not the case. The windows use case seems to be pretty close to reality.

We don't want to interact in between executors, but that's a side effect because we don't make unique paths. Just load the model zoo in two separate processes and this problem will pop up.

I'd recommend to store the files in a specific non unique directory and check if they exist (and match the hash) before downloading the archive from the internet. If it doesn't exist, download the zip file to a temporary dir. Then put a readwritelock (shared read, unique write) onto the non unique target path and extract the archive file into it. Right now, the code assumes that we have exclusive access - especially the code which tries to delete the archive file which might be used from a different process because it's also non unique (on mobile, otherwise I'd post the code).

[marcoabreu]

Thanks for elaborating on the home variable. That makes sense because we run as administrator on windows.

[szha]

This situation has nothing to do with a micro service...

[KellenSunderland]

I'd agree this is a sandboxing issue in the CI. We'll take a look at how we can improve this @szha.

[marcoabreu]

To clarify: do we expect our users to be able to run multiple instances of MXNet on the same host?