[TEST][FLAKY] test_op_grad_level2.py::test_conv2d_grad.py

[tqchen]

https://ci.tlcpack.ai/job/tvm/job/main/245/execution/node/218/log/

[tqchen]

cc @altanh @jroesch @antinucleon would be great if you can take a look

[altanh]

I suspect some recent PR might have broke something, this is the error: tests/python/relay/test_op_grad_level2.py::test_conv2d_grad Fatal Python error: Aborted.

Doesn't seem to me like a numerical issue with the gradient

[tqchen]

https://ci.tlcpack.ai/job/tvm/job/main/250/execution/node/233/log/ another one related

[altanh]

I can't reproduce this locally on the current main branch

[altanh]

per discussion with @tkonolige, we're pretty sure the abort is being caused by libomp conflicts between different 3rd party libraries (e.g. PyTorch and ONNX).

[tkonolige]

The error message is:

OMP: Error #15: Initializing libomp.dylib, but found libiomp5.dylib already initialized.
OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see http://openmp.llvm.org/

Pytorch loading:

dyld: loaded: <F7FFAF24-7A9F-35EA-B715-F2A2F250F575> /Users/tristan/Library/Python/3.8/lib/python/site-packages/torch/lib/libtorch_global_deps.dylib
dyld: loaded: <52F67CC7-A4B0-3F4D-A80D-7DC28D4A776A> /Users/tristan/Library/Python/3.8/lib/python/site-packages/torch/lib/../.dylibs/libiomp5.dylib

Onnx loading:

dyld: loaded: <C903042A-EFCF-3557-AB7C-155BA03165D0> /usr/local/lib/python3.8/site-packages/onnxruntime/capi/onnxruntime_pybind11_state.so
dyld: loaded: <FF7BABED-D8CA-3F78-BCE2-F0C293919D70> /usr/local/opt/libomp/lib/libomp.dylib

[altanh]

Relevant issue on onnxruntime GitHub: microsoft/onnxruntime#5369

[tqchen]

It would be great to propose a fix, given that the flaky error happens quite frequently.

Is this related to the fact that we are using pytorch for gradient testing? Ideally we sould move that to a separate set of test suite. By default, we should use numerical gradient checking that is independent from other frameworks

[altanh]

I agree. I think first we should address #7017 to confirm it's the same failure that is happening on CI, and then look into removing the dependencies. If we can't remove the dependency (like in the case of test_onnx.py and test_dlpack.py), I propose sandboxing based on dependency so that files with conflicting dependencies will always be run on separate pytest processes. If a single file uses two conflicting dependencies, I'm not sure how to proceed- we may need to build dependencies with special libomp configuration on the CI machine (at least we can cache this?)

[altanh]

@tkonolige found that pytest-xdist package supports passing --forked argument to pytest. This seems to fix the problem for running contrib tests.

[altanh]

We should keep this issue but rename to dependency libomp conflict I think (or open a new one), since it might arise in the future