Skip to content
This repository was archived by the owner on Nov 17, 2023. It is now read-only.

Flaky test: test_operator_gpu.test_rnnrelu_bidirectional #11410

Description

@larroy

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/PR-11382/7/pipeline/

======================================================================

FAIL: test_operator_gpu.test_rnnrelu_bidirectional

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 198, in runTest

    self.test(*self.arg)

  File "/usr/local/lib/python3.5/dist-packages/nose/util.py", line 620, in newfunc

    return func(*arg, **kw)

  File "/work/mxnet/tests/python/gpu/../unittest/common.py", line 157, in test_new

    orig_test(*args, **kwargs)

  File "/work/mxnet/tests/python/gpu/../unittest/test_operator.py", line 207, in test_rnnrelu_bidirectional

    check_rnn_consistency(fused, stack, T, N, I, H, 'add')

  File "/work/mxnet/tests/python/gpu/../unittest/test_operator.py", line 65, in check_rnn_consistency

    assert_allclose(mod1.get_input_grads()[0].asnumpy(), mod2.get_input_grads()[0].asnumpy(), rtol=1e-2, atol=1e-4)

  File "/usr/local/lib/python3.5/dist-packages/numpy/testing/nose_tools/utils.py", line 1396, in assert_allclose

    verbose=verbose, header=header, equal_nan=equal_nan)

  File "/usr/local/lib/python3.5/dist-packages/numpy/testing/nose_tools/utils.py", line 779, in assert_array_compare

    raise AssertionError(msg)

AssertionError: 

Not equal to tolerance rtol=0.01, atol=0.0001



(mismatch 0.01999999999999602%)

 x: array([[[ 7.377952e-03,  1.249200e-02, -2.486279e-03, ...,

         -1.402252e-03, -5.022895e-04, -7.681327e-04],

        [ 7.553479e-03,  8.460643e-03,  2.067674e-03, ...,...

 y: array([[[ 7.377951e-03,  1.249200e-02, -2.486280e-03, ...,

         -1.402253e-03, -5.022900e-04, -7.681326e-04],

        [ 7.553477e-03,  8.460643e-03,  2.067674e-03, ...,...

-------------------- >> begin captured logging << --------------------

common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1834194485 to reproduce.

--------------------- >> end captured logging << ---------------------


Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions