test_quantization.test_get_optimal_thresholds sometimes fails during testing.

[KellenSunderland]

Example Run

http://jenkins.mxnet-ci.amazon-ml.com/blue/rest/organizations/jenkins/pipelines/incubator-mxnet/branches/PR-11422/runs/20/nodes/656/log/?start=0

Failing Output

test_quantization.test_get_optimal_thresholds ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=916054564 to reproduce.
FAIL

======================================================================
FAIL: test_quantization.test_get_optimal_thresholds
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/work/mxnet/tests/python/quantization/common.py", line 157, in test_new
    orig_test(*args, **kwargs)
  File "/work/mxnet/tests/python/quantization/test_quantization.py", line 500, in test_get_optimal_thresholds
    assert_almost_equal(np.array([th_dict['layer1'][1]]), expected_threshold, rtol=0.001, atol=0.001)
  File "/work/mxnet/python/mxnet/test_utils.py", line 493, in assert_almost_equal
    raise AssertionError(msg)
AssertionError: 
Items are not equal:
Error 1.148570 exceeds tolerance rtol=0.001000, atol=0.001000.  Location of maximum error:(0,), a=11.327124, b=11.341299
 a: array([11.32712421])
 b: array([11.341299], dtype=float32)
-------------------- >> begin captured logging << --------------------
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=916054564 to reproduce.
--------------------- >> end captured logging << ---------------------

[frankfliu]

Thank you for submitting the issue! We are labeling it so MXNet community members can help resolve it.

[marcoabreu]

Commands:
ci/build.py -p ubuntu_cpu -i
PYTHONPATH=/work/mxnet/python python3 tools/flakiness_checker.py -s 1690031347 test_quantization.test_get_optimal_thresholds

======================================================================
FAIL: test_quantization.test_get_optimal_thresholds
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/work/mxnet/tests/python/quantization/common.py", line 172, in test_new
    orig_test(*args, **kwargs)
  File "/work/mxnet/tests/python/quantization/test_quantization.py", line 524, in test_get_optimal_thresholds
    assert_almost_equal(np.array([th_dict['layer1'][1]]), expected_threshold, rtol=0.001, atol=0.001)
  File "/work/mxnet/python/mxnet/test_utils.py", line 493, in assert_almost_equal
    raise AssertionError(msg)
AssertionError:
Items are not equal:
Error 1.607982 exceeds tolerance rtol=0.001000, atol=0.001000.  Location of maximum error:(0,), a=11.320015, b=11.339857
 a: array([11.32001483])
 b: array([11.339857], dtype=float32)
-------------------- >> begin captured logging << --------------------
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1690031347 to reproduce.
--------------------- >> end captured logging << ---------------------

[haojin2]

Confirm reproduction using the above seeds. Seems like a tolerance problem to me from the existing incidents, on average a single run takes 7.1 seconds, verifying the new tolerance in the background now.

[haojin2]

@KellenSunderland @marcoabreu Related fix is merged, should be good to close.

[haojin2]

@sandeep-krishnamurthy Related PR is merged, should be good to close.