[TEST][FLAKY] test_topi_conv2d_winograd.test_conv2d_nchw

[tmoreau89]

http://ci.tvm.ai:8080/blue/organizations/jenkins/tvm/detail/PR-3496/53/pipeline/239

Occurs on test_topi_conv2d_winograd.test_conv2d_nchw

AssertionError: 

Not equal to tolerance rtol=1e-05, atol=1e-07


Mismatch: 0.0018%

Max absolute difference: 0.00521026

Max relative difference: 1.04917928e-05

 x: array([[[[ 506.34094,  631.938  ,  750.3708 , ...,  764.49725,

           624.0597 ,  503.50046],

         [ 637.637  ,  792.98926,  950.3151 , ...,  953.4434 ,...

 y: array([[[[ 506.342991,  631.939629,  750.371327, ...,  764.495986,

           624.05856 ,  503.500831],

         [ 637.638467,  792.990849,  950.315452, ...,  953.442646,...

-------------------- >> begin captured stdout << ---------------------

Workload: (1, 128, 17, 192, 7, 1, 3, 1)

Running on target: cuda


--------------------- >> end captured stdout << ----------------------

[tmoreau89]

It could be due to #3553 @merrymercy @cbalint13

[cbalint13]

@tmoreau89 ,

Would like to address a hot patch in form of PR ?

• It fail with a very tiny limit on newly enabled (kernel > 3) options.
• Probably the mmoization was lost and on new random generated inputs tiny limit is triggered.

Below is the possible fix.

--- a/topi/tests/python/test_topi_conv2d_winograd.py
+++ b/topi/tests/python/test_topi_conv2d_winograd.py
@@ -81,7 +81,12 @@ def verify_conv2d_nchw(batch, in_channel, in_size, num_filter, kernel, stride, p
         else:
             func = tvm.build(s, [A, W, C], device, name="relu_%d_%d_%d_%d_%d_%d_%d_%d" % (batch, in_channel, in_size, num_filter, kernel, stride, padding, dilation))
             func(a, w, c)
-        tvm.testing.assert_allclose(c.asnumpy(), c_np, rtol=1e-5)
+
+        rtol = 1e-5
+        if (kernel > 3):
+          rtol = 2e-5
+
+        tvm.testing.assert_allclose(c.asnumpy(), c_np, rtol=rtol)

[tmoreau89]

That seems fine, do you mind issuing the PR with the hot fix, and adding myself and @merrymercy to the reviewers? Thanks

[liangfu]

This error still exist:

======================================================================
FAIL: test_topi_conv2d_winograd.test_conv2d_nchw
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/workspace/topi/tests/python/test_topi_conv2d_winograd.py", line 133, in test_conv2d_nchw
    verify_conv2d_nchw(1, 1, 1, 1, 3, 1, 1)
  File "/workspace/topi/tests/python/test_topi_conv2d_winograd.py", line 93, in verify_conv2d_nchw
    check_device(device)
  File "/workspace/topi/tests/python/test_topi_conv2d_winograd.py", line 89, in check_device
    tvm.testing.assert_allclose(c.asnumpy(), c_np, rtol=rtol)
  File "/workspace/python/tvm/testing.py", line 29, in assert_allclose
    np.testing.assert_allclose(actual, desired, rtol=rtol, atol=atol, verbose=True)
  File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 1510, in assert_allclose
    verbose=verbose, header=header, equal_nan=equal_nan)
  File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 836, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=1e-05, atol=1e-07
Mismatch: 100%
Max absolute difference: 5.92579793e-07
Max relative difference: 1.54095876e-05
 x: array([[[[0.038456]]]], dtype=float32)
 y: array([[[[0.038455]]]])

@cbalint13 would you please fix this?

[cbalint13]

@liangfu ,

It fails on verify_conv2d_nchw(1, 1, 1, 1, 3, 1, 1) with 3x3 kernel with smallest winograd tile size, meaning that it could fail before #3553 .

• Where can you see this failure exactly, is it Jenkins ?