Flaky test MKLDNN_BASE.MKLDNNSum

[marcoabreu]

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/master/1319/pipeline
Cpp: MKLDNN+GPU

[0;32m[----------] �[m2 tests from MKLDNN_BASE
�[0;32m[ RUN      ] �[mMKLDNN_BASE.MKLDNNSum
format: 8, 33
format: 50
format: 8, 33
format: 50
format: 8, 33
format: 50
Verifying: Normal NDArray [279936] with Normal NDArray [279936]
Verifying: Normal NDArray [279936] with Normal NDArray [279936]
Verifying: Normal NDArray [279936] with Normal NDArray [279936]
Verifying: Normal NDArray [279936] with Normal NDArray [279936]
Verifying: Normal NDArray [279936] with Normal NDArray [279936]
Verifying: Normal NDArray [279936] with Normal NDArray [279936]
Verifying: Normal NDArray [279936] with Normal NDArray [279936]
Verifying: MKLDNN NDArray [279936] with MKLDNN NDArray [279936]
Verifying: MKLDNN NDArray [279936] with MKLDNN NDArray [279936]
Verifying: MKLDNN NDArray [279936] with MKLDNN NDArray [279936]
Verifying: MKLDNN NDArray [279936] with MKLDNN NDArray [279936]
Verifying: MKLDNN NDArray [279936] with MKLDNN NDArray [279936]
Verifying: MKLDNN NDArray [279936] with MKLDNN NDArray [279936]
Verifying: MKLDNN NDArray [279936] with MKLDNN NDArray [279936]
Verifying: Reshaped MKLDNN NDArray [279934] with Reshaped MKLDNN NDArray [279934]
Verifying: Reshaped MKLDNN NDArray [279934] with Reshaped MKLDNN NDArray [279934]
Verifying: Reshaped MKLDNN NDArray [279934] with Reshaped MKLDNN NDArray [279934]
Verifying: MKLDNN NDArray with different memory layout 1/2 [279936] with MKLDNN NDArray with different memory layout 1/2 [279936]
Verifying: MKLDNN NDArray with different memory layout 1/2 [279936] with MKLDNN NDArray with different memory layout 1/2 [279936]
Verifying: MKLDNN NDArray with different memory layout 1/2 [279936] with MKLDNN NDArray with different memory layout 1/2 [279936]
Verifying: MKLDNN NDArray with different memory layout 1/2 [279936] with MKLDNN NDArray with different memory layout 1/2 [279936]
Verifying: MKLDNN NDArray with different memory layout 1/2 [279936] with MKLDNN NDArray with different memory layout 1/2 [279936]
Verifying: MKLDNN NDArray with different memory layout 1/2 [279936] with MKLDNN NDArray with different memory layout 1/2 [279936]
Verifying: MKLDNN NDArray with different memory layout 1/2 [279936] with MKLDNN NDArray with different memory layout 1/2 [279936]
Verifying: Reshaped MKLDNN NDArray with different memory layout 1/2 [279934] with Reshaped MKLDNN NDArray with different memory layout 1/2 [279934]
Verifying: Reshaped MKLDNN NDArray with different memory layout 1/2 [279934] with Reshaped MKLDNN NDArray with different memory layout 1/2 [279934]
Verifying: Reshaped MKLDNN NDArray with different memory layout 1/2 [279934] with Reshaped MKLDNN NDArray with different memory layout 1/2 [279934]
Verifying: Normal NDArray [34848] with Normal NDArray [34848]
Verifying: Normal NDArray [34848] with Normal NDArray [34848]
Verifying: Normal NDArray [34848] with Normal NDArray [34848]
Verifying: Normal NDArray [34848] with Normal NDArray [34848]
Verifying: Normal NDArray [34848] with Normal NDArray [34848]
Verifying: Normal NDArray [34848] with Normal NDArray [34848]
Verifying: Normal NDArray [34848] with Normal NDArray [34848]
Verifying: Normal NDArray [34848] with Normal NDArray [34848]
Verifying: Normal NDArray [34848] with Normal NDArray [34848]
Verifying: Normal NDArray [34848] with Normal NDArray [34848]
Verifying: Normal NDArray [34848] with Normal NDArray [34848]
Verifying: Normal NDArray [34848] with Normal NDArray [34848]
Verifying: Normal NDArray [34848] with Normal NDArray [34848]
Verifying: Normal NDArray [34848] with Normal NDArray [34848]
Verifying: Normal NDArray [34848] with Normal NDArray [34848]
Verifying: MKLDNN NDArray [34848] with MKLDNN NDArray [34848]
Verifying: MKLDNN NDArray [34848] with MKLDNN NDArray [34848]
Verifying: MKLDNN NDArray [34848] with MKLDNN NDArray [34848]
Verifying: MKLDNN NDArray [34848] with MKLDNN NDArray [34848]
Verifying: MKLDNN NDArray [34848] with MKLDNN NDArray [34848]
Verifying: MKLDNN NDArray [34848] with MKLDNN NDArray [34848]
Verifying: MKLDNN NDArray [34848] with MKLDNN NDArray [34848]
Verifying: MKLDNN NDArray [34848] with MKLDNN NDArray [34848]
Verifying: MKLDNN NDArray [34848] with MKLDNN NDArray [34848]
Verifying: MKLDNN NDArray [34848] with MKLDNN NDArray [34848]
Verifying: MKLDNN NDArray [34848] with MKLDNN NDArray [34848]
Verifying: MKLDNN NDArray [34848] with MKLDNN NDArray [34848]
Verifying: MKLDNN NDArray [34848] with MKLDNN NDArray [34848]
Verifying: MKLDNN NDArray [34848] with MKLDNN NDArray [34848]
Verifying: MKLDNN NDArray [34848] with MKLDNN NDArray [34848]
Verifying: Reshaped MKLDNN NDArray [34846] with Reshaped MKLDNN NDArray [34846]
Verifying: Reshaped MKLDNN NDArray [34846] with Reshaped MKLDNN NDArray [34846]
Verifying: Reshaped MKLDNN NDArray [34846] with Reshaped MKLDNN NDArray [34846]
Verifying: MKLDNN NDArray with different memory layout 1/2 [34848] with MKLDNN NDArray with different memory layout 1/2 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/2 [34848] with MKLDNN NDArray with different memory layout 1/2 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/2 [34848] with MKLDNN NDArray with different memory layout 1/2 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/2 [34848] with MKLDNN NDArray with different memory layout 1/2 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/2 [34848] with MKLDNN NDArray with different memory layout 1/2 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/2 [34848] with MKLDNN NDArray with different memory layout 1/2 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/2 [34848] with MKLDNN NDArray with different memory layout 1/2 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/2 [34848] with MKLDNN NDArray with different memory layout 1/2 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/2 [34848] with MKLDNN NDArray with different memory layout 1/2 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/2 [34848] with MKLDNN NDArray with different memory layout 1/2 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/2 [34848] with MKLDNN NDArray with different memory layout 1/2 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/2 [34848] with MKLDNN NDArray with different memory layout 1/2 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/2 [34848] with MKLDNN NDArray with different memory layout 1/2 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/2 [34848] with MKLDNN NDArray with different memory layout 1/2 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/2 [34848] with MKLDNN NDArray with different memory layout 1/2 [34848]
Verifying: Reshaped MKLDNN NDArray with different memory layout 1/2 [34846] with Reshaped MKLDNN NDArray with different memory layout 1/2 [34846]
Verifying: Reshaped MKLDNN NDArray with different memory layout 1/2 [34846] with Reshaped MKLDNN NDArray with different memory layout 1/2 [34846]
Verifying: Reshaped MKLDNN NDArray with different memory layout 1/2 [34846] with Reshaped MKLDNN NDArray with different memory layout 1/2 [34846]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: Reshaped MKLDNN NDArray with different memory layout 1/4 [34846] with Reshaped MKLDNN NDArray with different memory layout 1/4 [34846]
Verifying: Reshaped MKLDNN NDArray with different memory layout 1/4 [34846] with Reshaped MKLDNN NDArray with different memory layout 1/4 [34846]
Verifying: Reshaped MKLDNN NDArray with different memory layout 1/4 [34846] with Reshaped MKLDNN NDArray with different memory layout 1/4 [34846]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/4 [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: Reshaped MKLDNN NDArray with different memory layout 1/5 [34846] with Reshaped MKLDNN NDArray with different memory layout 1/5 [34846]
Verifying: Reshaped MKLDNN NDArray with different memory layout 1/5 [34846] with Reshaped MKLDNN NDArray with different memory layout 1/5 [34846]
Verifying: Reshaped MKLDNN NDArray with different memory layout 1/5 [34846] with Reshaped MKLDNN NDArray with different memory layout 1/5 [34846]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: MKLDNN NDArray with different memory layout 1/5 [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: Normal NDArray [96,2916] with Normal NDArray [96,2916]
Verifying: Normal NDArray [96,2916] with Normal NDArray [96,2916]
Verifying: Normal NDArray [96,2916] with Normal NDArray [96,2916]
Verifying: Normal NDArray [96,2916] with Normal NDArray [96,2916]
Verifying: Normal NDArray [96,2916] with Normal NDArray [96,2916]
Verifying: Normal NDArray [96,2916] with Normal NDArray [96,2916]
Verifying: Normal NDArray [96,2916] with Normal NDArray [96,2916]
Verifying: MKLDNN NDArray with different memory layout 2/1 [96,2916] with MKLDNN NDArray with different memory layout 2/1 [96,2916]
Verifying: MKLDNN NDArray with different memory layout 2/1 [96,2916] with MKLDNN NDArray with different memory layout 2/1 [96,2916]
Verifying: MKLDNN NDArray with different memory layout 2/1 [96,2916] with MKLDNN NDArray with different memory layout 2/1 [96,2916]
Verifying: MKLDNN NDArray with different memory layout 2/1 [96,2916] with MKLDNN NDArray with different memory layout 2/1 [96,2916]
Verifying: MKLDNN NDArray with different memory layout 2/1 [96,2916] with MKLDNN NDArray with different memory layout 2/1 [96,2916]
Verifying: MKLDNN NDArray with different memory layout 2/1 [96,2916] with MKLDNN NDArray with different memory layout 2/1 [96,2916]
Verifying: MKLDNN NDArray with different memory layout 2/1 [96,2916] with MKLDNN NDArray with different memory layout 2/1 [96,2916]
Verifying: Reshaped MKLDNN NDArray with different memory layout 2/1 [94,2916] with Reshaped MKLDNN NDArray with different memory layout 2/1 [94,2916]
Verifying: Reshaped MKLDNN NDArray with different memory layout 2/1 [94,2916] with Reshaped MKLDNN NDArray with different memory layout 2/1 [94,2916]
Verifying: Reshaped MKLDNN NDArray with different memory layout 2/1 [94,2916] with Reshaped MKLDNN NDArray with different memory layout 2/1 [94,2916]
Verifying: MKLDNN NDArray [96,2916] with MKLDNN NDArray [96,2916]
Verifying: MKLDNN NDArray [96,2916] with MKLDNN NDArray [96,2916]
Verifying: MKLDNN NDArray [96,2916] with MKLDNN NDArray [96,2916]
Verifying: MKLDNN NDArray [96,2916] with MKLDNN NDArray [96,2916]
Verifying: MKLDNN NDArray [96,2916] with MKLDNN NDArray [96,2916]
Verifying: MKLDNN NDArray [96,2916] with MKLDNN NDArray [96,2916]
Verifying: MKLDNN NDArray [96,2916] with MKLDNN NDArray [96,2916]
Verifying: Reshaped MKLDNN NDArray [94,2916] with Reshaped MKLDNN NDArray [94,2916]
Verifying: Reshaped MKLDNN NDArray [94,2916] with Reshaped MKLDNN NDArray [94,2916]
Verifying: Reshaped MKLDNN NDArray [94,2916] with Reshaped MKLDNN NDArray [94,2916]
Verifying: Normal NDArray [96,363] with Normal NDArray [96,363]
Verifying: Normal NDArray [96,363] with Normal NDArray [96,363]
Verifying: Normal NDArray [96,363] with Normal NDArray [96,363]
Verifying: Normal NDArray [96,363] with Normal NDArray [96,363]
Verifying: Normal NDArray [96,363] with Normal NDArray [96,363]
Verifying: Normal NDArray [96,363] with Normal NDArray [96,363]
Verifying: Normal NDArray [96,363] with Normal NDArray [96,363]
Verifying: Normal NDArray [96,363] with Normal NDArray [96,363]
Verifying: Normal NDArray [96,363] with Normal NDArray [96,363]
Verifying: Normal NDArray [96,363] with Normal NDArray [96,363]
Verifying: Normal NDArray [96,363] with Normal NDArray [96,363]
Verifying: Normal NDArray [96,363] with Normal NDArray [96,363]
Verifying: Normal NDArray [96,363] with Normal NDArray [96,363]
Verifying: Normal NDArray [96,363] with Normal NDArray [96,363]
Verifying: Normal NDArray [96,363] with Normal NDArray [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/1 [96,363] with MKLDNN NDArray with different memory layout 2/1 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/1 [96,363] with MKLDNN NDArray with different memory layout 2/1 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/1 [96,363] with MKLDNN NDArray with different memory layout 2/1 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/1 [96,363] with MKLDNN NDArray with different memory layout 2/1 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/1 [96,363] with MKLDNN NDArray with different memory layout 2/1 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/1 [96,363] with MKLDNN NDArray with different memory layout 2/1 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/1 [96,363] with MKLDNN NDArray with different memory layout 2/1 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/1 [96,363] with MKLDNN NDArray with different memory layout 2/1 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/1 [96,363] with MKLDNN NDArray with different memory layout 2/1 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/1 [96,363] with MKLDNN NDArray with different memory layout 2/1 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/1 [96,363] with MKLDNN NDArray with different memory layout 2/1 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/1 [96,363] with MKLDNN NDArray with different memory layout 2/1 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/1 [96,363] with MKLDNN NDArray with different memory layout 2/1 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/1 [96,363] with MKLDNN NDArray with different memory layout 2/1 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/1 [96,363] with MKLDNN NDArray with different memory layout 2/1 [96,363]
Verifying: Reshaped MKLDNN NDArray with different memory layout 2/1 [94,363] with Reshaped MKLDNN NDArray with different memory layout 2/1 [94,363]
Verifying: Reshaped MKLDNN NDArray with different memory layout 2/1 [94,363] with Reshaped MKLDNN NDArray with different memory layout 2/1 [94,363]
Verifying: Reshaped MKLDNN NDArray with different memory layout 2/1 [94,363] with Reshaped MKLDNN NDArray with different memory layout 2/1 [94,363]
Verifying: MKLDNN NDArray [96,363] with MKLDNN NDArray [96,363]
Verifying: MKLDNN NDArray [96,363] with MKLDNN NDArray [96,363]
Verifying: MKLDNN NDArray [96,363] with MKLDNN NDArray [96,363]
Verifying: MKLDNN NDArray [96,363] with MKLDNN NDArray [96,363]
Verifying: MKLDNN NDArray [96,363] with MKLDNN NDArray [96,363]
Verifying: MKLDNN NDArray [96,363] with MKLDNN NDArray [96,363]
Verifying: MKLDNN NDArray [96,363] with MKLDNN NDArray [96,363]
Verifying: MKLDNN NDArray [96,363] with MKLDNN NDArray [96,363]
Verifying: MKLDNN NDArray [96,363] with MKLDNN NDArray [96,363]
Verifying: MKLDNN NDArray [96,363] with MKLDNN NDArray [96,363]
Verifying: MKLDNN NDArray [96,363] with MKLDNN NDArray [96,363]
Verifying: MKLDNN NDArray [96,363] with MKLDNN NDArray [96,363]
Verifying: MKLDNN NDArray [96,363] with MKLDNN NDArray [96,363]
Verifying: MKLDNN NDArray [96,363] with MKLDNN NDArray [96,363]
Verifying: MKLDNN NDArray [96,363] with MKLDNN NDArray [96,363]
Verifying: Reshaped MKLDNN NDArray [94,363] with Reshaped MKLDNN NDArray [94,363]
Verifying: Reshaped MKLDNN NDArray [94,363] with Reshaped MKLDNN NDArray [94,363]
Verifying: Reshaped MKLDNN NDArray [94,363] with Reshaped MKLDNN NDArray [94,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: Reshaped MKLDNN NDArray with different memory layout 2/4 [94,363] with Reshaped MKLDNN NDArray with different memory layout 2/4 [94,363]
Verifying: Reshaped MKLDNN NDArray with different memory layout 2/4 [94,363] with Reshaped MKLDNN NDArray with different memory layout 2/4 [94,363]
Verifying: Reshaped MKLDNN NDArray with different memory layout 2/4 [94,363] with Reshaped MKLDNN NDArray with different memory layout 2/4 [94,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/4 [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: Reshaped MKLDNN NDArray with different memory layout 2/5 [94,363] with Reshaped MKLDNN NDArray with different memory layout 2/5 [94,363]
Verifying: Reshaped MKLDNN NDArray with different memory layout 2/5 [94,363] with Reshaped MKLDNN NDArray with different memory layout 2/5 [94,363]
Verifying: Reshaped MKLDNN NDArray with different memory layout 2/5 [94,363] with Reshaped MKLDNN NDArray with different memory layout 2/5 [94,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: MKLDNN NDArray with different memory layout 2/5 [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: Normal NDArray [10,96,54,54] with Normal NDArray [10,96,54,54]
Verifying: Normal NDArray [10,96,54,54] with Normal NDArray [10,96,54,54]
Verifying: Normal NDArray [10,96,54,54] with Normal NDArray [10,96,54,54]
Verifying: Normal NDArray [10,96,54,54] with Normal NDArray [10,96,54,54]
Verifying: Normal NDArray [10,96,54,54] with Normal NDArray [10,96,54,54]
Verifying: Normal NDArray [10,96,54,54] with Normal NDArray [10,96,54,54]
Verifying: Normal NDArray [10,96,54,54] with Normal NDArray [10,96,54,54]
Verifying: MKLDNN NDArray [10,96,54,54] with MKLDNN NDArray [10,96,54,54]
Verifying: MKLDNN NDArray [10,96,54,54] with MKLDNN NDArray [10,96,54,54]
Verifying: MKLDNN NDArray [10,96,54,54] with MKLDNN NDArray [10,96,54,54]
Verifying: MKLDNN NDArray [10,96,54,54] with MKLDNN NDArray [10,96,54,54]
Verifying: MKLDNN NDArray [10,96,54,54] with MKLDNN NDArray [10,96,54,54]
Verifying: MKLDNN NDArray [10,96,54,54] with MKLDNN NDArray [10,96,54,54]
Verifying: MKLDNN NDArray [10,96,54,54] with MKLDNN NDArray [10,96,54,54]
Verifying: Reshaped MKLDNN NDArray [8,96,54,54] with Reshaped MKLDNN NDArray [8,96,54,54]
Verifying: Reshaped MKLDNN NDArray [8,96,54,54] with Reshaped MKLDNN NDArray [8,96,54,54]
Verifying: Reshaped MKLDNN NDArray [8,96,54,54] with Reshaped MKLDNN NDArray [8,96,54,54]
Verifying: MKLDNN NDArray [10,96,54,54] with MKLDNN NDArray [10,96,54,54]
Verifying: MKLDNN NDArray [10,96,54,54] with MKLDNN NDArray [10,96,54,54]
Verifying: MKLDNN NDArray [10,96,54,54] with MKLDNN NDArray [10,96,54,54]
Verifying: MKLDNN NDArray [10,96,54,54] with MKLDNN NDArray [10,96,54,54]
Verifying: MKLDNN NDArray [10,96,54,54] with MKLDNN NDArray [10,96,54,54]
Verifying: MKLDNN NDArray [10,96,54,54] with MKLDNN NDArray [10,96,54,54]
Verifying: MKLDNN NDArray [10,96,54,54] with MKLDNN NDArray [10,96,54,54]
Verifying: Normal NDArray [96,3,11,11] with Normal NDArray [96,3,11,11]
Verifying: Normal NDArray [96,3,11,11] with Normal NDArray [96,3,11,11]
Verifying: Normal NDArray [96,3,11,11] with Normal NDArray [96,3,11,11]
Verifying: Normal NDArray [96,3,11,11] with Normal NDArray [96,3,11,11]
Verifying: Normal NDArray [96,3,11,11] with Normal NDArray [96,3,11,11]
Verifying: Normal NDArray [96,3,11,11] with Normal NDArray [96,3,11,11]
Verifying: Normal NDArray [96,3,11,11] with Normal NDArray [96,3,11,11]
Verifying: Normal NDArray [96,3,11,11] with Normal NDArray [96,3,11,11]
Verifying: Normal NDArray [96,3,11,11] with Normal NDArray [96,3,11,11]
Verifying: Normal NDArray [96,3,11,11] with Normal NDArray [96,3,11,11]
Verifying: Normal NDArray [96,3,11,11] with Normal NDArray [96,3,11,11]
Verifying: Normal NDArray [96,3,11,11] with Normal NDArray [96,3,11,11]
Verifying: Normal NDArray [96,3,11,11] with Normal NDArray [96,3,11,11]
Verifying: Normal NDArray [96,3,11,11] with Normal NDArray [96,3,11,11]
Verifying: Normal NDArray [96,3,11,11] with Normal NDArray [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/1 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/1 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/1 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/1 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/1 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/1 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/1 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/1 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/1 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/1 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/1 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/1 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/1 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/1 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/1 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/1 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/1 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/1 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/1 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/1 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/1 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/1 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/1 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/1 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/1 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/1 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/1 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/1 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/1 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/1 [96,3,11,11]
Verifying: Reshaped MKLDNN NDArray with different memory layout 4/1 [94,3,11,11] with Reshaped MKLDNN NDArray with different memory layout 4/1 [94,3,11,11]
Verifying: Reshaped MKLDNN NDArray with different memory layout 4/1 [94,3,11,11] with Reshaped MKLDNN NDArray with different memory layout 4/1 [94,3,11,11]
Verifying: Reshaped MKLDNN NDArray with different memory layout 4/1 [94,3,11,11] with Reshaped MKLDNN NDArray with different memory layout 4/1 [94,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/2 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/2 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/2 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/2 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/2 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/2 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/2 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/2 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/2 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/2 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/2 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/2 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/2 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/2 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/2 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/2 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/2 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/2 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/2 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/2 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/2 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/2 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/2 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/2 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/2 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/2 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/2 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/2 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/2 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/2 [96,3,11,11]
Verifying: Reshaped MKLDNN NDArray with different memory layout 4/2 [94,3,11,11] with Reshaped MKLDNN NDArray with different memory layout 4/2 [94,3,11,11]
Verifying: Reshaped MKLDNN NDArray with different memory layout 4/2 [94,3,11,11] with Reshaped MKLDNN NDArray with different memory layout 4/2 [94,3,11,11]
Verifying: Reshaped MKLDNN NDArray with different memory layout 4/2 [94,3,11,11] with Reshaped MKLDNN NDArray with different memory layout 4/2 [94,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: Reshaped MKLDNN NDArray [94,3,11,11] with Reshaped MKLDNN NDArray [94,3,11,11]
Verifying: Reshaped MKLDNN NDArray [94,3,11,11] with Reshaped MKLDNN NDArray [94,3,11,11]
Verifying: Reshaped MKLDNN NDArray [94,3,11,11] with Reshaped MKLDNN NDArray [94,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: Reshaped MKLDNN NDArray with different memory layout 4/5 [94,3,11,11] with Reshaped MKLDNN NDArray with different memory layout 4/5 [94,3,11,11]
Verifying: Reshaped MKLDNN NDArray with different memory layout 4/5 [94,3,11,11] with Reshaped MKLDNN NDArray with different memory layout 4/5 [94,3,11,11]
Verifying: Reshaped MKLDNN NDArray with different memory layout 4/5 [94,3,11,11] with Reshaped MKLDNN NDArray with different memory layout 4/5 [94,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: MKLDNN NDArray with different memory layout 4/5 [96,3,11,11] with MKLDNN NDArray with different memory layout 4/5 [96,3,11,11]
Verifying: In Place Copy [279936] with Normal NDArray [279936]
Verifying: In Place Copy [279936] with MKLDNN NDArray [279936]
Verifying: In Place Copy [279934] with Reshaped MKLDNN NDArray [279934]
Verifying: In Place Copy [279936] with MKLDNN NDArray with different memory layout 1/2 [279936]
Verifying: In Place Copy [279934] with Reshaped MKLDNN NDArray with different memory layout 1/2 [279934]
Verifying: In Place Copy [34848] with Normal NDArray [34848]
Verifying: In Place Copy [34848] with MKLDNN NDArray [34848]
Verifying: In Place Copy [34846] with Reshaped MKLDNN NDArray [34846]
Verifying: In Place Copy [34848] with MKLDNN NDArray with different memory layout 1/2 [34848]
Verifying: In Place Copy [34846] with Reshaped MKLDNN NDArray with different memory layout 1/2 [34846]
Verifying: In Place Copy [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: In Place Copy [34846] with Reshaped MKLDNN NDArray with different memory layout 1/4 [34846]
Verifying: In Place Copy [34848] with MKLDNN NDArray with different memory layout 1/4 [34848]
Verifying: In Place Copy [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: In Place Copy [34846] with Reshaped MKLDNN NDArray with different memory layout 1/5 [34846]
Verifying: In Place Copy [34848] with MKLDNN NDArray with different memory layout 1/5 [34848]
Verifying: In Place Copy [96,2916] with Normal NDArray [96,2916]
Verifying: In Place Copy [96,2916] with MKLDNN NDArray with different memory layout 2/1 [96,2916]
Verifying: In Place Copy [94,2916] with Reshaped MKLDNN NDArray with different memory layout 2/1 [94,2916]
Verifying: In Place Copy [96,2916] with MKLDNN NDArray [96,2916]
Verifying: In Place Copy [94,2916] with Reshaped MKLDNN NDArray [94,2916]
Verifying: In Place Copy [96,363] with Normal NDArray [96,363]
Verifying: In Place Copy [96,363] with MKLDNN NDArray with different memory layout 2/1 [96,363]
Verifying: In Place Copy [94,363] with Reshaped MKLDNN NDArray with different memory layout 2/1 [94,363]
Verifying: In Place Copy [96,363] with MKLDNN NDArray [96,363]
Verifying: In Place Copy [94,363] with Reshaped MKLDNN NDArray [94,363]
Verifying: In Place Copy [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: In Place Copy [94,363] with Reshaped MKLDNN NDArray with different memory layout 2/4 [94,363]
Verifying: In Place Copy [96,363] with MKLDNN NDArray with different memory layout 2/4 [96,363]
Verifying: In Place Copy [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: In Place Copy [94,363] with Reshaped MKLDNN NDArray with different memory layout 2/5 [94,363]
Verifying: In Place Copy [96,363] with MKLDNN NDArray with different memory layout 2/5 [96,363]
Verifying: In Place Copy [10,96,54,54] with Normal NDArray [10,96,54,54]
Verifying: In Place Copy [10,96,54,54] with MKLDNN NDArray [10,96,54,54]
Verifying: In Place Copy [8,96,54,54] with Reshaped MKLDNN NDArray [8,96,54,54]
Verifying: In Place Copy [10,96,54,54] with MKLDNN NDArray [10,96,54,54]
Verifying: In Place Copy [96,3,11,11] with Normal NDArray [96,3,11,11]
Verifying: In Place Copy [96,3,11,11] with MKLDNN NDArray with different memory layout 4/1 [96,3,11,11]
Verifying: In Place Copy [94,3,11,11] with Reshaped MKLDNN NDArray with different memory layout 4/1 [94,3,11,11]
Verifying: In Place Copy [96,3,11,11] with MKLDNN NDArray with different memory layout 4/2 [96,3,11,11]
Verifying: In Place Copy [94,3,11,11] with Reshaped MKLDNN NDArray with different memory layout 4/2 [94,3,11,11]
Verifying: In Place Copy [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
Verifying: In Place Copy [94,3,11,11] with Reshaped MKLDNN NDArray [94,3,11,11]
Verifying: In Place Copy [96,3,11,11] with MKLDNN NDArray [96,3,11,11]
unknown file: Failure
C++ exception with description "[00:17:20] /work/mxnet/src/ndarray/ndarray.cc:444: Check failed: delay_alloc 

Stack trace returned 10 entries:
[bt] (0) build/tests/mxnet_unit_tests(dmlc::StackTrace[abi:cxx11]()+0x1bc) [0xf9c75c]
[bt] (1) build/tests/mxnet_unit_tests(dmlc::LogMessageFatal::~LogMessageFatal()+0x28) [0xf9db08]
[bt] (2) build/tests/mxnet_unit_tests(mxnet::NDArray::Chunk::SetMKLMem(nnvm::TShape const&, int)+0x4ef) [0x455370f]
[bt] (3) build/tests/mxnet_unit_tests(mxnet::NDArray::GetMKLDNNData() const+0xa8) [0x45539c8]
[bt] (4) build/tests/mxnet_unit_tests(void mxnet::CopyFromToDnsImpl<mshadow::cpu, mshadow::cpu>(mxnet::NDArray const&, mxnet::NDArray const&, mxnet::RunContext)+0x2e8) [0x456e7e8]
[bt] (5) build/tests/mxnet_unit_tests(void mxnet::CopyFromToImpl<mshadow::cpu, mshadow::cpu>(mxnet::NDArray const&, mxnet::NDArray const&, mxnet::RunContext, std::vector<mxnet::Resource, std::allocator<mxnet::Resource> > const&)+0x4d0) [0x45764a0]
[bt] (6) build/tests/mxnet_unit_tests() [0x455fdff]
[bt] (7) build/tests/mxnet_unit_tests(mxnet::engine::ThreadedEngine::ExecuteOprBlock(mxnet::RunContext, mxnet::engine::OprBlock*)+0x48d) [0x29c03bd]
[bt] (8) build/tests/mxnet_unit_tests(std::_Function_handler<void (std::shared_ptr<dmlc::ManualEvent>), mxnet::engine::ThreadedEnginePerDevice::PushToExecute(mxnet::engine::OprBlock*, bool)::{lambda()#1}::operator()() const::{lambda(std::shared_ptr<dmlc::ManualEvent>)#1}>::_M_invoke(std::_Any_data const&, std::shared_ptr<dmlc::ManualEvent>&&)+0xe2) [0x29d9412]
[bt] (9) build/tests/mxnet_unit_tests(std::thread::_Impl<std::_Bind_simple<std::function<void (std::shared_ptr<dmlc::ManualEvent>)> (std::shared_ptr<dmlc::ManualEvent>)> >::_M_run()+0x4a) [0x29bfe1a]

" thrown in the test body.
�[0;31m[  FAILED  ] �[mMKLDNN_BASE.MKLDNNSum (4679 ms)

[marcoabreu]

@pengzhao-intel

[pengzhao-intel]

@jinhuang415 please help take a look for this issue.

[lupesko]

@jinhuang415 - bounce
@azai91 FYI

[jinhuang415]

We have reproduced this issue in local luster, looks only happens very occasionally, will update once there is further findings.

[jinhuang415]

@marcoabreu @zheng-da @azai91 @pengzhao-intel
We have found the root cause of this issue and here is the analysis:
The test failed at MKLDNNSum in place test case, and failed due to delay_alloc check failure in SetMKLMem(),

void NDArray::Chunk::SetMKLMem(const TShape &shape, int dtype) {
...
  if (shandle.dptr == nullptr) {
    CHECK(delay_alloc);         <<< check failure here
    CheckAndAlloc();
  }
...

it indicates the related NDArray's shandle is nullptr but delay_alloc is set to false, usually delay_alloc will only be set to false after shandle is allocated, so most probably there is race condition here (this issue is very hard to reproduce, only happens occasionally), and I tried to print delay_alloc 2 times before CHECK(delay_alloc) and it is found when issue happens delay_alloc is true at first but then become false for the second time! This confirms that delay_alloc must be set to false by another thread asynchronously and cause thread race condition (print will make the race condition easier to happen). By checking the test code it is found there may exist race condition for below 2 calls (commented lines)

TEST(MKLDNN_BASE, MKLDNNSum) {
 ...
    NDArrayAttrs orig_arr(in_arr.arr.Copy(in_arr.arr.ctx()), "In Place Copy");    <<< Copy will push operation to engine asynchronously which will alloc shandle for orig_arr.arr
    PrintVerifyMsg(orig_arr, in_arr);
    InitMKLDNNArray(&orig_arr.arr, input_mem->get_primitive_desc());     <<< may alloc shandle for orig_arr.arr which cause conflict
    orig_arr.arr.CopyFrom(*input_mem);
...
}

The detailed function calls is as below:

NDArray::Copy()
  --> CopyFromTo(*this, ret)
    --> Engine::Get()->PushAsync(...,   
           CopyFromToImpl<cpu, cpu>(from, to, ctx, requested) ...)
              --> CopyFromToDnsImpl()
                  --> to.GetMKLDNNData()
                     --> ptr_->SetMKLMem()
                       --> CHECK(delay_alloc) 
                       -->  CheckAndAlloc()
                          --> alloc shandle and set delay_alloc to false
                         

InitMKLDNNArray()
  --> InitDefaultArray(arr, is_rand)
    --> arr->data()
      --> CheckAndAlloc()
         --> alloc shandle and set delay_alloc to false

InitMKLDNNArray() happens in test main thread while CopyFromToImpl() happens asynchronously in Engine thread, so there may exist thread race condition here that InitMKLDNNArray() set delay_alloc to false and cause the CHECK failed for CopyFromToImpl().

A simple fix is to add barrier to wait for Copy operation to finish before further operations to get rid of the race condition. The proposed change is as below:

diff --git a/tests/cpp/operator/mkldnn.cc b/tests/cpp/operator/mkldnn.cc
index 4f8f6f1..59bd3a5 100644
--- a/tests/cpp/operator/mkldnn.cc
+++ b/tests/cpp/operator/mkldnn.cc
@@ -1308,6 +1308,7 @@ TEST(MKLDNN_BASE, MKLDNNSum) {
     auto input_mem = in_arr.arr.GetMKLDNNData();
     auto input_mem2 = in_arr2.arr.GetMKLDNNData();
     NDArrayAttrs orig_arr(in_arr.arr.Copy(in_arr.arr.ctx()), "In Place Copy");
+    orig_arr.arr.WaitToRead();
     PrintVerifyMsg(orig_arr, in_arr);
     InitMKLDNNArray(&orig_arr.arr, input_mem->get_primitive_desc());
     orig_arr.arr.CopyFrom(*input_mem);

[pengzhao-intel]

In short, the CPP test case mixed the usage of MXNET OP and C/C++ function where the MXNET OP is executed with async mode and C function will run directly.
So, this will bypass the dependency engine and then cause the data race.

In concluding, this is NOT the bug in MKLDNN or dependency engine.
Just a not common implementation of the test case.

[marcoabreu]

Sorry, I'm currently swamped with high priority tasks and I don't have time to review your task or assist with inqueries.

[anirudh2290]

Fixed by #12080