Flaky test: test_operator_gpu.test_countsketch

[eric-haibin-lin]

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/PR-10983/1/pipeline/

test_operator_gpu.test_exc_multiple_waits ... ok (0.0126s)

**test_operator_gpu.test_countsketch ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=0 to reproduce.**

**FAIL**

test_operator_gpu.test_cached ... SKIP: test fails intermittently. temporarily disabled till it gets fixed. tracked at https://github.com/apache/incubator-mxnet/issues/8049

test_operator_gpu.test_sparse_nd_setitem ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=584830846 to reproduce.

ERROR

test_operator_gpu.test_residual ... ERROR

test_operator_gpu.test_lstm_dropout ... ok (0.0739s)

test_operator_gpu.test_parameter_sharing ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1302250019 to reproduce.

ERROR

test_operator_gpu.test_exc_post_fail ... ok (0.0255s)

test_operator_gpu.test_bce_equal_ce2 ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=2007597932 to reproduce.

ERROR

test_operator_gpu.test_exc_mutable_var_fail ... ok (0.0051s)

test_operator_gpu.test_sparse_nd_slice ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=28702949 to reproduce.

ERROR

test_operator_gpu.test_logistic_loss_equal_bce ... ERROR

test_operator_gpu.test_ndarray_elementwise ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=0 to reproduce.

ERROR

test_operator_gpu.test_residual_bidirectional ... ERROR

test_operator_gpu.test_kl_loss ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1234 to reproduce.

ERROR

test_operator_gpu.test_parameter_str ... ok (0.0005s)

test_operator_gpu.test_ndarray_elementwisesum ... [06:12:50] src/operator/tensor/./.././../common/../operator/mxnet_op.h:576: Check failed: (err) == (cudaSuccess) Name: mxnet_generic_kernel ErrStr:an illegal memory access was encountered

/work/runtime_functions.sh: line 507:     7 Aborted                 (core dumped) python3.6 -m "nose" --with-timer --verbose tests/python/gpu

[srochel]

We should make a focussed effort to resolve all flaky tests in Q3 2018.

[aaronmarkham]

I think I hit this same issue.

test check_format for sparse ndarray ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1329389823 to reproduce.

FAIL

test_operator_gpu.test_trunc ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=367168103 to reproduce.

ERROR

test_operator_gpu.test_output ... [22:41:32] src/operator/tensor/./.././../common/../operator/mxnet_op.h:576: Check failed: (err) == (cudaSuccess) Name: mxnet_generic_kernel ErrStr:an illegal memory access was encountered

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/PR-11157/5/pipeline

[haojin2]

Was able to reproduce the issue with a different seed, the problem is that the absolute tolerance is very low (1e-12), I've bumped it up to 1e-5 to see if it could pass 1000 runs.

[anirudh2290]

assigned to haibin @haojin2 is working on this.

[haojin2]

From the reproduced error we can see that only part of the grad ndarray is filled:

======================================================================
FAIL: test_operator_gpu.test_countsketch
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/ubuntu/5-mxnet/tests/python/gpu/../unittest/common.py", line 157, in test_new
    orig_test(*args, **kwargs)
  File "/home/ubuntu/5-mxnet/tests/python/gpu/test_operator_gpu.py", line 103, in test_countsketch
    check_countsketch(in_dim, out_dim, n)
  File "/home/ubuntu/5-mxnet/tests/python/gpu/test_operator_gpu.py", line 88, in check_countsketch
    assert_almost_equal(a,arr_grad[0].asnumpy(),rtol=1e-3, atol=1e-5)
  File "/home/ubuntu/6-mxnet/python/mxnet/test_utils.py", line 493, in assert_almost_equal
    raise AssertionError(msg)
AssertionError: 
Items are not equal:
Error 159853.341227 exceeds tolerance rtol=0.001000, atol=0.000010.  Location of maximum error:(5, 9), a=6.016212, b=-0.027810
 a: array([[-0.08690866,  0.        , -0.        , ...,  0.        ,
         0.        , -0.        ],
       [ 0.        ,  0.        ,  0.        , ..., -0.        ,...
 b: array([[-0.08690866, -3.90360618, -1.36067092, ..., -0.4085128 ,
        -2.49076152,  0.51365918],
       [ 9.16543007, -3.62473965, -6.45960188, ..., -1.51162243,...
-------------------- >> begin captured logging << --------------------
common: INFO: Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=954783568 to reproduce.
common: INFO: 1 of 10000: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1220294681 to reproduce.
common: INFO: 2 of 10000: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1220294681 to reproduce.
common: INFO: 3 of 10000: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1220294681 to reproduce.
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 1 test in 5.961s

[haojin2]

Fix in #11780.

[haojin2]

@larroy Please take a look at my comments on how the test could fail without a sync.

[anirudh2290]

Reopening since this again failed here : http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fcentos-gpu/detail/PR-15118/13/pipeline

[larroy]

But all of them are failing, might not be specific to this one, seems memory corruption or hardware issue.