flaky test: test_operator.test_dropout

[haojin2]

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/PR-12311/1/pipeline

======================================================================

FAIL: test_operator.test_dropout

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 198, in runTest

    self.test(*self.arg)

  File "/work/mxnet/tests/python/unittest/common.py", line 172, in test_new

    orig_test(*args, **kwargs)

  File "/work/mxnet/tests/python/unittest/test_operator.py", line 5844, in test_dropout

    check_dropout_ratio(0.0, shape)

  File "/work/mxnet/tests/python/unittest/test_operator.py", line 5788, in check_dropout_ratio

    assert exe.outputs[0].asnumpy().min() == min_value

AssertionError: 

-------------------- >> begin captured logging << --------------------

common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=119185068 to reproduce.

--------------------- >> end captured logging << ---------------------

[haojin2]

@samskalicky Can you take a look at this?

[haojin2]

@mxnet-label-bot [Flaky, Test]

[samskalicky]

@haojin2 this looks like an issue with MKL, building with MKL=OFF works fine

[ChaiBapchya]

Flakiness, yet another time - http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/PR-13031/1/pipeline

[Chancebair]

Another failure - http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/master/1841/pipeline/

[Chancebair]

Created a PR to disable test: #13057

[mseth10]

I investigated this issue. The dropout symbol implementation calls a function BernoulliGenerate when MKL and OpenMP flags are on. This is where the flakiness comes from.

BernoulliGenerate uses multithreading and calls MKL library function viRngBernoulli to populate a mask vector. https://github.com/apache/incubator-mxnet/blob/master/src/operator/nn/dropout-inl.h#L82

When p=1.0, it is supposed to populate the mask vector r of length n with all 1s. But for a few values of seed internally generated, mask vector has n-1 1s and a 0 (located at a different index for a different flaky seed), which causes the error.

Also, the error only occurs when multi-threading used. It is not reproduced when single thread used.

I suspect that viRngBernoulli is not thread-safe.

@TaoLv @pengzhao-intel @ZhennanQin @xinyu-intel Can you please take a look? Would really like your inputs on the same. Thanks!

[pengzhao-intel]

@mseth10 thanks for the deep dive. I will look into the case and back to you soon.

[mseth10]

@pengzhao-intel Apparently, the issue is not with viRngBernoulli, it is with the streams being generated. In the present code, different seed is passed to vslNewStream for different threads. Making the seeds constant fixes the flaky issue. Please review. Fix in #13276 .