test_operator_gpu.test_rms fails

[indhub]

Description

test_operator_gpu.test_rms fails causing build failure.

Failed build:
https://builds.apache.org/view/Incubator%20Projects/job/incubator-mxnet/job/master/503/
https://builds.apache.org/blue/organizations/jenkins/incubator-mxnet/detail/master/514/pipeline

Error log:

[Python3: MKLML-GPU] ======================================================================
[Python3: MKLML-GPU] FAIL: test_operator_gpu.test_rms
[Python3: MKLML-GPU] ----------------------------------------------------------------------
[Python3: MKLML-GPU] Traceback (most recent call last):
[Python3: MKLML-GPU]   File "/usr/local/lib/python3.4/dist-packages/nose/case.py", line 198, in runTest
[Python3: MKLML-GPU]     self.test(*self.arg)
[Python3: MKLML-GPU]   File "/usr/local/lib/python3.4/dist-packages/nose/util.py", line 620, in newfunc
[Python3: MKLML-GPU]     return func(*arg, **kw)
[Python3: MKLML-GPU]   File "/workspace/tests/python/gpu/../unittest/test_optimizer.py", line 564, in test_rms
[Python3: MKLML-GPU]     compare_optimizer(opt1(**kwarg), opt2(**kwarg), shape, dtype)
[Python3: MKLML-GPU]   File "/workspace/tests/python/gpu/../unittest/test_optimizer.py", line 101, in compare_optimizer
[Python3: MKLML-GPU]     compare_ndarray_tuple(state1, state2, rtol=1e-4, atol=1e-5)
[Python3: MKLML-GPU]   File "/workspace/tests/python/gpu/../unittest/test_optimizer.py", line 72, in compare_ndarray_tuple
[Python3: MKLML-GPU]     compare_ndarray_tuple(s1, s2, rtol, atol)
[Python3: MKLML-GPU]   File "/workspace/tests/python/gpu/../unittest/test_optimizer.py", line 74, in compare_ndarray_tuple
[Python3: MKLML-GPU]     assert_almost_equal(t1.asnumpy(), t2.asnumpy(), rtol=rtol, atol=atol)
[Python3: MKLML-GPU]   File "/workspace/python/mxnet/test_utils.py", line 491, in assert_almost_equal
[Python3: MKLML-GPU]     raise AssertionError(msg)
[Python3: MKLML-GPU] AssertionError:
[Python3: MKLML-GPU] Items are not equal:
[Python3: MKLML-GPU] Error 319.562256 exceeds tolerance rtol=0.000100, atol=0.000010.  Location of maximum error:(0, 0, 2), a=0.007485, b=0.004156
[Python3: MKLML-GPU]  a: array([[[ 0.016     ,  0.016     ,  0.00748458,  0.016     ,  0.016     ],
[Python3: MKLML-GPU]         [ 0.01204856,  0.016     ,  0.00497273,  0.016     ,  0.016     ],
[Python3: MKLML-GPU]         [ 0.00171078,  0.016     ,  0.016     ,  0.016     ,  0.016     ],...
[Python3: MKLML-GPU]  b: array([[[ 0.016     ,  0.016     ,  0.00415614,  0.016     ,  0.016     ],
[Python3: MKLML-GPU]         [ 0.00951787,  0.016     ,  0.00399938,  0.016     ,  0.016     ],
[Python3: MKLML-GPU]         [ 0.00043585,  0.016     ,  0.01412016,  0.016     ,  0.016     ],...
[Python3: MKLML-GPU]

The failing code is from PR #7654.

Environment info (Required)

Failed on the build machine.
Build config: Python3: MKLML-GPU
Check https://builds.apache.org/blue/organizations/jenkins/incubator-mxnet/detail/master/503/pipeline

MXNet commit hash:
2ff9584

Steps to reproduce

Build and run unit test. See above for more details.

[indhub]

@ptrendx @piiswrong

[ptrendx]

Will look into that.*

[ptrendx]

The failure was uncovered now, because in the PR #7654 I added a lot more tests to test_optimizer.py file. However, I can reproduce this failure on a previous commit f56aaa9 if I test with the new set of tests (minus multi precision), so the problem is most probably with RMS itself. Adding @sbodenstein who added RMS as NNVM op.

[indhub]

This test has been disabled in the master branch now. No more builds will break because of this test.

We now need to discuss if the absence of this test if a blocker for cutting the release candidate for 0.12.0. Unless someone confirms in this ticket that it is not a blocker, I'll assume it is. Please let me know.

[frankfliu]

I enabled to test, and was able to reproduce the failure with seed: 2127559962:

test_operator_gpu.test_rms ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=2127559962 to reproduce.
FAIL

======================================================================
FAIL: test_operator_gpu.test_rms

Traceback (most recent call last):
File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/nose/util.py", line 620, in newfunc
return func(*arg, **kw)
File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/common.py", line 157, in test_new
orig_test(*args, **kwargs)
File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/test_optimizer.py", line 868, in test_rms
compare_optimizer(opt1(**kwarg), opt2(**kwarg), shape, dtype)
File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/test_optimizer.py", line 109, in compare_optimizer
assert_almost_equal(w1.asnumpy(), w2.asnumpy(), rtol=rtol, atol=atol)
File "/home/ubuntu/incubator-mxnet/python/mxnet/test_utils.py", line 493, in assert_almost_equal
raise AssertionError(msg)
AssertionError:
Items are not equal:
Error 1.976562 exceeds tolerance rtol=0.000100, atol=0.000010. Location of maximum error:(0, 3, 0), a=0.054413, b=0.054443
a: array([[[0.1688 , 0.356 , 0.861 , 0.3599 , 0.1854 ],
[0.05255, 0.4927 , 0.4006 , 0.1686 , 0.2196 ],
[0.765 , 0.1313 , 0.751 , 0.2019 , 0.08466],...
b: array([[[0.1688 , 0.356 , 0.861 , 0.3599 , 0.1854 ],
[0.05255, 0.4927 , 0.4006 , 0.1686 , 0.2196 ],
[0.765 , 0.1313 , 0.751 , 0.2019 , 0.08466],...
-------------------- >> begin captured logging << --------------------
common: INFO: Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=1945781048 to reproduce.
common: WARNING: *** test-level seed set: all "@with_seed()" tests run deterministically ***
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=2127559962 to reproduce.
--------------------- >> end captured logging << ---------------------

---

Ran 1 test in 2.574s