Skip to content
This repository was archived by the owner on Nov 17, 2023. It is now read-only.
This repository was archived by the owner on Nov 17, 2023. It is now read-only.

Flaky test_random.test_exponential_generator @ Python2: MKLDNN-CPU #9843

Closed
Closed
Flaky test_random.test_exponential_generator @ Python2: MKLDNN-CPU#9843
Assignees
anirudh2290
Labels
FlakyTest

Description

@marcoabreu
marcoabreu
opened on Feb 21, 2018

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/master/395/pipeline

======================================================================

FAIL: test_random.test_exponential_generator

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/usr/local/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest

    self.test(*self.arg)

  File "/workspace/tests/python/unittest/common.py", line 155, in test_new

    orig_test(*args, **kwargs)

  File "/workspace/tests/python/unittest/test_random.py", line 374, in test_exponential_generator

    verify_generator(generator=generator_mx_same_seed, buckets=buckets, probs=probs)

  File "/workspace/python/mxnet/test_utils.py", line 1891, in verify_generator

    str(buckets), str(probs)))

AssertionError: Generator test fails, Chi-square p=[0.21031861623067064, 0.023120840905593496, 0.02566284972432512, 0.0396308101152387, 0.031248307960405564], obs_freq=[array([200561, 200496, 199515, 200039, 199389]), array([198895, 200007, 200598, 200780, 199720]), array([200509, 200524, 199203, 200596, 199168]), array([199922, 200497, 200741, 199937, 198903]), array([199609, 200258, 199725, 201138, 199270])], expected_freq=[array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32)].

buckets=[(0.0, 0.22314355131420976), (0.22314355131420976, 0.5108256237659907), (0.5108256237659907, 0.916290731874155), (0.916290731874155, 1.6094379124341005), (1.6094379124341005, inf)], probs=[0.2, 0.2, 0.2, 0.2, 0.2]

-------------------- >> begin captured stdout << ---------------------

ctx=cpu(0), dtype=float16, Scale=0.1:

ctx=cpu(0), dtype=float16, Scale=1:



--------------------- >> end captured stdout << ----------------------

-------------------- >> begin captured logging << --------------------

common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1690699840 to reproduce.

--------------------- >> end captured logging << ---------------------

Activity

  1. marcoabreu
    added
    Test
    Flaky
    on Feb 21, 2018
  2. marcoabreu
    changed the title [-]Flaky test_random.test_exponential_generator[/-] [+]Flaky test_random.test_exponential_generator @ Python2: MKLDNN-CPU[/+] on Feb 21, 2018

  3. zheng-da commented on Feb 22, 2018

    @zheng-da
    zheng-da
    on Feb 22, 2018
    Contributor

    It seems this problem is related to #9856

  4. sxjscience commented on Feb 22, 2018

    @sxjscience
    sxjscience
    on Feb 22, 2018
    Member

    As shown in the log, the chi-square value is above the 0.05 threshold (indicates the check has passed) once in the 5 runs, i.e, 20% success rate. Currently we force the success rate to be 25% (https://github.com/apache/incubator-mxnet/blob/master/python/mxnet/test_utils.py#L1848), we could lower it to 10% to fix the test.

  5. marcoabreu commented on Feb 22, 2018

    @marcoabreu
    marcoabreu
    on Feb 22, 2018
    ContributorAuthor

    That might fix the test, but could we risk masking a problem that way? I'm afraid I'm unfamiliar with the topic, so would you mind explaining how an actual failure would look like and how we could select the best threshold?

  6. sxjscience commented on Feb 23, 2018

    @sxjscience
    sxjscience
    on Feb 23, 2018
    Member

    @marcoabreu A wrongly implemented random number generator would typically have success_rate = 0, i.e, producing p < 0.05 for all the 5 runs. However, since this tests the randomness, it's really tricky to set the threshold and my previous design is to force the success rate to be at least 25%.

  7. zheng-da commented on Feb 28, 2018

    @zheng-da
    zheng-da
    on Feb 28, 2018
    Contributor

    This test also fails constantly with a special seed.

    $ export MXNET_TEST_SEED=548415857
$ for i in {1..10}; do nosetests -v tests/python/gpu/test_operator_gpu.py:test_exponential_generator; done
[INFO] Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=798238534 to reproduce.
[WARNING] *** test-level seed set: all "@with_seed()" tests run deterministically ***
test_operator_gpu.test_exponential_generator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
FAIL

======================================================================
FAIL: test_operator_gpu.test_exponential_generator
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/usr/lib/python2.7/dist-packages/nose/util.py", line 620, in newfunc
    return func(*arg, **kw)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/common.py", line 155, in test_new
    orig_test(*args, **kwargs)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/test_random.py", line 369, in test_exponential_generator
    verify_generator(generator=generator_mx, buckets=buckets, probs=probs)
  File "/home/ubuntu/incubator-mxnet/python/mxnet/test_utils.py", line 1891, in verify_generator
    str(buckets), str(probs)))
AssertionError: Generator test fails, Chi-square p=[0.027708004423388152, 0.048739405069752051, 0.049679278693325266, 0.093148605055595635, 0.0099893338307405818], obs_freq=[array([198823, 200824, 200335, 199968, 200050]), array([200671, 199850, 198978, 199885, 200616]), array([200488, 200237, 199846, 198872, 200557]), array([199234, 199570, 200093, 200233, 200870]), array([200578, 198743, 200487, 200589, 199603])], expected_freq=[array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32)].
buckets=[(0.0, 0.22314355131420976), (0.22314355131420976, 0.51082562376599072), (0.51082562376599072, 0.916290731874155), (0.916290731874155, 1.6094379124341005), (1.6094379124341005, inf)], probs=[0.2, 0.2, 0.2, 0.2, 0.2]
-------------------- >> begin captured stdout << ---------------------
ctx=gpu(0), dtype=float16, Scale=0.1:
ctx=gpu(0), dtype=float16, Scale=1:
ctx=gpu(0), dtype=float32, Scale=0.1:
ctx=gpu(0), dtype=float32, Scale=1:

--------------------- >> end captured stdout << ----------------------
-------------------- >> begin captured logging << --------------------
common: INFO: Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=798238534 to reproduce.
common: WARNING: *** test-level seed set: all "@with_seed()" tests run deterministically ***
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 1 test in 3.485s

FAILED (failures=1)
[INFO] Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=422739486 to reproduce.
[WARNING] *** test-level seed set: all "@with_seed()" tests run deterministically ***
test_operator_gpu.test_exponential_generator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
FAIL

======================================================================
FAIL: test_operator_gpu.test_exponential_generator
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/usr/lib/python2.7/dist-packages/nose/util.py", line 620, in newfunc
    return func(*arg, **kw)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/common.py", line 155, in test_new
    orig_test(*args, **kwargs)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/test_random.py", line 369, in test_exponential_generator
    verify_generator(generator=generator_mx, buckets=buckets, probs=probs)
  File "/home/ubuntu/incubator-mxnet/python/mxnet/test_utils.py", line 1891, in verify_generator
    str(buckets), str(probs)))
AssertionError: Generator test fails, Chi-square p=[0.027708004423388152, 0.048739405069752051, 0.049679278693325266, 0.093148605055595635, 0.0099893338307405818], obs_freq=[array([198823, 200824, 200335, 199968, 200050]), array([200671, 199850, 198978, 199885, 200616]), array([200488, 200237, 199846, 198872, 200557]), array([199234, 199570, 200093, 200233, 200870]), array([200578, 198743, 200487, 200589, 199603])], expected_freq=[array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32)].
buckets=[(0.0, 0.22314355131420976), (0.22314355131420976, 0.51082562376599072), (0.51082562376599072, 0.916290731874155), (0.916290731874155, 1.6094379124341005), (1.6094379124341005, inf)], probs=[0.2, 0.2, 0.2, 0.2, 0.2]
-------------------- >> begin captured stdout << ---------------------
ctx=gpu(0), dtype=float16, Scale=0.1:
ctx=gpu(0), dtype=float16, Scale=1:
ctx=gpu(0), dtype=float32, Scale=0.1:
ctx=gpu(0), dtype=float32, Scale=1:

--------------------- >> end captured stdout << ----------------------
-------------------- >> begin captured logging << --------------------
common: INFO: Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=422739486 to reproduce.
common: WARNING: *** test-level seed set: all "@with_seed()" tests run deterministically ***
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 1 test in 4.569s

FAILED (failures=1)
[INFO] Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=1017842502 to reproduce.
[WARNING] *** test-level seed set: all "@with_seed()" tests run deterministically ***
test_operator_gpu.test_exponential_generator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
FAIL

======================================================================
FAIL: test_operator_gpu.test_exponential_generator
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/usr/lib/python2.7/dist-packages/nose/util.py", line 620, in newfunc
    return func(*arg, **kw)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/common.py", line 155, in test_new
    orig_test(*args, **kwargs)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/test_random.py", line 369, in test_exponential_generator
    verify_generator(generator=generator_mx, buckets=buckets, probs=probs)
  File "/home/ubuntu/incubator-mxnet/python/mxnet/test_utils.py", line 1891, in verify_generator
    str(buckets), str(probs)))
AssertionError: Generator test fails, Chi-square p=[0.027708004423388152, 0.048739405069752051, 0.049679278693325266, 0.093148605055595635, 0.0099893338307405818], obs_freq=[array([198823, 200824, 200335, 199968, 200050]), array([200671, 199850, 198978, 199885, 200616]), array([200488, 200237, 199846, 198872, 200557]), array([199234, 199570, 200093, 200233, 200870]), array([200578, 198743, 200487, 200589, 199603])], expected_freq=[array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32)].
buckets=[(0.0, 0.22314355131420976), (0.22314355131420976, 0.51082562376599072), (0.51082562376599072, 0.916290731874155), (0.916290731874155, 1.6094379124341005), (1.6094379124341005, inf)], probs=[0.2, 0.2, 0.2, 0.2, 0.2]
-------------------- >> begin captured stdout << ---------------------
ctx=gpu(0), dtype=float16, Scale=0.1:
ctx=gpu(0), dtype=float16, Scale=1:
ctx=gpu(0), dtype=float32, Scale=0.1:
ctx=gpu(0), dtype=float32, Scale=1:

--------------------- >> end captured stdout << ----------------------
-------------------- >> begin captured logging << --------------------
common: INFO: Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=1017842502 to reproduce.
common: WARNING: *** test-level seed set: all "@with_seed()" tests run deterministically ***
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 1 test in 3.078s

FAILED (failures=1)
[INFO] Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=300887491 to reproduce.
[WARNING] *** test-level seed set: all "@with_seed()" tests run deterministically ***
test_operator_gpu.test_exponential_generator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
FAIL

======================================================================
FAIL: test_operator_gpu.test_exponential_generator
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/usr/lib/python2.7/dist-packages/nose/util.py", line 620, in newfunc
    return func(*arg, **kw)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/common.py", line 155, in test_new
    orig_test(*args, **kwargs)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/test_random.py", line 369, in test_exponential_generator
    verify_generator(generator=generator_mx, buckets=buckets, probs=probs)
  File "/home/ubuntu/incubator-mxnet/python/mxnet/test_utils.py", line 1891, in verify_generator
    str(buckets), str(probs)))
AssertionError: Generator test fails, Chi-square p=[0.027708004423388152, 0.048739405069752051, 0.049679278693325266, 0.093148605055595635, 0.0099893338307405818], obs_freq=[array([198823, 200824, 200335, 199968, 200050]), array([200671, 199850, 198978, 199885, 200616]), array([200488, 200237, 199846, 198872, 200557]), array([199234, 199570, 200093, 200233, 200870]), array([200578, 198743, 200487, 200589, 199603])], expected_freq=[array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32)].
buckets=[(0.0, 0.22314355131420976), (0.22314355131420976, 0.51082562376599072), (0.51082562376599072, 0.916290731874155), (0.916290731874155, 1.6094379124341005), (1.6094379124341005, inf)], probs=[0.2, 0.2, 0.2, 0.2, 0.2]
-------------------- >> begin captured stdout << ---------------------
ctx=gpu(0), dtype=float16, Scale=0.1:
ctx=gpu(0), dtype=float16, Scale=1:
ctx=gpu(0), dtype=float32, Scale=0.1:
ctx=gpu(0), dtype=float32, Scale=1:

--------------------- >> end captured stdout << ----------------------
-------------------- >> begin captured logging << --------------------
common: INFO: Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=300887491 to reproduce.
common: WARNING: *** test-level seed set: all "@with_seed()" tests run deterministically ***
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 1 test in 3.074s

FAILED (failures=1)
[INFO] Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=520324218 to reproduce.
[WARNING] *** test-level seed set: all "@with_seed()" tests run deterministically ***
test_operator_gpu.test_exponential_generator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
FAIL

======================================================================
FAIL: test_operator_gpu.test_exponential_generator
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/usr/lib/python2.7/dist-packages/nose/util.py", line 620, in newfunc
    return func(*arg, **kw)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/common.py", line 155, in test_new
    orig_test(*args, **kwargs)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/test_random.py", line 369, in test_exponential_generator
    verify_generator(generator=generator_mx, buckets=buckets, probs=probs)
  File "/home/ubuntu/incubator-mxnet/python/mxnet/test_utils.py", line 1891, in verify_generator
    str(buckets), str(probs)))
AssertionError: Generator test fails, Chi-square p=[0.027708004423388152, 0.048739405069752051, 0.049679278693325266, 0.093148605055595635, 0.0099893338307405818], obs_freq=[array([198823, 200824, 200335, 199968, 200050]), array([200671, 199850, 198978, 199885, 200616]), array([200488, 200237, 199846, 198872, 200557]), array([199234, 199570, 200093, 200233, 200870]), array([200578, 198743, 200487, 200589, 199603])], expected_freq=[array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32)].
buckets=[(0.0, 0.22314355131420976), (0.22314355131420976, 0.51082562376599072), (0.51082562376599072, 0.916290731874155), (0.916290731874155, 1.6094379124341005), (1.6094379124341005, inf)], probs=[0.2, 0.2, 0.2, 0.2, 0.2]
-------------------- >> begin captured stdout << ---------------------
ctx=gpu(0), dtype=float16, Scale=0.1:
ctx=gpu(0), dtype=float16, Scale=1:
ctx=gpu(0), dtype=float32, Scale=0.1:
ctx=gpu(0), dtype=float32, Scale=1:

--------------------- >> end captured stdout << ----------------------
-------------------- >> begin captured logging << --------------------
common: INFO: Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=520324218 to reproduce.
common: WARNING: *** test-level seed set: all "@with_seed()" tests run deterministically ***
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 1 test in 3.156s

FAILED (failures=1)
[INFO] Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=994645507 to reproduce.
[WARNING] *** test-level seed set: all "@with_seed()" tests run deterministically ***
test_operator_gpu.test_exponential_generator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
FAIL

======================================================================
FAIL: test_operator_gpu.test_exponential_generator
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/usr/lib/python2.7/dist-packages/nose/util.py", line 620, in newfunc
    return func(*arg, **kw)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/common.py", line 155, in test_new
    orig_test(*args, **kwargs)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/test_random.py", line 369, in test_exponential_generator
    verify_generator(generator=generator_mx, buckets=buckets, probs=probs)
  File "/home/ubuntu/incubator-mxnet/python/mxnet/test_utils.py", line 1891, in verify_generator
    str(buckets), str(probs)))
AssertionError: Generator test fails, Chi-square p=[0.027708004423388152, 0.048739405069752051, 0.049679278693325266, 0.093148605055595635, 0.0099893338307405818], obs_freq=[array([198823, 200824, 200335, 199968, 200050]), array([200671, 199850, 198978, 199885, 200616]), array([200488, 200237, 199846, 198872, 200557]), array([199234, 199570, 200093, 200233, 200870]), array([200578, 198743, 200487, 200589, 199603])], expected_freq=[array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32)].
buckets=[(0.0, 0.22314355131420976), (0.22314355131420976, 0.51082562376599072), (0.51082562376599072, 0.916290731874155), (0.916290731874155, 1.6094379124341005), (1.6094379124341005, inf)], probs=[0.2, 0.2, 0.2, 0.2, 0.2]
-------------------- >> begin captured stdout << ---------------------
ctx=gpu(0), dtype=float16, Scale=0.1:
ctx=gpu(0), dtype=float16, Scale=1:
ctx=gpu(0), dtype=float32, Scale=0.1:
ctx=gpu(0), dtype=float32, Scale=1:

--------------------- >> end captured stdout << ----------------------
-------------------- >> begin captured logging << --------------------
common: INFO: Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=994645507 to reproduce.
common: WARNING: *** test-level seed set: all "@with_seed()" tests run deterministically ***
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 1 test in 4.483s

FAILED (failures=1)
[INFO] Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=1429331890 to reproduce.
[WARNING] *** test-level seed set: all "@with_seed()" tests run deterministically ***
test_operator_gpu.test_exponential_generator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
FAIL

======================================================================
FAIL: test_operator_gpu.test_exponential_generator
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/usr/lib/python2.7/dist-packages/nose/util.py", line 620, in newfunc
    return func(*arg, **kw)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/common.py", line 155, in test_new
    orig_test(*args, **kwargs)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/test_random.py", line 369, in test_exponential_generator
    verify_generator(generator=generator_mx, buckets=buckets, probs=probs)
  File "/home/ubuntu/incubator-mxnet/python/mxnet/test_utils.py", line 1891, in verify_generator
    str(buckets), str(probs)))
AssertionError: Generator test fails, Chi-square p=[0.027708004423388152, 0.048739405069752051, 0.049679278693325266, 0.093148605055595635, 0.0099893338307405818], obs_freq=[array([198823, 200824, 200335, 199968, 200050]), array([200671, 199850, 198978, 199885, 200616]), array([200488, 200237, 199846, 198872, 200557]), array([199234, 199570, 200093, 200233, 200870]), array([200578, 198743, 200487, 200589, 199603])], expected_freq=[array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32)].
buckets=[(0.0, 0.22314355131420976), (0.22314355131420976, 0.51082562376599072), (0.51082562376599072, 0.916290731874155), (0.916290731874155, 1.6094379124341005), (1.6094379124341005, inf)], probs=[0.2, 0.2, 0.2, 0.2, 0.2]
-------------------- >> begin captured stdout << ---------------------
ctx=gpu(0), dtype=float16, Scale=0.1:
ctx=gpu(0), dtype=float16, Scale=1:
ctx=gpu(0), dtype=float32, Scale=0.1:
ctx=gpu(0), dtype=float32, Scale=1:

--------------------- >> end captured stdout << ----------------------
-------------------- >> begin captured logging << --------------------
common: INFO: Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=1429331890 to reproduce.
common: WARNING: *** test-level seed set: all "@with_seed()" tests run deterministically ***
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 1 test in 3.407s

FAILED (failures=1)
[INFO] Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=1471819377 to reproduce.
[WARNING] *** test-level seed set: all "@with_seed()" tests run deterministically ***
test_operator_gpu.test_exponential_generator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
FAIL

======================================================================
FAIL: test_operator_gpu.test_exponential_generator
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/usr/lib/python2.7/dist-packages/nose/util.py", line 620, in newfunc
    return func(*arg, **kw)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/common.py", line 155, in test_new
    orig_test(*args, **kwargs)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/test_random.py", line 369, in test_exponential_generator
    verify_generator(generator=generator_mx, buckets=buckets, probs=probs)
  File "/home/ubuntu/incubator-mxnet/python/mxnet/test_utils.py", line 1891, in verify_generator
    str(buckets), str(probs)))
AssertionError: Generator test fails, Chi-square p=[0.027708004423388152, 0.048739405069752051, 0.049679278693325266, 0.093148605055595635, 0.0099893338307405818], obs_freq=[array([198823, 200824, 200335, 199968, 200050]), array([200671, 199850, 198978, 199885, 200616]), array([200488, 200237, 199846, 198872, 200557]), array([199234, 199570, 200093, 200233, 200870]), array([200578, 198743, 200487, 200589, 199603])], expected_freq=[array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32)].
buckets=[(0.0, 0.22314355131420976), (0.22314355131420976, 0.51082562376599072), (0.51082562376599072, 0.916290731874155), (0.916290731874155, 1.6094379124341005), (1.6094379124341005, inf)], probs=[0.2, 0.2, 0.2, 0.2, 0.2]
-------------------- >> begin captured stdout << ---------------------
ctx=gpu(0), dtype=float16, Scale=0.1:
ctx=gpu(0), dtype=float16, Scale=1:
ctx=gpu(0), dtype=float32, Scale=0.1:
ctx=gpu(0), dtype=float32, Scale=1:

--------------------- >> end captured stdout << ----------------------
-------------------- >> begin captured logging << --------------------
common: INFO: Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=1471819377 to reproduce.
common: WARNING: *** test-level seed set: all "@with_seed()" tests run deterministically ***
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 1 test in 3.576s

FAILED (failures=1)
[INFO] Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=454445406 to reproduce.
[WARNING] *** test-level seed set: all "@with_seed()" tests run deterministically ***
test_operator_gpu.test_exponential_generator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
FAIL

======================================================================
FAIL: test_operator_gpu.test_exponential_generator
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/usr/lib/python2.7/dist-packages/nose/util.py", line 620, in newfunc
    return func(*arg, **kw)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/common.py", line 155, in test_new
    orig_test(*args, **kwargs)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/test_random.py", line 369, in test_exponential_generator
    verify_generator(generator=generator_mx, buckets=buckets, probs=probs)
  File "/home/ubuntu/incubator-mxnet/python/mxnet/test_utils.py", line 1891, in verify_generator
    str(buckets), str(probs)))
AssertionError: Generator test fails, Chi-square p=[0.027708004423388152, 0.048739405069752051, 0.049679278693325266, 0.093148605055595635, 0.0099893338307405818], obs_freq=[array([198823, 200824, 200335, 199968, 200050]), array([200671, 199850, 198978, 199885, 200616]), array([200488, 200237, 199846, 198872, 200557]), array([199234, 199570, 200093, 200233, 200870]), array([200578, 198743, 200487, 200589, 199603])], expected_freq=[array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32)].
buckets=[(0.0, 0.22314355131420976), (0.22314355131420976, 0.51082562376599072), (0.51082562376599072, 0.916290731874155), (0.916290731874155, 1.6094379124341005), (1.6094379124341005, inf)], probs=[0.2, 0.2, 0.2, 0.2, 0.2]
-------------------- >> begin captured stdout << ---------------------
ctx=gpu(0), dtype=float16, Scale=0.1:
ctx=gpu(0), dtype=float16, Scale=1:
ctx=gpu(0), dtype=float32, Scale=0.1:
ctx=gpu(0), dtype=float32, Scale=1:

--------------------- >> end captured stdout << ----------------------
-------------------- >> begin captured logging << --------------------
common: INFO: Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=454445406 to reproduce.
common: WARNING: *** test-level seed set: all "@with_seed()" tests run deterministically ***
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 1 test in 3.255s

FAILED (failures=1)
[INFO] Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=53548602 to reproduce.
[WARNING] *** test-level seed set: all "@with_seed()" tests run deterministically ***
test_operator_gpu.test_exponential_generator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
FAIL

======================================================================
FAIL: test_operator_gpu.test_exponential_generator
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/usr/lib/python2.7/dist-packages/nose/util.py", line 620, in newfunc
    return func(*arg, **kw)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/common.py", line 155, in test_new
    orig_test(*args, **kwargs)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/test_random.py", line 369, in test_exponential_generator
    verify_generator(generator=generator_mx, buckets=buckets, probs=probs)
  File "/home/ubuntu/incubator-mxnet/python/mxnet/test_utils.py", line 1891, in verify_generator
    str(buckets), str(probs)))
AssertionError: Generator test fails, Chi-square p=[0.027708004423388152, 0.048739405069752051, 0.049679278693325266, 0.093148605055595635, 0.0099893338307405818], obs_freq=[array([198823, 200824, 200335, 199968, 200050]), array([200671, 199850, 198978, 199885, 200616]), array([200488, 200237, 199846, 198872, 200557]), array([199234, 199570, 200093, 200233, 200870]), array([200578, 198743, 200487, 200589, 199603])], expected_freq=[array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32)].
buckets=[(0.0, 0.22314355131420976), (0.22314355131420976, 0.51082562376599072), (0.51082562376599072, 0.916290731874155), (0.916290731874155, 1.6094379124341005), (1.6094379124341005, inf)], probs=[0.2, 0.2, 0.2, 0.2, 0.2]
-------------------- >> begin captured stdout << ---------------------
ctx=gpu(0), dtype=float16, Scale=0.1:
ctx=gpu(0), dtype=float16, Scale=1:
ctx=gpu(0), dtype=float32, Scale=0.1:
ctx=gpu(0), dtype=float32, Scale=1:

--------------------- >> end captured stdout << ----------------------
-------------------- >> begin captured logging << --------------------
common: INFO: Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=53548602 to reproduce.
common: WARNING: *** test-level seed set: all "@with_seed()" tests run deterministically ***
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 1 test in 3.210s

FAILED (failures=1)

  8. sxjscience commented on Feb 28, 2018

    @sxjscience
    sxjscience
    on Feb 28, 2018
    Member

    Should I revise the code to use a smaller threshold?

  9. azai91 commented on Jun 27, 2018

    @azai91
    azai91
    on Jun 27, 2018
    Contributor

    @marcoabreu the success rate has been lowered. do we know if this test is still causing us problems?

  10. marcoabreu commented on Jun 27, 2018

    @marcoabreu
    marcoabreu
    on Jun 27, 2018
    ContributorAuthor

    Haven't seen any problems recently. Thanks for following up!

  11. marcoabreu
    closed this as completedon Jun 27, 2018

  12. azai91 commented on Jun 27, 2018

    @azai91
    azai91
    on Jun 27, 2018
    Contributor

    kk ran this test 500x as well and didn't see a problem

    ubuntu@ip-172-31-11-93:~/incubator-mxnet-original$ for i in {1..500}; do nosetests tests/python/unittest/test_random.py:test_exponential_generator; if [ $? -ne 0 ]; then break; fi; done || exit 1
  13. anirudh2290
    self-assigned this
    on Jun 27, 2018

  14. piyushghai commented on Jan 7, 2019

    @piyushghai
    piyushghai
    on Jan 7, 2019
    Contributor

    Found a failure of this test on the CI pipeline on Windows-CPU.

    Details of failure captured here : http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fwindows-cpu/detail/PR-13678/30/pipeline

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

FlakyTest

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions