[Flaky Test] test_operator_gpu.test_ftrl

[xziya]

Description

test_ftrl() is located in https://github.com/apache/incubator-mxnet/blob/master/tests/python/unittest/test_optimizer.py#L671

Occurrences

• http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-gpu/detail/PR-18001/2/pipeline/429
• http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-gpu/detail/PR-18001/2/pipeline/418
• http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-gpu/detail/PR-17779/17/pipeline
• http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-gpu/detail/PR-17531/28/pipeline

[xziya]

@mxnet-label-bot add [gpu, optimizer]

[ptrendx]

Also seen in #17995.

[ptrendx]

Hmm, something is wrong with the random seed generation of tests - all those failing test_ftrl got the same seed:

common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=386103813 to reproduce.

FYI @roywei Since you are already looking at random-related stuff and @DickJC123 since you wrote the @with_seed facility.

[leezu]

@szha plans to switch the testing suite to pytest
This involves refactoring the randomness logic and may fix the issue. The new code would likely be inspired by the code in GluonNLP, where we transitioned to pytest already in the initial release and thus refactored the random-related stuff for pytest.

https://github.com/dmlc/gluon-nlp/blob/v0.9.x/conftest.py#L41

[szha]

Occurred again in http://jenkins.mxnet-ci.amazon-ml.com/blue/rest/organizations/jenkins/pipelines/mxnet-validation/pipelines/unix-cpu/branches/PR-18146/runs/2/nodes/365/steps/784/log/?start=0

[leezu]

As a sanity check for the pytest random seed infrastructure: Test failure can be reproduced deterministically using:

MXNET_TEST_SEED=1632189249 pytest tests/python/unittest/test_optimizer.py -k test_ftrl