Flaky Test Issue of GPU Operator

Description

kernel_error_check_imperative() and kernel_error_check_symbolic() in test_operator_gpu.py have flaky issues.

test_operator_gpu.test_kernel_error_checking ... Process SpawnProcess-1:

Traceback (most recent call last):

  File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap

    self.run()

  File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run

    self._target(*self._args, **self._kwargs)

  File "/work/mxnet/tests/python/gpu/test_operator_gpu.py", line 1832, in kernel_error_check_imperative

    c = (a / b).asnumpy()

  File "/work/mxnet/tests/python/unittest/../../../python/mxnet/ndarray/ndarray.py", line 1910, in asnumpy

    ctypes.c_size_t(data.size)))

  File "/work/mxnet/tests/python/unittest/../../../python/mxnet/base.py", line 210, in check_call

    raise MXNetError(py_str(_LIB.MXGetLastError()))

mxnet.base.MXNetError: [19:32:53] src/operator/tensor/././../mxnet_op.h:586: Check failed: err == cudaSuccess (9 vs. 0) Name: mxnet_generic_kernel_ex ErrStr:invalid configuration argument

Hi, would you mind looking a CI run where this occurred?

This fails at "/work/mxnet/tests/python/gpu/test_operator_gpu.py", line 1832, in kernel_error_check_imperative http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/PR-11502/25/pipeline/799

I cannot reproduce this error on my local machine, are you sure that this failure is not related to your PR? Seems like this is happening to other environments for the same build as well.

I am closing this issue, since I don't have this issue any more. My PR just passed CI test.

@haojin2

Agree with @haojin2 , the error seems to be related to the PR. Thanks everybody!

@zhanghang1989

I just had the exact same error in another PR. http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/PR-11482/17/pipeline/859#step-1530-log-1018

@zhanghang1989 what did you do to resolve the problem?

I had the exact same error too. Any pointers to the same?

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/PR-12749/28/pipeline/1003/#step-1612-log-1562

@szha

@szha Can we reopen this issue? Your latest PR on Notice #14043 has the same CI issue.

PR #15772
Pipeline - http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-gpu/detail/PR-15772/3/pipeline

Navigation Menu

Repository navigation

Flaky Test Issue of GPU Operator #11592

Description

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Flaky Test Issue of GPU Operator #11592

Description

Description

Activity

marcoabreu commented on Jul 6, 2018

zhanghang1989 commented on Jul 6, 2018

haojin2 commented on Jul 11, 2018

zhanghang1989 commented on Jul 11, 2018

marcoabreu commented on Jul 11, 2018

szha commented on Jul 27, 2018

ChaiBapchya commented on Nov 2, 2018

ChaiBapchya commented on Feb 1, 2019

ChaiBapchya commented on Aug 7, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions