[CI][NightlyTestsForBinaries] Test Large Tensor: CPU killing node instance

[perdasilva]

Description

It seems Test Large Tensor: CPU is killing the underlying CI node somehow:

build.py: 2019-05-16 06:17:45,292Z INFO Started container: ad87c775febf
+ NOSE_COVERAGE_ARGUMENTS='--with-coverage --cover-inclusive --cover-xml --cover-branches --cover-package=mxnet'
+ NOSE_TIMER_ARGUMENTS='--with-timer --timer-ok 1 --timer-warning 15 --timer-filter warning,error'
+ CI_CUDA_COMPUTE_CAPABILITIES='-gencode=arch=compute_52,code=sm_52 -gencode=arch=compute_70,code=sm_70'
+ CI_CMAKE_CUDA_ARCH_BIN=52,70
+ set +x
+ export PYTHONPATH=./python/
+ PYTHONPATH=./python/
+ nosetests-3.4 tests/nightly/test_large_array.py
....[INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1786473439 to reproduce.
Cannot contact mxnetlinux-cpu_8gdgtj05sa: java.lang.InterruptedException

Complete log:

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/NightlyTestsForBinaries/detail/master/312/pipeline/

[mxnet-label-bot]

Hey, this is the MXNet Label Bot.
Thank you for submitting the issue! I will try and suggest some labels so that the appropriate MXNet community members can help resolve it.
Here are my recommended labels: Test

[vdantu]

@mxnet-label-bot add [test]
@apeforest : Would you be able to help out here?

[perdasilva]

Just had this failure in branch v1.5.x. Seems like this error could be related to #14981 and might be fixed by #14990. I'm running a test to verify.

[Chancebair]

@ChaiBapchya @apeforest @access2rohit @anirudh2290 @larroy

The discussion has come up that this test needs to be refactored so that several gigs of memory are not needed to run tests. Pedro brought this up in an email subject "Tests with large inputs and rationalize resource usage. Better testing strategies..."

Could someone please take lead on this? The test is currently disabled in the Jenkins steps for now

[apeforest]

@larroy @Chancebair Any suggestion for updating the tests? We still need to test large tensors for CPU. Please advise what the best practice is.

[larroy]

@apeforest Im talking to @access2rohit to understand how can we test this better. I will update this.

[larroy]

The discussion was moved to devlist.

[larroy]

@Chancebair @perdasilva can we rerun the linked CI run? is the failure related to the test or something else? (what's the root cause?)

[perdasilva]

I think the tests were causing the machine to run out of ram and crash out. Should I re-run it?

[larroy]

how much ram do we have in that machine? If you are confident is not going to cause problems in your fleet I'd say let's run it to see if the problem persist.

[perdasilva]

I think the CPU instances are c5.4xlarge - so 32GB. There's been a PR to disable the CPU tests. I would suggest to re-enable it in a PR, copy the nightly tests for binaries Jenkinsfile content to be be of the PR Jenkins files and seeing if it works. I'm sorry I can't do it atm as I'm on sick leave =S

[szha]

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fcentos-cpu/detail/PR-18146/42/pipeline#step-96-log-126

[access2rohit]

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fcentos-cpu/detail/PR-18146/42/pipeline#step-96-log-126

@szha The link shared about has flags -DMSHADOW_INT64_TENSOR_SIZE=0 which is not Large Tensor Build. Currently Large Tensor Tests are only in nightly(Not CI that runs on every PR). Is it possible that you copied a different link ?