[bug] Autograd throws an exception that was not caught in MXNet 1.6

[yzh119]

Description

The Autograd module throws an exception that was not caught:

Error in sys.excepthook:

Original exception was:

After the execution of the program.

To Reproduce

Below is a minimal example to reproduce the bug:

from mxnet import nd
import mxnet as mx
x = mx.np.zeros((10,))
x.attach_grad()

class Op(mx.autograd.Function):
    def forward(self, x):
        out = x + 1
        return out

    def backward(self, grad):
        grad_x = grad
        return grad_x

op = Op()
with mx.autograd.record():
    y = op(x)
    y.sum().backward()

print(x.grad)

However, it could successfully print the result

[1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]
Error in sys.excepthook:

Original exception was:

Environment

----------Python Info----------
Version      : 3.7.6
Compiler     : GCC 7.3.0
Build        : ('default', 'Jan  8 2020 19:59:22')
Arch         : ('64bit', '')
------------Pip Info-----------
Version      : 20.0.2
Directory    : /home/***/anaconda3/lib/python3.7/site-packages/pip
----------MXNet Info-----------
Version      : 1.6.0
Directory    : /home/***/anaconda3/lib/python3.7/site-packages/mxnet
Num GPUs     : 1
Commit Hash   : 6eec9da55c5096079355d1f1a5fa58dcf35d6752
----------System Info----------
Platform     : Linux-5.7.8-200.fc32.x86_64-x86_64-with-fedora-32-Thirty_Two
system       : Linux
node         : LAPTOP-24KAE66Q.lan
release      : 5.7.8-200.fc32.x86_64
version      : #1 SMP Thu Jul 9 14:34:51 UTC 2020

[yzh119]

@sxjscience , @eric-haibin-lin do you have any idea on what happened? I met the same problem on an AWS p3 instance, and the error message persists after I upgrade my MXNet to 2.0 version.

[szha]

I think the numpy array support in autograd.function was missed. #18790

[yzh119]

@szha Thanks for your help, but I think the problem also exists for mxnet.ndarray's, not only for mxnet.numpy.array.

[szha]

Did you run the above as a script? Or in an interactive python shell? I want to see if the program terminated immediately after the execution.

[yzh119]

I ran the above code as a script.
If I ran it in interactive mode, the exception message appears as I exit:

>>> from mxnet import nd
>>> import mxnet as mx
>>> x = mx.np.zeros((10,))
>>> x.attach_grad()
>>> class Op(mx.autograd.Function):
...     def forward(self, x):
...         out = x + 1
...         return out
...     def backward(self, grad):
...         grad_x = grad
...         return grad_x
... 
>>> op = Op()
>>> with mx.autograd.record():
...     y = op(x)
...     y.sum().backward()
... 
>>> print(x.grad)
[1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]
>>> 
>>> 
Error in sys.excepthook:

Original exception was:

[szha]

#18768 probably have fixed it. Would you try the nightly build and see if this is still an issue?
pip install --pre mxnet-cu100 -f https://dist.mxnet.io/python

[yzh119]

@szha no, the problem still exists.

[szha]

In [3]: from mxnet import nd
   ...: import mxnet as mx
   ...: x = mx.np.zeros((10,))
   ...: x.attach_grad()
   ...: mx.npx.set_np()
   ...: class Op(mx.autograd.Function):
   ...:     def forward(self, x):
   ...:         out = x + 1
   ...:         return out
   ...:
   ...:     def backward(self, grad):
   ...:         grad_x = grad
   ...:         return grad_x
   ...:
   ...: op = Op()
   ...: with mx.autograd.record():
   ...:     y = op(x)
   ...:     y.sum().backward()
   ...:
   ...: print(x.grad)
[1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]

In [4]:
Do you really want to exit ([y]/n)?

[szha]

mx.npx.set_np() seems to be missing

[sxjscience]

Actually I cannot reproduce this error. @yzh119 Would you try again with the latest nightly version and with the following code snippet?

To install the nightly version:

# Install the version with CUDA 10.0
python3 -m pip install -U --pre "mxnet-cu100>=2.0.0b20200802" -f https://dist.mxnet.io/python

# Install the version with CUDA 10.1
python3 -m pip install -U --pre "mxnet-cu101>=2.0.0b20200802" -f https://dist.mxnet.io/python

# Install the version with CUDA 10.2
python3 -m pip install -U --pre "mxnet-cu102>=2.0.0b20200802" -f https://dist.mxnet.io/python

# Install the cpu-only version
python3 -m pip install -U --pre "mxnet>=2.0.0b20200802" -f https://dist.mxnet.io/python

from mxnet import nd
import mxnet as mx
mx.npx.set_np()
x = mx.np.zeros((10,))
x.attach_grad()

class Op(mx.autograd.Function):
    def forward(self, x):
        out = x + 1
        return out

    def backward(self, grad):
        grad_x = grad
        return grad_x

op = Op()
with mx.autograd.record():
    y = op(x)
    y.sum().backward()

print(x.grad)