mxnet.base.MXNetError: Error in operator transpose176: [02:14:06] src/operator/tensor/./matrix_op-inl.h:354: Check failed: shp.ndim() == param.axes.ndim() (-1 vs. 4)

I get an exception when I load model using MXNET as keras's backend. I can load the model correctly with Tensorflow , Theano and cntk as the backend, but I get the following error with MXNET:

Traceback (most recent call last):
File "/data/code/lemon-git/scripts/patch_mxnet/crash_checker.py", line 31, in
model = keras.models.load_model(file_path,custom_objects=ModelUtils.custom_objects())
File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/saving.py", line 496, in load_model
model = deserialize_model(f, custom_objects, compile)
File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/saving.py", line 302, in _deserialize_model
model = model_from_config(model_config, custom_objects=custom_objects)
File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/saving.py", line 535, in model_from_config
return deserialize(config, custom_objects=custom_objects)
File "/root/anaconda3/lib/python3.6/site-packages/keras/layers/init.py", line 55, in deserialize
printable_module_name='layer')
File "/root/anaconda3/lib/python3.6/site-packages/keras/utils/generic_utils.py", line 145, in deserialize_keras_object
list(custom_objects.items())))
File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/sequential.py", line 301, in from_config
model.add(layer)
File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/sequential.py", line 181, in add
output_tensor = layer(self.outputs[0])
File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/base_layer.py", line 470, in call
output = self.call(inputs, kwargs)
File "/root/anaconda3/lib/python3.6/site-packages/keras/layers/convolutional.py", line 175, in call
dilation_rate=self.dilation_rate)
File "/root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py", line 3705, in conv2d
padding_mode=padding, data_format=data_format)
File "/root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py", line 94, in func_wrapper
train_symbol = func(*args, kwargs)
File "/root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py", line 5045, in _convnd
padding, is_slice, out_size = _preprocess_padding_mode(padding_mode, x.shape,
File "/root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py", line 4395, in shape
return self.get_shape()
File "/root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py", line 4404, in get_shape
_, out_shape, _ = self.symbol.infer_shape_partial()
File "/root/anaconda3/lib/python3.6/site-packages/mxnet/symbol/symbol.py", line 1152, in infer_shape_partial
return self.infer_shape_impl(True, *args, **kwargs)
File "/root/anaconda3/lib/python3.6/site-packages/mxnet/symbol/symbol.py", line 1210, in infer_shape_impl
ctypes.byref(complete)))
File "/root/anaconda3/lib/python3.6/site-packages/mxnet/base.py", line 253, in check_call
raise MXNetError(py_str(LIB.MXGetLastError()))
mxnet.base.MXNetError: Error in operator transpose176: [02:14:06] src/operator/tensor/./matrix_op-inl.h:354: Check failed: shp.ndim() == param.axes.ndim() (-1 vs. 4) :
Stack trace:
[bt] (0) /root/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x4b09db) [0x7fd6920089db]
[bt] (1) /root/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x235e45c) [0x7fd693eb645c]
[bt] (2) /root/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x2620d12) [0x7fd694178d12]
[bt] (3) /root/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x26235fb) [0x7fd69417b5fb]
[bt] (4) /root/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(MXSymbolInferShapeEx+0x103e) [0x7fd6940dff7e]
[bt] (5) /root/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(MXSymbolInferShapePartialEx+0x82) [0x7fd6940e0672]
[bt] (6) /root/anaconda3/lib/python3.6/lib-dynload/../../libffi.so.6(ffi_call_unix64+0x4c) [0x7fd6c688bec0]
[bt] (7) /root/anaconda3/lib/python3.6/lib-dynload/../../libffi.so.6(ffi_call+0x22d) [0x7fd6c688b87d]
[bt] (8) /root/anaconda3/lib/python3.6/lib-dynload/ctypes.cpython-36m-x86_64-linux-gnu.so(ctypes_callproc+0x2ce) [0x7fd6c6d33ede]

The code is as follows:
script.py:

import os
import sys
import argparse

"""Parser of command args"""
parse = argparse.ArgumentParser()
parse.add_argument("--backend", type=str, help="name of backends")
parse.add_argument("--crash_dir", type=str, help="path")
flags, unparsed = parse.parse_known_args(sys.argv[1:])

bk = flags.backend
os.environ['KERAS_BACKEND'] = bk

from keras import backend as K
import keras
import traceback


def custom_objects():

    def no_activation(x):
        return x

    def leakyrelu(x):
        import keras.backend as K
        return K.relu(x, alpha=0.01)

    objects = {}
    objects['no_activation'] = no_activation
    objects['leakyrelu'] = leakyrelu
    return objects


print("INFO:Using {} as backend for states extraction| {} is wanted".format(K.backend(), bk))
files = os.listdir(flags.crash_dir)
pass_cnt = 0
for f in files:
    print(f)
    file_path = os.path.join(flags.crash_dir,f)
    try:
        print(f"Loading model. {pass_cnt+1} of {len(files)}")
        model = keras.models.load_model(file_path,custom_objects=custom_objects())
        pass_cnt += 1
    except:
        traceback.print_exc()

print(f"Backend:{bk} Total: {len(files)} Pass:{pass_cnt}")

This is a very simple code. Put the model in a folder, and then run the code to load the model with different backend.

You can run like this:(change mxnet to other backend like tensorflow )

python -u script.py --backend mxnet --crash_dir your/path/

My related library version is:

numpy 1.16.1
tensorboard 1.14.0 py36hf484d3e_0
tensorflow-estimator 1.14.0
tensorflow-gpu 1.14.0
Theano 1.0.4
cntk-gpu 2.7
mxnet-cu101 1.5.1.post0
mkl 2018.0.3 1
mkl-service 1.1.2 py36h17a0993_4
mkl_fft 1.0.1 py36h3010b51_0
mkl_random 1.0.1 py36h629b387_0
Keras 2.2.4
keras-applications 1.0.8 py_0
keras-mxnet 2.2.4.2
keras-preprocessing 1.1.0 py_1

You can download the model from this link:
https://1drv.ms/u/s!Aj6dGBsJFcs0jWiaHhB85gLYKQIQ?e=xdgTSX

This is the structure of the model

thanks in advance!

It is a bug in a shape inference of the Transpose operator. It was changed recently (after 1.5.1 release), but I don't believe your case is covered by those changes (so it would still fail). It should be an easy change, I will make PR to fix it.

@Justobe

@Justobe Actually, after closer examination of the shape inference of transpose, I believe the current version is correct and should fix your issue. Please try nightly build, which you can find e.g. here: https://repo.mxnet.io/dist/2019-12-30/dist/mxnet_cu101mkl-1.6.0b20191230-py2.py3-none-manylinux1_x86_64.whl

@ptrendx

@ptrendx I really appreciate your reply! I got another exception when I changed the version from 1.5 to 1.6 (both the version you provided and version mxnet-cu101_1.6.0b20191122). I reconfirmed that this exception will only occur on the MXNET backend.

/root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py:94: UserWarning: MXNet Backend performs best with channels_first format. Using channels_last will significantly reduce performance due to the Transpose operations. For performance improvement, please use this APIkeras.utils.to_channels_first(x_input)to transform channels_last data to channels_first format and also please change the image_data_format in keras.json to channels_first.Note: x_input is a Numpy tensor or a list of Numpy tensorRefer to: https://github.com/awslabs/keras-apache-mxnet/tree/master/docs/mxnet_backend/performance_guide.md
train_symbol = func(*args, **kwargs)
/root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py:97: UserWarning: MXNet Backend performs best with channels_first format. Using channels_last will significantly reduce performance due to the Transpose operations. For performance improvement, please use this APIkeras.utils.to_channels_first(x_input)to transform channels_last data to channels_first format and also please change the image_data_format in keras.json to channels_first.Note: x_input is a Numpy tensor or a list of Numpy tensorRefer to: https://github.com/awslabs/keras-apache-mxnet/tree/master/docs/mxnet_backend/performance_guide.md
test_symbol = func(*args, **kwargs)
/root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py:94: UserWarning: MXNet Backend uses channels_first format. Axis for BatchNorm should ideally be 1.Provided - -1. Performance can be significantly lower!
train_symbol = func(*args, **kwargs)
/root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py:97: UserWarning: MXNet Backend uses channels_first format. Axis for BatchNorm should ideally be 1.Provided - -1. Performance can be significantly lower!
test_symbol = func(*args, **kwargs)
Traceback (most recent call last):
File "crash_checker.py", line 44, in
model = keras.models.load_model(file_path,custom_objects=custom_objects())
File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/saving.py", line 496, in load_model
model = _deserialize_model(f, custom_objects, compile)
File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/saving.py", line 302, in _deserialize_model
model = model_from_config(model_config, custom_objects=custom_objects)
File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/saving.py", line 535, in model_from_config
return deserialize(config, custom_objects=custom_objects)
File "/root/anaconda3/lib/python3.6/site-packages/keras/layers/init.py", line 55, in deserialize
printable_module_name='layer')
File "/root/anaconda3/lib/python3.6/site-packages/keras/utils/generic_utils.py", line 145, in deserialize_keras_object
list(custom_objects.items())))
File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/sequential.py", line 301, in from_config
model.add(layer)
File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/sequential.py", line 181, in add
output_tensor = layer(self.outputs[0])
File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/base_layer.py", line 470, in call
output = self.call(inputs, **kwargs)
File "/root/anaconda3/lib/python3.6/site-packages/keras/layers/convolutional.py", line 175, in call
dilation_rate=self.dilation_rate)
File "/root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py", line 3705, in conv2d
padding_mode=padding, data_format=data_format)
File "/root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py", line 94, in func_wrapper
train_symbol = func(*args, **kwargs)
File "/root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py", line 5047, in _convnd
filter_dilation)
File "/root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py", line 4870, in _preprocess_padding_mode
for i in range(nd)])
ValueError: not enough values to unpack (expected 3, got 0)

@roywei

@roywei ^^

@Justobe

Hi @Justobe what's the backend used to train your model? Currently keras-mxnet does not support loading model trained using other backends. Mainly due to we made some special cases in Keras front end in order to make it work with mxnet backend. (ref). You may have to train your model directly using MNXet backend, save and load with MXNet backend should work.

@roywei

Hello @roywei, thanks for your reply.

These models are trained under TensorFlow and can be run correctly under CNTK and Theano. I have a lot of models, and only a few models (including this one) throw exceptions under the MXNET. I wonder if this is a bug in currently mxnet. I'm really looking forward that it could be fix if it's a bug. I enjoy using keras under the mxnet backend (this Issue17258 may have the same reason)
I do notice that you have made a lot of changes in keras-mxnet for mxnet. Will this affect other backends in keras-mxnet?(e.g. TensorFlow, CNTK). This is very exciting if keras-mxnet will not affect other backend and can support mxnet at the same time
Besides, I have another problem.I fail to use inceptionV3 (provided by keras) under mxnet, but it works fine with other backends. It seems that mxnet might have some unexpected behavior when I use API of keras.

@Justobe

@Justobe Actually, after closer examination of the shape inference of transpose, I believe the current version is correct and should fix your issue. Please try nightly build, which you can find e.g. here: https://repo.mxnet.io/dist/2019-12-30/dist/mxnet_cu101mkl-1.6.0b20191230-py2.py3-none-manylinux1_x86_64.whl

Hi @ptrendx It seems that mxnet has fixed the bug of transpose, but mxnet still does not support models trained by other backends.
Could you share me more information about the bug in transpose? e.g. commit link or PR link .
Looking forward to receiving your reply！ Thanks in advance.

The PR that fixed the issue in transpose shape inference is #15713.

@ptrendx

@ptrendx Thanks for your reply! Hope that mxnet would also support models trained by other backends in the future.

@ptrendx

Sorry for disturbing you again @ptrendx @roywei I find that this model works fine on versions before mxnet1.4.1(includes version 1.4.1). In mxnet 1.5.1, it crashes and throws an exception about the transpose operator. In mxnet 1.6, it still crashes but throws another exception messages like ValueError: not enough values to unpack (expected 3, got 0).
Is it possible that mxnet1.6 fixed the transpose bug but introduced other bugs, or there also exist some other bugs?

@ptrendx

@ptrendx @roywei gentle ping

Getting the same error while using transpose in dataset's transform in dataloader but when transpose is moved outside the transform, the issue is not present

Navigation Menu

Repository navigation

mxnet.base.MXNetError: Error in operator transpose176: [02:14:06] src/operator/tensor/./matrix_op-inl.h:354: Check failed: shp.ndim() == param.axes.ndim() (-1 vs. 4) #17250

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

mxnet.base.MXNetError: Error in operator transpose176: [02:14:06] src/operator/tensor/./matrix_op-inl.h:354: Check failed: shp.ndim() == param.axes.ndim() (-1 vs. 4) #17250

Description

Activity

ptrendx commented on Jan 8, 2020

ptrendx commented on Jan 8, 2020

Justobe commented on Jan 9, 2020

ptrendx commented on Jan 9, 2020

roywei commented on Jan 9, 2020

Justobe commented on Jan 10, 2020

Justobe commented on Feb 23, 2020

ptrendx commented on Feb 23, 2020

Justobe commented on Feb 23, 2020

Justobe commented on Mar 5, 2020

ChaiBapchya commented on Apr 20, 2020

djaym7 commented on Jun 9, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions