a very interesting problem

[universewill]

I use resent-152 to classify 3800 categories images. And I find a very interesting case: the accuracy of each epoch is decreasing during each epoch while the accuracy is increasing between epochs. I don't know why.

Notice that I shuffled data at the end of each epoch.
Here is the log:

INFO:root:Epoch[5] Batch [4]	Speed: 137.92 samples/sec	accuracy=0.051562
INFO:root:Epoch[5] Batch [8]	Speed: 137.79 samples/sec	accuracy=0.021484
INFO:root:Epoch[5] Batch [12]	Speed: 137.72 samples/sec	accuracy=0.013672
INFO:root:Epoch[5] Batch [16]	Speed: 137.89 samples/sec	accuracy=0.013672
INFO:root:Epoch[5] Batch [20]	Speed: 137.49 samples/sec	accuracy=0.005859
INFO:root:Epoch[5] Batch [24]	Speed: 137.66 samples/sec	accuracy=0.001953
INFO:root:Epoch[5] Batch [28]	Speed: 137.16 samples/sec	accuracy=0.001953
INFO:root:Epoch[5] Train-accuracy=0.015625
INFO:root:Epoch[5] Time cost=27.917
INFO:root:Epoch[6] Batch [4]	Speed: 137.18 samples/sec	accuracy=0.076563
INFO:root:Epoch[6] Batch [8]	Speed: 137.69 samples/sec	accuracy=0.058594
INFO:root:Epoch[6] Batch [12]	Speed: 137.75 samples/sec	accuracy=0.029297
INFO:root:Epoch[6] Batch [16]	Speed: 137.20 samples/sec	accuracy=0.005859
INFO:root:Epoch[6] Batch [20]	Speed: 138.05 samples/sec	accuracy=0.009766
INFO:root:Epoch[6] Batch [24]	Speed: 138.11 samples/sec	accuracy=0.005859
INFO:root:Epoch[6] Batch [28]	Speed: 138.11 samples/sec	accuracy=0.003906
INFO:root:Epoch[6] Train-accuracy=0.007812
INFO:root:Epoch[6] Time cost=27.911
INFO:root:Epoch[7] Batch [4]	Speed: 137.94 samples/sec	accuracy=0.114062
INFO:root:Epoch[7] Batch [8]	Speed: 137.92 samples/sec	accuracy=0.089844
INFO:root:Epoch[7] Batch [12]	Speed: 137.99 samples/sec	accuracy=0.070312
INFO:root:Epoch[7] Batch [16]	Speed: 137.76 samples/sec	accuracy=0.037109
INFO:root:Epoch[7] Batch [20]	Speed: 137.43 samples/sec	accuracy=0.023438
INFO:root:Epoch[7] Batch [24]	Speed: 136.92 samples/sec	accuracy=0.009766
INFO:root:Epoch[7] Batch [28]	Speed: 137.28 samples/sec	accuracy=0.007812
INFO:root:Epoch[7] Train-accuracy=0.007812
INFO:root:Epoch[7] Time cost=27.933
INFO:root:Epoch[8] Batch [4]	Speed: 138.18 samples/sec	accuracy=0.146875
INFO:root:Epoch[8] Batch [8]	Speed: 137.99 samples/sec	accuracy=0.130859
INFO:root:Epoch[8] Batch [12]	Speed: 137.80 samples/sec	accuracy=0.085938
INFO:root:Epoch[8] Batch [16]	Speed: 137.72 samples/sec	accuracy=0.068359
INFO:root:Epoch[8] Batch [20]	Speed: 137.98 samples/sec	accuracy=0.048828
INFO:root:Epoch[8] Batch [24]	Speed: 137.76 samples/sec	accuracy=0.027344
INFO:root:Epoch[8] Batch [28]	Speed: 137.62 samples/sec	accuracy=0.019531
INFO:root:Epoch[8] Train-accuracy=0.015625
INFO:root:Epoch[8] Time cost=27.877
INFO:root:Epoch[9] Batch [4]	Speed: 137.92 samples/sec	accuracy=0.225000
INFO:root:Epoch[9] Batch [8]	Speed: 137.97 samples/sec	accuracy=0.173828
INFO:root:Epoch[9] Batch [12]	Speed: 137.87 samples/sec	accuracy=0.099609
INFO:root:Epoch[9] Batch [16]	Speed: 137.93 samples/sec	accuracy=0.082031
INFO:root:Epoch[9] Batch [20]	Speed: 137.42 samples/sec	accuracy=0.058594
INFO:root:Epoch[9] Batch [24]	Speed: 138.03 samples/sec	accuracy=0.041016
INFO:root:Epoch[9] Batch [28]	Speed: 137.75 samples/sec	accuracy=0.033203
INFO:root:Epoch[9] Train-accuracy=0.031250
INFO:root:Epoch[9] Time cost=27.864
INFO:root:Epoch[10] Batch [4]	Speed: 137.32 samples/sec	accuracy=0.260937
INFO:root:Epoch[10] Batch [8]	Speed: 137.73 samples/sec	accuracy=0.195312
INFO:root:Epoch[10] Batch [12]	Speed: 137.00 samples/sec	accuracy=0.177734
INFO:root:Epoch[10] Batch [16]	Speed: 138.26 samples/sec	accuracy=0.115234
INFO:root:Epoch[10] Batch [20]	Speed: 138.08 samples/sec	accuracy=0.089844
INFO:root:Epoch[10] Batch [24]	Speed: 137.97 samples/sec	accuracy=0.082031
INFO:root:Epoch[10] Batch [28]	Speed: 138.18 samples/sec	accuracy=0.082031
INFO:root:Epoch[10] Train-accuracy=0.039062
INFO:root:Epoch[10] Time cost=27.892
INFO:root:Epoch[11] Batch [4]	Speed: 137.22 samples/sec	accuracy=0.315625
INFO:root:Epoch[11] Batch [8]	Speed: 137.96 samples/sec	accuracy=0.306641
INFO:root:Epoch[11] Batch [12]	Speed: 138.00 samples/sec	accuracy=0.236328
INFO:root:Epoch[11] Batch [16]	Speed: 137.58 samples/sec	accuracy=0.171875
INFO:root:Epoch[11] Batch [20]	Speed: 137.81 samples/sec	accuracy=0.148438
INFO:root:Epoch[11] Batch [24]	Speed: 137.80 samples/sec	accuracy=0.113281
INFO:root:Epoch[11] Batch [28]	Speed: 137.76 samples/sec	accuracy=0.097656
INFO:root:Epoch[11] Train-accuracy=0.101562
INFO:root:Epoch[11] Time cost=27.908

[wkcn]

What version of MXNet did you use?

For the old version of MXNet, Train-accuracy is the accuracy of the last batch.
For the latest MXNet, Train-accuracy is the accuracy of the whole epoch.

Please see the PR:-)
Track epoch metric separately (#12182)

[universewill]

I just don't understand why the accuracy is decreasing during each epoch. You can see the accuracy is decreasing for batches.
I think it has nothing to do with mxnet version?

INFO:root:Epoch[9] Batch [4]	Speed: 137.92 samples/sec	accuracy=0.225000
INFO:root:Epoch[9] Batch [8]	Speed: 137.97 samples/sec	accuracy=0.173828
INFO:root:Epoch[9] Batch [12]	Speed: 137.87 samples/sec	accuracy=0.099609
INFO:root:Epoch[9] Batch [16]	Speed: 137.93 samples/sec	accuracy=0.082031
INFO:root:Epoch[9] Batch [20]	Speed: 137.42 samples/sec	accuracy=0.058594
INFO:root:Epoch[9] Batch [24]	Speed: 138.03 samples/sec	accuracy=0.041016
INFO:root:Epoch[9] Batch [28]	Speed: 137.75 samples/sec	accuracy=0.033203
INFO:root:Epoch[9] Train-accuracy=0.031250
INFO:root:Epoch[9] Time cost=27.864

You can see the accuracy is decreasing: 0.22 -> 0.17 -> 0.09 -> 0.08 -> 0.058 -> 0.041 -> 0.033
My mxnet version is 1.3.0
@wkcn

[wkcn]

@universewill
Could you please provide the reproduced code?

[universewill]

This is my model code:

def my_model(num_classes = num_classes):
    sym, arg_params, aux_params = mx.model.load_checkpoint('resnet-152', 0)
    
    all_layers = sym.get_internals()
    print(all_layers.list_outputs()[-10:])
    feat_sym = all_layers['flatten0_output']
    
    # 用于冻结
    abandon_para_names = all_layers.list_outputs()[-60:]
    freeze_para_names =[k for k in arg_params.keys() if k not in abandon_para_names]
    
    # 额外增加两个smulti-task的共有层
    net = mx.sym.FullyConnected(data=feat_sym, num_hidden=1024, name='my_layer_0')
    net = mx.sym.FullyConnected(data=net, num_hidden=512, name='my_layer_1')
    net = mx.sym.FullyConnected(data=net, num_hidden=256, name='my_layer_2')
    net = mx.symbol.BatchNorm(net, name='my_layer')
    net = mx.symbol.FullyConnected(data=net, num_hidden=num_classes, name='my_layer_3')
    # 分类
    loss = mx.sym.SoftmaxOutput(data=net, name= 'softmax')
    mod = mx.mod.Module(loss, context=mx.gpu(0), fixed_param_names=freeze_para_names)
    
    new_args = dict({k:arg_params[k] for k in arg_params if k not in all_layers.list_outputs()[-60:]})
    return mod, arg_params, aux_params

This is my train code:

import logging
logging.getLogger().setLevel(logging.INFO)
logging.basicConfig(level=logging.INFO, filename='train.log')

mod, arg_params, aux_params = mx_model.my_model(num_classes)
train_iter = DataIter(image_root=data_root, batch_size = batch_size)

batch_size = 128
data_root = '/home/universe/jupyter/data'
train_iter = DataIter(image_root=data_root, batch_size = batch_size)

mod.bind(data_shapes= train_iter.provide_data, label_shapes= train_iter.provide_label, force_rebind=True)
curr_data_shapes = tuple(i.shape for i in mod._data_shapes)
print(curr_data_shapes)
new_data_shapes = tuple(i.shape for i in train_iter.next().data)
print(new_data_shapes)
initializer = mx.initializer.Xavier()
mod.init_params(initializer=initializer, arg_params=arg_params, aux_params=aux_params, allow_missing=True, allow_extra=True)
mod.init_optimizer(optimizer='adam', optimizer_params={'learning_rate': 0.0005}, force_init=True)

mod.fit(train_data=train_iter,
        batch_end_callback=mx.callback.Speedometer(batch_size, 4),
        epoch_end_callback=mx.callback.do_checkpoint("./checkpoints/my_model", 50),
        num_epoch=1500)

@wkcn

[wkcn]

@universewill Have you tested that 'train_iter' is random?

[universewill]

yeah, I printed the first batch of each epoch, this data is truely shuffled between each epoch
@wkcn

[andrewfayres]

@mxnet-label-bot [python]

[wkcn]

@universewill
I don't know. I think you could try to print the loss.