Validation Accuracy of mlp.py and lenet.py.

[II-Matto]

The reference validation accuracy provided (https://github.com/dmlc/mxnet/tree/master/example/mnist) is 97.8% and 99% for mlp.py and lenet.py repectively, but what I obtained is 97.16% and 98.97%. I was wondering what may cause the small gap between my results and the reference values.

BTW, I built mxnet with CUDA 7.0, mkl, and cuDNN R3 (release candidate).

[tqchen]

I guess it was due to recent change of default initializer to more conservative one uniform(0.01), was more aggressive before that causing divergence in some deeper archs.

@antinucleon Can you update the stats?

[antinucleon]

Fixed in #196

INFO:root:Batch [600]   Speed: 34028.16 samples/sec
INFO:root:Iteration[19] Train-accuracy=1.000000
INFO:root:Iteration[19] Time cost=1.764
INFO:root:Iteration[19] Validation-accuracy=0.990800

[II-Matto]

@tqchen @antinucleon Many thanks! One more question. How can I update my current install of mxnet? Can I just simply make and python setup.py install again with the newly cloned copy?

[antinucleon]

You can use python setup.py develop --user, then each time you type make clean; make, the python pkg will be auto updated

[II-Matto]

@antinucleon Since I already have mxnet installed, how should I manually update python pkg now with make clean before switching to python setup.py develop --user next time?

[tqchen]

You can simply install again, it should override the original files, as far as I understand

Tianqi

[II-Matto]

@tqchen Thanks. The original files should have been overridden (^_^). The lenet.py now gives validation accuracy of 99.10%. The mlp.py remains the same as 97.16% - I noticed this file was not modified in @antinucleon 's fix.

BTW, why changing from ReLU to tanh and increasing learning rate from 0.01 to 0.05 help boost the validation accuracy? What I understand is that larger learning rate indicates more aggressive search for a good solution.

After changing ReLU from tanh, mlp.py gives validation accuracy of 97.87%.

Sorry for my unfamiliarity with python resulting in my questions about updating mxnet. I have to be careful since I am working on a server used by many people. Again thanks for your help.

[tqchen]

Glad things works out for you! Thanks for using mxnet.

[tqchen]

With respect to learning rate, too large learning rate will results in divergence due to overshooting. So normally we need a moderate learning rate

[II-Matto]

@tqchen Thanks the team for creating mxnet :)