Data Shuffling inconsistency

[pluskid]

While looking at the code for the io module, I found some inconsistency, maybe potential bugs we need to fix:

• It seems data shuffling for MNIST iterator is only done once when the iterator is created, at here. No re-shuffling is done after each reset following the end of an epoch. I'm not sure this is intended or a bug, as we have a unit test in the python binding (see here) that explicitly assert this behavior. I believe re-shuffling after each epoch is helpful to improve convergence (though MNIST might be too trivial to see a difference).
• It seems data shuffling for image record iterator is only done at least after one epoch, at here. I have not tried to understand every bit of the IO code yet, so please correct if I'm wrong. This means no matter we turned on the shuffle bit or not, the first round is always not shuffled. It could hurt the performance a lot if the dataset is originally sorted according to category. Although it seems unlikely that a typical image dataset will be sorted in that way, but we should fix this to avoid potential bug.

[tqchen]

For the general ImageRecordIter we are doing shuffling locally per block. A block is a batch of image we load in(usually larger than batch size) as a unit in the data loader, and the shuffling going on for every block. Including the first epoch. So that logic https://github.com/dmlc/mxnet/blob/master/src/io/iter_image_recordio.cc#L350 is doing shuffling after one block is loaded in, and the iterator returns the images in the shuffled order.

It is harder to do a global shuffle, mainly due to cost random seek. So in general, what we can do is divide the data into blocks, load the blocks in random order, and shuffle the data within the block.

Current ImageRecordIter implemented within block shuffling, but did not yet implement random load of blocks

[pluskid]

Thanks for clarification. Block-wise in memory shuffling totally makes sense.