RuntimeWarning: invalid value encountered in divide

[cjfuture]

log20170604h20m24.txt
Uploading log20170604h20m24.txt…

when i run the vnet training, I met the warning in the epoch:
RuntimeWarning: invalid value encountered in divide
(self.gt[i, :] * self.union[i]) / ((self.union[i]) ** 2) - 2.0prob[i,1,:](self.intersection[i]) / (
/home/jamson/3D-caffe/VNet-master/pyLayer.py:65: RuntimeWarning: invalid value encountered in divide
(self.union[i]) ** 2))
but when i print the value of (self.uniion[i])**2, it is valid.
self.union[1]**2 1240996.0
self.union[0]**2 1020100.0
after this epoch, all the loss are 0. it seems that the diff of backpropogation in that epoch result in disruption of model.
is there something wrong? how can i handle this problem?
thanks a lot

[faustomilletari]

Apparently there is some Nan or something like that. Check the data that is fed to the network. Be sure that there is some voxels that are part of foreground. Fausto Milletarì Sent from my iPhone

…

On 4. Jun 2017, at 10:58, cjfuture ***@***.***> wrote: when i run the vnet training, I met the warning in the epoch: RuntimeWarning: invalid value encountered in divide (self.gt[i, :] * self.union[i]) / ((self.union[i]) ** 2) - 2.0prob[i,1,:](self.intersection[i]) / ( /home/jamson/3D-caffe/VNet-master/pyLayer.py:65: RuntimeWarning: invalid value encountered in divide (self.union[i]) ** 2)) but when i print the value of (self.uniion[i])**2, it is valid. self.union[1]**2 1240996.0 self.union[0]**2 1020100.0 after this epoch, all the loss are 0. it seems that the diff of backpropogation in that epoch result in disruption of model. is there something wrong? how can i handle this problem? thanks a lot — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[faustomilletari]

I don’t think so. one thing you can do is to edit pylayer.py and put an assertion that makes sure that every ground truth label volume has at least one white pixel. there have been issues where people didn’t pay attention at the resolution vs. pixel size parameters and ended up with volumes that were a very very small sub window of their original volumes. let’s say you have something having resolution 0.1 units per pixel (unit can be arbitrary: can be meters, cm, kilometers, inches etc..) and you have the original volume at pixel size of 100x100x100 if you now resample at 0.01 pixels per size and you want to have an input volume which is 128x128x128, we first resample your volume that was 100x100x100 to be 1000x1000x1000 and then crop a 128x128x128 patch from the center of it. you will have almost everything cut out from that patch. pay attention to this! Fausto

…

On Jun 6, 2017, at 2:36 AM, cjfuture ***@***.***> wrote: Thanks a lot. I am sure all the training samples and their gt have the foreground. Is there possibel the deformation of augmentation in DataManager.py cause this problem? how can i modify the code to ensure the training's maintenance even thought that error happened? thank you — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#41 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AMtsvjJPu0waBgxenxfURJM2Rkcds229ks5sBPN7gaJpZM4NvZUr>.

[happyzhouch]

@cjfuture Was the problem solved? Can you tell me how to solve it ?