Skip to content
This repository was archived by the owner on Nov 17, 2023. It is now read-only.
This repository was archived by the owner on Nov 17, 2023. It is now read-only.

BatchNorm running variables are not properly synchronized on CPU #18610

Description

@anko-intel

Description

During investigation of #14357 I have found that running_mean and running_var variables are sometimes not yet calculated when converted to NumPy.

Error Message

When runing the python test from #14357 (comment) sometimes there is information that consecutive reads of the same variable differes, which means that backward pass is calculated after the first read:
var1 = layer2.running_var.data().asnumpy()
It shouldn't happen as asnumpy() function should force all needed calculations to get running_var values.
I such case the test shows a message similar to the one below:

seed: 331752807
cpu(0) - DIFF in running_var reads:
   0 : [3.9974644 3.9977627 3.997775 ] 
   2 : [3.9962945 4.0025434 3.9987571]
cpu(0) [3.9962945 4.0025434 3.9987571] [10.001289  9.997483 10.002162]
gpu(0) [3.9962947 4.0025434 3.9987576] [10.001292  9.997481 10.002162]

Verified on master branch 9591436

To Reproduce

run python test from #14357 (comment)

What have you tried to solve it?

I already have the fix. I will put it to github soon.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions