You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Nov 17, 2023. It is now read-only.
There is a problem with a custom BERT model training with the later version of MXNet 1.5.0 (observed with cu90).
mlm_loss stops around 7.3 and nsp_acc stopps around 54.
mxnet-cu90 version which is older than 1.5.0b20190425 does not have this issue.
1.5.0b20190426 onward has this issue.
Environment info (Required)
Amazon SageMaker Notebook (ml.p3.16xlarge)
CUDA version: 9.0
dmlc/gluon-nlp#690
Description
There is a problem with a custom BERT model training with the later version of MXNet 1.5.0 (observed with cu90).
mlm_loss stops around 7.3 and nsp_acc stopps around 54.
mxnet-cu90 version which is older than 1.5.0b20190425 does not have this issue.
1.5.0b20190426 onward has this issue.
Environment info (Required)
Amazon SageMaker Notebook (ml.p3.16xlarge)
CUDA version: 9.0
Package used (Python/R/Scala/Julia):
Python 3.6