change layernorm code to pytorch's native layer norm

[dhpollack]

The current code basically recreates pytorch's native LayerNorm code. The only difference is that the default eps in the pytorch function is 1e-5 instead of 1e-12. PyTorch's native version is optimized for cudnn so it should be faster than this version.

[codecov-io]

Codecov Report

Merging #1089 into master will increase coverage by 0.04%.
The diff coverage is n/a.

@@ Coverage Diff @@ ## master #1089 +/- ## ========================================== + Coverage 79.61% 79.66% +0.04%  ========================================== Files 42 42 Lines 6898 6898 ========================================== + Hits 5492 5495 +3  + Misses 1406 1403 -3

Impacted Files	Coverage Δ
pytorch_transformers/modeling_bert.py	87.98% <ø> (ø)	⬆️
pytorch_transformers/file_utils.py	73.94% <0%> (+2.11%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e00b4ff...e13465f. Read the comment docs.

[honnibal]

I think your PR misses the point though? The models need to be 100% accurate reproductions of the Tensorflow code, right down to differences in eps values. Otherwise if you run the activations and get different results, you don't know whether there's a bug. You also can't reason about different results, and whether they matter.

[dhpollack]

@honnibal but looking at the code, every call of BertLayerNorm explicitly sets the eps, thus the actual values used in the BERT models does not change. Only the default value, but this default value is never used. Additionally, if APEX is available then you use FusedLayerNorm, which uses the same default eps of 1e-5 as the pytorch default LayerNorm. So you already have an inconsistency, but you solved this by explicitly setting the eps every time you use the layer.

[honnibal]

Oh right! Fair point, sorry.

[thomwolf]

Yes @dhpollack is right we can switch to PyTorch official LayerNorm.

What made me reimplement the LayerNorm when I was working on Bert last year was actually a typo in PyTorch's doc formula for computing the LayerNorm which indicated, at that time, that the epsilon was added to the square root of the variance instead of being added to the variance it-self. This typo is now corrected in pytorch/pytorch#8545.

Everything is right and we can drop these custom LayerNorms.

[julien-c]

Are we sure the names of the parameters are the same though? (eps vs. variance_epsilon)