This repository was archived by the owner on Jan 15, 2024. It is now read-only.
[Bugfix] TransformerXL LayerNorm eps and XLNet pretrained model config #1005
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@ ## master #1005 +/- ## ========================================== + Coverage 89.93% 89.95% +0.01% ========================================== Files 67 67 Lines 6340 6340 ========================================== + Hits 5702 5703 +1 + Misses 638 637 -1
|
leezu changed the title 
Member
| Job PR-1005/1 is complete. |
Member
| Job PR-1005/2 is complete. |
leezu force-pushed the fixtransformerxllayernormeps branch from 01de29d to d3b9e1d Compare Member
| Job PR-1005/3 is complete. |
np.unravel_index argument names changed in v1.16.0. scripts/language_model/conversion_utils/compare_transformerxl_pytorch_gluon_model.py requires v1.16.0 due to making use of the updated argument names. This was previously not correctly declared.
leezu force-pushed the fixtransformerxllayernormeps branch from d3b9e1d to 2de2a99 Compare Member
| Job PR-1005/4 is complete. |
szha approved these changes Nov 28, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
3 participants
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
Checklist
Essentials
Changes
1e-12to1e-5Comments
We previously asserted that our implementation computes the same result as the pytorch-transformers implementation. However, that implementation was wrong up until 1.2.0 release in which their implementation was fixed unintentionally / by accident in huggingface/transformers#1089.
This also updates the comparison scripts which broke to API breakage in the packages that we compare to (renaming / reordering of arguments etc.).
Also updates
gelutoapprox_geluin the XLNet pretrained model configuration, which should have been done in #988.cc @dmlc/gluon-nlp-team
@zburning FYI