[Dy2stat]support pure fp16 for dy2stat #36944
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
| Thanks for your contribution! |
added 8 commits November 2, 2021 12:49
Aurelius84 reviewed Nov 11, 2021
| | ||
| def _in_pure_fp16_guard(): | ||
| tracer = _dygraph_tracer() | ||
| if tracer: |
Contributor
There was a problem hiding this comment.
Suggested change
| if tracer: | |
| return tracer and tracer._amp_level == core.AmpLevel.O2 |
Aurelius84 previously approved these changes Nov 22, 2021
| """ | ||
| Tests model decorated by `dygraph_to_static_output` in static mode. For users, the model is defined in dygraph mode and trained in static mode. | ||
| """ | ||
| with fluid.dygraph.guard(place): |
| dygraph_loss = self.train(to_static=False) | ||
| self.assertTrue( | ||
| np.allclose(static_loss, dygraph_loss), | ||
| msg="static_loss: {} \n dygraph_loss: {}".format(static_loss, |
Contributor
There was a problem hiding this comment.
确保CI稳定没有问题，若放开tol，需要加NOTE
Zjq9409 pushed a commit to Zjq9409/Paddle that referenced this pull request Dec 10, 2021
* run dy2stat pure fp16 in Linear model * no use self._pure_fp16_inputs * add test and fix Adam error in dy2stat pure fp16 training * use paddle.optimizer.Adam * run test in gpu * change test time for CI * enlarge atol for test_resnet_pure_fp16 * refine code and enlarge atol * make custom_white_list and custom_black_list take effect for AMP and pure fp16 * check tracer is not None * use default atol * change filter_size * change atol and add some NOTE
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
PR types
New features
PR changes
Others
Describe
本PR在
CastPureFp16Inputs函数中加入了特判逻辑：对于动转静中所调用的run_program op直接跳过，不再进行后续的cast操作。因为run_program op只有FP32 kernel，且我们在动转静pure fp16训练时在调用run_program op时已经将run_program op的输入cast为fp16类型，所以在CastPureFp16Inputs应该跳过对于run_program op的处理，避免将其输入再cast回fp32类型。1e-3，以确保单测通过。对于动转静pure fp16训练，已经在mnist和resnet网络上测试了网络的收敛性， 均可以正常收敛。paddle.amp.auto_cast接口中黑白名单(custom_white_list参数和custom_black_list参数)的设置。在此PR之前在动转静AMP训练中设置这两个参数并不会生效。