Skip to content

[cherry-pick]Support FP16 in HybridParallel and Fix bugs in HybridOptimizer #36707

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 12 commits into from
Oct 26, 2021
Merged

[cherry-pick]Support FP16 in HybridParallel and Fix bugs in HybridOptimizer #36707

merged 12 commits into from
Oct 26, 2021

Conversation

@haohongxiang
Copy link
Contributor

@haohongxiang haohongxiang commented Oct 25, 2021

PR types

Others

PR changes

Others

Describe

[cherry-pick]Support FP16 in HybridParallel and Fix bugs in HybridOptimizer
haohongxiang and others added 8 commits October 25, 2021 20:18
@haohongxiang
fix bugs in HybridParallelClipGrad of hybrid_parallel_optimizer (#36237)
42308b8 
* fix bugs in HybridParallelClipGrad of hybrid_parallel_optimizer * update * update
@haohongxiang
fix bugs in mp_layers、pp_layers and HybridParallelClipGrad (#36144)
a27481e 
* fix calling bug of HybridParallelClipGrad * fix bugs of HybridParallelClipGrad * add unittest of pp with HybridParallelClipGrad * fix bugs in mp_layers.py * update * fix bugs in pp_layers.py * update
@ForFishes @haohongxiang
[HybridParallel]Rebuild code for pipeline (#36396)
30c6919 
* add no_sync for parameters sync * add pipeline for moe
@haohongxiang
[HybridParallel]Support fp16 in dygraph hybrid parallel (#36420)
7f22db3 
* [HybridParallel]Support fp16 in dygraph hybrid parallel * update * update * update for recompute * add unittest of pp+fp16 * add unittest of recompute+fp16 * update * modify ut
@haohongxiang
modify ut of cond (#36475)
fe5ce21
@haohongxiang
fix bugs of ClipGradByGlobalNorm in HybridParallel (#36555)
50f43d0 
* fix bugs of ClipGradByGlobalNorm * add unittests * add unittests
@haohongxiang
[HybridParallel]fix bug of check_inf in fleet_base.py (#36651)
8e889c2 
* fix bug of check_inf * fix allreduce
@zhaoyinglia @haohongxiang
support ClipGradByGlobalNorm in sharding (#36012)
985397e 
* support ClipGradByGlobalNorm in sharding * support ClipGradByGlobalNorm in sharding * test=allcase
@paddle-bot-old
Copy link

paddle-bot-old bot commented Oct 25, 2021

Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.
zhiqiu
zhiqiu approved these changes Oct 26, 2021
View reviewed changes
Copy link
Contributor

@zhiqiu zhiqiu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
ForFishes
ForFishes approved these changes Oct 26, 2021
View reviewed changes
Copy link
Member

@ForFishes ForFishes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
@fuyinno4 fuyinno4 merged commit 5b357e0 into PaddlePaddle:release/2.2 Oct 26, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

5 participants

@haohongxiang @zhiqiu @ForFishes @fuyinno4 @zhaoyinglia