[Unified Checkpoint] Support sharding_comm_overlap #9392
Conversation
| Thanks for your contribution! |
| self.optimizer = fleet.distributed_optimizer(self.optimizer) | ||
| | ||
| if self.args.enable_sharding_comm_overlap: | ||
| model.register_sharding_comm_overlap_hook(self.optimizer) |
There was a problem hiding this comment.
增加了一个判断条件，针对split_param开启了再打开
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@ ## develop #9392 +/- ## =========================================== - Coverage 52.96% 52.94% -0.02% =========================================== Files 676 676 Lines 107827 107836 +9 =========================================== - Hits 57109 57099 -10 - Misses 50718 50737 +19 ☔ View full report in Codecov by Sentry. |
lugimzzz force-pushed the develop branch 3 times, most recently from 3310514 to aecf9f1 Compare 
| model = self.model_wrapped | ||
| opt_state_dict = self.unified_checkpoint_handler.load_unified_optimizer( | ||
| model=self.model, | ||
| model=model, |
There was a problem hiding this comment.
emm，会不会 state_dict 的 name 前面又套了一层其他的东西
比如 model.model.embedding ?
paddlenlp/trainer/trainer.py Outdated
| opt_state_dict = None | ||
| else: | ||
| model = self.model | ||
| if hasattr(self.args, "enable_sharding_comm_overlap") and self.args.enable_sharding_comm_overlap: |
There was a problem hiding this comment.
emm，我觉得要不你搞一个 uc_with_pp_sharding_comm_overlap 之类的config，你内部单独用吧。
不和 enable_sharding_comm_overlap 搞在一起了。
There was a problem hiding this comment.
这样会更复杂，而且判断逻辑如果和training_args.py存在冲突或对不齐，就糟糕了。
|
|
* [Unified Checkpoint] Support sharding_comm_overlap (#9392)
3 participants
PR types
Function optimization
PR changes
Others
Description
Support sharding_comm_overlap.