Skip to content

feature/parallel_gpu #7293

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 8 commits into from
Jan 10, 2018
Merged

feature/parallel_gpu #7293

merged 8 commits into from
Jan 10, 2018
+89 −71

Conversation

@tonyyang-svail
Copy link

@tonyyang-svail tonyyang-svail commented Jan 8, 2018 •
edited
Loading

A simple implementation parallel_do supporting multigpu

... | ParallelDo |	Split input |	Copy parameter to multiple GPUs -	Wait ||||	Forward on multiple GPUs -	Wait |	Merge output -	Wait | ... | ParallelGradDo |	Split output@grad -	Wait ||||	Backward on multiple GPUs -	Wait |	AllReduce parameters -	Wait | ...

TODO:
@tonyyang-svail tonyyang-svail requested a review from reyoung January 8, 2018 07:52
@chengduoZH
Copy link
Contributor

chengduoZH commented Jan 9, 2018 •
edited
Loading 

... | ParallelDo |	Split input |	Copy parameter to multiple GPUs -	Wait ||||	Forward on multiple GPUs -	Wait

It seems that the first Wait is unnecessary. The current GPU begins forward calculation doesn't depend on whether another GPU has completed the transmission of parameters or not.
@tonyyang-svail
Copy link
Author

@chengduoZH thanks for the suggestion. In the future implementation, we will have IO stream and computation stream. Then the waiting becomes necessary.

Let's target the correctness of this op first. :)
reyoung
reyoung approved these changes Jan 10, 2018
View reviewed changes
Copy link
Collaborator

@reyoung reyoung left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool
@tonyyang-svail tonyyang-svail changed the title [WIP] feature/parallel_gpu feature/parallel_gpu Jan 10, 2018
@tonyyang-svail tonyyang-svail merged commit 4bcc0b6 into PaddlePaddle:develop Jan 10, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

MultiDevices

3 participants

@tonyyang-svail @chengduoZH @reyoung