Design Doc for PaddlePaddle TrainingJob k8s CRD #563

Omkaka09 · 2018-01-23T08:22:30Z

hi, @typhoonzero @Yancey1989 , I'd like you to review this issue. In this issue. This is the design doc for paddlepaddle cluster job.

Yancey0623

Sorry for the later review, and how about writing a design doc to describe how to implement a TrainingJobControler and how to implement the simple state machine?

Yancey0623 · 2018-01-25T07:00:36Z

go/pkg/trainingjober/trainingjober.go

@@ -0,0 +1,62 @@
+// trainingjober is a package for managing a TrainingJob


Sorry I don't think trainingjober is a suitable name, we need a Controller to manager the TrainingJob, so how about rename TrainingJober to TrainingJobController ?

Yancey0623 · 2018-01-25T07:02:12Z

go/pkg/trainingjober/trainingjober.go

+type trainingJobEvent struct {
+// pet is the TrainingJobEventType of TrainingJob
+pet trainingJobEventType
+// The job transfer the information fo job


The job transfer the information fo job , Do you mean job is the reference to TrainingJob ?

Yancey0623 · 2018-01-25T07:03:47Z

go/pkg/trainingjober/trainingjober.go

+job *v1alpha1.TrainingJob
+}
+
+// TrainingJober is to manager a specific TrainingJob


I think we need a TrainingjobController to manager a TrainingJob.

We have a controller which @m3ngyang committed in another PR #565. We use controller to manager all TrainingJob of a specific cluster and use TrainingJober to manager a specific TrainingJob.

Omkaka09 · 2018-01-29T03:30:45Z

@Yancey1989 ok~ I will write a design doc to describe the definition of Controller、TrainingJober、TrainingJob and the state machine of a TrainingJob.

typhoonzero · 2018-02-07T03:38:57Z

go/pkg/design_doc.md

@@ -0,0 +1,172 @@
+# Design Doc for PaddlePaddle TrainingJob k8s CRD


Design docs must put under docs/design

Thanks for your advice, I will put it under docs/design.

typhoonzero · 2018-02-07T06:10:28Z

doc/design/traingjob_crd_design.md

+The whole life cycle of TrainingJob is managed through the two layer control of Controller and TrainingJober. As 
+shown in the following figure:
+
+![image](https://github.com/qizheng09/figure/blob/master/paddlecloud/overall_design.png?raw=true)


Put the image in this repo. And, if you are using tools like "OmniGraffle" or something like that, you should put the source file in the repo too, so we can modify the graph later for updates.

This figure should have some explanation of how it runs and what components it consists of.

typhoonzero · 2018-02-07T06:12:55Z

doc/design/traingjob_crd_design.md

+
+Here is an example yaml for a PaddlePaddle cluster job.
+
+```yaml


Can you please try to merge this to current design: https://github.com/PaddlePaddle/cloud/blob/develop/doc/edl/README.md, so we only have one total design here.

typhoonzero · 2018-02-07T06:14:25Z

doc/design/traingjob_crd_design.md

+ image: "paddlepaddle/paddlecloud-job"
+ # base port of pserver
+ port: 7164
+ # ports num default 1


These comments seem not necessary, we can explain the details of the yaml file in another markdown file.

typhoonzero · 2018-02-07T06:15:10Z

doc/design/traingjob_crd_design.md

+
+The example will create a PaddlePaddle cluster job with a master, 2 pserver and 2 trainer.
+
+### TrainingJober


TrainingJober is a confusing name.

m3ngyang · 2018-02-08T08:41:15Z

too many commits for one pr, you'd better rebase and merge the commits.

typhoonzero · 2018-02-11T02:24:00Z

doc/edl/README.md

 A PaddlePaddle training job contains several trainer instances,
-several parameter server instances, and one master instance. We would
-like to automatically scale the number of training instances and the
+several parameter server instances, and one master instance. 


Background section should only tell us about "Why we need to do this", designs and thoughts need to put to bottom sections.

typhoonzero · 2018-02-11T02:26:16Z

doc/edl/README.md

+- Create and delete TrainingJob
+- TrainingJob life cycle management
+
+**The goals of Autoscale Controller including:**


We can call the controller "controller", and the Autoscaler is not a Kubernetes controller actually, It's just a goroutine running at the background to do autoscaling.

typhoonzero · 2018-02-11T02:27:31Z

doc/edl/README.md


 ```yaml
-apiVersion: paddlepaddle.org/v1
+apiVersion: paddlepaddle.org/v1alpha1


We can use paddlepaddle.org/v1 so some code won't need big changes.

typhoonzero · 2018-02-11T02:32:41Z

doc/edl/README.md

- v1.9.
+![](state_machine.jpg)

-We will support TPR first, because some of our clusters is using


We may still need this line somewhere.

Omkaka09 · 2018-02-12T14:51:48Z

@typhoonzero , I fix the comments you mentioned above. How do you think about this commit?

…ngqi_readme

Update EDL design doc

m3ngyang · 2018-02-13T11:46:50Z

doc/edl/README.md

+// TrainingJobUpdater is to manager a specific TrainingJob
+type TrainingJobUpdater struct {
+// job is the job the TrainingJobUpdater manager.
+job *v1.TrainingJob


code format problem

m3ngyang · 2018-02-19T03:21:01Z

doc/edl/README.md


 Kubrenetes provide [CustomResourceDefinition (CRD)](https://kubernetes.io/docs/concepts/api-extension/custom-resources/#custom-resources)and [custom controller](https://kubernetes.io/docs/concepts/api-extension/custom-resources/#custom-controllers)
-we can use these feature to develop EDL controller, so it can flexiblly running in our
+we can use these feature to develop EDL controller, so it can flexibly running in our


running -> run

typhoonzero

LGTM++

Omkaka09 added 2 commits January 23, 2018 16:11

add trainingjober struct

e08fdaf

code format

6549582

Yancey0623 reviewed Jan 25, 2018

View reviewed changes

Omkaka09 added 3 commits February 5, 2018 15:10

Merge remote-tracking branch 'upstream/develop' into develop

11dcb44

add design doc

534361e

remove code

6c27bfa

Omkaka09 changed the title ~~add trainingjober struct~~ Design Doc for PaddlePaddle TrainingJob k8s CRD Feb 5, 2018

typhoonzero reviewed Feb 7, 2018

View reviewed changes

Omkaka09 added 3 commits February 7, 2018 11:46

Merge remote-tracking branch 'upstream/develop' into develop

cf301b2

move design doc to doc/design

299e43b

move design doc to doc/design

e76fda6

typhoonzero reviewed Feb 7, 2018

View reviewed changes

Omkaka09 added 2 commits February 8, 2018 16:34

merge design to edl design

fdce3d2

fix pic url

25e91f6

Omkaka09 added 3 commits February 8, 2018 16:47

fix pic url

56f9d65

fix pic url

f5bd77d

fix pic url

d4c43c2

typhoonzero reviewed Feb 11, 2018

View reviewed changes

fix design doc

f6e5467

Omkaka09 and others added 4 commits February 12, 2018 23:06

adjust header

9c313de

Merge branch 'develop' of https://github.com/qizheng09/cloud into zhe…

9ccc3a6

…ngqi_readme

update edl readme

590d0e5

Merge pull request #1 from typhoonzero/zhengqi_readme

60f010a

Update EDL design doc

m3ngyang reviewed Feb 13, 2018

View reviewed changes

fix code format and typo

f8f1467

m3ngyang reviewed Feb 19, 2018

View reviewed changes

fix typo and change colors of pictures

0170d38

typhoonzero approved these changes Feb 23, 2018

View reviewed changes

typhoonzero merged commit 002506f into PaddlePaddle:develop Feb 23, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Design Doc for PaddlePaddle TrainingJob k8s CRD #563

Design Doc for PaddlePaddle TrainingJob k8s CRD #563

Uh oh!

Omkaka09 commented Jan 23, 2018 •

edited

Loading

Yancey0623 left a comment

Yancey0623 Jan 25, 2018

Yancey0623 Jan 25, 2018

Yancey0623 Jan 25, 2018

Omkaka09 Jan 29, 2018

Omkaka09 commented Jan 29, 2018 •

edited

Loading

typhoonzero Feb 7, 2018

Omkaka09 Feb 7, 2018

typhoonzero Feb 7, 2018

typhoonzero Feb 7, 2018

typhoonzero Feb 7, 2018

typhoonzero Feb 7, 2018

typhoonzero Feb 7, 2018

m3ngyang commented Feb 8, 2018

typhoonzero Feb 11, 2018

typhoonzero Feb 11, 2018

typhoonzero Feb 11, 2018

typhoonzero Feb 11, 2018

Omkaka09 commented Feb 12, 2018

m3ngyang Feb 13, 2018

m3ngyang Feb 19, 2018

typhoonzero left a comment

Labels

4 participants

		@@ -0,0 +1,62 @@
		// trainingjober is a package for managing a TrainingJob

		@@ -0,0 +1,172 @@
		# Design Doc for PaddlePaddle TrainingJob k8s CRD


		Here is an example yaml for a PaddlePaddle cluster job.

		```yaml


		The example will create a PaddlePaddle cluster job with a master, 2 pserver and 2 trainer.

		### TrainingJober

Design Doc for PaddlePaddle TrainingJob k8s CRD #563

Design Doc for PaddlePaddle TrainingJob k8s CRD #563

Uh oh!

Conversation

Omkaka09 commented Jan 23, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Yancey0623 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Omkaka09 commented Jan 29, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

m3ngyang commented Feb 8, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Omkaka09 commented Feb 12, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

typhoonzero left a comment

Choose a reason for hiding this comment

Labels

4 participants

Omkaka09 commented Jan 23, 2018 •

edited

Loading

Omkaka09 commented Jan 29, 2018 •

edited

Loading