add VarDesc design #3835

jacquesqiao · 2017-09-03T16:06:52Z

add VarDesc: #3776

Superjomn · 2017-09-03T16:13:42Z

doc/design/var_desc.md

+create a variable with a tensor value.
+
+```python
+a = Variable("X", shape=[784, 10], data_type=INT32, value=0)


INT32 -> pd.int32

Superjomn · 2017-09-03T16:15:22Z

doc/design/var_desc.md

+## Background
+PaddlePaddle divides the description of neural network computation graph into two stages: compile time and runtime.
+
+The data structure to describe the compile time graph should be able to be serialized for distributing training. So we use proto message OpDesc to describe computation and VarDesc to describe data.


distributing training -> distributed training

reyoung · 2017-09-03T18:55:29Z

doc/design/var_desc.md

+ INT64 = 3;
+ FP16 = 4;
+ FP32 = 5;
+ DOUBLE = 6


Unify float names, please. Either FP16, FP32, FP64, or half, float, double. Do not mix them together.

wangkuiyi · 2017-09-03T22:00:06Z

doc/design/var_desc.md

+ }
+
+ Type element_type = 1;
+ repeated int dims = 2; // [UNK, UNK, 6000] is saved as [-1, -1, 6000]


[UNK, 640, 480] as [-1, 640, 480]

wangkuiyi · 2017-09-03T22:00:28Z

doc/design/var_desc.md

+ Type element_type = 1;
+ repeated int dims = 2; // [UNK, UNK, 6000] is saved as [-1, -1, 6000]
+ optional int lod_level [default=0] = 3;
+ repeated int32 int16_val = 4 [packed = true]; // INT16


LoDTensorDesc doesn't have values.

wangkuiyi · 2017-09-03T22:01:27Z

doc/design/var_desc.md

+ LOD_TENSOR = 6;
+ }
+
+ message Value {


VarDesc doesn't have value.

wangkuiyi · 2017-09-03T22:01:41Z

doc/design/var_desc.md

+ INT64 = 3;
+ FP16 = 4;
+ FP32 = 5;
+ DOUBLE = 6


DOUBEL = FP64

wangkuiyi · 2017-09-03T22:21:14Z

doc/design/var_desc.md

+There is a class `Variable` in python to help create Variable.
+
+```python
+class Variable(object):


The following example shows how a Variable is to be used in Python programs:

def flatten_size(X, num_flatten_dims): prod = 1 # of last num_flatten_dims for i in xrange(num_flatten_dims): prod = prod * X.dims[-i-1] return prod def layer.fc(X, output_size, num_flatten_dims): W = tensor(elem_type=FP32, dims=[flatten_size(X, num_flatten_dims), output_size]) b = tensor(elem_type=FP32, diims=[output_size]) y = operator.fc(X, W, b) return y x = var(dim=[-1, 640, 480]) y = layer.fc(x, output_size=100) paddle.train(y, ...) print(y)

import VarDesc import framework class Var(object): def __init__(self, name, dims, type): self._name = name self.op = None _var_desc = VarDesc(name=name, dims=dims, data_type=type) self._var = framework.CreateVar(_var_desc) def dims(self): return self._var.dims() def type(self): return self._var.type()

The following example shows how a Variable is to be used in Python programs:

import paddle as pd def flatten_size(X, num_flatten_dims): prod = 1 # of last num_flatten_dims for i in xrange(num_flatten_dims): prod = prod * X.dims[-i-1] return prod def layer.fc(X, output_size, num_flatten_dims): W = Var(type=FP32, dims=[flatten_size(X, num_flatten_dims), output_size]) b = Var(type=FP32, dims=[output_size]) out = Var(type=FP32) y = operator.fc(X, W, b, output=out) # fc will put fc op input into out pd.InferShape(y) return out x = var(dim=[-1, 640, 480]) y = layer.fc(x, output_size=100) z = layer.fc(y, output_size=200) paddle.train(z, ...) print(y)

@wangkuiyi @reyoung

几点达成一致：

Var是python端的一个class。封装了一个cpp端的VarDesc。

Eval(targets=[]) targets是是一个Var数组。

Var中需要保存生成这个Var的Op。

多个Var可以同名，但是包含不同的Op，这样他们的内存是共同的，但是追踪依赖的时候会有区别。

reyoung · 2017-09-03T23:46:12Z

doc/design/var_desc.md

+or create a Variable with a string value
+
+```python
+a = Variable("X", data_type=pd.STRING, value="aa")


If Variable only contains the VarDesc*, we cannot implement Block.eval(targets=[]). Only use Variable cannot specify what operators should be run to get that Variable since Variable can be written by many operators.

For example, an SGD operator reads the weight tensor and gradient tensor of one parameter and writes the weight tensor. The same weight tensor is the input and the output of SGD operator. That weight tensor is also written by Load operator or Random operator. If the user specifies Block.eval(weight), what operators should be run?

So we should add a field to identify which operator generates that Variable. The implementation could be

class Var(object): def __init__(self): self.var_desc = ... self.op = ...

So if we assume the Block is a linear list of operators, the Block.eval could be

class Block(object): def __init__(self): self.ops = [] # a list of operators def eval(block, targets=[]): last_op_idx = get_last_op_in_block(block, targets) needed_var_names = set(get_var_names(targets)) ops = self.op[0: last_op_idx+1] sub_block = [] for op in reverse(ops): if any of op.outputs in needed_var_names: needed_var_names.extends(op.inputs) sub_block.append(op) sub_block = reverse(sub_block) sub_block.run()

helinwang · 2017-09-04T21:56:43Z

doc/design/var_desc.md

+
+```proto
+message VarDesc {
+ required string name = 1;


Do we need to use required field? https://stackoverflow.com/a/31814967/852385

good point! proto3 even remove required.

But to be compatible with the current code, I want to use required in this PR and create another pr to change all the proto at once.

I see, thanks!

QiJune · 2017-09-05T02:30:45Z

doc/design/var_desc.md

+1. Computation graph should be able to be saved to a file.
+1. In distributed training, the graph will be serialized and send to multiple workers.
+
+The computation graph is constructed by Data Node and Operation Node. The concept to represent them is in the table below.


What are the Node and Edge of computation graph?

data and operator are both Node, the Edge is the relationship between data and operator: input/output relation.

reyoung · 2017-09-06T20:27:13Z

doc/design/var_desc.md

+
+PaddlePaddle use proto message to describe compile time graph for
+
+1. Computation graph should be able to be saved to a file.


be saved to a file --> to be serialized

reyoung · 2017-09-06T20:29:06Z

doc/design/var_desc.md

+In Python API, layer will take Variable as Input, and return Variable as Output. There should be a class `Variable` in python to help create and manage Variable.
+
+```python
+image = Variable(dims=[-1, 640, 480])


-1 is not good for user. Maybe UNK or BatchSize here.

reyoung

I am not sure whether the code in this design doc is runnable. It seems many details should be considered when implementing.

Just LGTM now, but this documentation should be in a flux.

Superjomn · 2017-09-06T22:41:04Z

doc/design/var_desc.md

+ if initializer is not None:
+ AddInitialOperator(self, initializer)
+
+ def dims(self):


shape ? shape of a tensor, not dims

Superjomn · 2017-09-06T22:41:56Z

doc/design/var_desc.md

+ def dims(self):
+ return self._var.dims()
+
+ def data_type(self):


type in __init__, data_type here, need a unified name for data type

Superjomn · 2017-09-06T22:43:08Z

doc/design/var_desc.md

+# add an initialize Operator to block to init this Variable
+
+class Variable(object):
+ def __init__(self, name, dims, type, initializer):


make name = None as default? no name is passed in the demo below.

Superjomn · 2017-09-06T22:43:58Z

doc/design/var_desc.md

+
+class Variable(object):
+ def __init__(self, name, dims, type, initializer):
+ self._block = get_default_block()


block and name do not need to be a protected member, make the public is ok.

Superjomn · 2017-09-06T22:45:59Z

doc/design/var_desc.md

+ return prod
+
+def layer.fc(X, output_size, num_flatten_dims):
+ W = Variable(pd.random_uniform(), type=FP32, dims=[flatten_size(X, num_flatten_dims), output_size])


FP32 is a strange name, what is it mean?

use lower-wise and full name is ok, for example, type=pd.float32 is more clear.

Superjomn · 2017-09-06T22:47:46Z

doc/design/var_desc.md

+ pd.InferShape(y)
+ return out
+
+x = Variable(dims=[-1, 640, 480])


-1 -> None, None is more clear to be a placeholder here, -1 looks vague, if -1 is ok, what do -2, -200 mean here

Superjomn · 2017-09-06T22:48:07Z

doc/design/var_desc.md

+y = layer.fc(x, output_size=100)
+z = layer.fc(y, output_size=200)
+
+paddle.eval(targets=[z], ...)


add var description design

633fcc9

jacquesqiao requested review from Superjomn and wangkuiyi September 3, 2017 16:07

jacquesqiao mentioned this pull request Sep 3, 2017

Unify Python and C++ InferShape for VarDesc and Tensor. #3776

Closed

Superjomn reviewed Sep 3, 2017

View reviewed changes

update python variable define

842daac

jacquesqiao requested a review from reyoung September 3, 2017 17:43

reyoung reviewed Sep 3, 2017

View reviewed changes

wangkuiyi reviewed Sep 3, 2017

View reviewed changes

reyoung reviewed Sep 3, 2017

View reviewed changes

jacquesqiao added 5 commits September 4, 2017 08:45

update vardesc

3b29da3

typo

2695d96

complete the demo code

87ee694

update the design of variable

2ea6f47

typo

2666d66

helinwang reviewed Sep 4, 2017

View reviewed changes

QiJune reviewed Sep 5, 2017

View reviewed changes

jacquesqiao added 3 commits September 4, 2017 22:10

use block in demo code

d1fe875

typo

9b36536

move DataType outof LoDTensorDesc

a7704b6

jacquesqiao mentioned this pull request Sep 6, 2017

add var desc proto #3933

Merged

reyoung reviewed Sep 6, 2017

View reviewed changes

reyoung approved these changes Sep 6, 2017

View reviewed changes

Superjomn reviewed Sep 6, 2017

View reviewed changes

jacquesqiao changed the title ~~add var description design~~ add VarDesc design Sep 7, 2017

jacquesqiao merged commit 45cca9f into PaddlePaddle:develop Sep 7, 2017

heavengate pushed a commit to heavengate/Paddle that referenced this pull request Aug 16, 2021

update weights path, test=document_fix (PaddlePaddle#3835)

c865c88


		PaddlePaddle use proto message to describe compile time graph for

		1. Computation graph should be able to be saved to a file.

add VarDesc design #3835

add VarDesc design #3835

Uh oh!

Conversation

jacquesqiao commented Sep 3, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wangkuiyi Sep 3, 2017 • edited by jacquesqiao Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

jacquesqiao Sep 3, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

jacquesqiao Sep 3, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

reyoung Sep 3, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

reyoung left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Labels

6 participants

jacquesqiao commented Sep 3, 2017 •

edited

Loading

wangkuiyi Sep 3, 2017 •

edited by jacquesqiao

Loading

jacquesqiao Sep 3, 2017 •

edited

Loading

jacquesqiao Sep 3, 2017 •

edited

Loading

reyoung Sep 3, 2017 •

edited

Loading