basic impl of operator

cmake/flags.cmake

@@ -101,7 +101,8 @@ set(COMMON_FLAGS
  -fPIC
  -fno-omit-frame-pointer
  -Wall
- -Wextra
+# -Wextra # Because map implementation of protobuf 3.1 has
+ # several warnings in GCC Wextra. Disable Wextra until we upgrade our protobuf library.
  -Werror
  -Wnon-virtual-dtor
  -Wdelete-non-virtual-dtor

doc/design/attribute.md

@@ -0,0 +1,112 @@
+# Design Doc: Operator Attributes
+
+## Background
+
+An operator could have attributes. For example, CosineOp could have a float typed attribute scale, which changes the output range from [-1,1] to [-scale,scale]. The default value of scale is `1.0`.
+
+Attributes is defined by a name and a type. An instance of an attribute has a value of that type.
+
+As part of the network description, attribute need to be serialized. So we need a protobuf message that describes an attribute, say `Attribute`.
+
+An operator could parse the Attribute and save them into its private data member.
+
+## Protobuf Implementation
+
+There are two frameworks implement `Attribute` concept in `protobuf`. They are [`caffe2`](https://github.com/caffe2/caffe2/blob/master/caffe2/proto/caffe2.proto#L98) and [`tensorflow`](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/framework/attr_value.proto#L16).
+
+* Caffe2 uses `proto2` syntax. It treats all attributes as a list, and each attribute contains a `name`. Each time caffe2 read an attribute is searching a variable in a list. It is slow if the number of attributes is large. Caffe2 also mark all field as `optional`. It doesn't ensure `one of` attribute value is set.
+* By using `proto3` syntax in tensorflow, the attribute implementation in tensorflow is using `map`, and `oneof` keywords. Looking up from attribute map in tensorflow is fast.
+
+Paddle is using `protobuf 3` as its dependency library. By simplify `tensorflow`'s implementation, Paddle's Attribute protobuf message schema could be
+
+```protobuf
+message Attribute {
+ message ListValue {
+ repeated int32 ints = 1;
+ repeated float floats = 2;
+ repeated string strings = 3;
+ }
+
+ oneof value {
+ ListValue list = 1;
+ int32 i = 2;
+ float f = 3;
+ string s = 4;
+ }
+}
+```
+
+In `OperatorDescription` message, there should be a field like this:
+
+```protobuf
+message OperatorDescription {
+ map<string, Attribute> attrs;
+}
+```
+
+## CPP implementation
+
+### AttributeReader
+
+In CPP, it should be a helper class for reading `map<string, Attribute>`. The reading methods in that helper class should accept a template parameter, which is the type of Attribute. That helper class we named `AttributeReader`.
+
+The interface of `AttributeReader` is like this:
+
+```cpp
+using AttributeMap = google::protobuf::Map<std::string, Attribute>;
+class AttributeReader {
+ public:
+ explicit AttributeReader(const AttributeMap& attrs) : attrs_(attrs) {}
+
+ // Get a plain type T attribute, which name is `name`
+ template <typename T>
+ T Get(const std::string& name) const;
+
+ // Get attribute with a array of type T, which name is `name`
+ template <typename T>
+ void GetArray(const std::string& name, std::vector<T>* vec) const;
+
+ // Is that `name` attribute with type T in map or not.
+ // T could be int, float, string and std::vector of them
+ template <typename T>
+ bool Contains(const std::string& name) const;
+
+ private:
+ const AttributeMap& attrs_;
+};
+```
+
+### Attribute in Operator
+
+Each operator parse and store its attribute into private member data when `InitializeAttribute`. That method will be invoked by `CreateOperator `. User can use `PADDLE_ENFORCE` to validate attribute. Also, use `Contains` method, user can set default value of attributes.
+
+```cpp
+class OperatorBase {
+ public:
+ virtual void InitializeAttribute(const AttributeReader& attrs) = 0;
+};
+
+class CosineOp : public OperatorBase {
+ public:
+ void InitializeAttribute(const AttributeReader& attrs) {
+ if (attrs.Contains<float>("scale")) {
+ scale_ = attrs.Get<float>("scale");
+ PADDLE_ENFORCE(scale_ > 0.0f, "Scale of consine op should be larger than 0.0");
+ }
+ }
+
+ private:
+ float scale_ {1.0};
+};
+```
+
+`InitializeAttribute` will be invoked by `CreateOperator`. Since `InitializeAttribute ` could throw an EnforceNotMet, a `unique_ptr` is used to make code exception-safe.
+
+```cpp
+std::unique_ptr<OperatorBase> CreateOperator(const OperatorDescription& desc) {
+ std::unique_ptr<OperatorBase> op(OperatorRegister.Create(
+ desc.type(), desc.inputs(), desc.outputs()));
+ op->InitializeAttribute(AttributeReader(desc.attrs()));
+ return std::move(op);
+}
+```

doc/design/operator.md

@@ -0,0 +1,149 @@
+# Operator Design
+
+`Operator` in PaddlePaddle mainly describes how to do an operation with `Variable`. It does not actually contain any data or state, but with the key to find these `Variables/State` from `Scope`.
+
+Op will get/update data/state from Scope when running. It should not change the information inside op so all it's `Run()` function should be `const`
+
+`Operator` has a template parameter `DeviceContext`. DeviceContext is used to specify on which device this op will run. Each Op will implement multi Op according to different Context.
+
+#### We design the Operator class with tree layer.
+
+1. OperatorBase
+1. Operator
+1. CustomOperator
+
+
+#### `OperatorBase`
+
+`OperatorBase` is the base class of an Operator, but without any specific type information. So that NetworkBase can treat all Ops as OperatorBase and do not need to consider the type or context of these ops. It can just call `Run()` of Op:
+
+```cpp
+class NetworkBase {
+public:
+ Error Run(Scope* scope, Context* context) {
+ for (auto& op : operators_) {
+ Error err = op->Run(scope, context);
+ if (!err.isOk()) {
+ return err;
+ }
+ }
+ return Error();
+ }
+
+private:
+ vector<unique_ptr<OperatorBase>> operators_;
+};
+```
+
+#### `Operator`
+
+`Operator` is the operator with DeviceContext information. It's the middle layer that handles the context converting and data prefetching work.
+
+It has two Run() function.
+
+1. `Error Run(Scope* scope, Context* context) const final`
+
+This Run() is derived from OperatorBase, It's used to convert BaseContext to CPUContext or GPUContext and get Variables from scope and then passed them to the next Run()
+
+2. `Error Run(std::vector<Variable*>& inputs, std::vector<Variable*>& outputs, DeviceContext* context) const overrid`
+
+This Run() should be derived by Customer Operator class and put there calculate logic in the function. It has all the Variable and Context with right type to run.
+
+#### `CustomOperator`
+
+`CustomOperator` is the operator that should be implemented such as FcLayer. They are derived from the second `Operator` with certain `DeviceContext`, they know where they are running and how to do the right operation.
+
+
+The following is the pseudocode for these three layers.
+
+
+```cpp
+#pragma once
+
+namespace paddle {
+namespace framework {
+
+// OperatorBase provide base element of an Operator without any template.
+class OperatorBase {
+public:
+ explicit OperatorBase(const OpDesc& desc);
+ virtual ~OperatorBase() {}
+
+ // initialize Attributes of this OP from proto message desc.attrs()
+ // you should derive this function to init the attr you need in OP.
+ virtual Error InitializeAttributes(const AttrbuteMap& attrs) = 0;
+ virtual Error Run(Scope* scope, Context* context) const = 0;
+
+private:
+ std::string type_;
+ std::vector<std::string> inputs_;
+ std::vector<std::string> outputs_;
+};
+
+// Operator is the class your should derive when implement a new Operator.
+template <typename DeviceContext>
+class Operator : public OperatorBase {
+public:
+ explicit Operator(const OpDesc& desc): OperatorBase(desc) {}
+
+ // This function will get all input and output Vars from scope and ten call
+ // Run(std::vector<Variable> inputs, std::vector<Variable> outputs, T* context)
+ Error Run(Scope* scope, Context* context) const final {
+ DeviceContext* dev_context = dynamic_cast<DeviceContext*>(context);
+ if (dev_context == nullptr) {
+ return Error("dynamic_cast devContext failed!");
+ }
+
+ std::vector<Variable*> input_vars;
+ std::vector<Variable*> output_vars;
+
+ input_vars.reserve(inputs_.size());
+ for(auto& input: inputs_) {
+ input_vars.push_back(scope->getOrCreateVariable(input));
+ }
+ output_vars.reserve(outputs_.size());
+ for(auto& input: outputs_) {
+ output_vars.push_back(scope->getOrCreateVariable(input));
+ }
+
+ return Run(input_vars, output_vars, dev_context);
+ }
+
+ // when implement an Op, your should implement this function.
+ virtual Error Run(std::vector<Variable*>& inputs,
+ std::vector<Variable*>& outputs,
+ DeviceContext* context) const = 0;
+};
+
+
+// Sample Operator implement. Show how to implement a Cosine Operator.
+template <typename DeviceContext>
+class CosineOp final : public Operator<DeviceContext> {
+public:
+ CosineOp(const OpDesc& desc):
+ Operator<DeviceContext>(desc) {};
+
+ // init attrs that are needed by this Operator, check the legality here.
+ Error InitializeAttributes(const AttrbuteMap& attrs) {
+ scale_ = attrs.get<float>("scale");
+ if (scale_ <= 0.0) {
+ return Error("scale of CosineOp must be larger than 0.0, get %f", scale_);
+ }
+ return Error();
+ }
+
+ // Add the actual calculate logic in this function.
+ Error Run(std::vector<Variable*>& inputs,
+ std::vector<Variable*>& outputs,
+ DeviceContext* context) const override {
+ // TODO(to be implement)
+ return Error();
+ }
+
+private:
+ float scale_;
+};
+
+} // namespace framework
+} // namespace paddle
+```

paddle/CMakeLists.txt

@@ -15,6 +15,7 @@ if(Boost_FOUND)
  add_subdirectory(memory)
  add_subdirectory(platform)
  add_subdirectory(framework)
+ add_subdirectory(operators)
 endif()
 
 if(WITH_C_API)

paddle/framework/CMakeLists.txt

@@ -1,7 +1,10 @@
 # ddim lib
+proto_library(attribute SRCS attribute.proto)
 cc_library(ddim SRCS ddim.cc)
+cc_library(operator SRCS operator.cc DEPS attribute)
 cc_test(ddim_test SRCS ddim_test.cc DEPS ddim)
 nv_test(dim_test SRCS dim_test.cu DEPS ddim)
 cc_test(variable_test SRCS variable_test.cc)
 cc_test(scope_test SRCS scope_test.cc)
 cc_test(enforce_test SRCS enforce_test.cc)
+cc_test(attribute_reader_test SRCS attribute_reader_test.cc DEPS attribute protobuf)

paddle/framework/attribute.proto

@@ -0,0 +1,23 @@
+syntax="proto3";
+package paddle.framework;
+
+// Attribute for OperatorDescription.
+// It represent a variant type of value.
+// The int, float, string and repeated of them are supported.
+message Attribute {
+ message ListValue {
+ repeated int32 ints = 1;
+ repeated float floats = 2;
+ repeated string strings = 3;
+ }
+
+ oneof value {
+ // Since proto3 not support repeated filed in `oneof`, we use
+ // ListValue to support repeated.
+ ListValue list = 1;
+
+ int32 i = 2;
+ float f = 3;
+ string s = 4;
+ }
+}