【Hackathon No.67】remove reference to operator.h in phi [part 1] #50624

RedContritio · 2023-02-17T14:17:03Z

PR types

Others

PR changes

Others

Describe

由于此前 <fluid/framework/operator.h> 和 <fluid/framework/phi_utils.h> 存在循环依赖，且 phi_utils.h 存在大量对 fluild 的依赖，因此此前的很多解耦合功能只是将引用放进了 phi_utils.h 中，没有真实解决依赖问题。

直接依赖的地方分别是 paddle/phi/kernels/funcs/math_function.h 和 paddle/phi/kernels/funcs/blas/blas.h。

本 pr 解决 math_functions.h。

先将 operator.h 的所有依赖展开，然后将展开项中的 phi_utils.h 展开，得到列表如下：

// operator.h #include "paddle/fluid/framework/attribute.h" #include "paddle/fluid/framework/block_desc.h" #include "paddle/fluid/framework/convert_utils.h" #include "paddle/fluid/framework/lod_tensor.h" #include "paddle/fluid/framework/op_info.h" #include "paddle/fluid/framework/op_kernel_type.h" #include "paddle/fluid/framework/phi_utils.h" #include "paddle/fluid/framework/scope.h" #include "paddle/fluid/framework/selected_rows_utils.h" #include "paddle/fluid/framework/unused_var_check.h" #include "paddle/fluid/memory/malloc.h" #include "paddle/phi/core/compat/arg_map_context.h" #include "paddle/phi/core/compat/op_utils.h" #include "paddle/phi/core/kernel_context.h" #include "paddle/phi/core/kernel_factory.h" #include "paddle/utils/flat_hash_map.h" // phi_utils.h #include "paddle/fluid/framework/framework.pb.h" #include "paddle/fluid/framework/op_kernel_type.h" #include "paddle/fluid/framework/operator.h" // 循环依赖 #include "paddle/fluid/framework/tensor.h" #include "paddle/fluid/framework/variable.h" #include "paddle/fluid/platform/macros.h" #include "paddle/fluid/platform/place.h" #include "paddle/phi/api/lib/utils/tensor_utils.h" #include "paddle/phi/common/backend.h" #include "paddle/phi/core/compat/arg_map_context.h" #include "paddle/phi/core/kernel_factory.h" #include "paddle/utils/flat_hash_map.h" #include "paddle/utils/small_vector.h" #ifdef PADDLE_WITH_XPU #include "paddle/fluid/platform/device/xpu/xpu_op_list.h" #endif #ifdef PADDLE_WITH_CUSTOM_DEVICE #include "paddle/phi/backends/custom/custom_device_op_list.h" #endif

从列表中删除 operator.h，取交集，随后逐个将能替代的全部替换成 phi 内的头文件即可。

（也就是说，每次引入 operator.h，都会引入上面这个头文件集合）

展开 operator.h 引入的依赖，并放到实际使用的地方
添加 VisitPlace 到 phi/core/utils/visit_place.h 中
替换掉改动文件中的 fluid/framework.proto 中定义的类型及方法，以避免引入额外的 fluid 头文件

paddle-bot · 2023-02-17T14:17:08Z

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

RedContritio · 2023-02-17T14:26:07Z

这里有一个类型引用 paddle::framework::ExecutionContext，该类型仅声明在 fluid/framework/operator.h 中。

Paddle/paddle/phi/kernels/funcs/blas/blas.h

Line 17 in e6af9bd

#include "paddle/fluid/framework/operator.h"

Paddle/paddle/phi/kernels/funcs/blas/blas.h

Lines 582 to 587 in e6af9bd

     template <typename DeviceContext, typename T>  
   inline BlasT<DeviceContext, T> GetBlas(  
   const paddle::framework::ExecutionContext& exe_ctx) {  
   return BlasT<DeviceContext, T>(  
   exe_ctx.template device_context<DeviceContext>());  
   }  
 

我是否应该创建 phi/core/operator.h （及 phi/core/operator.cc）以定义该类型，并在原有的 fluid/framework/operator.h 中使用 using 来导出 phi::ExecutionContext 使成为 paddle::framework::ExecutionContext。
或者考虑将 operator.h （及 .cc）整个移动到 phi 中？

@YuanRisheng

RedContritio · 2023-02-25T10:26:20Z

@YuanRisheng 辛苦先 review 这一部分的，另一部分对这部分有依赖，且改动有点多

paddle/phi/kernels/funcs/math_function.cc

paddle/phi/kernels/funcs/math_function.h

paddle/phi/kernels/funcs/math_function_impl.h

paddle/phi/kernels/impl/activation_grad_impl.h

paddle/phi/kernels/set_value_grad_kernel.h

paddle/fluid/framework/data_layout_transform.h

luotao1 · 2023-02-27T03:24:43Z

辛苦先 review 这一部分的，另一部分对这部分有依赖，且改动有点多

好的，可以等这个 PR 合入后尽快提交下一个 PR，奖金以最后完成的 PR 为准。

luotao1 · 2023-03-02T08:27:34Z

需要按照黑客松流程在总issue下回复状态：

【PaddlePaddle Hackathon 第四期】任务总览 #50629

paddle/phi/kernels/funcs/math_function.h

paddle/phi/common/place.h

RedContritio · 2023-03-04T04:39:04Z

@xiegegege 问一下这个 CI 没过是因为啥

出错关键字可以搜索 "Traceback"

2023-03-04 12:25:14 Error: Can not import paddle core while this file exists: /usr/local/python3.7.0/lib/python3.7/site-packages/paddle/fluid/libpaddle.so
2023-03-04 12:25:14 Traceback (most recent call last):
2023-03-04 12:25:14 File "", line 1, in
2023-03-04 12:25:14 File "/usr/local/python3.7.0/lib/python3.7/site-packages/paddle/init.py", line 27, in
2023-03-04 12:25:14 from .framework import monkey_patch_variable
2023-03-04 12:25:14 File "/usr/local/python3.7.0/lib/python3.7/site-packages/paddle/framework/init.py", line 17, in
2023-03-04 12:25:14 from . import random # noqa: F401
2023-03-04 12:25:14 File "/usr/local/python3.7.0/lib/python3.7/site-packages/paddle/framework/random.py", line 16, in
2023-03-04 12:25:14 import paddle.fluid as fluid
2023-03-04 12:25:14 File "/usr/local/python3.7.0/lib/python3.7/site-packages/paddle/fluid/init.py", line 36, in
2023-03-04 12:25:14 from . import framework
2023-03-04 12:25:14 File "/usr/local/python3.7.0/lib/python3.7/site-packages/paddle/fluid/framework.py", line 34, in
2023-03-04 12:25:14 from . import core
2023-03-04 12:25:14 File "/usr/local/python3.7.0/lib/python3.7/site-packages/paddle/fluid/core.py", line 355, in
2023-03-04 12:25:14 raise e
2023-03-04 12:25:14 File "/usr/local/python3.7.0/lib/python3.7/site-packages/paddle/fluid/core.py", line 269, in
2023-03-04 12:25:14 from . import libpaddle
2023-03-04 12:25:14 ImportError: libpaddle_full_api_shared.so: cannot open shared object file: No such file or directory

paddle/phi/core/operator.h

paddle/phi/kernels/impl/activation_impl.h

paddle/phi/kernels/set_value_grad_kernel.h

luotao1 · 2023-03-06T07:44:01Z

https://www.paddlepaddle.org.cn/documentation/docs/zh/develop/dev_guides/code_contributing_path_cn.html#span-id-caution1-pr-span

建议：每次提交时，保持尽量少的 commit。可以通过git rebase -i HEAD~3将最新的 3 个 commit 合并成一个（可以根据实际情况修改该数值），再 Push 到远程仓库

@RedContritio 建议提交的时候，将本地的commit压缩一下，便于审核人看，现在commit太多了。

RedContritio · 2023-03-06T15:23:49Z

https://www.paddlepaddle.org.cn/documentation/docs/zh/develop/dev_guides/code_contributing_path_cn.html#span-id-caution1-pr-span 建议：每次提交时，保持尽量少的 commit。可以通过git rebase -i HEAD~3将最新的 3 个 commit 合并成一个（可以根据实际情况修改该数值），再 Push 到远程仓库

@RedContritio 建议提交的时候，将本地的commit压缩一下，便于审核人看，现在commit太多了。

了解了，已经在（今日中午）最新的 develop 上重建了 PR 分支，并根据功能进行 commit，最终保留三个 commit。

RedContritio · 2023-03-07T18:06:00Z

参考此前的 review 意见重新整理了一遍，剩下几个（没过的） CI 感觉是合理范围。

APPROVAL：20文件
Coverage：迁移常态覆盖率不够
Model-benchmark：感觉玄学，重跑过但是没差
OP-benchmark：超时两次了，同玄学

luotao1 · 2023-03-08T01:53:38Z

Model-benchmark：感觉玄学，重跑过但是没差
OP-benchmark：超时两次了，同玄学

这两个不用管，等 @YuanRisheng 审核通过后，可以申请豁免

XiaoguangHu01

LGTM

paddle-bot bot added contributor External developers status: proposed labels Feb 17, 2023

luotao1 assigned luotao1 and YuanRisheng Feb 21, 2023

RedContritio changed the title ~~[PHI decoupling] remove reference to fluid/framework/operator.h in phi~~ 【Hackathon No.67】[PHI decoupling] remove reference to operator.h in phi Feb 22, 2023

RedContritio mentioned this pull request Feb 22, 2023

【PaddlePaddle Hackathon 第四期】任务总览 #50629

Closed

RedContritio changed the title ~~【Hackathon No.67】[PHI decoupling] remove reference to operator.h in phi~~ 【Hackathon No.67】remove reference to operator.h in phi [part 1] Feb 23, 2023

YuanRisheng reviewed Feb 27, 2023

View reviewed changes

RedContritio requested a review from YuanRisheng March 2, 2023 09:19

YuanRisheng reviewed Mar 2, 2023

View reviewed changes

paddle/phi/kernels/funcs/math_function.h Outdated Show resolved Hide resolved

paddle/phi/common/place.h Outdated Show resolved Hide resolved

RedContritio requested a review from YuanRisheng March 2, 2023 22:54

YuanRisheng reviewed Mar 6, 2023

View reviewed changes

paddle/phi/core/operator.h Outdated Show resolved Hide resolved

paddle/phi/kernels/impl/activation_impl.h Outdated Show resolved Hide resolved

paddle/phi/kernels/set_value_grad_kernel.h Outdated Show resolved Hide resolved

luotao1 assigned Ligoml Mar 6, 2023

Ligoml mentioned this pull request Mar 7, 2023

【PaddlePaddle Hackathon 第四期】任务总览 #51281

Closed

RedContritio added 4 commits March 7, 2023 08:39

add visit_place to phi/core/utils

a69d46d

remove reference to operator.h in phi/kernels/funcs/math_function.h

91cf6d5

update data type from framework.proto to phi

8ce056f

fix enforce error in fluid

bb1256c

RedContritio requested a review from YuanRisheng March 7, 2023 17:52

YuanRisheng approved these changes Mar 8, 2023

View reviewed changes

XiaoguangHu01 approved these changes Mar 9, 2023

View reviewed changes

YuanRisheng merged commit 9ffdb2b into PaddlePaddle:develop Mar 9, 2023

YuanRisheng mentioned this pull request Mar 24, 2023

[phi] Decoupled phi from fluid tracking issue #47615

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

【Hackathon No.67】remove reference to operator.h in phi [part 1] #50624

【Hackathon No.67】remove reference to operator.h in phi [part 1] #50624

Uh oh!

RedContritio commented Feb 17, 2023 •

edited

Loading

paddle-bot bot commented Feb 17, 2023

RedContritio commented Feb 17, 2023 •

edited

Loading

RedContritio commented Feb 25, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

luotao1 commented Feb 27, 2023

luotao1 commented Mar 2, 2023

Uh oh!

Uh oh!

RedContritio commented Mar 4, 2023 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

luotao1 commented Mar 6, 2023

RedContritio commented Mar 6, 2023

RedContritio commented Mar 7, 2023 •

edited

Loading

luotao1 commented Mar 8, 2023

XiaoguangHu01 left a comment

Labels

5 participants

【Hackathon No.67】remove reference to operator.h in phi [part 1] #50624

【Hackathon No.67】remove reference to operator.h in phi [part 1] #50624

Uh oh!

Conversation

RedContritio commented Feb 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR types

PR changes

Describe

paddle-bot bot commented Feb 17, 2023

RedContritio commented Feb 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

RedContritio commented Feb 25, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

luotao1 commented Feb 27, 2023

luotao1 commented Mar 2, 2023

Uh oh!

Uh oh!

RedContritio commented Mar 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

luotao1 commented Mar 6, 2023

RedContritio commented Mar 6, 2023

RedContritio commented Mar 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

luotao1 commented Mar 8, 2023

XiaoguangHu01 left a comment

Choose a reason for hiding this comment

Labels

5 participants

RedContritio commented Feb 17, 2023 •

edited

Loading

RedContritio commented Feb 17, 2023 •

edited

Loading

RedContritio commented Mar 4, 2023 •

edited

Loading

RedContritio commented Mar 7, 2023 •

edited

Loading