use elementwise to optimize gelu backward implementation on GPU #38263
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
| Thanks for your contribution! |
Contributor
| PR名字描述稍微详细一点，跟上个PR对应 |
Zjq9409 changed the title 
Contributor Author
Done. |
ZzSean reviewed Dec 21, 2021
paddle/fluid/operators/gelu_op.cu Outdated
| tanh(kAlpha * x * (one + static_cast<MPType>(0.044715) * x * x)); | ||
| auto ans = | ||
| half * x * ((one - tanh_out * tanh_out) * | ||
| (kAlpha + static_cast<MPType>(0.1070322243) * x * x)) + |
paddle/fluid/operators/gelu_op.cu Outdated
| auto tanh_out = | ||
| tanh(kAlpha * x * (one + static_cast<MPType>(0.044715) * x * x)); | ||
| auto ans = | ||
| half * x * ((one - tanh_out * tanh_out) * |
JamesLim-sy reviewed Dec 21, 2021
paddle/fluid/operators/gelu_op.cu Outdated
| #include "paddle/fluid/operators/amp/fp16_type_traits.h" | ||
| #include "paddle/fluid/operators/elementwise/elementwise_op_broadcast.cu.h" | ||
| #include "paddle/fluid/operators/gelu_op.h" | ||
| #include "paddle/fluid/platform/float16.h" |
Contributor
There was a problem hiding this comment.
这个头文件引用已经在paddle/fluid/operators/amp/fp16_type_traits.h 引用过了，可以删除
Zjq9409 commented Dec 22, 2021
paddle/fluid/operators/gelu_op.cu Outdated
| (one + tanh(static_cast<MPType>(0.79788456) * x * | ||
| (one + static_cast<MPType>(0.044715) * x * x))); | ||
| MPType half = static_cast<MPType>(0.5); | ||
| MPType decimal = static_cast<MPType>(0.044715); |
Contributor Author
There was a problem hiding this comment.
// gelu(x) = 0.5 * x * (1 + tanh(sqrt(2 / \pi) * (x + 0.044715 * x^{3})))
gelu的公式，其中“0.044715”是一个固定的常量值
Contributor
There was a problem hiding this comment.
0.044715
可以用宏定义的方式表达这个常量值, 比如：
#define GELU_CONSTANT 0.044715 同时将这个宏定义放在通用文件夹中，比如gelu_op.h 中，同步修改使用了0.044715 的代码
JamesLim-sy reviewed Dec 22, 2021
paddle/fluid/operators/gelu_op.cu Outdated
| MPType dout = static_cast<MPType>(arg_dout); | ||
| MPType one = static_cast<MPType>(1); | ||
| MPType half = static_cast<MPType>(0.5); | ||
| MPType decimal = static_cast<MPType>(0.044715); |
paddle/fluid/operators/gelu_op.cu Outdated
| (one + tanh(static_cast<MPType>(0.79788456) * x * | ||
| (one + static_cast<MPType>(0.044715) * x * x))); | ||
| MPType half = static_cast<MPType>(0.5); | ||
| MPType decimal = static_cast<MPType>(0.044715); |
Contributor
There was a problem hiding this comment.
0.044715
可以用宏定义的方式表达这个常量值, 比如：
#define GELU_CONSTANT 0.044715 同时将这个宏定义放在通用文件夹中，比如gelu_op.h 中，同步修改使用了0.044715 的代码
paddle/fluid/operators/gelu_op.cu Outdated
| MPType kBeta = kAlpha * decimal * static_cast<MPType>(3); | ||
| auto tanh_out = tanh(kAlpha * x * (one + decimal * x * x)); | ||
| auto temp = (one - tanh_out * tanh_out) * (kAlpha + kBeta * x * x); | ||
| auto ans = half * x * temp + half * (one + tanh_out); |
Contributor
There was a problem hiding this comment.
这部分的计算中应该存在多次重复计算的参数，比如:x^3 ，可以把这类参数挑出来，减少计算量
zmxdream pushed a commit to zmxdream/Paddle that referenced this pull request Dec 25, 2021
…lePaddle#38263) * optimize gelu backward * optimize gelu backward * optimize code * Number to expression * Replacement number
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
PR types
Performance optimization
PR changes
OPs
Describe
使用elementwise优化gelu算子GPU反向计算，前向计算+反向计算优化后性能数据如下：
