optimize elementwise_mul_grad using new interfaces #37728
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Update forked PaddlePaddle
Update my fork
update from PaddlePaddle
Update forked paddle repo
Update USERNAME/paddle
update Paddle USERNAME repo
update username repo
update local paddlepaddle
update paddlepaddle
… elem_mul_grad
| Thanks for your contribution! |
ZzSean reviewed Dec 3, 2021
| std::vector<int> reduce_dims = GetReduceDim(x->dims(), out->dims(), axis); | ||
| gpuStream_t stream = ctx.cuda_device_context().stream(); | ||
| | ||
| framework::Tensor wayto_dx; |
| wayto_dx.Resize(dout->dims()); | ||
| default_elementwise_mul<DeviceContext, T>(ctx, dout, y, &wayto_dx); | ||
| | ||
| const framework::Tensor* const_to_dx = |
Zjq9409 reviewed Dec 6, 2021
| }; | ||
| | ||
| template <typename T> | ||
| struct MulFunctor<paddle::platform::complex<T>> { |
Contributor
There was a problem hiding this comment.
这个函数可以删除，可以判断当复数形式时从原来的y(y.real, y.imag)构造y_conj(y.real, -y.imag);传入乘法就行
Contributor Author
There was a problem hiding this comment.
此函数放在这里确实不合适，与MulFunctor语义冲突。已修改
AshburnLee commented Dec 10, 2021
| inline HOSTDEVICE T operator()(const T& a, const T& b) const { return a * b; } | ||
| }; | ||
| template <typename T> | ||
| struct MulDxDyFunctor<paddle::platform::complex<T>> { |
Contributor Author
There was a problem hiding this comment.
函数MulGradDY接受4个参数，如果使用这个函数，就要将调用者改为函数ElemwiseGradCompute，该函数最终调用kernel是优化前的kernel，并且意义不同，此处的功能是elemwiseCompute而非ElemwiseGradCompute。所以我认为，此处复用不了MulGradDY，故提供了MulDxDyFunctor。
Zjq9409 reviewed Dec 13, 2021
| inline HOSTDEVICE T operator()(const T& a, const T& b) const { return a * b; } | ||
| }; | ||
| template <typename T> | ||
| struct MulDxDyFunctor<paddle::platform::complex<T>> { |
Contributor
There was a problem hiding this comment.
名字可以修改一下，并且提取到elementwise_functor.h公共文件中
| std::vector<int> reduce_dims = GetReduceDim(x->dims(), out->dims(), axis); | ||
| gpuStream_t stream = ctx.cuda_device_context().stream(); | ||
| | ||
| framework::Tensor dx_tmp; |
Contributor Author
There was a problem hiding this comment.
done. 改为dx_origin_dims，表示reduce之前的dx结果。
| template <typename DeviceContext, typename T> | ||
| typename std::enable_if< | ||
| std::is_same<DeviceContext, platform::CPUDeviceContext>::value>::type | ||
| default_elementwise_mul_grad(const framework::ExecutionContext& ctx, |
Contributor
There was a problem hiding this comment.
default_elementwise_mul_grad和elementwise_mul_grad代码存在重复
Zjq9409 reviewed Dec 13, 2021
| | ||
| template <typename T> | ||
| struct MulDxDyFunctor { | ||
| inline HOSTDEVICE T operator()(const T& a, const T& b) const { return a * b; } |
… elem_mul_grad
… elem_mul_grad
… elem_mul_grad
… elem_mul_grad
… elem_mul_grad
JamesLim-sy reviewed Jan 5, 2022
| } | ||
| } | ||
| } | ||
| */ |
| template <typename DeviceContext, typename T> | ||
| typename std::enable_if< | ||
| std::is_same<DeviceContext, platform::CUDADeviceContext>::value>::type | ||
| default_elementwise_mul_grad(const framework::ExecutionContext& ctx, |
Contributor
There was a problem hiding this comment.
这部分的代码根据Zjq9409的最新合入PR修改一下
Zjq9409 reviewed Jan 5, 2022
| dx[col] = plat::complex<double>(y[col].real, -y[col].imag) * o; | ||
| dy[col] = plat::complex<double>(x[col].real, -x[col].imag) * o; | ||
| col += blockDim.x * gridDim.x; | ||
| } |
Contributor
There was a problem hiding this comment.
SimpleElemwiseMulGradCUDAKernel函数代码可以删除
JamesLim-sy approved these changes Jan 5, 2022
Zjq9409 approved these changes Jan 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
PR types
Performance optimization
PR changes
OPs
Describe
功能
该PR使用新的接口优化了elementwise_mul的反向计算
opbenchmark 表现
关于CI-opbenchmark 性能较develop性能变差的3个配置：使用reduce接口优化后，前3个配置较dev下降15%左右；适配多输出代码优化后，该3个配置与dev打平，其他超越（1.85x~12.16x)或打平。整体上，不差于dev和竞品。