"accelerate elementwise_add_grad, add reduce functor"

[dzhwinter]

fix #7862

[tonyyang-svail]

@dzhwinter could you briefly explain why the original version is slow?

[dzhwinter]

The previous version use broadcast, it is a quite low efficient operation.
After enhance, the result shows, the elementwise_add_grad lower than convolution.

-------------------------> Profiling Report <------------------------- Place: CPU	Total Time:41762.5ms	Total Memory:17689.2MB	Sorted by total time in descending order in the same thread Event Calls Total Min. Max. Ave. Total Memory.Min Memory. Max Memory. thread0::conv2d_grad 13 15602.1 276.188 5248.83 1200.16 9069.18 0.0078125 392.148 thread0::conv2d 13 8975.5 219.889 3366.09 690.423 338.152 12.2539 392.004 thread0::dropout 10 3036.16 0.300329 1213.68 303.616 1906.18 0.132812 784.008 thread0::elementwise_add_grad 16 2279.28 0.030858 544.775 142.455 8960.14 0.0195312 392.008 thread0::batch_norm_grad 14 1926.31 0.390424 462.51 137.594 8961.79 0.0742188 392.012 thread0::relu_grad 14 1689.34 0.087101 397.12 120.667 8961.71 0.0664062 392.004 thread0::pool2d_grad 5 1247.38 22.8108 617.521 249.475 9020.14 12.2539 392.004 thread0::batch_norm 14 1125.46 0.435152 264.698 80.39 1122.16 0.0742188 392.012 thread0::elementwise_add 16 991.63 0.031354 248.922 61.9769 730.156 0.015625 392.004 thread0::dropout_grad 10 808.702 0.048284 379.893 80.8702 8961.64 0.0664062 392.004 thread0::relu 14 795.466 0.042214 189.24 56.819 1514.17 0.0664062 392.004 thread0::adam 60 516.043 0.013363 238.108 8.60072 17688.7 0 0 thread0::fill_zeros_like 66 445.392 0.003328 185.057 6.74837 8961.57 0.00390625 392.004 thread0::pool2d 5 342.792 7.17049 180.343 68.5584 4258.21 3.06641 98.0039 thread0::mul_grad 3 74.9344 0.593411 71.8969 24.9781 8960.16 0.269531 52.0703 thread0::mul 3 44.4806 0.180829 43.4023 14.8269 8959.48 0.015625 0.0664062 thread0::elementwise_mul 60 0.363544 0.004298 0.022641 0.00605907 17688.7 0.00390625 0.00390625 thread0::softmax 1 0.278477 0.278477 0.278477 0.278477 8960.05 0.015625 0.015625 thread0::fill_constant 61 0.260437 0.002431 0.025904 0.00426946 8960.11 0.00390625 0.00390625

[dzhwinter]

This solution involves in the reduce primitive, its CPU version implement is quite easy, as the test result shown above, which demonstrate the performance well. But the GPU kernel is hard to implement because of GPU threads overwriting problem.

I had dig into the reduce kernel in https://github.com/zchee/cuda-sample/blob/master/6_Advanced/reduction/reduction_kernel.cu , but in my local machine, the reduce kernel is still not implemented correctly, so this PR has been delayed such a long time.

Currently, this issue partly has been fixed in without using the reduce kernel. Please check the detail in #8402

[paddle-bot-old]

Since you haven't replied for a long time, we have closed this issue/pr.
If the problem is not solved or there is a follow-up one, please reopen it at any time and we will continue to follow up.
由于您长期未回复，我们将关闭这个issue/pr。
若问题未解决或有后续问题，请随时重新打开，我们会继续跟进。