add depend when doing fuse_all_optimizer on program

[zhiqiu]

PR types

Bug fixes

PR changes

Others

Describe

add depend when doing fuse_all_optimizer on program

If fuse_all_optimizer is applied to a program, it rewrites the program with fused tensors as below,

fused_grads = coalesce_tensor(grad0, grad1, ...) fused_params = coalesce_tensor(param0, param1, ...) ... grad0 = grad_op0(...) grad1 = grad_op1(...) ... fused_params = adam(fused_params, fused_grads, ...) 

If the operators in the program are executed op-by-op, it is ok.
However, it the program is executed by an Executor that may run op parallel, the adam and grad_op0, grad_op1 may run in any order since there are no explicit dependencies. And it is wrong.

This PR adds explicit dependency in that case, by adding an empty depend op.

fused_grads = coalesce_tensor(grad0, grad1, ...) fused_params = coalesce_tensor(param0, param1, ...) ... grad0 = grad_op0(...) grad1 = grad_op1(...) ... fused_params = depend(fused_params, grad0, grad1, ...) fused_params = adam(fused_params, fused_grads, ...)