[user-streams] Move user object bytecode generation after calling user compiler

[mlazos]

This move needs to occur in order to allow AOTAutograd to indicate if more streams/events need to be created for the backward.

Stack from ghstack (oldest at bottom):

• [user-streams] Insert backward syncs #167747
• [user-streams] Stash graph created objects in keep_alive list for backwards #167705
• -> [user-streams] Move user object bytecode generation after calling user compiler #167704
• [user-streams] Assign streams to gradient accum in bwd #167513

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @chenyang78 @kadeng @chauhang @amjames @Lucaskabela

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/167704

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (3 Unrelated Failures)

As of commit deff21b with merge base e5eb89e ():

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• inductor / inductor-cpu-test / test (cpu_inductor_torchbench, 2, 2, linux.2xlarge.amx) (gh) (trunk failure)
  stable_diffusion_unet
• inductor / inductor-cpu-test / test (dynamic_cpu_inductor_torchbench, 2, 2, linux.2xlarge.amx) (gh) (trunk failure)
  stable_diffusion_unet
• inductor / inductor-test / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) (gh) (trunk failure)
  stable_diffusion_unet

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[pytorch-bot]

The label module: dynamo is only applicable to issues and has been removed. Please only use this label on issues.

[pytorch-bot]

The label module: dynamo is only applicable to issues and has been removed. Please only use this label on issues.

[pytorch-bot]

The label module: dynamo is only applicable to issues and has been removed. Please only use this label on issues.

[pytorch-bot]

The label module: dynamo is only applicable to issues and has been removed. Please only use this label on issues.

[williamwen42]

Can you comment why this move needs to be done? (In my nested graph break work, the order of codegen is extremely important)

[pytorch-bot]

The label module: dynamo is only applicable to issues and has been removed. Please only use this label on issues.

[mlazos]

Can you comment why this move needs to be done? (In my nested graph break work, the order of codegen is extremely important)

Updated the PR desc.

[pytorch-bot]

The label module: dynamo is only applicable to issues and has been removed. Please only use this label on issues.

[pytorch-bot]

The label module: dynamo is only applicable to issues and has been removed. Please only use this label on issues.