[WIP] Integrate OptimizerInBackward into SFT distributed recipe

[joecummings]

Currently works, just need to debug resume from checkpoint.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchtune/2818

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 Cancelled Jobs, 1 Unrelated Failure

As of commit 4123ba1 with merge base f24a8de ():

CANCELLED JOBS - The following jobs were cancelled. Please retry:

• GPU tests / gpu_test (3.10, stable) (gh)
• GPU tests / gpu_test (3.9, stable) (gh)

FLAKY - The following job failed but was likely due to flakiness present on trunk:

• GPU tests / gpu_test (3.11, stable) (gh) (detected as infra flaky with no log or failing log classifier)

This comment was automatically generated by Dr. CI and updates every 15 minutes.