Revert D113962248: Optimizer scratch-buffer stashing in MemoryStashingManager (#4495) - #4495
Open
doIIarplus wants to merge 1 commit into
Open
Revert D113962248: Optimizer scratch-buffer stashing in MemoryStashingManager (#4495)#4495doIIarplus wants to merge 1 commit into
doIIarplus wants to merge 1 commit into
Conversation
Contributor
|
@doIIarplus has exported this pull request. If you are a Meta employee, you can view the originating Diff in D114959521. |
wz337
approved these changes
Aug 6, 2026
wz337
left a comment
Contributor
There was a problem hiding this comment.
Review automatically exported from Phabricator review in Meta.
…gManager (meta-pytorch#4495) Summary: D113962248 added the torchrec half of "free DistributedShampoo scratch buffers during the optimizer-state stash window": the `_ScratchBufferOptimizer` protocol, `scratch_buffers()` discovery, `_release_optimizer_scratch_buffers`, a second `_optimizer_scratch_buffer_restore_callbacks` list, the `restore_optimizer_state(restore_scratch_buffer=...)` parameter, and `scratch_buffers()` fan-out through `CombinedOptimizer` / `KeyedOptimizerWrapper` / `OptimizerWrapper` / `SemiSyncOptimizer`. None of that machinery is needed. `_global_dist_buffer` is pure scratch that DistributedShampoo allocates, fills, and consumes entirely within `update_params()`; it never leaves the optimizer and is not part of `state_dict`. Routing its free/re-alloc through torchrec meant a cross-repo protocol plus wrapper fan-out and a parallel callback list, purely so an external caller could drive a resize the owner could do itself. Reverting so the feature can be re-implemented inside `hpc/optimizers/distributed_shampoo/dev/` behind `DDPDistributedConfig.free_dist_buffer_between_steps`. Middle of a 3-diff revert stack (on top of the D113962249 revert). Two deliberate carve-outs -- this is NOT a mechanical `sl backout`: 1. Kept the BUCK dep fix. D113962248 also changed `//caffe2:_torch` -> `fbsource//third-party/pypi/torch:torch` in `torchrec/optim/BUCK` and `torchrec/optim/tests/BUCK`. `_torch` is a private target that should not be depended on directly, so those hunks are left in place. 2. Kept the `staged_cpu_view_for` contiguity fix and its `TestCheckpointWhileStashed` tests. D113962248 also replaced the `view.set_(storage, offset, shape, stride)` reconstruction with a contiguous `reshape/narrow/view`. That is an unrelated correctness fix: `chunked_copy_` fills the pinned buffer in logical/row-major order, so rebuilding the view with the source's original (possibly transposed) stride hands the DCP stager transposed values -> silently corrupt optimizer state on resume. Reverting it would reintroduce that bug. Original commit changeset: 37123fd3d19f Original Phabricator Diff: D113962248 Reviewed By: wz337, hjmshi Differential Revision: D114959521
doIIarplus
force-pushed
the
export-D114959521
branch
from
August 7, 2026 04:00
fd2d9ad to
15bfdd3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
D113962248 added the torchrec half of "free DistributedShampoo scratch buffers
during the optimizer-state stash window": the
_ScratchBufferOptimizerprotocol,scratch_buffers()discovery,_release_optimizer_scratch_buffers, a second_optimizer_scratch_buffer_restore_callbackslist, therestore_optimizer_state(restore_scratch_buffer=...)parameter, andscratch_buffers()fan-out throughCombinedOptimizer/KeyedOptimizerWrapper/
OptimizerWrapper/SemiSyncOptimizer.None of that machinery is needed.
_global_dist_bufferis pure scratch thatDistributedShampoo allocates, fills, and consumes entirely within
update_params(); it never leaves the optimizer and is not part ofstate_dict.Routing its free/re-alloc through torchrec meant a cross-repo protocol plus
wrapper fan-out and a parallel callback list, purely so an external caller could
drive a resize the owner could do itself.
Reverting so the feature can be re-implemented inside
hpc/optimizers/distributed_shampoo/dev/behindDDPDistributedConfig.free_dist_buffer_between_steps.Middle of a 3-diff revert stack (on top of the D113962249 revert).
Two deliberate carve-outs -- this is NOT a mechanical
sl backout:Kept the BUCK dep fix. D113962248 also changed
//caffe2:_torch->fbsource//third-party/pypi/torch:torchintorchrec/optim/BUCKandtorchrec/optim/tests/BUCK._torchis a private target that should not bedepended on directly, so those hunks are left in place.
Kept the
staged_cpu_view_forcontiguity fix and itsTestCheckpointWhileStashedtests. D113962248 also replaced the
view.set_(storage, offset, shape, stride)reconstruction with a contiguous
reshape/narrow/view. That is an unrelatedcorrectness fix:
chunked_copy_fills the pinned buffer in logical/row-majororder, so rebuilding the view with the source's original (possibly transposed)
stride hands the DCP stager transposed values -> silently corrupt optimizer
state on resume. Reverting it would reintroduce that bug.
Original commit changeset: 37123fd3d19f
Original Phabricator Diff: D113962248
Reviewed By: wz337, hjmshi
Differential Revision: D114959521