fix(cosine): remove erroneous restart, match PyTorch CosineAnnealingLR behavior#5191
Open
DeathSurfing wants to merge 3 commits into
Open
fix(cosine): remove erroneous restart, match PyTorch CosineAnnealingLR behavior#5191DeathSurfing wants to merge 3 commits into
DeathSurfing wants to merge 3 commits into
Conversation
Remove the modulo operation that wraps current_iter back to 0 after num_iters steps, causing the learning rate to abruptly jump back to max_lr. Now current_iter increases monotonically, matching PyTorch's CosineAnnealingLR behavior which explicitly states 'without restarts, T_cur = t and increases monotonically with each call to step()'. Update test_lr_change to verify the smooth full-period cosine curve without artificial restarts.
Update the CosineAnnealingLrSchedulerConfig doc to reflect that this scheduler performs cosine annealing without restarts (matching PyTorch's CosineAnnealingLR). Clarify that the LR continues along the cosine curve past num_iters without resetting, and update the num_iters field doc to describe it as the number of iterations to reach min_lr.
Update CosineAnnealingLrScheduler doc to clarify that while the formula originates from the SGDR paper, the implementation performs cosine annealing without periodic restarts. The iteration counter increases monotonically, continuing along the cosine curve past num_iters.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #5128
The
CosineAnnealingLrSchedulerwas wrapping its iteration counter with% (self.num_iters + 1), causing the learning rate to abruptly jump back tomax_lrafter everynum_iters + 1steps. This created an erroneous half-cosine restarting waveform instead of the standard cosine annealing schedule.Changes
% (self.num_iters + 1)wrapping fromstep(). Thecurrent_iternow increases monotonically (viawrapping_add(1)), matching PyTorch'sCosineAnnealingLRwhich explicitly states "performs cosine annealing without restarts, soT_cur = tand increases monotonically with each call tostep()"test_lr_changenow verifies a full 5-step smooth cosine curve rather than the previous 4-step sequence that expected an artificial restartVerification
cargo test -p burn-optim -- lr_scheduler::cosine)cargo test -p burn-optim)% (self.num_iters + 1)); the rest is documentation and test updatesReference
PyTorch CosineAnnealingLR docs