[QUESTION] Will the data get re-shuffled if the sequence length is modified during training?

[SefaZeng]

Your question
If I change the seq-length during training, like 4096 -> 8192, or 4096 -> 1024, will the data loader reset the order of the dataset or continue with the old order?
I want to change the seq-length after training maybe 100k steps, and I need the model to continue train with the left data, will Megatron-LM do this by default?

[Advait251206]

Short answer: no, Megatron-LM does not automatically reshuffle or reset dataset order when you change sequence length mid-training — but changing seq-length is not a “transparent” operation either.

What happens depends on how you change the sequence length and whether you restart the data loader / training state.

---

1. What actually controls data order in Megatron-LM

Megatron-LM data ordering is determined by:

• dataset index mapping
• data sampler (deterministic or shuffled)
• global step + microbatch offset
• distributed rank partitioning

So order is already fixed at dataset construction time, not sequence length.

---

2. If you ONLY change seq_length (no restart)

If you modify sequence length dynamically during training:

✔ Data order:

• unchanged
• sampler continues from same global step

⚠ BUT data format changes

Because sequences are:

• packed differently
• truncated or extended differently
• re-chunked into new boundaries

So even though order is preserved:

the content per sample changes

---

3. Case A: 4096 → 8192 (increase context length)

What happens:

• dataset is re-packed into longer sequences
• fewer samples per batch
• tokens previously split across samples may now be merged

Effect:

• training continues logically
• but sample boundaries shift
• effective “view” of dataset changes

---

4. Case B: 4096 → 1024 (decrease context length)

What happens:

• sequences are aggressively re-chunked
• one original sample becomes multiple smaller ones
• remaining tokens are re-segmented

Effect:

• stronger reshuffling at token-boundary level
• but still NOT dataset index reshuffle

---

5. Does Megatron-LM reset dataset or sampler?

❌ No automatic reset

Unless you explicitly:

• restart dataloader
• reinitialize dataset
• reload checkpoint with sampler state reset

Megatron will:

continue from current dataset iterator state

---

6. Critical subtlety: “order” vs “token stream”

Megatron-LM training is closer to:

continuous token stream slicing

not:

fixed dataset examples

So when seq-length changes:

• dataset order remains stable
• token grouping changes

---

7. What about checkpoint resume after change?

If you:

• change seq-length
• reload checkpoint

Then:

✔ model state continues correctly

⚠ dataset position depends on:

• saved dataloader state
• consumed_samples / global_step

If those are preserved:

training resumes from correct position

If not:

data may restart or shift

---

8. Best practice (important)

When changing sequence length mid-training:

✔ Recommended approach:

1. save checkpoint
2. stop training
3. rebuild dataset with new seq-length
4. restart training with:
   • same checkpoint
   • same consumed samples offset

---

9. Why Megatron behaves this way

Because:

• it optimizes for large-scale deterministic training
• dataset is treated as streaming corpus
• reshuffling mid-stream would break reproducibility

So:

sequence length is a packing strategy, not a dataset ordering parameter

---

10. Bottom line

• ❌ No automatic dataset reshuffling on seq-length change
• ✔ Data order stays deterministic unless sampler is reset
• ⚠ Token grouping and sample boundaries WILL change
• ✔ Proper continuation requires controlled restart with dataset rebuild

---

If this answer helped or pointed you in the right direction, I'd appreciate it if you could mark it as the accepted answer so it's easier for others with the same issue to find.

Also, if you found my contribution useful, I'd appreciate it if you could check out my GitHub profile, follow me, and star any repositories you find interesting.

GitHub: https://github.com/Advait251206