Skip to content

[main] moe(feat): support chunkwise context parallelism for GDN#5637

Draft
yuzhongw-nvidia wants to merge 544 commits into
NVIDIA:mainfrom
yuzhongw-nvidia:yuzhongw/fla_cp
Draft

[main] moe(feat): support chunkwise context parallelism for GDN#5637
yuzhongw-nvidia wants to merge 544 commits into
NVIDIA:mainfrom
yuzhongw-nvidia:yuzhongw/fla_cp

Conversation

@yuzhongw-nvidia

@yuzhongw-nvidia yuzhongw-nvidia commented Jul 3, 2026

Copy link
Copy Markdown
Contributor
  • I, the PR author, have personally reviewed every line of this PR.

What does this PR do?

pr for dev: #3282

depends on: #5392

⚠️ For major changes (either in lines of code or in its impact), please make sure to first share a design doc with the team. If you're unsure what's the best way to do so, contact @NVIDIA/mcore-oncall.

Issue tracking

For PRs from open-source community contributors:

  • New features: a linked issue is required. Please open a feature request and reference it here before submitting the PR.
  • Small updates (bug fixes, minor improvements): a linked issue is recommended and will accelerate the PR review process.

Linked issue:

Contribution process

Pre-checks

  • I have added relevant unit tests
  • I have added relevant functional tests
  • I have added proper typing to my code Typing guidelines
  • I have added relevant documentation
  • I have run the autoformatter.sh on my PR

Code review

Feel free to message or comment @NVIDIA/mcore-oncall to help accelerate your merge into main. The less complex your PR is, the faster it will be approved and merged!

All PRs start as draft. If you open a non-draft PR, it will be automatically converted to draft.

Step 1: Mark PR as "Ready for Review"

  1. When your PR is ready, click Ready for Review.
  2. An oncall reviewer is auto-assigned and expert reviewers are notified based on your changes.
    • Some PRs may jump straight to step 2. This is determined by .github/CODEOWNERS.

⚠️ Only mark as ready once merge-conflicts are resolved and the CI is passing.
Final Review might get declined if these requirements are not fulfilled.

Step 2: Final Review

For PRs that change megatron/core, once all expert reviewers have approved, the Final Review label is applied automatically and final reviewers are assigned.

For PRs outside megatron/core, this step is skipped.

Step 3: Approved

Once all required reviewers have approved, the Approved label is applied automatically.

Merge

Any member of mcore-engineers will be able to merge your PR.

Wohox and others added 30 commits February 5, 2026 14:40
@Wohox
[Dev] Fix EP Overlap missing record stream for shared expert (NVIDIA#…
ef336ca 
…3244)
@shjwudp @yaox12
Restore missing linear-cross-entropy option accidentally removed from…
ec94d63 
… arguments.py (NVIDIA#3266)

Co-authored-by: Xin Yao <xiny@nvidia.com>
@eternally-z @yaox12
Fix reload_model_params failure when loading MoE models with explicit…
500e080 
… state_dict (NVIDIA#3243)

Co-authored-by: Xin Yao <xiny@nvidia.com>
@ko3n1g
ci: Disable moe20 tests (NVIDIA#3312)
433c169 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g
ci: Pin down setuptools to lt 82 (NVIDIA#3316)
fd4801e 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@IanBoyanZhang @yaox12
[None][Fix] Prevent resource leak warnings (NVIDIA#3216)
52eabf0 
Co-authored-by: Xin Yao <xiny@nvidia.com>
@Wohox
[Dev] Fix backward dw dependency (NVIDIA#3338)
c0030d6
@ko3n1g @chtruong814
ci: Rely exclusively on GitHub CI (NVIDIA#3341)
2c2e749 
Signed-off-by: oliver könig <okoenig@nvidia.com>
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Co-authored-by: Charlie Truong <chtruong@nvidia.com>
@ko3n1g
[dev] ci: skip queue in merge-gate (NVIDIA#3344)
98f6f81 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g
Revert "[None][Fix] Prevent resource leak warnings (NVIDIA#3216)" (NV…
28b130f 
…IDIA#3366)
@chtruong814 @ko3n1g
ci: Fix dev branch merge queue (NVIDIA#3397)
e868e8f 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: oliver könig <okoenig@nvidia.com>
Co-authored-by: oliver könig <okoenig@nvidia.com>
@xuwchen @BestJuly
[Dev] Add Qwen3-VL support with Megatron-FSDP (NVIDIA#2842)
c4b910f 
Co-authored-by: Li Tao <lit@nvidia.com>
@kunlunl
Add absorbed-mla (NVIDIA#3193)
6059f36
@chtruong814
cp: Remove gpu sanity check (NVIDIA#3420) into dev (NVIDIA#3421)
9f2ca96 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@ko3n1g
[dev] ci: Fix merge queue (NVIDIA#3385)
1dcf0da 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g
[dev] cp: Cherrypick CI changes (NVIDIA#3543)
cd1c215 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@Victarry
[Dev] Fix MoE aux loss tracker hang with MTP enabled (NVIDIA#3400)
aa86018
@chtruong814
ci: Remove multi-approval action from dev branch (NVIDIA#3576)
2b4b9c4 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@FDecaYed
Merge branch 'main' into dev
0ab47fa
@ko3n1g @HaochenYuan
@Phlip79 @duncanriach @yobibyte @chtruong814 @wdykas @github-actions @1195343015 @yuzhongw-nvidia @lhb8125 @youngeunkwon0405 @santhnm2 @jiemingz @tgkyrie @dimapihtar @yaox12 @rkarimimahab @sancha @deepakn94 @sbhavani @ahmadki @BestJuly @maanug-nv @mvirts @asolergi-nv @HollowMan6 @buptzyb @shengf-nv @venmugil @mathemakitten @jaredcasper @parthmannan @tdene @Autumn1998 @wplf @yashaswikarnati @nschank @Glabred @janEbert @rj42 @juntaowww @Wohox @chrisgrimm @ChenhanYu @ericharper @xiaoxi-wangfj @shjwudp @c1lovez1 @skydoorkai @yeyu-nvidia @kwyss-nvidia @jon-barker @AAnoosheh @sidsingh-nvidia @megnvidia @thecaptain789 @ISEEKYAN @xuwchen @jstjohn @lmcafee-nvidia @claude @ArEsKay3 @Victarry @cursoragent @kevalmorabia97 @shanmugamr1992 @vasunvidia @sajadn @kunlunl @laixinn @dndnda @jenchen13 @yueshen2016 @faradawn @cspades @arendu @seonjinn @sbak5 @mkhona-nvidia
[dev] pull main 260220 (NVIDIA#3574)
a1a73f8 
Signed-off-by: oliver könig <okoenig@nvidia.com>
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: Hongbin Liu <hongbinl@nvidia.com>
Signed-off-by: Youngeun Kwon <youngeunk@nvidia.com>
Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com>
Signed-off-by: Jimmy Zhang <jiemingz@nvidia.com>
Signed-off-by: Santosh Bhavani <santosh.bhavani@live.com>
Signed-off-by: Deepak Narayanan <dnarayanan@nvidia.com>
Signed-off-by: Hollow Man <hollowman@opensuse.org>
Signed-off-by: Robin Zhang <robinz@nvidia.com>
Signed-off-by: jinliangl <jinliangl@nvidia.com>
Signed-off-by: Maanu Grover <maanug@nvidia.com>
Signed-off-by: dimapihtar <dpihtar@gmail.com>
Signed-off-by: xiaoxi-wangfj <690912414@qq.com>
Signed-off-by: skydoorkai <htsantaclara@163.com>
Signed-off-by: Asha Anoosheh <aanoosheh@nvidia.com>
Signed-off-by: meg miranda <mmiranda@nvidia.com>
Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com>
Signed-off-by: sajadn <snorouzi@nvidia.com>
Signed-off-by: lit <lit@nvidia.com>
Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>
Signed-off-by: Cory Ye <cye@nvidia.com>
Signed-off-by: adithyare <adithyare@nvidia.com>
Signed-off-by: Soumye Singhal <soumyes@cw-dfw-cs-001-dc-01.cm.cluster>
Signed-off-by: Ahmad Kiswani <kiswani.ahmad@gmail.com>
Signed-off-by: mikail <mkhona@nvidia.com>
Co-authored-by: HaochenYuan <106647990+HaochenYuan@users.noreply.github.com>
Co-authored-by: Philip Petrakian <ppetrakian@nvidia.com>
Co-authored-by: oliver könig <okoenig@nvidia.com>
Co-authored-by: Duncan Riach <33532941+duncanriach@users.noreply.github.com>
Co-authored-by: yobi byte <yobibyte@users.noreply.github.com>
Co-authored-by: Charlie Truong <chtruong@nvidia.com>
Co-authored-by: wdykas <73254672+wdykas@users.noreply.github.com>
Co-authored-by: root <root@gpu-h100-0348.cm.cluster>
Co-authored-by: root <root@gpu-h100-0193.cm.cluster>
Co-authored-by: root <root@gpu-h100-0082.cm.cluster>
Co-authored-by: root <root@gpu-h100-0495.cm.cluster>
Co-authored-by: William Dykas <wdykas@cw-pdx-cs-001-vscode-02.cm.cluster>
Co-authored-by: root <root@gpu-h100-0213.cm.cluster>
Co-authored-by: root <root@gpu-h100-0435.cm.cluster>
Co-authored-by: root <root@gpu-h100-0188.cm.cluster>
Co-authored-by: root <root@gpu-h100-0032.cm.cluster>
Co-authored-by: root <root@gpu-h100-0023.cm.cluster>
Co-authored-by: root <root@gpu-h100-0368.cm.cluster>
Co-authored-by: root <root@gpu-h100-0203.cm.cluster>
Co-authored-by: root <root@gpu-h100-0229.cm.cluster>
Co-authored-by: root <root@gpu-h100-0123.cm.cluster>
Co-authored-by: root <root@gpu-h100-0217.cm.cluster>
Co-authored-by: root <root@gpu-h100-0496.cm.cluster>
Co-authored-by: root <root@gpu-h100-0261.cm.cluster>
Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Jiayi Yan <66017932+1195343015@users.noreply.github.com>
Co-authored-by: Yuzhong Wang <yuzhongw@nvidia.com>
Co-authored-by: Hongbin Liu <lhb8125@users.noreply.github.com>
Co-authored-by: Youngeun Kwon <youngeunk@nvidia.com>
Co-authored-by: Keshav Santhanam <ksanthanam@nvidia.com>
Co-authored-by: Jimmy Zhang <133159885+jiemingz@users.noreply.github.com>
Co-authored-by: tgkyrie <74066353+tgkyrie@users.noreply.github.com>
Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>
Co-authored-by: Xin Yao <xiny@nvidia.com>
Co-authored-by: rkarimimahab <rkarimimahab@nvidia.com>
Co-authored-by: Rabeeh Mahabadi <rkarimimahab@nb-hel-cs-001-vscode-02.cm.cluster>
Co-authored-by: Sanjeev Satheesh <sasatheesh@nvidia.com>
Co-authored-by: Deepak Narayanan <dnarayanan@nvidia.com>
Co-authored-by: Santosh Bhavani <santosh.bhavani@live.com>
Co-authored-by: Ahmad Kiswani <kiswani.ahmad@gmail.com>
Co-authored-by: Li Tao <lit@nvidia.com>
Co-authored-by: Maanu Grover <109391026+maanug-nv@users.noreply.github.com>
Co-authored-by: mvirts <mvirts@gmail.com>
Co-authored-by: Antoni-Joan Solergibert <asolergibert@nvidia.com>
Co-authored-by: ℍ𝕠𝕝𝕝𝕠𝕨 𝕄𝕒𝕟 <hollowman@opensuse.org>
Co-authored-by: Robin Zhang <robinz@nvidia.com>
Co-authored-by: Sheng Fu <shengf@nvidia.com>
Co-authored-by: Venmugil Elango <498703+venmugil@users.noreply.github.com>
Co-authored-by: mathemakitten <helenn@nvidia.com>
Co-authored-by: Jared Casper <155158+jaredcasper@users.noreply.github.com>
Co-authored-by: Parth Mannan <38387286+parthmannan@users.noreply.github.com>
Co-authored-by: Teodor-Dumitru Ene <34819528+tdene@users.noreply.github.com>
Co-authored-by: Tong Liu <tongliu@nvidia.com>
Co-authored-by: Li Jinliang <jinliangl@nvidia.com>
Co-authored-by: Jinliang Li <jinliangl@pool0-01676.cm.cluster>
Co-authored-by: Jinliang Li <jinliangl@cw-dfw-cs-001-vscode-01.cm.cluster>
Co-authored-by: Yashaswi Karnati <144376261+yashaswikarnati@users.noreply.github.com>
Co-authored-by: Nick Schank <nick@reflection.ai>
Co-authored-by: Jeffrey Chen <jeffrey@reflection.ai>
Co-authored-by: janEbert <janpabloe@nvidia.com>
Co-authored-by: rj42 <lbkzman@gmail.com>
Co-authored-by: Juntao Wang <juntaow@nvidia.com>
Co-authored-by: Pingtian Li <158665726+Wohox@users.noreply.github.com>
Co-authored-by: Chris Grimm <chris@reflection.ai>
Co-authored-by: Chenhan D. Yu <5185878+ChenhanYu@users.noreply.github.com>
Co-authored-by: Eric Harper <eharper@nvidia.com>
Co-authored-by: xiaoxi-wangfj <690912414@qq.com>
Co-authored-by: Jianbin Chang <shjwudp@gmail.com>
Co-authored-by: c1lovez1 <141424951+c1lovez1@users.noreply.github.com>
Co-authored-by: Zhang Haitao <htsantaclara@163.com>
Co-authored-by: yeyu-nvidia <yeyu@nvidia.com>
Co-authored-by: kwyss-nvidia <kwyss@nvidia.com>
Co-authored-by: Jon Barker <jbarker@nvidia.com>
Co-authored-by: Asha Anoosheh <aanoosheh@nvidia.com>
Co-authored-by: Siddharth Singh <136645615+sidsingh-nvidia@users.noreply.github.com>
Co-authored-by: megnvidia <mmiranda@nvidia.com>
Co-authored-by: thecaptain789 <257642323+thecaptain789@users.noreply.github.com>
Co-authored-by: thecaptain789 <thecaptain789@users.noreply.github.com>
Co-authored-by: litianjian <litianjian@bytedance.com>
Co-authored-by: Yan Bai <baiyan1996@icloud.com>
Co-authored-by: xuwchen <xuwenc@nvidia.com>
Co-authored-by: John St. John <jstjohn@users.noreply.github.com>
Co-authored-by: Lawrence McAfee <85179052+lmcafee-nvidia@users.noreply.github.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Robert Kirby <ArEsKay3@users.noreply.github.com>
Co-authored-by: Siddharth Singh <sidsingh@nvidia.com>
Co-authored-by: Robert Kirby <rkirby@cw-dfw-cs-001-vscode-01.cm.cluster>
Co-authored-by: Teodor-Dumitru Ene <teodord.ene@gmail.com>
Co-authored-by: Dennis(Zhenhuan) Liu <denliu@nvidia.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com>
Co-authored-by: Shanmugam Ramasamy <111910568+shanmugamr1992@users.noreply.github.com>
Co-authored-by: vasunvidia <108759426+vasunvidia@users.noreply.github.com>
Co-authored-by: Philip Petrakian <pgpetrak@gmail.com>
Co-authored-by: Sajad Norouzi <sajad.n@gmail.com>
Co-authored-by: Kunlun Li <94586211+kunlunl@users.noreply.github.com>
Co-authored-by: xielaixin <xielx@shanghaitech.edu.cn>
Co-authored-by: Robert Kirby <rkirby@nvidia.com>
Co-authored-by: Ming <93323717+dndnda@users.noreply.github.com>
Co-authored-by: liming127 <liming127@meituan.com>
Co-authored-by: Jon Barker <jbarker@oci-hsg-cs-001-vscode-01.cm.cluster>
Co-authored-by: helen ngo <helen.ngo14@gmail.com>
Co-authored-by: Jenny Chen <jennifchen@nvidia.com>
Co-authored-by: yueshen2016 <39203804+yueshen2016@users.noreply.github.com>
Co-authored-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>
Co-authored-by: Cory Ye <44509866+cspades@users.noreply.github.com>
Co-authored-by: Adi Renduchintala <adithya.r@gmail.com>
Co-authored-by: Soumye Singhal <soumyes@cw-dfw-cs-001-dc-01.cm.cluster>
Co-authored-by: Seonjin Na <sna@nvidia.com>
Co-authored-by: Seonmyeong Bak <sbak@nvidia.com>
Co-authored-by: Mikail Khona (NVIDIA) <mkhona@nvidia.com>
@LiuXTao @yuzhongw-nvidia @yanring
[dev] fix(moe): fix the bug where gate was not sliced when kv_head < …
2e4a5d4 
…tp_size. (NVIDIA#3529)

Co-authored-by: xiaotaoliu <xiaotaoliu@tencent.com>
Co-authored-by: Yuzhong Wang <yuzhongw@nvidia.com>
Co-authored-by: Zijie Yan <zijiey@nvidia.com>
@kunlunl
Add unit test for THD (NVIDIA#3608)
d0e0cf0
@Victarry @claude
[Dev] feat(checkpoint): zero-copy storage sharing in CheckpointWithou…
bc9298c 
…tOutput (NVIDIA#3641)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@xiaoyao0115 @kunlunl
[Dev] Add E2E support for THD format (NVIDIA#2924)
5c613ab 
Signed-off-by: xiaoyao0115 <1804647152@qq.com>
Signed-off-by: tailaim <tailaim@nvidia.com>
Co-authored-by: kunlunl <kunlunl@nvidia.com>
@Victarry @claude
fix: skip FSDP DTensor boundary validation under fake process group (N…
5dadaf1 
…VIDIA#3668)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@chtruong814
ci: Remove cudagraph codeowners entry in dev branch (NVIDIA#3712)
2176c4a 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@FDecaYed @skyw
[dev] refactor to support emerging optimizers beyond muon (NVIDIA#3618)
31f5294 
Signed-off-by: Hao Wu <skyw@nvidia.com>
Co-authored-by: Hao Wu <skyw@nvidia.com>
@BestJuly
[Dev] Move some processing into a function so can be compiled (NVIDIA…
a268231 
…#3220)
@yanring @buptzyb
[Dev] Refactor MoE loss logging (NVIDIA#2569)
f983b21 
Co-authored-by: Robin Zhang <robinz@nvidia.com>
@jingqiny-99 @Victarry
[dev] feat(mHC): Add basic pytorch implementation of manifold hyper c…
0b0074e 
…onnection(mHC). (NVIDIA#2943)

Co-authored-by: Jingqin Yang <jingqiny@login-eos01.eos.clusters.nvidia.com>
Co-authored-by: root <root@eos0478.eos.clusters.nvidia.com>
Co-authored-by: Dennis Liu <denliu@nvidia.com>
hxbai and others added 29 commits June 16, 2026 23:51
@hxbai
[Dev] restore DSv4 tflops calc in training and fix the packed seq case (
9af7c79 
NVIDIA#5358)

Signed-off-by: Hongxiao Bai <hongxiaob@nvidia.com>
@yuzhongw-nvidia
[dev] moe(perf): Restore fused GDN THD all-to-all on dev (NVIDIA#5389)
3346fa8 
Signed-off-by: Yuzhong Wang <yuzhongw@nvidia.com>
@github-actions @claude
Merge remote-tracking branch 'origin/main' into main2dev/22_06_2026
c2a9a60 
Nightly sync of main into dev (22_06_2026). Resolves 16 conflicts
preserving dev features (pre-push guard: 0 dropped dev lines); brings
in main's inference shard-spec API additively. Supersedes NVIDIA#5429.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
@Baibaifan
[Fix] Fix MoE router z-loss compatibility with TE CUDA Graph capture. (…
3fae48f 
…NVIDIA#5401)

Signed-off-by: yangfan.bai <yangfan.bai@shopee.com>
Co-authored-by: yangfan.bai <yangfan.bai@shopee.com>
@ko3n1g @claude
test: mark ep_a2a_overlap activation-offloading test flaky_in_dev (NV…
94f70b6 
…IDIA#5450)

Signed-off-by: oliver könig <okoenig@nvidia.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@ko3n1g @claude
test: mark TestParallelTransformerBlockCudagraphs::test_gpu_cudagraph…
8c0aa6c 
… flaky_in_dev (NVIDIA#5475)

Signed-off-by: oliver könig <okoenig@nvidia.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@ko3n1g @claude
test: mark gated_delta_net selective-recompute test flaky_in_dev (NVI…
6c6ce4e 
…DIA#5476)

Signed-off-by: oliver könig <okoenig@nvidia.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions @claude
fix: post-CI corrections for sync test/impl reconciliation
cc083e1 
1) test_optimizer.py: reverted to dev. The sync kept dev's
   multi_latent_attention.py (split q/kv down-proj, no
   _synthesize_fused_qkv_down_weight), but auto-merged main's
   test asserting the fused linear_qkv_down_proj.weight key.

2) training.py: guard the dev-only sequence_packing_scheduler config
   access with getattr (lines in train_step and train()). main's new
   MIMO schedule-plumbing test (NVIDIA#5333) passes an empty SimpleNamespace
   config; the reconciled training.py keeps dev's packing path, so the
   access must tolerate a config lacking the attribute. Real configs are
   unaffected (getattr returns the same value).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
@xiaoyao0115
fix sequence packing wrapper for eval (NVIDIA#5483)
15486f7 
Signed-off-by: xiaoyao0115 <1804647152@qq.com>
@ISEEKYAN
[dev] Megatron Lite (4/4) shared attention (NVIDIA#5427)
4463f78 
Signed-off-by: Yan Bai <bayan@nvidia.com>
@lhb8125
fix: restore fused group MLP offload in main2dev sync (NVIDIA#5493)
f6c77e7 
Signed-off-by: hongbinl <hongbinl@nvidia.com>
Signed-off-by: svcnvidia-nemo-ci <svc-nvidia-nemo-ci@nvidia.com>
@chtruong814 @tdene
@AAnoosheh @jalbericiola @pthombre @claude @janEbert @Phlip79 @mathemakitten @yashaswikarnati @github-actions @Wong4j @balasaajay @ko3n1g @asolergi-nv @deepakn94 @ilml @santhnm2 @sraman-rgb @wujingyue @HollowMan6 @lhb8125 @lmcafee-nvidia @wdykas
chore: nightly sync main into dev (22_06_2026) (NVIDIA#5430)
72cef25 
Signed-off-by: Teodor-Dumitru Ene <teodord.ene@gmail.com>
Signed-off-by: Asha Anoosheh <aanoosheh@nvidia.com>
Signed-off-by: Pranav Prashant Thombre <pthombre@nvidia.com>
Signed-off-by: janEbert <janpabloe@nvidia.com>
Signed-off-by: Philip Petrakian <ppetrakian@nvidia.com>
Signed-off-by: Helen Ngo <helenn@nvidia.com>
Signed-off-by: ykarnati <ykarnati@nvidia.com>
Signed-off-by: Shijie Wang <jaywan@nvidia.com>
Signed-off-by: Ajay Balasa <abalasa@nvidia.com>
Signed-off-by: oliver könig <okoenig@nvidia.com>
Signed-off-by: Antoni-Joan Solergibert <asolergibert@nvidia.com>
Signed-off-by: ilml <tolong@nvidia.com>
Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com>
Signed-off-by: sraman <sraman@nvidia.com>
Signed-off-by: Jingyue Wu <wujingyue@gmail.com>
Signed-off-by: Hollow Man <hollowman@opensuse.org>
Signed-off-by: hongbinl <hongbinl@nvidia.com>
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: Lawrence McAfee <lmcafee@nvidia.com>
Signed-off-by: wdykas <wdykas@nvidia.com>
Signed-off-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Signed-off-by: svcnvidia-nemo-ci <svc-nvidia-nemo-ci@nvidia.com>
Co-authored-by: Teodor-Dumitru Ene <34819528+tdene@users.noreply.github.com>
Co-authored-by: Asha Anoosheh <aanoosheh@nvidia.com>
Co-authored-by: Jorge Albericio <jalbericiola@nvidia.com>
Co-authored-by: Pranav Thombre <pthombre@nvidia.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: janEbert <janpabloe@nvidia.com>
Co-authored-by: Philip Petrakian <ppetrakian@nvidia.com>
Co-authored-by: mathemakitten <helenn@nvidia.com>
Co-authored-by: Yashaswi Karnati <144376261+yashaswikarnati@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Shijie <505749828@qq.com>
Co-authored-by: Ajay <abalasa@nvidia.com>
Co-authored-by: oliver könig <okoenig@nvidia.com>
Co-authored-by: Antoni-Joan Solergibert <asolergibert@nvidia.com>
Co-authored-by: Deepak Narayanan <dnarayanan@nvidia.com>
Co-authored-by: Tom Long <tolong@nvidia.com>
Co-authored-by: Keshav Santhanam <ksanthanam@nvidia.com>
Co-authored-by: Teodor-Dumitru Ene <teodord.ene@gmail.com>
Co-authored-by: Siddhartha Raman Sundara Raman <sraman@nvidia.com>
Co-authored-by: Jingyue Wu <wujingyue@gmail.com>
Co-authored-by: ℍ𝕠𝕝𝕝𝕠𝕨 𝕄𝕒𝕟 <hollowman@opensuse.org>
Co-authored-by: Hongbin Liu <lhb8125@users.noreply.github.com>
Co-authored-by: Charlie Truong <chtruong@nvidia.com>
Co-authored-by: Lawrence McAfee <85179052+lmcafee-nvidia@users.noreply.github.com>
Co-authored-by: wdykas <73254672+wdykas@users.noreply.github.com>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
@hxbai
[dev] [DeepSeek-v4] Packed Sequence (THD) support for DSv4 Hybrid Att…
056d9c0 
…ention (NVIDIA#5011)

Signed-off-by: Hongxiao Bai <hongxiaob@nvidia.com>
@xiaoyao0115
Improve default dynamic CP packing scheduler (NVIDIA#5154)
d2e7ec5 
Signed-off-by: tailaim <tailaim@nvidia.com>
@yuzhongw-nvidia
[dev] moe(perf): Pre-GDR kernel fusion (NVIDIA#5361)
d963266 
Signed-off-by: Yuzhong Wang <yuzhongw@nvidia.com>
@kunlunl @shyoshyo
Preserve DSA output across fused inverse RoPE (NVIDIA#5526)
04bbfe6 
Signed-off-by: kunlunl <kunlunl@nvidia.com>
Co-authored-by: Kaixiang Lei <5780122+shyoshyo@users.noreply.github.com>
@guihong-nv
Enable Deepseek-v4 hybrid_model in dev branch Part (2/N) (NVIDIA#5485)
3e89f3c 
Signed-off-by: guihong-nv <guihongl@nvidia.com>
@Wohox @claude
[dev] Add experimental decoupled compact LayerWise DDP layout for Muon (
3b9c248 
NVIDIA#5388)

Signed-off-by: pingtianl <pingtianl@nvidia.com>
Signed-off-by: Pingtian Li <pingtianl@nvidia.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@ISEEKYAN
[dev] Sync Megatron Lite with the latest implementation (NVIDIA#5577)
5e846ab 
Signed-off-by: Yan Bai <bayan@nvidia.com>
@HaochenYuan
[Dev] fix padding mask docstring (NVIDIA#5598)
66dc96f 
Signed-off-by: HaochenYuan <haocheny@nvidia.com>
@yuzhongw-nvidia
Add GDN context parallel communication modes
6031238 
Signed-off-by: Yuzhong Wang <yuzhongw@nvidia.com>
@yuzhongw-nvidia
Make GDN conv padding configurable
d749b6c 
Signed-off-by: Yuzhong Wang <yuzhongw@nvidia.com>
@yuzhongw-nvidia
Refactor and rename GDN CP layout handling
d8b0afa 
Signed-off-by: Yuzhong Wang <yuzhongw@nvidia.com>
@yuzhongw-nvidia
Address GDN CP review guards
df90299 
Signed-off-by: Yuzhong Wang <yuzhongw@nvidia.com>
@yuzhongw-nvidia
Fix THD contiguous CP rank partition
098d7d9 
Signed-off-by: Yuzhong Wang <yuzhongw@nvidia.com>
@yuzhongw-nvidia
Require flash-linear-attention CP API
3d39e26 
Signed-off-by: Yuzhong Wang <yuzhongw@nvidia.com>
@yuzhongw-nvidia
Document THD CP routing future work
a9fad1c 
Signed-off-by: Yuzhong Wang <yuzhongw@nvidia.com>
@yuzhongw-nvidia
Update Mamba MoE config golden
50a0a89 
Signed-off-by: Yuzhong Wang <yuzhongw@nvidia.com>
@yuzhongw-nvidia
fix(thd): use padded boundaries for TE attention
0beec47 
Signed-off-by: Yuzhong Wang <yuzhongw@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Jul 3, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.