Skip to content

Route A=4 all-to-all through helpers in the 63-256 MiB window #4572

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
srinathb-meta wants to merge 19 commits into
base: main
Choose a base branch
from
Open

Route A=4 all-to-all through helpers in the 63-256 MiB window #4572

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
5b3c5fc
Add sharded relay reduce-scatter C++ implementation
Aug 17, 2026
d061500
Add sharded relay reduce-scatter distributed utilities
Aug 17, 2026
ff71ee5
Add sharded relay reduce-scatter tests and benchmarks
Aug 17, 2026
6d83a90
Add sharded relay all-to-all distributed utilities
Aug 17, 2026
6d3695d
Add sharded relay all-to-all tests and benchmarks
Aug 17, 2026
0dbc459
Add sharded relay all-gather distributed utilities
Aug 17, 2026
4b6d208
Add sharded relay all-gather tests and benchmarks
Aug 17, 2026
f9e5135
Enable 4-active sharded relay allreduce in the distributed layer
Aug 17, 2026
ba0ae82
Add 4-active sharded relay allreduce tests and benchmark sweep
Aug 17, 2026
cc6b2e6
Add 4-active sharded relay reduce-scatter tests and benchmark sweep
Aug 17, 2026
132f2b1
Add 4-active sharded relay all-to-all tests, util sizing, and benchma…
Aug 17, 2026
d63ec3e
Size 4-active all-gather helper buffers and document 2-or-4 support
Aug 17, 2026
56d0fb0
Add 4-active sharded relay all-gather tests and benchmark sweep
Aug 17, 2026
7812177
Route the allreduce util through the shared flat pack/unpack helpers
Aug 17, 2026
7794336
Expose out-of-place allreduce through the sharded-relay Python API
Aug 17, 2026
a99b516
Add sharded-relay message-size sweep for all collectives (2- & 4-rank)
Aug 17, 2026
b742d88
Add parallel and single-group relay collective sweeps + file-based re…
Aug 17, 2026
c2ca39a
Link-balance the sharded relay schedules (up to 3.2x vs NCCL)
Aug 17, 2026
ec23323
Route A=4 all-to-all through helpers in the 63-256 MiB window
Aug 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Loading