Overhaul span compression by nnethercote · Pull Request #163857 · rust-lang/rust

nnethercote · 2026-10-06T04:22:45Z

Currently there are span representations if you have a context and no parent, or a parent and no context. But if you have both a context and a parent, the span is always interned. In incremental builds this is something like 20-40% of all spans.

This commit uses a new compression scheme that gives small perf wins for non-incremental and bigger perf wins for incremental. The following table briefly summarizes the formats.

-----------------------------------------------------------------
              old                        new
-----------------------------------------------------------------
InlineCtxt    lo32,len15,ctxt16,parent0  lo32,len15,ctxt16,parent0
InlineParent  lo32,len15,ctxt0,parent16  lo32,len14,ctxt0,parent15
InlinePair    n/a                        lo24,len7,ctxt15,parent16
PartialInt.   ctxt16                     everything else
FullInt.      everything else            n/a

Things to note.

The old scheme had a 32-bit field and two 16-bit fields. The new scheme has one 64-bit field and uses repr(packed(4)). (Keeping the alignment at 4 is important to keep AST node sizes low.)
The new scheme distinguishes the four formats with a single variable-length prefix in the high bits. This is simpler than the old combination of tag bits and special values.
The new SpanField type is used to describe the various field widths for each format, and provide operations on them.
The new InlineParent is very slightly worse than the old one, with one less bit for each of len and parent. This doesn't affect many spans.
The new InlinePair format handles most cases where both a context and a parent are present. It has to compromise some on lo and len, but still gets a lot of them. This is the big performance win.
The partially/fully interned cases are merged, because the Interned format can now always hold a full context inline, and the new Interned is equivalent to the old PartiallyInterned format. (This requires limiting ctxts to 29 bits, but we'll hit other problems, such as OOM, long before that limit is reached.) This also means we don't need to store the ctxt field in the interner, and we use the new SpanDataNoCtxt type for that.

r? @petrochenkov

nnethercote · 2026-10-06T04:23:42Z

@bors try @rust-timer queue

Overhaul span compression

rust-bors · 2026-10-06T05:28:18Z

☀️ Try build successful (CI)
Build commit: d2adbd4 (d2adbd4acd6d3b565c7a003a9fc9439c67d42cf6)
Base parent: b57eb9a (b57eb9a5fd94f30697f51fc88fbe898e67c7aae6)

rust-timer · 2026-10-06T06:11:41Z

Finished benchmarking commit (d2adbd4): comparison URL.

Overall result: ❌✅ regressions and improvements - please read:

Benchmarking means the PR may be perf-sensitive. It's automatically marked not fit for rolling up. Overriding is possible but disadvised: it risks changing compiler perf.

Next, please: If you can, justify the regressions found in this try perf run in writing along with @rustbot label: +perf-regression-triaged. If not, fix the regressions and do another perf run. Neutral or positive results will clear the label automatically.

@bors rollup=never rustc-perf
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	0.9%	[0.5%, 1.5%]	16
Improvements ✅ (primary)	-1.0%	[-4.0%, -0.1%]	261
Improvements ✅ (secondary)	-0.8%	[-5.6%, -0.1%]	244
All ❌✅ (primary)	-1.0%	[-4.0%, -0.1%]	261

Max RSS (memory usage)

Results (primary -1.5%, secondary -1.1%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	2.5%	[2.3%, 2.9%]	3
Improvements ✅ (primary)	-1.5%	[-3.3%, -0.5%]	32
Improvements ✅ (secondary)	-2.2%	[-4.1%, -1.0%]	10
All ❌✅ (primary)	-1.5%	[-3.3%, -0.5%]	32

Cycles

Results (primary -3.5%, secondary -3.5%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	4.1%	[3.9%, 4.4%]	2
Improvements ✅ (primary)	-3.5%	[-9.8%, -1.6%]	71
Improvements ✅ (secondary)	-4.1%	[-9.5%, -1.7%]	23
All ❌✅ (primary)	-3.5%	[-9.8%, -1.6%]	71

Binary size

This perf run didn't have relevant results for this metric.

Bootstrap: 489.693s -> 492.198s (0.51%)
Artifact size: 409.33 MiB -> 408.67 MiB (-0.16%)

nnethercote · 2026-10-06T08:41:30Z

icounts results look good. Cycles results are even better, probably because max-rss/faults/cache-misses/branch-misses are all improved.

nnethercote · 2026-10-06T08:43:08Z

LLM disclosure: I used an LLM for some of the ideas and analysis in this PR. I wrote the code and text myself.

nnethercote · 2026-10-07T03:12:57Z

I made some small changes that shouldn't affect perf, let's check:

@bors try @rust-timer queue

Overhaul span compression

rust-bors · 2026-10-07T04:12:13Z

☀️ Try build successful (CI)
Build commit: 03aba7c (03aba7c9e40d02c0476b996cdc81cbe1bd24e247)
Base parent: 8d1a764 (8d1a76430406c877b35d0b627e7f796dcf0dfeca)

rust-timer · 2026-10-07T05:26:13Z

Finished benchmarking commit (03aba7c): comparison URL.

Overall result: ❌✅ regressions and improvements - please read:

Benchmarking means the PR may be perf-sensitive. It's automatically marked not fit for rolling up. Overriding is possible but disadvised: it risks changing compiler perf.

Next, please: If you can, justify the regressions found in this try perf run in writing along with @rustbot label: +perf-regression-triaged. If not, fix the regressions and do another perf run. Neutral or positive results will clear the label automatically.

@bors rollup=never rustc-perf
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	0.9%	[0.5%, 1.6%]	16
Improvements ✅ (primary)	-1.0%	[-4.0%, -0.1%]	262
Improvements ✅ (secondary)	-0.8%	[-5.5%, -0.1%]	250
All ❌✅ (primary)	-1.0%	[-4.0%, -0.1%]	262

Max RSS (memory usage)

Results (primary -1.3%, secondary -1.0%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	2.4%	[2.4%, 2.4%]	1
Improvements ✅ (primary)	-1.3%	[-2.5%, -0.5%]	34
Improvements ✅ (secondary)	-1.7%	[-2.2%, -1.2%]	5
All ❌✅ (primary)	-1.3%	[-2.5%, -0.5%]	34

Cycles

Results (primary -3.6%, secondary -0.6%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	2.4%	[2.0%, 2.9%]	2
Regressions ❌ (secondary)	5.3%	[2.8%, 8.0%]	8
Improvements ✅ (primary)	-3.8%	[-10.0%, -1.6%]	58
Improvements ✅ (secondary)	-3.1%	[-6.7%, -1.9%]	19
All ❌✅ (primary)	-3.6%	[-10.0%, 2.9%]	60

Binary size

Results (primary -0.1%, secondary -0.1%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-0.1%	[-0.2%, -0.0%]	59
Improvements ✅ (secondary)	-0.1%	[-0.2%, -0.0%]	53
All ❌✅ (primary)	-0.1%	[-0.2%, -0.0%]	59

Bootstrap: 489.893s -> 492.261s (0.48%)
Artifact size: 408.60 MiB -> 408.66 MiB (0.02%)

nnethercote · 2026-10-07T06:12:13Z

New perf results are basically identical to the old ones.

joshtriplett · 2026-10-08T05:03:28Z

In addition to the performance optimization, the new design also seems much more understandable, and it's proving to be a good base to build on for further optimizations.

nnethercote · 2026-10-08T10:37:33Z

I did some rough measurements today.

In non-incremental builds InlineCtxt is used for approximately 100% of spans, and a tiny number become Interned.
In incremental builds typically ~1% of spans become Interned, down from 15-30%.

petrochenkov · 2026-10-12T15:06:22Z

-    len_with_tag_or_marker: u16,
-    ctxt_or_parent_or_marker: u16,
-}
+#[repr(packed(4))]


Does #[repr(packed(1))] improve anything further?
Since we are using unaligned accesses anyway, why not try reducing the span-containing types further (probably not in this PR though).
IIRC, we tried this in the past with the old encoding, and it didn't change much.

View changes since the review

I tried it, it's not worth it.

Instruction counts on benchmarks were slightly worse (< 1%) on some benchmarks.

librustc_driver.so size increased by 1.28 MiB.

It shrunk a few AST and HIR nodes, but none of the important ones. Here's the diff:

diff --git a/compiler/rustc_ast/src/ast.rs b/compiler/rustc_ast/src/ast.rs index 4c7ae87556f..9d932c7df62 100644 --- a/compiler/rustc_ast/src/ast.rs +++ b/compiler/rustc_ast/src/ast.rs @@ -4473,15 +4473,15 @@ mod size_asserts { static_assert_size!(Expr, 64); static_assert_size!(ExprKind, 32); static_assert_size!(FieldDef, 80); - static_assert_size!(Fn, 192); + static_assert_size!(Fn, 184); static_assert_size!(FnDecl, 24); - static_assert_size!(FnHeader, 76); - static_assert_size!(FnSig, 96); + static_assert_size!(FnHeader, 72); + static_assert_size!(FnSig, 88); static_assert_size!(ForeignItem, 72); static_assert_size!(ForeignItemKind, 16); static_assert_size!(GenericArg, 24); static_assert_size!(GenericArgs, 40); - static_assert_size!(GenericBound, 80); + static_assert_size!(GenericBound, 72); static_assert_size!(GenericParam, 80); static_assert_size!(Generics, 40); static_assert_size!(Impl, 80); @@ -4489,7 +4489,7 @@ mod size_asserts { static_assert_size!(ItemKind, 88); static_assert_size!(Lifetime, 16); static_assert_size!(LitKind, 24); - static_assert_size!(Local, 96); + static_assert_size!(Local, 88); static_assert_size!(MetaItem, 80); static_assert_size!(MetaItemKind, 40); static_assert_size!(MetaItemLit, 40); @@ -4502,7 +4502,7 @@ mod size_asserts { static_assert_size!(QSelf, 24); static_assert_size!(Stmt, 32); static_assert_size!(StmtKind, 16); - static_assert_size!(TraitImplHeader, 64); + static_assert_size!(TraitImplHeader, 56); static_assert_size!(Ty, 56); static_assert_size!(TyKind, 40); // tidy-alphabetical-end diff --git a/compiler/rustc_hir/src/hir.rs b/compiler/rustc_hir/src/hir.rs index dfcb8871fe9..1f668e7007b 100644 --- a/compiler/rustc_hir/src/hir.rs +++ b/compiler/rustc_hir/src/hir.rs @@ -5299,7 +5299,7 @@ mod size_asserts { static_assert_size!(Res, 12); static_assert_size!(Stmt<'_>, 32); static_assert_size!(StmtKind<'_>, 16); - static_assert_size!(TraitImplHeader<'_>, 48); + static_assert_size!(TraitImplHeader<'_>, 40); static_assert_size!(TraitItem<'_>, 88); static_assert_size!(TraitItemKind<'_>, 48); static_assert_size!(Ty<'_>, 48);

Currently there are span representations if you have a context and no parent, or a parent and no context. But if you have both a context *and* a parent, the span is always interned. In incremental builds this is something like 20-40% of all spans. This commit uses a new compression scheme that gives small perf wins for non-incremental and bigger perf wins for incremental. The following table briefly summarizes the formats. ``` ----------------------------------------------------------------- old new ----------------------------------------------------------------- InlineCtxt lo32,len15,ctxt16,parent0 lo32,len15,ctxt16,parent0 InlineParent lo32,len15,ctxt0,parent16 lo32,len14,ctxt0,parent15 InlinePair n/a lo24,len7,ctxt15,parent16 PartialInt. ctxt16 everything else FullInt. everything else n/a ``` Things to note. - The old scheme had a 32-bit field and two 16-bit fields. The new scheme has one 64-bit field and uses `repr(packed(4))`. (Keeping the alignment at 4 is important to keep AST node sizes low.) - The new scheme distinguishes the four formats with a single variable-length prefix in the high bits. This is simpler than the old combination of tag bits and special values. - The new `SpanField` type is used to describe the various field widths for each format, and provide operations on them. - The new `InlineParent` is very slightly worse than the old one, with one less bit for each of len and parent. This doesn't affect many spans. - The new `InlinePair` format handles most cases where both a context and a parent are present. It has to compromise some on `lo` and `len`, but still gets a lot of them. This is the big performance win. - The partially/fully interned cases are merged, because the Interned format can now always hold a full context inline, and the new `Interned` is equivalent to the old PartiallyInterned format. (This requires limiting ctxts to 29 bits, but we'll hit other problems, such as OOM, long before that limit is reached.) This also means we don't need to store the ctxt field in the interner, and we use the new `SpanDataNoCtxt` type for that.

rustbot · 2026-10-12T23:06:20Z

This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

nnethercote · 2026-10-12T23:08:18Z

I addressed all the suggestions except for repr(packed(1)), as explained above.

@rustbot ready

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Oct 6, 2026

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Oct 6, 2026

This comment has been minimized.

Sign in to view

rust-bors Bot pushed a commit that referenced this pull request Oct 6, 2026

Auto merge of #163857 - nnethercote:overhaul-span-compression, r=<try>

d2adbd4

Overhaul span compression

This comment has been minimized.

Sign in to view

rustbot added perf-regression Performance regression. and removed S-waiting-on-perf Status: Waiting on a perf run to be completed. labels Oct 6, 2026

bjorn3 reviewed Oct 6, 2026

View reviewed changes

Comment thread compiler/rustc_data_structures/src/stable_hash.rs Outdated

nnethercote force-pushed the overhaul-span-compression branch from 302cea2 to 1596682 Compare October 6, 2026 08:43

rustbot assigned petrochenkov Oct 6, 2026

nnethercote marked this pull request as ready for review October 6, 2026 08:45

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Oct 6, 2026

nnethercote force-pushed the overhaul-span-compression branch 2 times, most recently from 7d10c65 to e40a818 Compare October 7, 2026 03:07

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Oct 7, 2026

This comment has been minimized.

Sign in to view

rust-bors Bot pushed a commit that referenced this pull request Oct 7, 2026

Auto merge of #163857 - nnethercote:overhaul-span-compression, r=<try>

03aba7c

Overhaul span compression

This comment has been minimized.

Sign in to view

rustbot removed the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Oct 7, 2026

joshtriplett reviewed Oct 8, 2026

View reviewed changes

Comment thread compiler/rustc_data_structures/src/stable_hash.rs Outdated

joshtriplett mentioned this pull request Oct 8, 2026

Optimize Span operations, especially the hot Span::to #163971

Open

petrochenkov reviewed Oct 12, 2026

View reviewed changes

petrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Oct 12, 2026

nnethercote force-pushed the overhaul-span-compression branch from e40a818 to d25eebb Compare October 12, 2026 23:06

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Oct 12, 2026

Conversation

nnethercote commented Oct 6, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nnethercote commented Oct 6, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

rust-bors Bot commented Oct 6, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

rust-timer commented Oct 6, 2026

Overall result: ❌✅ regressions and improvements - please read:

Uh oh!

Uh oh!

nnethercote commented Oct 6, 2026

Uh oh!

nnethercote commented Oct 6, 2026

Uh oh!

nnethercote commented Oct 7, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

rust-bors Bot commented Oct 7, 2026

Uh oh!

This comment has been minimized.

rust-timer commented Oct 7, 2026

Overall result: ❌✅ regressions and improvements - please read:

Uh oh!

nnethercote commented Oct 7, 2026

Uh oh!

Uh oh!

joshtriplett commented Oct 8, 2026

Uh oh!

nnethercote commented Oct 8, 2026

Uh oh!

petrochenkov Oct 12, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rustbot commented Oct 12, 2026

Uh oh!

nnethercote commented Oct 12, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

nnethercote commented Oct 6, 2026 •

edited by rustbot

Loading

petrochenkov Oct 12, 2026 •

edited by rustbot

Loading