Optimize Span operations, especially the hot `Span::to` by joshtriplett · Pull Request #163971 · rust-lang/rust

joshtriplett · 2026-10-08T07:18:17Z

This stack of changes incrementally optimizes Span operations, and in particular Span::to, until at the end Span::to has a hot path that runs entirely via registers with no stack frame.

I recommend reviewing commit by commit, ignoring the first commit.

This is based on @nnethercote's PR overhauling span compression, because some of it depends on that, either directly or just because all of my benchmarking happened atop that.

In Span::prepare_to_combine, skip calling source_callsite() as it can't help.

prepare_to_combine, as its last fallback, checked a.ctxt == b.span().source_callsite().ctxt(). Immediately prior to that, it has a fast-path of checking a.ctxt.is_root().

source_callsite() recurses until it finds a root context. So, if it returns at all (rather than panicking), it'll return a root context. This means a.ctxt == b.span().source_callsite().ctxt() cannot be true unless a.ctxt.is_root() was already true. So, the call to source_callsite will do a pile of work but not change the outcome.

tt-muncher hits this case, and recurses deeply into it.

Remove this case, and let a.ctxt.is_root() be the only check here.

Add a Span::new_ordered constructor that skips the lo > hi check.

Valid spans have lo <= hi, so if we know we got the values from an already valid span, we don't need to recheck them. This eliminates a check and conditional move in to, which is a hot path.

Rework Span::try_metavars and Span::prepare_to_combine to reduce spilling.

Rather than passing in full decoded SpanData to the try_metavars helper, just pass in contexts, and get back Options for optionally replacing the corresponding span. This allows the caller to avoid materializing the decoded SpanData on the stack; the spill showed up in profiles.

As a minor rework, reorder Span::prepare_to_combine to avoid redundant conditions.

After the previous change, if try_metavars returns (None, None), we definitely won't have matching contexts and we need the error case. Otherwise, we definitely have matching contexts (since that led try_metavars to return one or both Some values), so we don't need to recheck that.

Span::to: Add a fast path where both spans are inline

InlineCtxt is by far the most common case. Checking for that case is fast (a prefix check on a span). And in that case, we know we don't have parents, we can do a quick comparison of the contexts, and we know the context will fit back in an inline span.

As a result, this fast path manages to entirely skip setting up a stack frame or saving registers.

Currently there are span representations if you have a context and no parent, or a parent and no context. But if you have both a context *and* a parent, the span is always interned. In incremental builds this is something like 20-40% of all spans. This commit uses a new compression scheme that gives small perf wins for non-incremental and bigger perf wins for incremental. The following table briefly summarizes the formats. ``` ----------------------------------------------------------------- old new ----------------------------------------------------------------- InlineCtxt lo32,len15,ctxt16,parent0 lo32,len15,ctxt16,parent0 InlineParent lo32,len15,ctxt0,parent16 lo32,len14,ctxt0,parent15 InlinePair n/a lo24,len7,ctxt15,parent16 PartialInt. ctxt16 everything else FullInt. everything else n/a ``` Things to note. - The old scheme had a 32-bit field and two 16-bit fields. The new scheme has one 64-bit field and uses `repr(packed(4))`. (Keeping the alignment at 4 is important to keep AST node sizes low.) - The new scheme distinguishes the four formats with a single variable-length prefix in the high bits. This is simpler than the old combination of tag bits and special values. - The new `SpanField` type is used to describe the various field widths for each format, and provide operations on them. - The new `InlineParent` is very slightly worse than the old one, with one less bit for each of len and parent. This doesn't affect many spans. - The new `InlinePair` format handles most cases where both a context and a parent are present. It has to compromise some on `lo` and `len`, but still gets a lot of them. This is the big performance win. - The partially/fully interned cases are merged, because the Interned format can now always hold a full context inline, and the new `Interned` is equivalent to the old PartiallyInterned format. (This requires limiting ctxts to 29 bits, but we'll hit other problems, such as OOM, long before that limit is reached.) This also means we don't need to store the ctxt field in the interner, and we use the new `SpanDataNoCtxt` type for that.

…t help `prepare_to_combine`, as its last fallback, checked `a.ctxt == b.span().source_callsite().ctxt()`. Immediately prior to that, it has a fast-path of checking `a.ctxt.is_root()`. `source_callsite()` recurses until it finds a root context. So, if it returns at all (rather than panicking), it'll return a root context. This means `a.ctxt == b.span().source_callsite().ctxt()` cannot be true unless `a.ctxt.is_root()` was already true. So, the call to source_callsite will do a pile of work but not change the outcome. tt-muncher hits this case, and recurses deeply into it. Remove this case, and let `a.ctxt.is_root()` be the only check here. This saves more than 2% instructions on tt-muncher incr-full and more on incr-unchanged.

Valid spans have `lo <= hi`, so if we know we got the values from an already valid span, we don't need to recheck them. This eliminates a check and conditional move in `to`, which is a hot path.

Rather than passing in full decoded SpanData to the `try_metavars` helper, just pass in contexts, and get back `Option`s for optionally replacing the corresponding span. This allows the caller to avoid materializing the decoded SpanData on the stack.

If `try_metavars` returns `(None, None)`, we definitely won't have matching contexts and we need the error case. Otherwise, we definitely have matching contexts (since that led `try_metavars` to return one or both `Some` values), so we don't need to recheck that.

`Span::to` is a hot path. `InlineCtxt` is by far the most common case. Checking for that case is fast (a prefix check on a span). And in that case, we know we don't have parents, we can do a quick comparison of the contexts, and we know the context will fit back in an inline span. As a result, this fast path manages to entirely skip setting up a stack frame or saving registers.

joshtriplett · 2026-10-08T07:20:02Z

Benchmarking this with @nnethercote's PR as the base:

@bors try parent=d2adbd4acd6d3b565c7a003a9fc9439c67d42cf6 @rust-timer queue

joshtriplett · 2026-10-08T07:20:53Z

Correct hash this time:

@bors try parent=03aba7c9e40d02c0476b996cdc81cbe1bd24e247 @rust-timer queue

Optimize Span operations, especially the hot `Span::to`

rust-bors · 2026-10-08T08:20:24Z

☀️ Try build successful (CI)
Build commit: 045d996 (045d996ac17ca1d63bf50530245ed323e2a5dd18)
Base parent: 03aba7c (03aba7c9e40d02c0476b996cdc81cbe1bd24e247)

rust-timer · 2026-10-08T09:06:20Z

Finished benchmarking commit (045d996): comparison URL.

Overall result: ✅ improvements - no action needed

Benchmarking means the PR may be perf-sensitive. It's automatically marked not fit for rolling up. Overriding is possible but disadvised: it risks changing compiler perf.

@bors rollup=never rustc-perf
@rustbot label: -S-waiting-on-perf -perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	0.6%	[0.6%, 0.7%]	6
Improvements ✅ (primary)	-0.5%	[-1.2%, -0.3%]	138
Improvements ✅ (secondary)	-0.8%	[-3.0%, -0.3%]	79
All ❌✅ (primary)	-0.5%	[-1.2%, -0.3%]	138

Max RSS (memory usage)

Results (primary -0.4%, secondary -0.4%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	0.8%	[0.4%, 2.2%]	24
Regressions ❌ (secondary)	0.8%	[0.4%, 2.8%]	47
Improvements ✅ (primary)	-0.9%	[-2.8%, -0.4%]	56
Improvements ✅ (secondary)	-1.1%	[-6.1%, -0.4%]	85
All ❌✅ (primary)	-0.4%	[-2.8%, 2.2%]	80

Cycles

Results (primary -0.2%, secondary -0.5%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	0.9%	[0.4%, 5.0%]	72
Regressions ❌ (secondary)	1.0%	[0.4%, 6.0%]	121
Improvements ✅ (primary)	-1.0%	[-6.3%, -0.4%]	89
Improvements ✅ (secondary)	-2.4%	[-14.2%, -0.4%]	103
All ❌✅ (primary)	-0.2%	[-6.3%, 5.0%]	161

Binary size

This perf run didn't have relevant results for this metric.

Bootstrap: 492.261s -> 488.235s (-0.82%)
Artifact size: 408.66 MiB -> 406.28 MiB (-0.58%)

joshtriplett · 2026-10-08T09:18:42Z

r? nnethercote

nnethercote

Mostly looks good. A couple of minor things to look at.

View changes since this review

… latter

joshtriplett · 2026-10-09T07:58:47Z

@rustbot ready

petrochenkov · 2026-10-12T15:38:37Z

Blocked on #163857.
@rustbot blocked

nnethercote and others added 6 commits October 7, 2026 14:06

Span: Add a new_ordered constructor that skips the lo > hi check

4793d75

Valid spans have `lo <= hi`, so if we know we got the values from an already valid span, we don't need to recheck them. This eliminates a check and conditional move in `to`, which is a hot path.

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Oct 8, 2026

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Oct 8, 2026

This comment has been minimized.

Sign in to view

rust-bors Bot pushed a commit that referenced this pull request Oct 8, 2026

Auto merge of #163971 - joshtriplett:span-optimization, r=<try>

045d996

Optimize Span operations, especially the hot `Span::to`

This comment has been minimized.

Sign in to view

rustbot removed the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Oct 8, 2026

rustbot assigned nnethercote Oct 8, 2026

joshtriplett marked this pull request as ready for review October 8, 2026 09:18

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Oct 8, 2026

petrochenkov self-assigned this Oct 8, 2026

nnethercote reviewed Oct 8, 2026

View reviewed changes

Comment thread compiler/rustc_span/src/span_encoding.rs

Comment thread compiler/rustc_span/src/lib.rs Outdated

nnethercote added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Oct 8, 2026

Span: Add documentation on new and new_ordered; steer towards the…

ef0c660

… latter

Span::new_ordered: Add a debug_assert! on lo <= hi

5acb8ca

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Oct 9, 2026

rustbot added S-blocked Status: Blocked on something else such as an RFC or other implementation work. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Oct 12, 2026

Navigation Menu

Repository navigation

Optimize Span operations, especially the hot `Span::to` - #163971

Optimize Span operations, especially the hot `Span::to`#163971
joshtriplett wants to merge 8 commits into
rust-lang:mainfrom
joshtriplett:span-optimization

joshtriplett commented Oct 8, 2026 •

edited

Loading

Uh oh!

joshtriplett commented Oct 8, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

joshtriplett commented Oct 8, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

rust-bors Bot commented Oct 8, 2026

Uh oh!

This comment has been minimized.

rust-timer commented Oct 8, 2026

Uh oh!

joshtriplett commented Oct 8, 2026

Uh oh!

nnethercote left a comment •

edited by rustbot

Loading

Uh oh!

Uh oh!

Uh oh!

joshtriplett commented Oct 9, 2026

Uh oh!

petrochenkov commented Oct 12, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

joshtriplett commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

joshtriplett commented Oct 8, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

joshtriplett commented Oct 8, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

rust-bors Bot commented Oct 8, 2026

Uh oh!

This comment has been minimized.

rust-timer commented Oct 8, 2026

Overall result: ✅ improvements - no action needed

Instruction count

Max RSS (memory usage)

Cycles

Binary size

Uh oh!

joshtriplett commented Oct 8, 2026

Uh oh!

nnethercote left a comment • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

joshtriplett commented Oct 9, 2026

Uh oh!

petrochenkov commented Oct 12, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

joshtriplett commented Oct 8, 2026 •

edited

Loading

nnethercote left a comment •

edited by rustbot

Loading