4000% performance regression with "-C target-cpu=x86-64-v3" and fat LTO

[im-0]

The problem:

bench_mat4_transform_point3
                        time:   [18.883 ns 18.986 ns 19.093 ns]
                        change: [+4284.5% +4303.0% +4320.2%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 4 outliers among 100 measurements (4.00%)
  3 (3.00%) high mild
  1 (1.00%) high severe

I encountered this issue while working on benchmarks from nalgebra crate after recompiling it with RUSTFLAGS="-C target-cpu=native".

Here is the smaller reproducer with more details in README: https://github.com/im-0/rust-fat-lto-perf-degradation (it still uses nalgebra and criterion as dependencies). This reproducer requires codegen-units = 32, but some benchmarks from nalgebra suffer even with just lto = "fat" and default value of codegen-units (which is 16).

I expected to see this happen: no or small performance degradation

Instead, this happened: more than 4000% performance degradation

Meta

rustc --version --verbose:

rustc 1.89.0 (29483883e 2025-08-04) (Fedora 1.89.0-2.fc42)
binary: rustc
commit-hash: 29483883eed69d5fb4db01964cdf2af4d86e9cb2
commit-date: 2025-08-04
host: x86_64-unknown-linux-gnu
release: 1.89.0
LLVM version: 20.1.8

and

rustc 1.89.0 (29483883e 2025-08-04)
binary: rustc
commit-hash: 29483883eed69d5fb4db01964cdf2af4d86e9cb2
commit-date: 2025-08-04
host: x86_64-unknown-linux-gnu
release: 1.89.0
LLVM version: 20.1.7

[workingjubilee]

uh.
that's... unexpected.
could you tell what CPU you benched on?
Ryzen 9950X if one digs into the link. Interesting.

[im-0]

Just tested on i7-8565U 😖

(full output is in my reproducer repo)

I believe this is reproducible on all relatively modern x86_64 CPUs.

[edwloef]

Even worse on my machine 😆 I'm running an i7-13700H

[zamazan4ik]

Also reproduced it locally with AMD Ryzen 9 5900x, Fedora 42, Rustc Stable 1.90 + Rustc Nightly 1.92 - on both compilers the same thing):

                    *** With profile fat-lto-32cu ***

+ RUSTFLAGS='-C target-cpu=x86-64-v2'
+ cargo run --quiet --profile fat-lto-32cu -- --bench --save-baseline baseline
+ RUSTFLAGS='-C target-cpu=x86-64-v3'
+ cargo run --quiet --profile fat-lto-32cu -- --bench --baseline baseline
bench_mat4_transform_point3
                        time:   [18.522 ns 18.557 ns 18.607 ns]
                        change: [+4383.2% +4395.5% +4406.3%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 8 outliers among 100 measurements (8.00%)
  1 (1.00%) high mild
  7 (7.00%) high severe

[TDecking]

I've created a version that only depends on criterion.

main.rs.txt

The fast version has an empty loop body, while the slow version has extra instructions that weren't removed.

[zamazan4ik]

Yeah, it's kinds interesting. My guess here is that with CU32 and FatLTO we are "just" unfortunate about how the application is divided into codegen units, and so we lost some critical for the benchmark optimization. I think the next step here should be trying to get optimization remarks from LLVM about that.

[im-0]

The fast version has an empty loop body, while the slow version has extra instructions that weren't removed.

Interesting... Doesn't this mean that most nalgebra's benchmarks measure an empty loop and thus produce nonsensical results?

I tried to fix the benchmark by adding black_box() call for both arguments:

diff --git a/src/main.rs b/src/main.rs
index 49c4f6d1a8a6..2963b882089c 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -1,6 +1,7 @@
 // SPDX-License-Identifier: Apache-2.0 OR MIT
 
 use std::env::args;
+use std::hint::black_box;
 use std::process::{Command, exit};
 
 use criterion::{Criterion, criterion_group};
@@ -37,6 +38,6 @@ fn bench_mat4_transform_point3(criterion: &mut Criterion) {
     );
 
     criterion.bench_function("bench_mat4_transform_point3", |b| {
-        b.iter(|| mat.transform_point(&pt));
+        b.iter(|| black_box(mat).transform_point(black_box(&pt)));
     });
 }

This changed the numbers, but there is a significant regression still in fat LTO + 32cu case:

[zamazan4ik]

Yesterday I did the same thing with black_box for the benchmarks, and I got the same test results at best as you. If you leave one black_box instead of two, you will get somewhere around +2000% performance degradation :)

I tried to compare flamegraphs between two versions but honestly didn't find anything helpful for the benchmark.

Doesn't this mean that most nalgebra's benchmarks measure an empty loop and thus produce nonsensical results?

I guess here we can ping nalgebra and ask them directly - maybe they have useful insights about that (not sure if you are the nalgebra developer - sorry if you are!)

[TDecking]

These are the loops viewed through IDA.

[im-0]

@zamazan4ik no, I am not a nalgebra developer. I just tried to add few things with benchmarks there. And then found myself descending into this rabbit hole instead.

I will try to check the disassembly of their benchmarks (just to make sure), and then will create an issue there.