Add #[inline] to all aarch64 and wasm32 functions - #44
Merged
Conversation
The aarch64 macro `vld_n_replicate_k!` generates ~300 public functions (all load/store variants) with `#[target_feature(enable = "neon")]` but no `#[inline]`. Similarly, all 17 wasm32 functions lack `#[inline]`. Without `#[inline]`, Rust does not emit function bodies in crate metadata for cross-crate inlining. Every NEON/WASM load and store becomes a `bl`/`call` instruction in the caller's hot loop instead of inlining to a single `ldr q`/`str q` instruction. This was empirically confirmed in garb (pixel swizzle crate): 44 `bl` calls to safe_unaligned_simd in the release assembly, reduced to 0 after adding `#[inline]`. The x86 modules already had `#[inline]` on all ~260 functions and were unaffected.
okaneco
approved these changes
Feb 28, 2026
okaneco
added
A-aarch64
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The
vld_n_replicate_k!macro generates ~300 aarch64 functions with#[target_feature(enable = "neon")]but no#[inline]. Similarly, all 17 wasm32 functions lack#[inline]. The x86/x86_64 modules already have#[inline]on every function.Without
#[inline], rustc does not include function bodies in crate metadata, preventing cross-crate inlining. Every NEON/WASM load and store becomes a function call (bl/call) instead of a single instruction (ldr q/str q).Impact
Benchmarked with garb (pixel swizzle crate) across Linux, macOS, and Windows aarch64 runners:
Before the fix, garb's NEON paths were slower than scalar on all aarch64 platforms. After: 2–4x faster than naive scalar loops. The release assembly went from 44
blcalls tosafe_unaligned_simdfunctions down to 0.Changes
src/aarch64.rs: Add#[inline]to bothloadandstorearms of thevld_n_replicate_k!macro (2 lines, affects ~300 generated fns)src/wasm32.rs: Add#[inline]to all 17 hand-written functions