RISC-V: riscv_hwprobe-based feature detection on Linux / Android

[a4lg]

This PR implements full riscv_hwprobe-based feature detection on newer Linux kernel and implements OS-independent extension implication logic to make extension handling easier.

This PR is a superset of #1769.

This PR is based on #1762 by @taiki-e. Commits with @taiki-e's code (with or without modification) are marked by Co-Authored-By.

While the OS-independent logic in this PR uses iterators, I confirmed that LLVM is smart enough to optimize them into a series of bit-manipulation operations (multi-feature masking then multi-feature enablement).

RFC: Where to Put imply_features()?

The only reason why I originally made this PR as a draft is, I was not sure where to put new OS-independent RISC-V extension implication logic (imply_features()).

Responses

@taiki-e: suggested the path like src/detect/os/riscv.rs.

Decision

@taiki-e's suggestion is adopted here (on PR version 7).

RFC: Automatic Loops?

Is it allowed to derive Eq for cache::Initializer? Using this way, we can continue implication until the feature flags converge (will loop 2 times on normal cases, up to 4 times on mostly adversarial cases).

If it's not allowed, I proved that (in the current state), looping group! definitions of Zvk* 3 times and group! definitions of Zk* 2 times is sufficient to ensure convergence (I tentatively introduced manual loops in the PR v4).

Decision

For now, I assume that it is okay to do that (on PR version 6).

Main Differences between #1762

1. Use key RISCV_HWPROBE_KEY_IMA_EXT_0 after checking that RISCV_HWPROBE_BASE_BEHAVIOR_IMA is set on the key RISCV_HWPROBE_KEY_BASE_BEHAVIOR.
   Because RISCV_HWPROBE_KEY_IMA_EXT_0 lists extensions compatible to RISCV_HWPROBE_BASE_BEHAVIOR_IMA, it must be checked (current Linux requires RV[32|64]IMA + some extra features but this difference makes the resulting code much robust when Linux kernel decided to lower its requirements),

2. Use key RISCV_HWPROBE_KEY_MISALIGNED_SCALAR_PERF, not just RISCV_HWPROBE_KEY_CPUPERF_0.
   @taiki-e's proposal only uses RISCV_HWPROBE_KEY_CPUPERF_0 but this key is considered deprecated (because it is incorrectly implemented as a bitmask).
   Considering the fact that some versions of the Linux kernel do not support RISCV_HWPROBE_KEY_MISALIGNED_SCALAR_PERF, RISCV_HWPROBE_KEY_CPUPERF_0-based unaligned scalar memory access performance checking is kept as a fallback.

3. Attempt to enable vectors only once and use the most up to date information
   This PR first uses prctl with PR_RISCV_V_GET_CONTROL (whether vector is enabled on the current thread when a runtime feature detection is requested) and falls back to the auxiliary vector (whether a vector extension, V, is enabled on the program starts) for workaround for userland emulation of QEMU (as of version 9.2.3).
   This PR attempts to use the latest (up to date) information for vector enablement and makes two differences when the program is ran on the real Linux kernel:

   1. When the program starts with vectors disabled and later enabled, this difference is properly handled (or vice versa; unlikely but possible when the crate is compiled as a Rust-based library).
   2. When only vector extension subset(s) are supported, this PR accurately reports their existence.

   And this PR won't try to use the auxiliary vector for V enablement once riscv_hwprobe is confirmed that capable of checking vector extensions (to ensure that we only test the vector status on one timing (either program startup or on the first feature detection), not two (program startup and on the first feature detection)).

4. Simplify feature enablement including uses of enable_feature through:

   1. Removal of enable_features and the value argument,
   2. Addition of the OS-independent RISC-V extension implication logic and
   3. More common closure for feature enablement (test which tests the value of RISCV_HWPROBE_KEY_IMA_EXT_0).

   Macros imply! and group! inside new implication function makes writing extension implication pretty easy and making implication a separate function (imply_features) will make computing various flags on the feature detection logic unnecessary. It'll make the logic hard to break when multiple feature detection logic is used (see code snippet 1).
   Complex implication logic also works as expected as in code snippet 2.

5. Imply I, M, A, Zicsr and Zifencei when we find the IMA base using riscv_hwprobe.
   Due to historical reasons, the I extension does not preserve backward compatibility but rather in reverse. The I extension in the ISA manual version 2.2 (RV32I/RV64I version 2.0) is splitted as three extensions I (RV32I/RV64I version 2.1), Zicsr, Zifencei and one non-extension "Counters" (with a few amendants) in the ISA manual version 20190608 and "Counters" are ratified as two extensions Zicntr (originally a part of the I extension) and Zihpm (amended parts in the non-extension "Counters") as in the ISA manual version 20240411.
   The thing is, Linux's base behavior defines that the I extension of the IMA base is of the ISA manual version 2.2 (later splitted to various extensions). If riscv_hwprobe succeeds and has the IMA base, the author chose to enable not just I but also Zicsr and Zifencei.

   1. Zifencei was initially excluded because the Linux documentation states that fence.i is not expected to be run on the user mode. Although no traps will be generated, its effect is unreliable unless SMP is disabled.
      However, this is only the normal path. If Concurrent Modification and Execution of Instructions (CMODX) is enabled, fence.i can be valid on the Linux userland, making this implication useful on certain cases. So, it is now implied on the PR version 9.
   2. Zihpm is excluded for not being a part of the original I extension as in the ISA manual version 2.2).
   3. Zicntr is excluded (on the PR version 2 or later) while it seems safe to imply that but Linux 6.15 will include (as of rc1) the separate constant RISCV_HWPROBE_EXT_ZICNTR to detect the existence of the Zicntr extension. So, the author chose more pessimistic assumption.
   4. Zicsr should be safe (so implied) because Linux depends on the privileged architecture, which depends on the Zicsr extension. If no other extensions with CSRs are enabled, it is almost equivalent to not having the Zicsr extension.

Same as #1762

1. Temporally remove privileged extensions
2. Add test

Code Snippets

1. From #1762: Multiple manual feature enablement

enable_features(
    &mut value,
    &[
        Feature::v,
        Feature::zve32f,
        Feature::zve32x,
        Feature::zve64d,
        Feature::zve64f,
        Feature::zve64x,
    ],
    has_v,
);

It enables V and its subsets. If this is alone, that would be okay but:

// Standard Vector Extensions
// v and zve{32,64}* extensions are detected by hwcap.
// enable_feature(Feature::v, ima_ext_0 & RISCV_HWPROBE_IMA_V != 0);
enable_feature(Feature::zvfh, ima_ext_0 & RISCV_HWPROBE_EXT_ZVFH != 0);
enable_feature(Feature::zvfhmin, ima_ext_0 & RISCV_HWPROBE_EXT_ZVFHMIN != 0);
// enable_feature(Feature::zve32x, ima_ext_0 & RISCV_HWPROBE_EXT_ZVE32X != 0);
// enable_feature(Feature::zve32f, ima_ext_0 & RISCV_HWPROBE_EXT_ZVE32F != 0);
// enable_feature(Feature::zve64x, ima_ext_0 & RISCV_HWPROBE_EXT_ZVE64X != 0);
// enable_feature(Feature::zve64f, ima_ext_0 & RISCV_HWPROBE_EXT_ZVE64F != 0);
// enable_feature(Feature::zve64d, ima_ext_0 & RISCV_HWPROBE_EXT_ZVE64D != 0);

Despite that commented out in #1762 (which is okay), handling separate vector subsets will need additional logic. Just removing comments here makes the code partially incorrect (depending on how has_v is computed) for being Zicsr not implied by Zve32x.

This PR removes the needs to write extension implication logic for multiple times.

2. From this PR: Complex implication

imply!(zcf => zca & f);
imply!(zcd => zca & d);
// ... omitted for being irrelevant ...
// Relatively complex implication rules from the "C" extension.
imply!(c => zca);
imply!(c & d => zcd);
#[cfg(target_arch = "riscv32")]
imply!(c & f => zcf);

It accurately represents what features to enable depending on the situation.

History

Version 1 (2025-04-11)

The first proposal.

Version 2 (2025-04-11)

See diff

• Use more pessimistic assumption about the Linux's IMA base.
  Excluded the Zicntr extension from implication.
• Renamed has_v inside fine-grained detection logic using riscv_hwprobe to has_vectors
is_vectors_enabled is more accurate name but it should be sufficient (denoting whether the vector extension or its subset(s) are enabled).
• Minor fix and clarification in comments.
  It sets V purely depending on the auxiliary vector only if no fine-grained vector extension detection is available (PR v1 stated that this occurs when riscv_hwprobe is unavailable but that was incorrect).

Version 3 (2025-04-11)

See diff

• Comment clarification
• An attempt to ensure convergence of feature flags (insufficient)

There's still a case where implying without a loop won't converge. We need to loop a portion several times.

Version 4 (2025-04-11)

See diff

• Ensure convergence of feature flags (formally proven)
  But this is a tentative change. If I'm allowed to derive Eq for cache::Initializer, OS-independent RISC-V extension implication logic will get a lot simpler (loop over until it converges; termination is guaranteed because we are never removing features).
• Reflect a change suggested by @taiki-e (Thanks for reviewing!)

Version 5 (2025-04-12)

See diff

• Rebase
• Tentative version with the Supm extension removed (to be too OS-dependent; thanks @Amanieu!).

Version 6 (2025-04-12)

See diff

• A clarification in a comment.
  I as in the ISA manual version 2.2 is not the I extension with version 2.2 (actually, the ISA manual version 2.2 defines version 2.0 of RV32I and RV64I and splitted I base indicates version 2.1 of RV32I and RV64I).
• Ensure convergence of feature flags through use of the loop.
  Normally, it loops twice. Termination of the loop (in finite time) can be easily proved (from two facts: (1) we have finite number of feature flags and (2) we never unset any feature flags) and in fact, even on the worst case (including the case where the feature flags are completely broken), the maximum loop count is currently 4 (formally proven).
  To implement this, the author added automatic derive of PartialEq and Eq to cache::Initializer.

Version 7 (2025-04-12)

See diff

• Various adjustments in the commit message.
• Improve documentation of performance hints (unaligned-scalar-mem and unaligned-vector-mem).
• Minor fix to a comment.
• Move imply_features to a separate module as suggested by @taiki-e.
  Now, it's an official proposal (leaving from the draft status).

Version 8 (2025-04-12)

See diff

• Add double quotes around documentation of unaligned-scalar-mem and unaligned-vector-mem
  (make them "unaligned-scalar-mem" and "unaligned-vector-mem" for consistency).

Version 9 (2025-04-12)

See diff

• Imply the Zifencei extension from the Linux IMA base.
  The author initially excluded this because fence.i of the Zifencei instruction is normally invalid on Linux ABI but found this is only normally true. If Concurrent Modification and Execution of Instructions (CMODX) is enabled, fence.i can be valid on the Linux userland. Even if CMODX is not enabled, it will not cause any traps so "use at your risk" policy should work (fence.i is generally unreliable on SMP-enabled systems with preemptive multi-threading, not just Linux).

Version 10 (2025-04-12)

See diff

• Bug Fix: Add missing implication: Zvfh → Zfhmin
• Doc: Improve capitalization consistency (fix description of the Zawrs extension)
• Add functional implications:
  • Zvknhb → Zvknha
  • Zbc → Zbkc
• If a dependency / implication is not explicitly stated in the specification but adding implication is useful for feature discovery (and does not form an extension group with "reverse implication"), such an implication is commented with a brief reason.
  • Zvbb → Zvkb (from PR v1; comment removed in PR v11 as there's a strong evidence now)
  • Zvknhb → Zvknha (from PR v10)
  • Zbc → Zbkc (from PR v10)
  • Zvfh → Zvfhmin (from PR v1)
  • Zkr → Zicsr (from PR v1)

Version 11 (2025-04-13)

See diff

This is functionally equivalent to the version 10 but incorporates minor changes.

• Tidying: Found a strong evidence to support Zvbb → Zvkb (remove "defined as subset" comment which have denoted that there was a weak evidence only).
• Tidying: Various comments (grammar and clarification).
• Tidying: Move Zvknhb → Zvknha implication after group definitions inside imply_features.
  It makes the worst iteration count minimum.
  Although the big loop inside this function is designed to be ordering-free (not introducing a bug when we move imply! and group! macro uses and it's free to add implications in any order),
  at least my contribution is designed also to minimize iteration count.

Version 12 (2025-04-13)

See diff

Oh no... How can I miss that section for years!?

• Bug Fix: Remove excess implication: Zkr → Zicsr (considered not an errata)
  Although the seed CSR must be accessed through CSR instructions, originally defined in the Zicsr extension, scalar cryptography spec defines its own seed CSR access instruction (a subset of Zicsr).
  I (somehow) did not catch this for years.

Version 13 (2025-04-15)

See diff

Despite that this is functionally equivalent to the version 12, it is now tested.

• Add tests.
  • Simple direct / indirect cases.
  • Complex case.
  • Whether the extension group is working as expected (2 tests).
  • Mostly adversarial case which maximizes the iteration count in the current design.

Version 14 (2025-04-16)

See diff

• Reflected a change suggested by @Amanieu (thanks!).

[rustbot]

r? @Amanieu

rustbot has assigned @Amanieu.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

[taiki-e]

This looks good to me.

The only reason why I make this PR a draft is, I'm not sure where to put new OS-independent RISC-V extension implication logic (imply_features()).

AArch64 code uses the way of putting things referenced by multiple OSes in src/detect/os, and use that module if the OS needs it. I think the same way should be fine.

https://github.com/rust-lang/stdarch/blob/master/crates/std_detect/src/detect/os/aarch64.rs

stdarch/crates/std_detect/src/detect/mod.rs

Lines 54 to 57 in 6780277

	} else if #[cfg(all(target_os = "freebsd", feature = "libc"))] {
	#[cfg(target_arch = "aarch64")]
	#[path = "os/aarch64.rs"]
	mod aarch64;

stdarch/crates/std_detect/src/detect/mod.rs

Lines 60 to 63 in 6780277

	} else if #[cfg(all(target_os = "openbsd", target_arch = "aarch64", feature = "libc"))] {
	#[allow(dead_code)] // we don't use code that calls the mrs instruction.
	#[path = "os/aarch64.rs"]
	mod aarch64;

Although not yet included in std_detect ¹, FreeBSD and OpenBSD also support auxv-based detection (via elf_aux_info) on RISC-V, so we can use imply_features on those OSes as well.

https://github.com/freebsd/freebsd-src/blob/main/sys/riscv/include/elf.h#L75
https://github.com/openbsd/src/blob/master/sys/arch/riscv64/include/elf.h#L41

Footnotes

1. I had generally implemented this for FreeBSD before, but abandoned it because I could not create a test environment. https://github.com/taiki-e/stdarch/commit/43b6f01b5b181cf7680cad0ffa8d19cf9569d372 ↩

[a4lg]

AArch64 code uses the way of putting things referenced by multiple OSes in src/detect/os, and use that module if the OS needs it. I think the same way should be fine.

https://github.com/rust-lang/stdarch/blob/master/crates/std_detect/src/detect/os/aarch64.rs

stdarch/crates/std_detect/src/detect/mod.rs

Lines 54 to 57 in 6780277

	} else if #[cfg(all(target_os = "freebsd", feature = "libc"))] {
	#[cfg(target_arch = "aarch64")]
	#[path = "os/aarch64.rs"]
	mod aarch64;

stdarch/crates/std_detect/src/detect/mod.rs

Lines 60 to 63 in 6780277

	} else if #[cfg(all(target_os = "openbsd", target_arch = "aarch64", feature = "libc"))] {
	#[allow(dead_code)] // we don't use code that calls the mrs instruction.
	#[path = "os/aarch64.rs"]
	mod aarch64;

Although not yet included in std_detect 1, FreeBSD and OpenBSD also support auxv-based detection (via elf_aux_info) on RISC-V, so we can use imply_features on those OSes as well.

Thanks for your suggestion! I'll try and will likely push the new version tomorrow.

[a4lg]

Adopted @taiki-e's suggestion and now, it's no longer a draft but a complete proposal.

[a4lg]

PR version 9 is rebased against the latest commit (after #1769 is merged).

See no diff

[a4lg]

Author Re-Review Complete (Version 11)

All of related extensions inside the ISA manual are reviewed and found that all implications inside imply_features (as of PR version 10 and feature-equivalent 11) should be okay. In fact, I suspect that the ISA manual (version 20240411) has another errata: implication Zvks → Zve64x (which I didn't included in this PR intentionally).

The Zvknhb and Zvbc Vector Crypto Extensions ― and accordingly the composite extensions Zvkn and Zvks ― require a Zve64x base, or application (V) base Vector Extension.

Zvks → Zve64x cannot be explained from the fact that Zvknhb and Zvbc require Zve64x (in fact, all members of Zvks only depend on the Zve32x base, not Zve64x). So, the word accordingly seems wrong (I suspect that Zvks is a typo of Zvksc because Zvksc depends on Zvbc which requires the Zve64x extension).

[a4lg]

Author Re-Re-Review Complete (Version 12)

And concluded that Zkr → Zicsr implication is actually not required (missing implication in the manual is not an errata).

I am still wondering how I could miss that section (defining its own seed CSR access instruction; a subset of Zicsr) for years.

I double checked Zvks → Zve64x (I intentionally excluded from this PR because it seemed to be an errata) and it seems... this is a true errata and already fixed in the latest ISA manual draft as follows:

The Zvknhb and Zvbc Vector Crypto Extensions ― and accordingly the composite extensions Zvkn, Zvknc, Zvkng, and Zvksc ― depend on Zve64x.

[Amanieu]

LGTM other than a minor nit!