rules_rust_prost internal crate_universe does not inherit exec_properties, causing local execution of Linux ELF build scripts on macOS during RBE.

[santhoshct]

When cross-compiling from macOS to Linux using RBE, the CargoBuildScriptRun actions for our main workspace crates (@crates//) correctly dispatch to our remote Linux workers. However, the CargoBuildScriptRun actions for rules_rust_prost's internal tools (e.g., @rules_rust++i2+rrc__*) attempt to execute locally on macOS, despite being compiled as Linux ELF binaries by the resolved toolchain.

Both the working main crates and the broken prost crates share the exact same config hash, meaning toolchain resolution is correct. The issue appears to be that the internal crate_universe instantiated by the rules_rust_prost module extension does not inherit the global execution platform constraints or exec_properties (unlike our user-controlled render_config).

Questions:

Is there a way to pass a render_config or custom exec_properties to the internal crate_universe used by rules_rust_prost?

Is rules_rust_prost hardcoding tags = ["local"] or stripping remote execution properties during its bootstrapping phase?

[santhoshct]

Environment

• rules_rust: 0.68.1
• rules_rust_prost: 0.68.1
• Bazel: 8.3.0
• Host: macOS aarch64 (Apple Silicon)
• RBE: Buildbarn (Linux x86_64 workers, Ubuntu 24.04)

Reproduction

# This works — main crate build scripts go to RBE
bazel build //my-rust-lib:lib --config=rbe

# This fails — prost crate build scripts run locally as darwin-sandbox
bazel build @crates//:protoc-gen-prost__protoc-gen-prost --config=rbe

Key Evidence

Building the prost protoc plugin shows: 3 darwin-sandbox, 24 remote

The CargoBuildScriptRun actions for @rules_rust++i2+rrc__* crates run in darwin-sandbox (macOS), but the runner binary they invoke is a Linux ELF:

file .../cargo_build_script_runner/runner
# ELF 64-bit LSB pie executable, x86-64

The toolchain resolution and exec platform resolution disagree. The Rust toolchain resolves to rust_linux_x86_64__x86_64-unknown-linux-gnu__stable (compiles Linux binaries), but the exec platform resolves to macOS (runs actions locally). For our main @crates// build scripts, both resolve consistently to Linux/RBE.

What we tried (none worked for prost crates)

• --extra_toolchains=@rust_toolchains//:all — toolchain resolves correctly, exec platform still macOS
• --remote_default_exec_properties — actions never reach the remote executor
• --strategy_regexp='.*CargoBuildScriptRun.*=remote' — exec platform decision overrides strategy
• --modify_execution_info=CargoBuildScriptRun=+no-remote — forces local but binary is Linux ELF

[santhoshct]

Update: Root Cause Identified

After extensive debugging, we've narrowed the issue. Here's the complete picture:

Why this doesn't surface in rules_rust CI

rules_rust tests RBE from Linux CI machines (BazelCI with rbe_preconfig(toolchain = "ubuntu2204")). The host platform is Linux, same as the RBE workers. So @platforms//host:host = Linux, and there's never a macOS-vs-Linux exec platform mismatch. The cfg = "exec" transition in rust_prost_toolchain works because exec = Linux = RBE.

Why this breaks on macOS-host + Linux-RBE

When developing on macOS with Linux RBE workers:

1. The prost toolchain's cfg = "exec" on prost_plugin transitions plugins to the toolchain implementation's exec platform, which resolves to @platforms//host:host (macOS)
2. In the macOS exec config, the macOS Rust toolchain is selected (exec=macOS, target=macOS)
3. This should produce macOS Mach-O binaries that run locally via --remote_local_fallback
4. BUT Bazel's output deduplication symlinks the build script binary (_bs_) from the Linux RBE config to the macOS config, because the action keys appear identical
5. The macOS config then tries to execute a Linux ELF binary → "cannot execute binary file"

Current workaround

With --remote_local_fallback and natural toolchain resolution (not forcing Linux Rust toolchains), we get 29,635 total processes (2,206 remote) with only 1 failure from the symlink deduplication edge case.

The deeper issue

This appears to be a Bazel action deduplication bug: cargo_build_script binaries get shared via symlink across exec configs that should produce platform-specific binaries. The action keys don't account for the target triple difference between the Linux and macOS exec configs.

[santhoshct]

Update: Root Cause Narrowed to Bazel Eager Toolchain Loading

After extensive debugging, we isolated the exact mechanism:

The Chain of Events

1. register_toolchains in rules_rust_prost's MODULE.bazel causes Bazel to eagerly load the default_prost_toolchain implementation during startup — before any target is analyzed.

2. The toolchain implementation has prost_plugin with cfg = "exec". During loading, this creates an exec configuration where the exec platform resolves to @platforms//host:host (macOS on dev machines).

3. Within that macOS exec config, the plugin's crate dependencies have cargo_build_script rules. The script attribute on those rules also has cfg = "exec", which transitions the build script binary (_bs_) to a second exec platform resolution.

4. This second resolution picks //:ubuntu-24-04 (Linux) because --extra_execution_platforms puts it first. So _bs_ is compiled as Linux ELF.

5. The CargoBuildScriptRun action (from step 2's macOS exec config) tries to execute _bs_ on macOS. Linux binary on macOS → crash.

Why exec_compatible_with Doesn't Help

We patched both the default prost toolchain and our own to add exec_compatible_with = ["@platforms//os:linux", "@platforms//cpu:x86_64"]. This correctly prevents the toolchains from matching for macOS exec platforms. But register_toolchains still loads the toolchain implementation (including evaluating cfg = "exec" transitions) regardless of whether the toolchain will match any configuration.

The exec_compatible_with is a selection-time constraint, not a load-time constraint.

What Works

• //my-rust-lib:lib --config=rbe — full Rust library build on RBE (228 remote actions) ✅
• //... --config=rbe with --remote_local_fallback — 30,000+ actions, 2,200+ remote, 1 failure from this edge case

The Fix Needs to Be in Bazel

This appears to be a Bazel architecture limitation. register_toolchains eagerly analyzes toolchain implementations, which triggers cfg = "exec" transitions and creates exec configurations that may not match any actual build. Lazy toolchain loading (only analyzing implementations when they're selected for a specific target) would prevent this.

Should this be filed as a separate Bazel issue, or is there a known mechanism to defer toolchain implementation loading?