Split PowdrConfig into per-stage GenerateConfig + SelectConfig

[georgwiese]

Summary

This PR makes the staged APC pipeline (generate-apcs → select-apcs → setup)
land on main as a single, self-contained change, and reworks the config types
that flow through it. It supersedes #3752 — that PR introduced the stage
split against a still-shared PowdrConfig; this one folds in the stage split
and splits the config along the same boundary, so #3752 should be closed in
favor of this one. No separate merge of #3752 is needed.

Context. #3750 (merged) introduced the per-stage subcommands and
--artifacts-dir caching, but the pipeline + cache logic lived entirely in
cli-openvm-riscv/src/main.rs as a CLI-only Pipeline — unusable by other
callers. #3752 then split the fused select-apcs into generate-apcs +
select-apcs, still against the CLI-local pipeline and a shared PowdrConfig.
This PR lifts the pipeline into the library (StagedPipeline in
openvm-riscv/src/pipeline.rs) and splits the config so it can be hashed as a
cache key, which is what lets openvm-eth reuse the same staged cache
instead of re-implementing it. The CLI becomes a thin shim over the shared
StagedPipeline. (sp1 adopts only the config split, not the cache — see
Related PRs.)

There are three parts:

1. The staged pipeline → library (the point of this PR): the
   generate → select → setup runner and its on-disk cache, previously a
   CLI-local Pipeline (powdr-openvm-riscv: per-stage CLI subcommands + --artifacts-dir caching #3750) split into two stages (powdr-openvm-riscv: split generate-apcs out of select-apcs #3752), now live in the
   powdr-openvm-riscv library as StagedPipeline. The CLI and openvm-eth
   share one implementation; sp1 keeps its own minimal build path and does not
   adopt the cache (see Related PRs).
2. The config split (new here): replace the monolithic PowdrConfig with
   per-stage GenerateConfig + SelectConfig + PgoConfig, which lets the
   library hash the config as a cache key itself and drops the error-prone
   cache_key argument from StagedPipeline.
3. The stage split (carried over from powdr-openvm-riscv: split generate-apcs out of select-apcs #3752): the one-shot select-apcs
   becomes a cached generate-apcs (build + rank candidates) feeding a cached
   select-apcs (trim the ranking), so an --autoprecompiles sweep under
   --pgo cell reuses one generate-cache entry instead of rebuilding every
   candidate per run.

---

Part 1 — the staged pipeline

select-apcs used to build APC candidates and pick the top N in one shot.
This PR splits those into two cached stages:

Command	Stages run
generate-apcs	Profile + build & rank APC candidates
select-apcs	… + trim the ranking to --autoprecompiles (after --skip)
setup	… + assemble the program with selected APCs
execute	… + run the guest in interpreted mode
prove	… + STARK proof, optionally with --recursion (compression)

--pgo and --max-columns shape the ranking, so they live on the generate
stage; --autoprecompiles and --skip are the trim, so they live on select.
A new --apc-candidates <N> caps how many candidates generate-apcs builds.

--apc-candidates resolution:

• --pgo cell always builds every eligible candidate (its density ranking
  needs each candidate's post-opt cost), so a user-set --apc-candidates is
  warned-about and ignored. Crucially, the cap is not a function of
  --autoprecompiles, so the generate hash is identical across a sweep.
• --pgo instruction|none: standalone generate-apcs builds all by default;
  in a fused pipeline (select-apcs and beyond) the cap defaults to
  --autoprecompiles + --skip. Pin it explicitly to over-build once and sweep
  selection cheaply.

Cache hierarchy (generate → select → setup): each stage hashes only its
own args plus a fingerprint of the transpiled guest VmExe. A later-stage flag
(--input, --mock, --recursion, --metrics) never invalidates an
earlier-stage cache; editing the guest invalidates everything downstream of it.
The cache is opt-in via the global --artifacts-dir <DIR>; with no dir, every
stage runs inline.

See cli-openvm-riscv/README.md for the full
surface and --profile-input vs --input semantics.

---

Part 2 — the config split

PowdrConfig bundled fields from every pipeline stage into one struct. That
forced generate-stage code to share a hash space with select-stage flags —
making it impossible for the library to safely hash the config as a cache key,
which in turn forced StagedPipeline to expose a stringly-typed
cache_key: &impl Hash argument that each external caller (CLI, openvm-eth,
sp1) fingerprinted on its own. Three callers, three conventions — easy to drop
a field and silently serve stale cache hits.

This PR splits PowdrConfig along the natural pipeline boundary:

• GenerateConfig (autoprecompiles/src/lib.rs) holds only the fields the
  generate stage reads (apc_candidates, superblock_max_bb_count,
  apc_max_instructions, apc_exec_count_cutoff, apc_candidates_dir_path,
  should_use_optimistic_precompiles, degree_bound). Derives Hash.
• SelectConfig holds autoprecompiles + skip. Derives Hash.
• PgoConfig holds pgo_type, max_columns, and the opaque inputs
  byte string (serialized stdin). Derives Hash, so it participates in both
  the generate and select/setup hashes.
• degree_bound keeps being passed plain into setup (no new wrapper).
• GenerateConfig::with_select_defaults(pgo, select) is the single policy
  definition for the apc_candidates cap default. Callers chain it in one line
  when they know both halves — compile_exe, the CLI shim, openvm-eth, sp1 all
  share it.
• InstructionPgo / NonePgo short-circuit on
  apc_candidates == Some(0) (mirroring Cell), set by with_select_defaults
  when select.autoprecompiles == 0. The stale autoprecompiles == 0 field
  check is gone.

StagedPipeline's API drops cache_key; callers supply only an
input_fp: &impl Hash covering anything hidden behind the closures (PGO stdin
contents, RPC chain id, …). The library hashes (guest_hash, generate, pgo)
for generate and (guest_hash, generate, pgo, select) for select/setup.
Generate's hash omits select by construction, so a Cell-PGO --autoprecompiles N sweep automatically reuses the candidate ranking — Cell's
with_select_defaults returns apc_candidates = None regardless of selection
size, so the generate hash is identical across the sweep.

compile_exe becomes a thin wrapper over the staged generate/select/setup
functions (no caching). PowdrConfig and default_powdr_openvm_config are
deleted.

---

Verifying the cache

All commands below use the setup subcommand (runs generate → select → setup
with no proving, so they're fast) and RUST_LOG=info to surface the cache log
lines. cached() logs cache hit: <stage>/<hash> on a hit and runs the stage
inline on a miss. The hashes are toolchain/guest-specific — what matters is
which stage logs a hit. Substitute any guest (e.g. guest-keccak) for
heavier, more realistic candidate counts. The cargo run invocation is spelled
out in full on each line (rather than stashed in a $BIN variable) so the
blocks paste cleanly into both bash and zsh.

Because the cache is hierarchical and short-circuits at the outermost hit,
an identical rerun logs only cache hit: setup/<h> (select and generate are
never consulted), whereas a run that only changes selection logs
cache hit: generate/<h> and re-runs select + setup.

1. Cell PGO: --autoprecompiles sweep reuses one generate entry

rm -rf /tmp/powdr-cache

# Cold: builds & ranks every eligible candidate.
RUST_LOG=info cargo run -r --bin powdr_openvm_riscv -- --artifacts-dir /tmp/powdr-cache \
    setup guest --profile-input 10 --autoprecompiles 2 --pgo cell

# Different --autoprecompiles: generate is reused, only select + setup re-run.
RUST_LOG=info cargo run -r --bin powdr_openvm_riscv -- --artifacts-dir /tmp/powdr-cache \
    setup guest --profile-input 10 --autoprecompiles 10 --pgo cell

Run 1 logs Generating autoprecompiles for all N blocks in parallel (cold
build). Run 2 logs:

INFO cache hit: generate/<h>      <- ranking reused across the sweep
INFO Adjust the program ...       <- select + setup re-run (cheap)
INFO Setup completed.

A third identical rerun of run 2 instead logs a single
cache hit: setup/<h> — the whole pipeline short-circuits.

2. Instruction PGO: sweeping rebuilds unless --apc-candidates is pinned

Under --pgo instruction|none the build cap defaults to
--autoprecompiles + --skip, so it lives in the generate hash and a naive
sweep rebuilds every iteration:

rm -rf /tmp/powdr-cache-instr
# ap=2 then ap=5, no --apc-candidates → generate hash differs each time.
RUST_LOG=info cargo run -r --bin powdr_openvm_riscv -- --artifacts-dir /tmp/powdr-cache-instr \
    setup guest --profile-input 10 --autoprecompiles 2 --pgo instruction
RUST_LOG=info cargo run -r --bin powdr_openvm_riscv -- --artifacts-dir /tmp/powdr-cache-instr \
    setup guest --profile-input 10 --autoprecompiles 5 --pgo instruction

Both runs log Generating up to N autoprecompiles in parallel — no
cache hit: generate. Pin --apc-candidates to the top of the sweep to share
one generate entry:

rm -rf /tmp/powdr-cache-instr2
RUST_LOG=info cargo run -r --bin powdr_openvm_riscv -- --artifacts-dir /tmp/powdr-cache-instr2 \
    setup guest --profile-input 10 --autoprecompiles 2 --apc-candidates 5 --pgo instruction
RUST_LOG=info cargo run -r --bin powdr_openvm_riscv -- --artifacts-dir /tmp/powdr-cache-instr2 \
    setup guest --profile-input 10 --autoprecompiles 5 --apc-candidates 5 --pgo instruction

Now the second run logs cache hit: generate/<h>.

3. Later-stage flags don't invalidate earlier stages

# After any cached setup run above, re-proving with a new --input / --mock
# hits the setup cache rather than rebuilding the program:
RUST_LOG=info cargo run -r --bin powdr_openvm_riscv -- --artifacts-dir /tmp/powdr-cache \
    prove guest --profile-input 10 --autoprecompiles 10 --pgo cell --input 123 --mock

--input / --mock are not part of any stage hash, so this logs
cache hit: setup/<h> and proceeds straight to the (mock) prover.

All three scenarios above were run locally on this branch (release build,
guest crate) and produced exactly the logged behavior described.

---

Related PRs

• powdr powdr-openvm-riscv: per-stage CLI subcommands + --artifacts-dir caching #3750 (merged): introduced the per-stage subcommands +
  --artifacts-dir caching as a CLI-local Pipeline.
  powdr-openvm-riscv: per-stage CLI subcommands + --artifacts-dir caching #3750
• powdr powdr-openvm-riscv: split generate-apcs out of select-apcs #3752 (superseded by this PR): split generate-apcs out of
  select-apcs. powdr-openvm-riscv: split generate-apcs out of select-apcs #3752

Downstream adopters of the split configs:

• powdr-labs/openvm-eth#8 —
  Adopt powdr's split GenerateConfig + SelectConfig + StagedPipeline. Also
  adopts the now-library StagedPipeline (openvm-eth depends on
  powdr-openvm).
• succinctlabs/sp1#2830 —
  Adopt powdr's split GenerateConfig + SelectConfig — the minimal
  migration. Deliberately does not adopt the staged cache: every sp1
  consumer builds APCs once per config, so only the dev CLI would have
  benefited. sp1 doesn't depend on powdr-openvm, so it couldn't share the
  library StagedPipeline anyway; the cache can be re-added later as an
  additive layer.

The .github/actions/patch-openvm-eth ref is pinned to the matching openvm-eth
branch; the sp1 fork pin is updated in CI. Both compile against this branch.

Test plan

• cargo build --release --bin powdr_openvm_riscv clean
• cargo clippy -p powdr-autoprecompiles -p powdr-openvm -p powdr-openvm-riscv -p cli-openvm-riscv --all-targets --features metrics -- -D warnings clean
• cargo test --release -p powdr-autoprecompiles --lib (33 passed)
• Cache behavior verified locally via the Verifying the cache
  commands — Cell sweep reuses generate, instruction sweep needs
  --apc-candidates pinned, identical rerun / later-stage flags hit setup
• openvm-eth + sp1 compile against this branch (path-patch verified locally)
• Snapshot tests guest_machine_pgo_modes, keccak_machine_pgo_modes for
  all 3 PGO modes (ranking unchanged by the split) — covered by CI
• CI green

🤖 Generated with Claude Code

[georgwiese]

Renamed, because it's more than just a config (contains ExecutionProfile). There is a new struct called PgoConfig, which contains the information needed to compute the PGO data.

[georgwiese]

Having to repeat these here many times is annoying and can be fixed, see #3764. I#m not sure if that should be part of this PR though, because it causes a significant diff in the tests of openvm-riscv/src/lib.rs