Assemble the fMRIPrep testbench dataset drawing from OpenNeuro

[oesteban]

Objective

Assemble Benchmark v1.0: a curated, version-pinned, held-out set of OpenNeuro datasets that serves as the reference corpus for the verifier (#1, nipreps/fmriprep#3663). Selection is driven primarily by coverage of fMRIPrep's conditional branches, with acquisition diversity layered on top, and the set is published with reproducible manifests.

Rationale

The verifier's equivalence decisions are only as meaningful as the data they run on: if the benchmark exercises only a fraction of fMRIPrep's code paths, an "equivalent" reconstruction could still diverge on an untested branch. Selecting datasets to exercise every major branch makes equivalence a statement about the whole pipeline, not a happy path.

The idea of a broad, OpenNeuro-spanning testbench is not original to this issue — it has a direct antecedent in our own work. The original fMRIPrep paper (Esteban et al., 2019) validated the pipeline by running it across essentially every OpenNeuro dataset available at the time — on the order of ~100 datasets, then almost entirely fMRI. Benchmark v1.0 revisits that idea at today's scale: OpenNeuro now holds thousands of datasets, so exhaustive coverage of a small archive gives way to deliberate coverage of fMRIPrep's branch space across a far larger and more heterogeneous corpus.

Note

Datasets are held out from KB and agent development and version-pinned, so the benchmark cannot be overfit and stays reproducible.

Methodological approach

Datasets are drawn from OpenNeuro (Markiewicz et al., 2021) — queried programmatically with the NiQuery skill (nipreps/niquery#42) and sourced via the OpenNeuroStudies DataLad collection — and selected against the KB operation inventory and fMRIPrep's conditional structure:

1. Branch coverage (primary). Using the NiQuery skill (Develop a *NiQuery* skill for querying OpenNeuro datasets niquery#42) to search OpenNeuro by acquisition and metadata criteria, choose datasets so every major code path is exercised — fieldmap present vs absent (and each SDC estimation type), single- vs multi-echo, single- vs multi-run/-session, with/without slice-timing metadata, surface vs volume-only outputs — and map each dataset to the branches/operations it triggers.
2. Acquisition diversity (secondary). Layer in a spread of field strengths, vendors, sequences, and populations so generalization is also tested.
3. Pinning, holdout, manifests. Version-pin each dataset (OpenNeuro snapshot / DataLad, via the OpenNeuroStudies collection), hold the set out from KB and agent development, and publish reproducible manifests (dataset ids, versions, the branch/operation map, checksums).

Outcomes/impact

1. A published, version-pinned OpenNeuro reference set with reproducible manifests.
2. An explicit branch/operation coverage map over that set.

This is the corpus over which the accuracy/recall benchmark (#1) and the numerical-variability CI (nipreps/fmriprep#3663) run, and the shared benchmark the ablation (Goal 2) and every engine transplant (Goal 3) are measured on.

Direct upstream issues

(issues this one depends on / builds upon)

• Develop a *NiQuery* skill for querying OpenNeuro datasets niquery#42

Direct downstream issues

(issues that depend on this one)

• fMRIPrep on Claude Code's /workflow (nipype ablation) agentic-files#3
• Implementation Accuracy & Recall Benchmark #1
• Numerical-precision CI check for fMRIPrep (Verificarlo) fmriprep#3663

[HarperZ9]

A benchmark manifest here could be especially useful if it separates four locks that tend to get conflated:

1. Dataset lock: OpenNeuro dataset id, snapshot/version, DataLad/git-annex refs, file checksums, BIDS validator version, and any exclusion notes.
2. Coverage lock: the branch/operation map that explains why each dataset is in Benchmark v1.0.
3. Invocation lock: fMRIPrep LTS image digest, command line, relevant env vars, seeds, threads, CPU/GPU/runtime constraints, FreeSurfer version, and tool/plugin versions.
4. Holdout seal: a machine-readable statement that the dataset was not used for KB/agent development, plus the date/hash of the benchmark manifest that downstream verifier runs consumed.

For the branch map, I would make the unit small enough to support the byte/numerical split in #1 and #2. For example:

{
  "dataset": "openneuro:dsXXXXXX@snapshot",
  "subject_selector": "...",
  "coverage": [
    {
      "branch_id": "sdc.fieldmap.pepolar",
      "operation_refs": ["kb:fmriprep:sdc:pepolar"],
      "derivative_classes": ["boldref", "mask", "confounds"],
      "expected_tier": "byte|numerical",
      "reason": "exercises fieldmap-present path"
    }
  ]
}

That lets the CI gate in nipreps/fmriprep#3663 answer two separate questions:

• Did we run the same benchmark corpus and invocation as the last accepted release?
• If an output moved from byte-stable to numerical-only, is that a legitimate determinism-map update or a regression?

A Project Telos / Gather / Crucible pattern that maps well here is to distinguish SOURCE_LEAD from VERIFIED_EVIDENCE. The manifest can say "this dataset is intended to cover branch X" as a source lead, but the verifier run should produce a separate receipt proving the branch was actually exercised and the output tier decision was MATCH, DRIFT, or UNVERIFIABLE.

Suggested acceptance checks for Benchmark v1.0:

• every selected dataset resolves from the manifest alone on a clean machine
• every major fMRIPrep branch/operation in scope maps to at least one dataset, or has an explicit gap record
• every dataset has a reason for inclusion tied to branch coverage, not only acquisition diversity
• manifest hash is embedded in byte/numerical verifier reports
• checksum or snapshot drift fails closed before equivalence scoring starts
• held-out status is represented as an explicit field rather than a prose-only note
• generated reports can list uncovered branches separately from failed equivalence outputs

That would make the benchmark reusable outside the initial team while preserving the project?s stronger claim: equivalence is measured against a pinned corpus that actually exercises the intended fMRIPrep behavior surface.