Diagnose the Enzyme LTS Hessian compiler failure

[ChrisRackauckas-Claude]

The LTS job in #1328 repeatedly fails with an LLVM verifier error while the same regression passes locally. This temporary diagnostic workflow checks out the exact failing commit, then compares an Enzyme-only analytic Hessian test, the unchanged regression, and the official AD command on hosted runners. It prints CPU details and uploads logs/manifests; Julia 1.13.0 provides a stable-version control.

Diagnostic PR: keep as a draft; do not merge. Ignore until reviewed by @ChrisRackauckas.

Local verification

Commands were run from the shared workspace. latest-enzyme-lts is the isolated environment with Enzyme 0.13.203, ChainRulesCore, ForwardDiff, and the subject OptimizationBase checkout.

GROUP=QA JULIA_PKG_PRECOMPILE_AUTO=0 TMPDIR="$PWD/review-tmp" timeout 3600 ~/.juliaup/bin/julia +1.12 --project=enzyme-hosted-diagnostic -e 'using Pkg; Pkg.test()'
TMPDIR="$PWD/review-tmp" timeout 3600 ~/.juliaup/bin/julia +1.10.12 --startup-file=no --check-bounds=auto --compiled-modules=yes --depwarn=yes --code-coverage=@"$PWD/enzyme-hosted-diagnostic" --project=enzyme-hosted-diagnostic/.diagnostics/enzyme enzyme-hosted-diagnostic/.diagnostics/enzyme/standalone.jl
OPTIMIZATION_SUBJECT="$PWD/Optimization.jl" TMPDIR="$PWD/review-tmp" timeout 3600 ~/.juliaup/bin/julia +1.10.12 --startup-file=no --check-bounds=auto --compiled-modules=yes --depwarn=yes --code-coverage=@"$PWD/Optimization.jl" --project=latest-enzyme-lts enzyme-hosted-diagnostic/.diagnostics/enzyme/fixture.jl

QA:                        21/21 passed, 1m07.5s
Standalone Enzyme Hessian: 108/108 passed, 4.0s
Enzyme Lagrangian Hessian: 134/134 passed, 1m23.5s

Runic, actionlint, typos, and git diff --check passed. The full subject AD suite also passed locally with automatic precompilation enabled: 938/938 in 6m34.4s. Its 238 comparable dependency versions match the failing CI run.

Isolation findings and diagnostic correction

The Enzyme-only test reproduced the identical LLVM verifier failure on Julia 1.10.12 / AMD EPYC 7763 (Zen 3), without importing any SciML package. Julia 1.13.0 passed 108/108 on the same CPU model. The unchanged LTS regression passed 134/134 on an Intel Xeon 6973P-C runner. Locally, configuring Enzyme's target as znver3 reproduces the exact %623 / %tempphi.sroa.156.1 dominance failure from the subject CI log. Ordinary reverse differentiation of the reduced objective passes; the nested Hessian path is being reduced further.

The first diagnostic wrapper incorrectly let tee hide Julia's abort, producing a green job. The workflow now explicitly selects shell: bash, which enables pipefail. Local regression evidence using a failing command in the same pipeline:

before FAIL: A failing Julia-command substitute was reported as successful exit: 0
after PASS: failure propagated, exit: 1

The original standalone test is retained. A separate driver adds an invokelatest barrier to identify the first failing case; that driver passes all 108 local assertions on the native CPU. First-run green job status must not be interpreted as successful execution of the crashing LTS standalone case.

The earlier Julia -C checks do not override GPUCompiler's physical-host target used by Enzyme. Target control for the local reproducer uses a separate Enzyme checkout; no dependency source changes are part of this PR. The obsolete fork-hosted run was cancelled after the SciML run reproduced the failure.

GPU and full downstream suites were not run locally; no production code, assertions, or tolerances are changed. Enzyme is already a repository test dependency and is MIT-licensed; this only pins it in the isolated diagnostic environment. Documentation was not built locally because no documentation or public API changes.

Links:

• Subject PR: Cap Enzyme Lagrangian Hessian batches at eight on every Julia version #1328
• Enzyme-only LTS failure (read log, not first-run green status): https://github.com/SciML/Optimization.jl/actions/runs/34698110920/job/103564998784
• Repeated LTS failure: https://github.com/SciML/Optimization.jl/actions/runs/34684825479/job/103554419756

🤖 Generated with Codex CLI 0.153.4 (model: gpt-6-astra; local session: 01a08039-86d4-7683-9601-30fd79501e27).