ci: replace CI with sovereign-ci.yml caller

.github/workflows/ci.yml

@@ -1,80 +1,21 @@
-# Unified CI — self-hosted clean-room runners for sovereignty (§2.1.1)
-# Spec: docs/specifications/unified-ci-pipeline.md
-#
-# Calls the centralized reusable gate workflow in paiml/.github.
-# Branch protection requires "unified / gate" to pass before merge.
-
+# Sovereign CI — calls reusable workflow from paiml/.github
+# Change once in paiml/.github → applies to all repos
 name: CI
 
 on:
-  pull_request_target:
-    branches: [main, master]
   push:
     branches: [main, master]
-  workflow_dispatch:  # manual trigger for testing
+  pull_request:
+    branches: [main, master]
+  workflow_dispatch:
 
-# One CI run per branch/PR; cancel stale runs on same branch
 concurrency:
   group: ci-${{ github.event.pull_request.number || github.ref }}
   cancel-in-progress: true
 
 jobs:
-  unified:
-    uses: paiml/.github/.github/workflows/unified-gate.yml@b6c24635217fe302ca1a67352b7de540fb3e02e7 # pin to SHA
+  ci:
+    uses: paiml/.github/.github/workflows/sovereign-ci.yml@main
     with:
       repo: ${{ github.event.repository.name }}
-      pr_sha: ${{ github.event.pull_request.head.sha || github.sha }}
     secrets: inherit
-
-  test:
-    runs-on: [self-hosted, clean-room]
-    steps:
-      - uses: actions/checkout@v4
-      - uses: dtolnay/rust-toolchain@stable
-      - uses: actions/cache@v4
-        with:
-          path: |
-            ~/.cargo/registry
-            ~/.cargo/git
-            target
-          key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
-      - run: cargo test --no-default-features --features parallel --lib
-
-  coverage:
-    runs-on: [self-hosted, clean-room]
-    steps:
-      - uses: actions/checkout@v4
-      - uses: dtolnay/rust-toolchain@stable
-      - uses: actions/cache@v4
-        with:
-          path: |
-            ~/.cargo/registry
-            ~/.cargo/git
-            target
-          key: ${{ runner.os }}-cargo-cov-${{ hashFiles('**/Cargo.lock') }}
-      - run: cargo install cargo-llvm-cov --locked || true
-      - run: cargo llvm-cov test --no-default-features --features parallel --lib --lcov --output-path lcov.info
-      - uses: codecov/codecov-action@v4
-        with:
-          files: lcov.info
-        continue-on-error: true
-
-  security:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v4
-      - uses: dtolnay/rust-toolchain@stable
-      - run: cargo install cargo-audit --locked || true
-      - run: cargo audit
-
-  gate:
-    name: gate
-    runs-on: [self-hosted, clean-room]
-    if: always()
-    needs: [test, coverage, security]
-    steps:
-      - name: Check all jobs
-        run: |
-          if [ "${{ needs.test.result }}" = "failure" ]; then
-            exit 1
-          fi

CLAUDE.md

@@ -1140,3 +1140,41 @@ The RAG index includes 335 documents across:
 Index auto-updates via post-commit hooks and `ora-fresh` on shell login.
 To manually check freshness: `ora-fresh`
 To force full reindex: `batuta oracle --rag-index --force`
+
+## Blackwell Training Infrastructure (trueno#200, trueno#203)
+
+### Blackwell sm_121 JIT Bug (trueno#200)
+
+**Problem**: `cuModuleLoadData` / `cuModuleLoadDataEx` fails with `CUDA_ERROR_UNKNOWN` on Blackwell (sm_121) GPUs when called during active GPU work (concurrent kernels, active streams, etc.). This specifically affects **backward (training) kernels** -- forward kernels work after a pre-warming phase.
+
+**Root Cause**: The NVIDIA JIT compiler on Blackwell has a bug where PTX-to-SASS compilation via the driver API fails non-deterministically when the GPU is already under load. This does NOT affect cuBLAS calls or pre-compiled cubin modules.
+
+**Workaround -- `from_ptx_direct`**: A Blackwell-safe PTX loading path that skips `cuModuleLoadDataEx` entirely:
+- Compiles PTX to cubin offline or at initialization time (before any GPU work)
+- Loads only pre-compiled cubin blobs during training
+- Forward PTX kernels work after pre-warming (loading all kernel variants before training starts)
+
+**Key Distinction**:
+- **Forward kernels**: Work after pre-warming (all variants loaded before first training step)
+- **Backward kernels**: Crash during training because they are compiled on-demand when the GPU is already active
+- **Inference (NOT affected)**: Uses cuBLAS and SIMD paths, no custom PTX compilation at runtime
+
+### Dimension-Independent Kernels Plan (trueno#203)
+
+**Current Architecture**: Dynamic PTX generation with dimensions (M, K, N) baked into the PTX source. This produces **50+ kernel variants** (one per unique shape) and requires JIT compilation for each new shape encountered at runtime.
+
+**Target Architecture**: Dimension-independent kernels that accept M, K, N as runtime parameters. This reduces the total kernel count to **~15 types** (GEMM, softmax, layernorm, attention, backward variants, etc.), each compiled once.
+
+**Pre-Compiled cubin Pipeline** (the real fix for JIT issues):
+```
+build.rs → nvcc (offline) → cubin blobs → include_bytes!() → zero JIT at runtime
+```
+
+- `build.rs` invokes `nvcc` to compile PTX to cubins for target architectures (sm_80, sm_89, sm_121)
+- cubin blobs are embedded in the binary via `include_bytes!()`
+- Runtime loads cubins directly -- no JIT compilation, no `cuModuleLoadDataEx`
+- Eliminates the Blackwell JIT bug entirely since no runtime PTX compilation occurs
+
+**Provable Contract**: `dimension-independent-kernels-v1.yaml`
+
+**Impact on Entrenar**: Training is currently blocked by the backward kernel JIT crash (trueno#200). The dimension-independent kernel architecture (trueno#203) is the permanent fix. Until then, inference via `apr run` is fully operational (uses cuBLAS/SIMD, not custom PTX).

contracts/dimension-independent-kernels-v1.yaml

@@ -0,0 +1,134 @@
+# Dimension-Independent Kernels Contract v1.0.0
+# THE SOURCE OF TRUTH for eliminating JIT compilation during GPU training
+#
+# STATUS: Authoritative — all trueno-gpu training kernels MUST implement this contract
+# CONSUMERS:
+#   - trueno-gpu/src/kernels/backward/* (backward pass kernels)
+#   - trueno-gpu/src/kernels/elementwise/rope/batched.rs (RoPE forward+backward)
+#   - trueno-gpu/src/kernels/elementwise/transform/* (layout transform kernels)
+#   - entrenar/src/autograd/cuda_forward/cache.rs (kernel pre-warming)
+#   - entrenar/src/autograd/cuda_backward/cache.rs (backward kernel cache)
+#
+# Problem Statement:
+#   trueno-gpu kernels bake dimensions (M, K, N, hidden_dim, num_heads, etc.)
+#   as compile-time immediates in PTX. When entrenar encounters a new dimension
+#   at training time, it JIT-compiles new PTX — but on Blackwell (sm_121),
+#   cuModuleLoadDataEx poisons the CUDA context during active GPU work
+#   (trueno#200). Even with from_ptx_direct fallback, JIT during training adds
+#   latency and creates fragile cache dependencies.
+#
+# Solution:
+#   All training kernels MUST pass dimensions as .param runtime parameters,
+#   NOT as baked immediates. This means:
+#   - ONE PTX per kernel type (not per dimension combination)
+#   - Pre-compile ~15-20 cubins at startup
+#   - Zero JIT during training
+#
+# References:
+#   - trueno#200: Blackwell cuModuleLoadData fails during active GPU work
+#   - trueno#203: Dimension-independent kernels architecture
+#   - S18.10: 8 training failures from JIT during GPU work
+#   - S18.13: ELI5 — PyTorch ships pre-compiled, we JIT-compile
+#   - S18.14: Dimension-independent kernels plan
+
+metadata:
+  version: "1.0.0"
+  created: "2026-03-22"
+  author: "PAIML Engineering"
+  description: "All GPU training kernels must be dimension-independent (runtime params, not baked PTX)"
+  references:
+    - "trueno#200: Blackwell cuModuleLoadDataEx poisons CUDA context"
+    - "trueno#203: Dimension-independent kernels architecture"
+    - "bashrs S18.14: Dimension-independent kernels plan"
+  issues:
+    - "https://github.com/paiml/trueno/issues/200"
+    - "https://github.com/paiml/trueno/issues/203"
+
+# Kernels audited as BAKING dimensions (2026-03-22):
+#
+# BAKED (20 kernels):
+#   backward/gemm.rs: GemmBackwardA/B tiled/tiled_unrolled (tile_size, n/k baked)
+#   backward/rms_norm.rs: RmsNormBackward (hidden_dim structure), BatchedRmsNormBackward (all baked)
+#   backward/softmax.rs: SoftmaxBackward (row_size structure), BatchedSoftmaxBackward (all baked)
+#   backward/layer_norm.rs: LayerNormBackward (hidden_dim structure)
+#   elementwise/rope/batched.rs: BatchedRopeKernel, BatchedRopeBackwardKernel (all baked, NO params)
+#   elementwise/transform/layout.rs: InterleavedToBatched, BatchedToInterleaved, Extract/CopySingleHead
+#   elementwise/transform/element_wise.rs: BatchedScale, BatchedSoftmax
+#   elementwise/transform/transpose.rs: Transpose, BatchedTranspose
+#
+# OK (10 kernels):
+#   backward/activations.rs: Relu/Gelu/SiluBackward (n is runtime param)
+#   backward/gemm.rs: GemmBackwardA/B naive (m,n,k all runtime params)
+#   backward/cross_entropy.rs: FusedCrossEntropy, FusedCausalCrossEntropy (vocab_size runtime)
+#   optimizer/adamw.rs: AdamWStep, AdamStep (all runtime params)
+#   optimizer/clip.rs: GradientClip (all runtime params)
+
+tests:
+  - id: FALSIFY-DIM-001
+    name: "Kernel PTX has no baked dimensions"
+    description: >
+      For each training-critical kernel type, emit PTX with constructor args
+      (M=4, K=64, N=64) and (M=8, K=128, N=256). If the PTX text differs,
+      the kernel bakes dimensions as immediates.
+      Test file: trueno-gpu/src/kernels/tests/dimension_independence.rs
+    prediction: FAIL_BEFORE_FIX
+    if_fails: "Kernel bakes dimensions — needs refactor to use runtime params"
+    severity: critical
+    current_status: "20 kernels FAIL, 10 PASS"
+
+  - id: FALSIFY-DIM-002
+    name: "Same PTX for different dimensions (backward kernels)"
+    description: >
+      For the 7 backward kernel types used in training, generate PTX with
+      two different dimension sets. PTX source MUST be byte-identical.
+      Kernels: silu_backward, batched_rms_norm_backward, batched_softmax_backward,
+      gemm_backward_a, gemm_backward_b, layer_norm_backward, cross_entropy.
+    prediction: PASS_AFTER_FIX
+    if_fails: "Backward kernel PTX differs between dimensions"
+    severity: critical
+
+  - id: FALSIFY-DIM-003
+    name: "Total unique training kernel PTX count <= 20"
+    description: >
+      Enumerate all kernel types used during a Qwen3-4B training step.
+      Count distinct PTX templates. Must be <= 20 to make pre-compilation
+      feasible. Currently ~50+ variants due to dimension baking.
+    prediction: PASS_AFTER_FIX
+    if_fails: "Too many kernel variants for pre-compilation"
+    severity: high
+
+  - id: FALSIFY-DIM-004
+    name: "Layout transform kernels are dimension-independent"
+    description: >
+      InterleavedToBatched, BatchedToInterleaved, BatchedTranspose,
+      BatchedSoftmax, and TransposeKernel must produce identical PTX
+      regardless of constructor dimensions. These are the kernels seen
+      JIT-compiling in Run 12 logs.
+    prediction: FAIL_BEFORE_FIX
+    if_fails: "Layout kernel still bakes dimensions"
+    severity: critical
+    current_status: "All 5 FAIL — these are the Run 12 JIT offenders"
+
+  - id: FALSIFY-DIM-005
+    name: "RoPE kernels are dimension-independent"
+    description: >
+      BatchedRopeKernel and BatchedRopeBackwardKernel must produce identical
+      PTX regardless of num_heads, head_dim, or theta. Currently they have
+      ZERO runtime params for these values — everything is baked.
+    prediction: FAIL_BEFORE_FIX
+    if_fails: "RoPE kernel still bakes head_dim/num_heads/theta"
+    severity: critical
+    current_status: "Both FAIL — no runtime params at all"
+
+  - id: FALSIFY-DIM-006
+    name: "Declared params are actually loaded"
+    description: >
+      Several kernels (BatchedRmsNormBackward, BatchedSoftmaxBackward,
+      TransposeKernel, BatchedTransposeKernel) declare .param entries but
+      never call ld.param to load them. The closure uses baked immediates
+      instead. For each kernel, verify that every declared .param has a
+      corresponding ld.param instruction in the PTX body.
+    prediction: FAIL_BEFORE_FIX
+    if_fails: "Phantom params — declared but never loaded"
+    severity: high
+    current_status: "8+ kernels have phantom params"

docs/specifications/trueno-gpu-spec.md

@@ -1429,3 +1429,105 @@ impl PerformanceGate {
 2. Implement PTX builder (TG-001)
 3. Write acceptance tests for each kernel
 4. Begin GEMM optimization journey
+
+---
+
+## 14. Dimension-Independent Kernel Architecture (trueno#203)
+
+### 14.1 Current Architecture: Dynamic PTX with Baked Dimensions
+
+The current kernel generation system bakes matrix dimensions (M, K, N) directly into PTX source code. Each unique shape produces a distinct PTX module that must be JIT-compiled by the CUDA driver:
+
+```
+Current Flow:
+┌─────────────────┐    ┌──────────────────┐    ┌──────────────────┐
+│  Kernel Request  │───▶│ Generate PTX     │───▶│ cuModuleLoadData │
+│  (M=2560, K=896) │    │ with baked dims  │    │ (JIT compile)    │
+└─────────────────┘    └──────────────────┘    └──────────────────┘
+                                                        │
+                        50+ kernel variants              │
+                        (one per unique shape)            ▼
+                                                 ┌──────────────┐
+                                                 │ GPU Execution │
+                                                 └──────────────┘
+```
+
+**Problems with this approach**:
+- **50+ kernel variants**: Every new dimension triplet generates a new PTX module
+- **JIT overhead**: Each variant requires runtime PTX-to-SASS compilation (~50-200ms per kernel)
+- **Blackwell JIT bug (trueno#200)**: `cuModuleLoadDataEx` fails during active GPU work on sm_121
+- **Cache pressure**: Large number of compiled kernels consumes GPU code cache
+
+### 14.2 Target Architecture: Dimension-Independent Kernels
+
+Kernels accept M, K, N as runtime parameters via kernel arguments rather than compile-time constants. This reduces the total kernel count to approximately **15 kernel types**:
+
+| # | Kernel Type | Direction | Notes |
+|---|-------------|-----------|-------|
+| 1 | GEMM (tiled) | Forward | General matrix multiply |
+| 2 | GEMM (NF4 fused) | Forward | Dequant + matmul fused |
+| 3 | Softmax | Forward | Warp shuffle reduction |
+| 4 | LayerNorm (RMS) | Forward | Fused RMSNorm |
+| 5 | RoPE | Forward | Rotary position embedding |
+| 6 | QK-Norm | Forward | Per-head RMS normalization |
+| 7 | SiLU | Forward | Activation function |
+| 8 | GEMM backward | Backward | dL/dW, dL/dx |
+| 9 | Softmax backward | Backward | Jacobian-vector product |
+| 10 | LayerNorm backward | Backward | dL/dgamma, dL/dbeta, dL/dx |
+| 11 | RoPE backward | Backward | Reverse rotation |
+| 12 | LoRA forward | Forward | Low-rank adapter matmul |
+| 13 | LoRA backward | Backward | Adapter gradient accumulation |
+| 14 | Cross-entropy loss | Forward | Log-softmax + NLL |
+| 15 | Adam update | Optimizer | Fused parameter update |
+
+```
+Target Flow:
+┌─────────────────┐    ┌──────────────────────┐    ┌──────────────────┐
+│  build.rs       │───▶│ nvcc --cubin          │───▶│ include_bytes!() │
+│  (compile time) │    │ 15 kernels × 3 archs  │    │ (embedded blobs) │
+└─────────────────┘    └──────────────────────┘    └──────────────────┘
+                                                            │
+                                                            ▼
+┌─────────────────┐    ┌──────────────────────┐    ┌──────────────────┐
+│  Kernel Request  │───▶│ cuModuleLoadData     │───▶│ cuLaunchKernel   │
+│  (M=2560, K=896) │    │ (pre-compiled cubin) │    │ (M,K,N as args)  │
+└─────────────────┘    └──────────────────────┘    └──────────────────┘
+                        Zero JIT compilation          Dims via params
+```
+
+### 14.3 Pre-Compilation Pipeline
+
+```bash
+# build.rs pseudocode
+for arch in [sm_80, sm_89, sm_121] {
+    for kernel in [gemm, softmax, layernorm, ...] {
+        nvcc --cubin --gpu-architecture={arch} {kernel}.ptx -o {kernel}_{arch}.cubin
+    }
+}
+```
+
+The pre-compiled cubin blobs are embedded via `include_bytes!()`:
+```rust
+const GEMM_SM80: &[u8] = include_bytes!("cubins/gemm_sm80.cubin");
+const GEMM_SM89: &[u8] = include_bytes!("cubins/gemm_sm89.cubin");
+const GEMM_SM121: &[u8] = include_bytes!("cubins/gemm_sm121.cubin");
+```
+
+### 14.4 Blackwell Compatibility
+
+This architecture **eliminates the Blackwell JIT bug entirely**:
+- No `cuModuleLoadDataEx` calls at runtime (no PTX JIT)
+- Pre-compiled cubins load via `cuModuleLoadData` which reads binary, no compilation
+- Safe to load modules at any time, even during active GPU work
+- Forward and backward kernels both work without pre-warming
+
+### 14.5 Provable Contract
+
+Contract file: `provable-contracts/contracts/dimension-independent-kernels-v1.yaml`
+
+Key assertions:
+- FALSIFY-DIK-001: All 15 kernel types accept M/K/N as runtime params
+- FALSIFY-DIK-002: Zero `cuModuleLoadDataEx` calls after initialization
+- FALSIFY-DIK-003: cubin blobs present for sm_80, sm_89, sm_121
+- FALSIFY-DIK-004: Dimension-independent GEMM produces identical results to baked-dimension GEMM
+- FALSIFY-DIK-005: Backward kernels work during active GPU training (no JIT crash)