|
| 1 | +# Why the self-improvement loop stopped improving — a 34-night audit |
| 2 | + |
| 3 | +**Date:** 2026-08-11 · **Data:** `planning/self_improve_ledger_aocc.jsonl`, |
| 4 | +952 records / 680 proposals / 34 nights (2026-07-09 .. 2026-08-11), archived |
| 5 | +at rotation to `planning/done/self_improve_ledger_aocc_2026-08-11.jsonl`. |
| 6 | + |
| 7 | +This document exists so that no future session has to re-derive any of it. |
| 8 | +It records what the loop's first 34 nights actually produced, why, and what |
| 9 | +was changed in response (PRs #299–#302). Read it alongside `GOAL.md` §2. |
| 10 | + |
| 11 | +--- |
| 12 | + |
| 13 | +## 1. The headline |
| 14 | + |
| 15 | +**Over 34 nights and 680 proposals the loop produced one measurable |
| 16 | +improvement, worth +0.005 mean AOCC.** |
| 17 | + |
| 18 | +| quantity | value | |
| 19 | +|---|---| |
| 20 | +| first-of-night baseline, 34 nights | mean 0.3398, sd 0.0061 | |
| 21 | +| linear trend | +0.00015/night, t = **+1.39** (not significant) | |
| 22 | +| first 5 nights → last 5 nights | 0.3433 → **0.3420** | |
| 23 | +| proposals | 680 | |
| 24 | +| accepts | 72 (10.6%) | |
| 25 | +| codify slots that reached a 12-seed A/B | 5 | |
| 26 | +| codify slots that survived it | **0** | |
| 27 | + |
| 28 | +The one improvement — dropping the `Restart` analyzer on 2026-07-30 — |
| 29 | +claimed **+0.0384** in its local A/B and delivered **+0.0049** in the nightly |
| 30 | +baseline (n=22 before vs 12 after, SE 0.0018; real, but ~8× smaller than |
| 31 | +claimed). That +0.0049 is essentially the *entire* 34-night drift. Treat |
| 32 | +every future A/B headline number as inflated several-fold; this is textbook |
| 33 | +winner's curse and it is now a measured local constant, not a worry. |
| 34 | + |
| 35 | +--- |
| 36 | + |
| 37 | +## 2. The accept rate was exactly what pure noise predicts |
| 38 | + |
| 39 | +``` |
| 40 | +proposal deltas: n=680 mean -0.00057 sd 0.01011 |
| 41 | +eps_accept: 0.005 <- 0.5 sd of the noise |
| 42 | +P(delta > 0.005): 0.301 |
| 43 | +screening accepts: 210/680 = 0.309 <- matches |
| 44 | +confirm survival: 72/210 = 0.343 (independent redraw predicts 0.30) |
| 45 | +joint: 0.30 x 0.34 = 0.106 vs observed accept rate 0.106 |
| 46 | +``` |
| 47 | + |
| 48 | +The screening bar sat at **half a standard deviation** of the measurement |
| 49 | +noise, so roughly 30% of *zero-effect* proposals cleared it, and the |
| 50 | +confirmation gate re-rolled at the same bar against a fresh |
| 51 | +`randomize_iteration`. There is no room left in the observed accept rate for |
| 52 | +a real signal. The loop was not finding weak improvements; it was sampling |
| 53 | +its own noise twice. |
| 54 | + |
| 55 | +--- |
| 56 | + |
| 57 | +## 3. Four defects, all in the instrument |
| 58 | + |
| 59 | +### 3.1 The hold-out leg measured a different metric entirely |
| 60 | + |
| 61 | +`SelfImprover._measure()` branches on `metric == "aocc"`. |
| 62 | +`_measure_holdout()` did not — it went straight to |
| 63 | +`holdout_harness_config()` → the **composite** harness. |
| 64 | + |
| 65 | +| | mean | |
| 66 | +|---|---| |
| 67 | +| `seed_training_score` (AOCC) | 0.3402 | |
| 68 | +| `seed_holdout_score` (actually `composite_score`) | 0.0339 | |
| 69 | +| composite ledger's own baseline mean | **0.0452** ← same scale | |
| 70 | + |
| 71 | +So the 8.5× "instance-family generalization gap" that headed `GOAL.md` §5.1 |
| 72 | +as the project's number-one research priority from 2026-07-30 was **a unit |
| 73 | +mismatch**. After the fix a live quick-battery run measures **0.3383 |
| 74 | +training vs 0.3342 hold-out** — a 0.004 gap. It also explains |
| 75 | +`overfit=False` on all 66 hold-out records: the gate was differencing two |
| 76 | +incommensurable quantities. |
| 77 | + |
| 78 | +The ratio evidence had pointed the right way before the cause was found — |
| 79 | +the *untuned* seed spec dropped by the same factor as the tuned one |
| 80 | +(0.0996 vs 0.1166), which already ruled out overfitting. The wrong |
| 81 | +conclusion drawn from it at first was "the hold-out battery is just harder"; |
| 82 | +the real answer was in the code. |
| 83 | + |
| 84 | +### 3.2 No accept decision ever crossed an instance-family boundary |
| 85 | + |
| 86 | +The §6.4 confirmation gate's hold-out leg was guarded by |
| 87 | +`and self.config.metric != "aocc"` — added precisely because of §3.1. |
| 88 | + |
| 89 | +``` |
| 90 | +confirm_holdout_seed across 138 confirm records: {None: 138} |
| 91 | +accepts citing a hold-out base seed: 0 / 72 |
| 92 | +``` |
| 93 | + |
| 94 | +The confirmation only ever drew a fresh `randomize_iteration` *inside* base |
| 95 | +seed 42. |
| 96 | + |
| 97 | +### 3.3 The nightly never passed `--base-seed` |
| 98 | + |
| 99 | +``` |
| 100 | +base_seed across all 952 records: {42: 952} |
| 101 | +``` |
| 102 | + |
| 103 | +`--base-seed` existed and defaulted to 42; the workflow simply never set it. |
| 104 | +Codify-scan's "k ≥ 2 distinct nights" gate — and the `--min-fresh-nights` |
| 105 | +resurrection gate added 2026-08-08 — were therefore counting **k |
| 106 | +re-measurements of one instance draw** as k independent confirmations. |
| 107 | + |
| 108 | +This is the mechanical cause of the 0/5 codify hit rate. Five slots |
| 109 | +(`Sensitivity.update_interval`, `Sensitivity drop_analyzer`, `NelderMead |
| 110 | +drop_heuristic`, `JSO drop_heuristic`, `NLSHADE_LBC add_heuristic`) each had |
| 111 | +a tight pooled CI excluding zero, and each died at 12 seeds. The transfer |
| 112 | +coefficient is measurable directly: |
| 113 | + |
| 114 | +``` |
| 115 | +corr(training_delta, holdout_delta) = +0.175 (n = 66) |
| 116 | +positive on training 32/66 positive on holdout 33/66 |
| 117 | +``` |
| 118 | + |
| 119 | +r = 0.175 — the nightly measurement explained 3% of the variance in what |
| 120 | +happened on a different seed. |
| 121 | + |
| 122 | +### 3.4 `--sync-eval` was unreachable from the loop |
| 123 | + |
| 124 | +Shipped 2026-08-09 on `scripts/ioh_benchmark.py`, where it cut measurement |
| 125 | +noise 1.6×, but never plumbed into `scripts/self_improve.py run`. The |
| 126 | +process generating *all* the evidence was the one not using it. Measured |
| 127 | +cost when finally wired: **2.6 s vs 2.7 s** on the quick battery. |
| 128 | + |
| 129 | +--- |
| 130 | + |
| 131 | +## 4. The deeper problem: a scalar objective over heterogeneous regimes |
| 132 | + |
| 133 | +Fixing precision is necessary but not sufficient. On 2026-08-11 (PR #298) |
| 134 | +the NL-SHADE-LBC arm produced the first per-dimension split where both CIs |
| 135 | +exclude zero, in opposite directions: |
| 136 | + |
| 137 | +| dim | Δmean | CI95 | |
| 138 | +|---|---|---| |
| 139 | +| d2 | **−0.0241** | [−0.0401, −0.0080] | |
| 140 | +| d5 | **+0.0080** | [+0.0007, +0.0154] | |
| 141 | +| scalar composite | −0.0080 | "lean-negative" | |
| 142 | + |
| 143 | +And the same arm leaned *positive* at 2-D×200 evals while losing at |
| 144 | +2-D×1000 — a budget effect on top of the dimension effect. |
| 145 | + |
| 146 | +If effects are routinely that heterogeneous, the population-mean objective |
| 147 | +has a **flat optimum by construction**. No amount of extra measurement |
| 148 | +precision helps, because the surface really is level; the loop was climbing |
| 149 | +an average of two hills that point in opposite directions. |
| 150 | + |
| 151 | +The power arithmetic underlines it: between-seed paired delta sd is ~0.025, |
| 152 | +so detecting a +0.005 *population* effect at 80% power needs **~190 seeds |
| 153 | +per proposal** — about 45 minutes per decision at 6.8 s/iteration. Chasing |
| 154 | +sub-0.01 scalar effects was never going to work at this budget. The way out |
| 155 | +is not more seeds; it is asking a better-posed question. |
| 156 | + |
| 157 | +--- |
| 158 | + |
| 159 | +## 5. What was changed (2026-08-11) |
| 160 | + |
| 161 | +| PR | change | |
| 162 | +|---|---| |
| 163 | +| **#299** | hold-out routes through the AOCC path (§3.1); confirm gate crosses a base seed (§3.2); nightly rotates `--base-seed` over 7 values (§3.3); `--sync-eval` reachable and on (§3.4); `eps_accept` 0.005 → 0.0125 (2σ of the post-sync-eval floor), relax floor 0.001 → 0.006 | |
| 164 | +| **#300** | `--aocc-extra-dims`; nightly battery `dims=(2,)` → `(2, 5)`; cost 1.34× | |
| 165 | +| **#301** | `--accept-stat rank` — Wilcoxon signed-rank + Hodges-Lehmann. **Available, not enabled** | |
| 166 | +| **#302** | `--cell-by dim` — per-dim deltas/CIs reported and gated; on in the nightly | |
| 167 | + |
| 168 | +Under the null, the joint screen+confirm false-positive rate goes from ~9% |
| 169 | +to ~0.05%; power for a true +0.02 effect stays ~77%. Effects below ~0.01 |
| 170 | +were never shippable — five A/Bs proved that — so the loop should stop |
| 171 | +spending nights on them. |
| 172 | + |
| 173 | +`#301` is deliberately **not** enabled. Tonight already changes the accept |
| 174 | +regime three ways; a fourth simultaneous change to the accept *rule* would |
| 175 | +make the next few weeks unattributable. That discipline is the whole lesson |
| 176 | +of §3. |
| 177 | + |
| 178 | +--- |
| 179 | + |
| 180 | +## 6. Reading the ledger across this boundary |
| 181 | + |
| 182 | +Records before and after 2026-08-11 are **not comparable**, in three |
| 183 | +independent ways: single vs rotated base seed, async vs sync evaluation, and |
| 184 | +d2-only vs (d2, d5) battery — the last of which moves the score *level* |
| 185 | +(d2 alone ~0.369, (d2, d5) ~0.309), not just its noise. |
| 186 | + |
| 187 | +Each iteration record now carries `base_seed`, `sync_eval`, |
| 188 | +`aocc_extra_dims`, `accept_stat`, and `per_cell`, precisely so a consumer can |
| 189 | +group rather than blindly pool. **Codify-scan does not yet honour these** — |
| 190 | +that is queued in `TODO.md` and is the highest-value remaining item, because |
| 191 | +until it does, cross-night evidence still mixes regimes. |
| 192 | + |
| 193 | +The pre-boundary ledger was rotated into `planning/done/` at this point. The |
| 194 | +bandit posterior is preserved: the nightly passes `--prime-include-archives`, |
| 195 | +which replays archived ledgers. What resets is codify-scan's evidence base, |
| 196 | +which is the intent — 34 nights of single-draw evidence should count as one |
| 197 | +night's worth, not 34. |
| 198 | + |
| 199 | +--- |
| 200 | + |
| 201 | +## 7. What to do next |
| 202 | + |
| 203 | +1. **Teach codify-scan to read `per_cell` and to group by |
| 204 | + `base_seed` / `sync_eval` / `aocc_extra_dims`.** Without this the new |
| 205 | + instrumentation is recorded but not actionable. |
| 206 | +2. **Let the rotated-seed ledger accumulate**, then A/B `--accept-stat rank` |
| 207 | + against the mean rule on accept rate and codify survival. |
| 208 | +3. **Budget-phase cells** — needs AOCC recomputed on trajectory slices; |
| 209 | + `trace_evals` / `trace_fx` are already recorded. |
| 210 | +4. **Dimension/budget-gated arm activation** — the shippable form of the d5 |
| 211 | + gains that keep showing up (NLSHADE_LBC, and probably CMA-ES; re-measure |
| 212 | + that one at d5 first). |
| 213 | +5. **Re-earn the codify backlog.** Every slot rejected on pre-2026-08-11 |
| 214 | + evidence was rejected for the right reason (it did not replicate), but on |
| 215 | + an instrument that could not have told the difference. They are open |
| 216 | + questions again, not settled ones — though the burden of proof is on |
| 217 | + anyone who wants to re-litigate them. |
0 commit comments