Skip to content

feat(benchmark): dimension-gated arm activation — NLSHADE_LBC enters Rewarding_Restart at d≥5 - #304

Merged
haraldschilly merged 1 commit into
masterfrom
claude/focused-carson-se2f3d
Aug 13, 2026
Merged

feat(benchmark): dimension-gated arm activation — NLSHADE_LBC enters Rewarding_Restart at d≥5#304
haraldschilly merged 1 commit into
masterfrom
claude/focused-carson-se2f3d

Conversation

@haraldschilly

Copy link
Copy Markdown
Owner

codify-slot: NLSHADE_LBC/add_heuristic (shipped in its gated form; the 2026-08-11 rejection was "rejected as an unconditional add")

What

Daily improvement session (GOAL.md §3/§4). Today's codify scan surfaced 0 actionable candidates (13 total: 5 already codified, 8 rejection-suppressed; last night's two accepts reinforce already-codified slots), so this session executes the §4 attack that PR #298 queued: dimension-gated arm activation (GOAL §5.1(d)).

  • StrategySpec.create_strategy now honours two reserved heuristic-kwargs keys, gate_min_dim / gate_max_dim: the arm is only instantiated when gate_min_dim <= problem.dim <= gate_max_dim (either bound optional). The keys are stripped before the heuristic constructor sees them; curated MutationRules name (class, param) explicitly, so the reserved keys cannot be accidentally mutated by the nightly loop.
  • First user: (NLSHADE_LBC, {"NP_init": "auto", "k_rank": 3.0, "gate_min_dim": 5}) in Rewarding_Restart — the exact structural-catalog kwargs NLSHADE_LBC add to Rewarding_Restart: REJECTED unconditionally — first CI-significant per-dim split (d5 gain, d2 loss); log entry + revert #298 measured. At d<5 the spec is arm-for-arm identical to master.
  • Unit tests pin: gate below threshold, admit at threshold, key stripping, gate_max_dim, no mutation of the spec's kwargs dict.

Evidence

Why gated#298's 12-seed standard A/B measured the unconditional add at d2 −0.0241 [−0.0401, −0.0080] vs d5 +0.0080 [+0.0007, +0.0154]: both CIs exclude zero, in opposite directions. The scalar objective could not express the gain; the gate ships it without the d2 loss.

Ledger nightsadd_heuristic NLSHADE_LBC: 3 confirmed accepts across 2 nights (2026-07-09, 2026-08-10), pooled CI95 [+0.0075, +0.0192] — all on the RoundRobin_Random control, which is why no codify slot could reach the candidate spec and a direct A/B is the vehicle.

Local paired A/B (this PR) — 12-seed canonical roster, --standard, --sync-eval both sides, before = master 92b603e:

strategy dim before after Δmean CI95 verdict
Rewarding_Restart d2 0.4330 0.4297 −0.0033 [−0.0131, +0.0065] flat (gate off — spec identical to master)
Rewarding_Restart d5 0.3026 0.3088 +0.0062 [−0.0001, +0.0125] positive, boundary
RoundRobin_Random (control) d2 0.3618 0.3613 −0.0005 [−0.0019, +0.0009] flat
RoundRobin_Random (control) d5 0.2834 0.2846 +0.0011 [−0.0052, +0.0074] flat

At d5 the gated spec is byte-identical to the unconditional spec #298 measured, so #298's d5 row is an independent replication of the same effect. Fixed-effect inverse-variance pooling of the two 12-seed estimates (+0.0080, se 0.0033; +0.0062, se 0.0029):

pooled d5 effect: +0.0070, CI95 [+0.0027, +0.0112] — excludes zero.

The quick battery (d2-only) was not re-measured: with the gate off the spec is identical to master there, and today's d2 row confirms flat.

Guardrail compliance

  • One codify/attack slot in this PR; no batching of independently-evidenced changes.
  • composite_score formula and default batteries untouched (the legacy composite registry uses no gate keys, so its behavior is unchanged).
  • Frozen presets untouched; the gate is opt-in per spec, following the --extra-highdim extend-don't-edit pattern.

Validation

  • uv run pytest -q: 2014 passed, 1 skipped (coco not installed)
  • uv run ruff format --check .: clean
  • uv run ruff check on touched source files: clean (the 5 pre-existing tests/test_benchmark.py findings on master are unchanged)
  • planning/SELF_IMPROVEMENT_LOG.md dated 2026-08-12 entry, TODO.md and planning/GOAL.md §5.1(d) updated in this PR

Follow-ups queued (in TODO.md)

  • Re-measure the CMA-ES arm at d5 with this instrument — if it splits like NLSHADE_LBC, the same gate ships it.
  • Codify-scan reading per_cell (GOAL §5.1(c)) so cell-conditional proposals become automatic.

🤖 Generated with Claude Code

https://claude.ai/code/session_01W7zJ73CD59Matb6HdSdrp2


Generated by Claude Code

…ewarding_Restart at d>=5

StrategySpec.create_strategy honours two reserved heuristic-kwargs keys,
gate_min_dim / gate_max_dim, which decide whether the arm is instantiated
at all based on problem.dim (keys are stripped before the heuristic
constructor).  This is the shippable form PR #298 asked for: the same
arm measured d2 -0.0241 [-0.0401,-0.0080] vs d5 +0.0080 [+0.0007,+0.0154],
so an unconditional add cannot express the gain.

First user: (NLSHADE_LBC, {NP_init: auto, k_rank: 3.0, gate_min_dim: 5})
in Rewarding_Restart.  12-seed standard paired A/B (--sync-eval both
sides): d5 +0.0062 [-0.0001,+0.0125], d2 flat (gate off, spec identical
to master), control flat; pooled with #298's independent d5 estimate:
+0.0070 [+0.0027,+0.0112].

Full pytest green (2014 passed); ruff format clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7zJ73CD59Matb6HdSdrp2
@haraldschilly
haraldschilly marked this pull request as ready for review August 13, 2026 19:31
@haraldschilly
haraldschilly merged commit fac26b8 into master Aug 13, 2026
8 checks passed
@haraldschilly
haraldschilly deleted the claude/focused-carson-se2f3d branch August 13, 2026 19:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants