feat(benchmark): dimension-gated arm activation — NLSHADE_LBC enters Rewarding_Restart at d≥5 - #304
Merged
Merged
Conversation
…ewarding_Restart at d>=5 StrategySpec.create_strategy honours two reserved heuristic-kwargs keys, gate_min_dim / gate_max_dim, which decide whether the arm is instantiated at all based on problem.dim (keys are stripped before the heuristic constructor). This is the shippable form PR #298 asked for: the same arm measured d2 -0.0241 [-0.0401,-0.0080] vs d5 +0.0080 [+0.0007,+0.0154], so an unconditional add cannot express the gain. First user: (NLSHADE_LBC, {NP_init: auto, k_rank: 3.0, gate_min_dim: 5}) in Rewarding_Restart. 12-seed standard paired A/B (--sync-eval both sides): d5 +0.0062 [-0.0001,+0.0125], d2 flat (gate off, spec identical to master), control flat; pooled with #298's independent d5 estimate: +0.0070 [+0.0027,+0.0112]. Full pytest green (2014 passed); ruff format clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W7zJ73CD59Matb6HdSdrp2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
codify-slot: NLSHADE_LBC/add_heuristic (shipped in its gated form; the 2026-08-11 rejection was "rejected as an unconditional add")
What
Daily improvement session (GOAL.md §3/§4). Today's codify scan surfaced 0 actionable candidates (13 total: 5 already codified, 8 rejection-suppressed; last night's two accepts reinforce already-codified slots), so this session executes the §4 attack that PR #298 queued: dimension-gated arm activation (GOAL §5.1(d)).
StrategySpec.create_strategynow honours two reserved heuristic-kwargs keys,gate_min_dim/gate_max_dim: the arm is only instantiated whengate_min_dim <= problem.dim <= gate_max_dim(either bound optional). The keys are stripped before the heuristic constructor sees them; curatedMutationRules name (class, param) explicitly, so the reserved keys cannot be accidentally mutated by the nightly loop.(NLSHADE_LBC, {"NP_init": "auto", "k_rank": 3.0, "gate_min_dim": 5})inRewarding_Restart— the exact structural-catalog kwargs NLSHADE_LBC add to Rewarding_Restart: REJECTED unconditionally — first CI-significant per-dim split (d5 gain, d2 loss); log entry + revert #298 measured. At d<5 the spec is arm-for-arm identical to master.gate_max_dim, no mutation of the spec's kwargs dict.Evidence
Why gated — #298's 12-seed standard A/B measured the unconditional add at d2 −0.0241 [−0.0401, −0.0080] vs d5 +0.0080 [+0.0007, +0.0154]: both CIs exclude zero, in opposite directions. The scalar objective could not express the gain; the gate ships it without the d2 loss.
Ledger nights —
add_heuristic NLSHADE_LBC: 3 confirmed accepts across 2 nights (2026-07-09, 2026-08-10), pooled CI95 [+0.0075, +0.0192] — all on theRoundRobin_Randomcontrol, which is why no codify slot could reach the candidate spec and a direct A/B is the vehicle.Local paired A/B (this PR) — 12-seed canonical roster,
--standard,--sync-evalboth sides, before = master 92b603e:Rewarding_RestartRewarding_RestartRoundRobin_Random(control)RoundRobin_Random(control)At d5 the gated spec is byte-identical to the unconditional spec #298 measured, so #298's d5 row is an independent replication of the same effect. Fixed-effect inverse-variance pooling of the two 12-seed estimates (+0.0080, se 0.0033; +0.0062, se 0.0029):
pooled d5 effect: +0.0070, CI95 [+0.0027, +0.0112] — excludes zero.
The quick battery (d2-only) was not re-measured: with the gate off the spec is identical to master there, and today's d2 row confirms flat.
Guardrail compliance
composite_scoreformula and default batteries untouched (the legacy composite registry uses no gate keys, so its behavior is unchanged).--extra-highdimextend-don't-edit pattern.Validation
uv run pytest -q: 2014 passed, 1 skipped (coco not installed)uv run ruff format --check .: cleanuv run ruff checkon touched source files: clean (the 5 pre-existingtests/test_benchmark.pyfindings on master are unchanged)planning/SELF_IMPROVEMENT_LOG.mddated 2026-08-12 entry,TODO.mdandplanning/GOAL.md§5.1(d) updated in this PRFollow-ups queued (in TODO.md)
per_cell(GOAL §5.1(c)) so cell-conditional proposals become automatic.🤖 Generated with Claude Code
https://claude.ai/code/session_01W7zJ73CD59Matb6HdSdrp2
Generated by Claude Code