Skip to content

Add osaurus bench --gauntlet: measured model-capability probes writing the ledger#1916

Open
mimeding wants to merge 1 commit into
osaurus-ai:mainfrom
mimeding:mlx-perf/gauntlet
Open

Add osaurus bench --gauntlet: measured model-capability probes writing the ledger#1916
mimeding wants to merge 1 commit into
osaurus-ai:mainfrom
mimeding:mlx-perf/gauntlet

Conversation

@mimeding

@mimeding mimeding commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

What

Adds osaurus bench --gauntlet, a seven-probe model verification workflow:

  1. Non-empty load/stream response
  2. Template-marker leakage across content and reasoning channels
  3. Forced tool call with exact function name and schema-valid arguments
  4. Multi-character stop handling with an unstopped control run
  5. Long-generation degeneration detection, including two-token loops and character runs
  6. MTP-versus-AR comparison evidence
  7. Prefill/decode performance snapshot

Results are field-merged into the capability ledger without deleting existing verdicts, block reasons, digests, or unknown fields.

Safety contract

The gauntlet records per-probe evidence and a production-serving candidate result. It does not change productionServing in v1 because the runtime cannot yet bind that promotion to a verified weights digest. Failed or unproven critical probes exit nonzero. MTP equivalence remains untested unless the runtime can prove per-request MTP activation.

Validation

  • BenchGauntletTests: 20 passed
  • DegenerationDetectorTests: 12 passed
  • Scoped SwiftLint: passed
  • git diff --check: passed
  • Independent runtime review findings addressed: no vacuous stop/template passes, no stale ledger promotion, no destructive record replacement, and no two-token degeneration blind spot

Risk and rollback

CLI and evidence-writing scope only. Runtime model behavior, prompts, templates, samplers, and generation defaults are unchanged. Reverting the single feature commit removes the workflow.

@mimeding

mimeding commented Jul 7, 2026

Copy link
Copy Markdown
Contributor Author

Adversarial-review hardening pushed (e275312): stop-sequence probe now requires positive evidence (counted to the boundary) and fails on leaked stop text; mtp-equivalence and the canary guard against vacuous empty-transcript passes via real usage token counts; --tune-prefill/--gauntlet mutually exclusive. 15 new tests.

@mimeding mimeding force-pushed the mlx-perf/gauntlet branch from 5adb9c1 to 8a8367b Compare July 10, 2026 23:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant