NVMAI: Qwen 3.6 35B-A3B port with measured decode optimizations - #105
Open
Pummelchen wants to merge 310 commits into
Open
NVMAI: Qwen 3.6 35B-A3B port with measured decode optimizations#105Pummelchen wants to merge 310 commits into
Pummelchen wants to merge 310 commits into
Conversation
…wift test --no-parallel directly
…runs and runner scripts
- Remove all Gemma support (arch family, tool pipeline, repack sources, kernel names, fixtures); NVMAI is Qwen 3.6 only - Fix the audit findings across runtime, kernels, repack, server, and app: propagate command-buffer failures, restore runtime schema validation (verified against the real checkpoints), server timeouts/backpressure/ socket hardening, repack resume+path-security, dead-code removal - Trusted-install load policy: skip the full weights re-hash when verified-install.json exists (8-10x faster CLI/server startup) and warm the resident range with F_RDADVISE - Expert cache default 16 -> 32 slots, 64 accepted - Env-gated prefill phase diagnostics (TURBO_FIELDFARE_PHASES) Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
…ted 2x2 methodology - Restore Scripts/test.sh (serial suite) and the markdown link check that CI referenced but had deleted; release build now fails on warnings - Benchmark harness: replayable cache-warm sends that verify hits, PID-managed server lifecycle (no pkill self-match), answer verification, drop the impossible cache-on x mtp-on cell; README corrected with the methodology caveat - launch_8bit.sh: kill scoped to NVMAIServer only, wait-for-port-free - Qwen-only third-party notices; .gitignore dead patterns removed; AUDIT_TRACKER.md ledger of the audit and its resolution Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
Measured on the 4-bit model with a 1280-token prompt: prefill TTFT is 3.2x faster at a single 4096-token chunk than at 128-token chunks (13.6s vs 43.3s) and ~13% faster than the 1024 default (14.5s vs 16.7s). Larger chunks amortize the per-chunk route readbacks and fetch each distinct expert once per layer instead of re-fetching across chunks. Short prompts are unaffected (single chunk either way). Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
The two Scripts files were CI-only helpers; their logic now lives directly in ci.yml (swift test --no-parallel, and an inlined Ruby link check), so the folder is removed without losing the CI gates. Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
- logit.metal: write final_m/final_inv_d defaults before the cross-SIMD merge (threadgroup 0 lane 0 overwrites them) to silence the sometimes-uninitialized diagnostic; math unchanged - Remove unused constants/functions left over from the dead-kernel removal (attention kAttnMaxFullQPerKV/kAttnFullQPerThreadgroup, fused kFusedGroupSize/kFusedMaxD/fused_fc_d/fused_fc_n, prefill prefill_moe_int4_gemv_row_tg) — combined shader source now compiles warning-free Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
Measured on the 4-bit M3 (128-token decode, 5 interleaved rounds): the default (F_RDADVISE) policy is consistently ~10.6% faster than off (10.02 vs 9.06 tok/s; default beat off in all 5 rounds). The 18 GB expert pool does not fit the page cache, so the targeted read-ahead genuinely overlaps with GPU work. The M5/8-bit measurement found it neutral; the default now matches the deployment host's measured optimum. Host-dependent, so the runtime default follows the measurement. Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
…ot validate A receipt whose recorded model directory no longer matches (e.g. the model was moved after install) previously aborted the load once sizeCheckTrustedReceipt became the default policy. A non-validating receipt is now treated as 'no receipt': the weights file is fully re-hashed and the layer-layout check is skipped, restoring 6-bit and 8-bit installs that live at a different path than install time. Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
The HTTP server was not actually streaming: StreamingStopMatcher.utf8Boundary returned the start of the pending buffer when nothing needed retaining, so generated text accumulated in the matcher and only emerged as one full-text tail event (SSE delivered a single cumulative frame in a burst at completion, client TTFT ~= wall, and spec clients saw duplicated text). The boundary now emits the whole pending buffer, so per-token deltas stream incrementally. SSEOutbox.push handed a frame to the awaiting drainer without removing it from the queue, so every frame enqueued while the drainer was suspended was written twice; push now delivers each frame exactly once. Verified on a 64-token generation: 64 incremental deltas, no duplication, first content byte after prefill, and the prompt-cache continuation probe now hits (38 KV tokens restored) — the cache column of the 2x2 harness is measurable for the first time. benchmark: nvmai_benchmark.py reads decode rates from the server footer log (per-cell temperature 0 for MTP), parses MTP stats, and adds a two-turn continuation probe for the multi-prefix cache cell.
Three-case measurement (10k-token prefill, 1024-token generation, combined) using server-footer decode rates, RSS sampling for KV footprint, and per-chunk client timestamps for decode drift. Characterizes KV pressure at ~8.3k tokens: prefill ~25 ms/token, decode ~11% slower than short context, ~67 KB/token KV memory.
NVMAI_KERNEL_STATS=1 logs each decode command buffer's GPU span (gpuStartTime/gpuEndTime) by role at the request footer: attn_router, shared_expert, moe_phase1_hit, moe_phase1_2_routed, head_fused/head_logits, embed. Zero cost when off (two env-gated branches), no pipeline changes. First measurement (128-token greedy, 4-bit M3): GPU is ~29% of decode wall; attention+router is 78% of GPU (24.3 ms/token), the logits head 20% (6.2), and the routed MoE experts are negligible (0.5 ms/token combined) — hidden behind the pread/IO overlap. This gives the kernel micro-optimization funnel its per-shader ranking and shows the fused greedy head never engages on this path (head_fused count 0).
The decode cb1 (input norm through router) is split into attnCB (norm + QKV GEMV + epilogue, or the linear/gated attention), softmaxCB (full-attention layers), and tailCB (O-proj + residual + post-norm + router) — same kernels, same queue order, one wait on the last CB, so NVMAI_KERNEL_STATS now reports the per-block GPU split. Measured on the 4-bit M3 (warm, baseline kernel): attention block ~16.0 ms/token vs router+tail ~3.2 ms/token — the router is not the cost; the attention GEMVs are. Decode rate is unchanged (11.4 vs 11.2 tok/s warm), so the split is measurement-only overhead. A QKV dispatch sweep (rows_per_tg 4/8/16) confirmed 8 is optimal (16 and 4 both ~8% slower); the block is bandwidth-bound (~690 MB/token of 4-bit weight reads at ~42% of peak), so the remaining headroom is inner-loop vectorization, not dispatch shape.
The expert-cache miss fills ran serially (K12 serialized them for correctness after an earlier concurrentPerform was reverted). The decode fetch is single-flight — each miss preads into its own plan-reserved slot, pread on a shared fd is thread-safe, and only the bookkeeping is shared (small lock) — so the fills parallelize safely across the mostly-idle CPU cores (the server used ~0.4 of 8 cores). Measured on the 4-bit M3 (interleaved, 512-token greedy): IO wall 41.2 -> 30.9 ms/token and decode 9.98 -> 12.80 tok/s (+28%), every round faster. Greedy output is byte-identical to serial (deterministic check), sampling stable. NVMAI_PARALLEL_IO=0 disables; the prefill uses a separate tile fetch, so it is unaffected.
… 2x2 Adds the parallel expert pread fills to the feature list (with the NVMAI_PARALLEL_IO knob), the per-kernel GPU diagnostics env vars, and replaces the 'matrix being regenerated' caveat with the completed 4-bit regeneration (6.45/6.66/4.47 tok/s, MTP ~31% slower) plus the current ~12.8 tok/s long-generation decode.
Regenerated the matrix on the current build (parallel pread fills, SSE fix, footer-derived rates, answer verification, continuation probe): 4-bit 7.69/7.39/5.95, 6-bit 4.33/4.37/2.72, 8-bit 3.81/3.78/3.09 tok/s. MTP confirmed slower at every quant (-23%/-37%/-19%); the multi-prefix cache hits on multi-turn continuation in every cell. All 12/12 answers verified.
…nt-measured stats
512-token greedy coding-prompt benchmark (server footer, 3 runs after warmup): 4-bit 9.9, 6-bit 4.9, 8-bit 4.1 tok/s. Adds benchmark/nvmai_longgen.py for reproduction and notes the workload dependence (same build: 12.8 tok/s on prose vs 9.9 on the coding prompt).
The decode rate is a function of expert-cache locality: code ~9.9, essay ~12.4, counting ~13.4, and maximally repetitive digit cycles ~16.0 tok/s (4-bit, 512-token greedy) — the measured peak where expert preads nearly vanish. Adds benchmark/nvmai_maxthroughput.py for reproduction.
Three tables (4-bit, 6-bit, 8-bit), each sorted highest to lowest across the workload-locality envelope (digit cycles / counting / essay / coding, 512-token greedy): 4-bit 15.9/13.4/11.2/8.0, 6-bit 6.0/5.9/5.8/4.7, 8-bit 5.4/5.2/4.4/3.8 tok/s. Notes the flattening at higher bits (weight-read cost dominates) and keeps the reproduction script (now multi-quant).
A standalone tool that measures the fused QKV GEMV's achieved memory bandwidth in isolation (synthetic buffers, gpuStartTime/EndTime), plus benchmark variants (bandwidth probe, unroll2, unroll2_half) in the metal source. Findings: the kernel alone reaches ~70% of the M3's peak (67-71% across runs), the dequant ALU costs ~17 points (bandwidth probe 84-87%), and unrolling two blocks gains ~10 (65 -> 76 GB/s) — but in the real decode the attention block is not QKV-bandwidth-bound: warm steady-state attn time is identical with and without the unroll (12.7 ms/token), so no kernel change was kept. The tool remains the Stage-B instrument for future kernel work.
Every experiment after v4.1 -- the v4.2 control plane, Metal I/O, the v4.2.1 data-plane probes, and v4.3 predictive prefetch -- targeted exposed expert I/O, which the v4.2 128-token qualification measures at 18.8% of decode. That caps the whole programme at +23.2% and explains why twenty correctly-measured experiments all landed inside a plus or minus 4% band. Two controlled results show the component does not behave as its metric suggests: the event path raised I/O-hidden from 12.85% to 20.73% and still lost 2-3%, and v4.3 removed real demand misses and lost 3.9% at 4-bit. At batch 1 there is no independent work to hide I/O behind, so arriving sooner only moves the wait into the next dependent command. Records a new finding: verifyGreedyPair routes the width-2 MTP verify through executePrefillChunk, so it pays prefill tile scheduling, per-tile argument buffers and command buffers, and a GDN snapshot, and never uses the v4.1 hit/fixup decode scheduler. The 8-bit case at 92.6% acceptance implies a 1.965x verify cost against a union model predicting under 1.3x, which refutes the standing "acceptance is the lever" conclusion. Plans Gate 0 (decompose the unmeasured 78.7%), Track A (ANE prefill, the only measured >2x), Track B (make width-2 decode cost what it should), and Track C (fewer bytes, gated). Documents what to stop doing, including Metal I/O. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Decomposes the decode token at the published v4.1 benchmark profile: 512 tokens, native 262K context, 8 GiB budget, 8-bit KV, prompt cache on, temperature 0.6, one discarded warmup and a fresh server process per measured run. Parses the server's own NVMAI_RUNNER_STATS and NVMAI_KERNEL_STATS footers into per-role GPU time, inter-command gaps, host wait, and busy_per_token, then applies the Gate 0 decision rule from docs/v4.4-decode-width-plan.md. Changes no runtime behavior. The preflight refuses to start when another process already holds the GPU. That is not hypothetical: the first run of this script measured 6.6 tok/s against a published 16.45 with busy_per_token at 66-72 ms against a historical ~28 ms, and the cause was a background game holding the GPU at 62-78% utilization. Nothing in the tok/s number reveals that on its own, and a 2.4x error is large enough to mislead an entire investigation, so the check belongs in the harness. Override with --allow-busy-gpu. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both quantizations reproduce the published v4.1 benchmark on an idle machine (4-bit 16.656 tok/s against 16.45; 8-bit 8.376 against 8.75), so nothing after v4.1 regressed decode. Three findings change the plan. The control plane is 2.9-3.0% of the token and router readback alone is 0.010 ms, 0.017%. v4.2's closing recommendation targets under one percent; that avenue closes on arithmetic rather than difficulty. Exposed expert I/O is 14.2% at 4-bit but 30.9% at 8-bit, so this document's own generalization of the 18.8% figure to both quantizations was overstated for 8-bit, where only 15.9% of I/O is hidden and real room remains. The largest single gap is head_logits->embed at 15.45 ms (4-bit) and 15.49 ms (8-bit) -- identical across quantizations, so it is not weight-bandwidth work. Cause located in source: sampleOnce commits a private command buffer and waits on it once per token outside every instrumented role; the generic sample kernel selects top-k in k full passes over a 262,144-entry vocabulary from a single 256-thread threadgroup; and the three-stage SampleTopK64 reduction is gated on topK == 64 while GenerationDefaults.topK is 20, so the fast path never fires in production. Added as Track S, ahead of everything else. Also corrects this document's own Gate 0 rule, which specified an absolute GPU-busy threshold and therefore misfires on 8-bit; read as a share of the token, both quantizations are dependency-stalled. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The three-stage SampleTopK64 reduction was gated on `topK == 64` while
GenerationDefaults.topK is 20, so every shipped token fell through to the
generic kernel instead. That kernel extracts Top-K in k full passes over a
262,144-entry vocabulary, with an inner dedup loop per pass, from a single
256-thread threadgroup -- and it runs in its own command buffer that the
decode loop commits and waits on once per token, outside every instrumented
role. It cost 15.45 ms/token at 4-bit and 15.50 ms at 8-bit: identical in
absolute terms across quantizations, because it is not weight-bandwidth work.
Stage 1 already keeps the top 64 of every 1,024-entry tile, so the top k of
the vocabulary is a strict subset of what the reduction computes for any
k <= 64. The final stage now takes k and cuts there.
The cut lands before the Top-P scan, not after, and that ordering is what
makes the two kernels agree: the generic path only ever extracts k slots, so
the cumulative mass Top-P compares against is the mass of the top k, not of
the top 64. Cutting after would keep 30 candidates at k=20 on a distribution
whose 0.95 nucleus is reached at rank 30.
Interleaved A/B/B/A, one binary, one machine state, fresh server per run, a
discarded warmup per arm, at the published v4.1 profile:
4-bit 16.892 -> 21.971 tok/s (+30.07%), sampler gap 15.46 -> 1.40 ms
8-bit 8.418 -> 9.375 tok/s (+11.36%), sampler gap 15.50 -> 1.44 ms
All sixteen runs emitted the identical completion for their quantization
across both arms, so this is the same token stream produced faster, not a
sampling change. The 8-bit margin is thinner because the saving is a fixed
~14 ms against a token roughly twice as long; treat +11% as the floor of a
range, since that arm carries the widest single outlier in either matrix.
NVMAI_SAMPLER_PATH=tiled|generic selects the arm and fails closed, so the
control stays reproducible. SampleTopK64Tests pins token-for-token agreement
with the generic kernel across k in {1,2,7,20,33,63,64}, two temperatures and
three seeds at the production vocabulary, plus a narrow-nucleus case that
distinguishes the two Top-K/Top-P orderings.
780 tests pass serially, tools/lint.sh is clean, the release build is
warning-free, and tools/golden-baseline.sh --check 4 is byte-identical --
greedy uses the fused head and never reaches this path.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
sampleOnce now records its command buffer on the runner's NVMAI_KERNEL_STATS
timeline as role "sample", so the sampler's GPU span joins the role summary
and gap accounting instead of hiding inside head_logits->embed -- the blind
spot that concealed a 15.45 ms/token kernel until Gate 0 traced it by hand.
Diagnostics only; the digest and golden baseline are unchanged.
The corrected 4-bit split at the published profile: 46.37 ms token, 30.38 ms
GPU busy (65.5%), residual sampler cost 1.437 ms/token (3.1%) -- 0.910 ms of
kernel execution and 0.527 ms of commit/wait round trip.
That measurement closes Track S step 3 by its own gate ("remove the round
trip if step 1 shows it is material"): 1.14% of the token is below what the
qualification protocol on this machine can resolve, and the fused-greedy-head
experiment already measured the same idea class -- moving sampling work into
the head pass -- at ~3% slower before rejecting it. Folding sampling onto the
head command buffer would also thread the sampler through the LogitProducer
contract shared by the CLI, server, decode service and MTP; that blast radius
is not worth a sub-noise return. The remaining kernel cost is dominated by
the single-threadgroup logit_softcap_softmax; the shape of a future fix is
recorded in docs/v4.4-decode-width-plan.md, not built.
780 tests pass serially, tools/lint.sh is clean, the release build is
warning-free, and tools/golden-baseline.sh --check 4 is byte-identical.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…sured B1 attributed the MTP verify pass with per-pass phase clocks recorded in StreamingMTPDecoder.advance and printed by the server's mtp-phases footer. At 8-bit the 40-layer backbone alone cost 2.25x a scalar token and the whole pass 2.59x, against a union model predicting under 1.3x. Three structural defects, all in the width-4096 prefill tile path serving a width-2 verify: per-tile planning issued two sequential fetch awaits per layer and degraded the hit rate from 81% to 75% (497 MB of expert reads per pass against a 1.585x union model); the shared expert was waited on synchronously so no GPU work overlapped the fetch; and the grouped tile kernels cost ~71 ms/pass GPU at width 2 against ~25 for the decode kernels. B2 replaces that stage for the t==2 verify with the pair schedule: one union cache plan over both rows' experts, one parallel miss fetch overlapped with a non-blocking shared expert, decode phase-1/phase-2 per row with per-row acts/y buffers so whole-buffer hazard tracking cannot falsely serialize the rows, and a two-row lm_head (encodeLogitsPair) that reads the model's largest weight once per pass instead of twice. NVMAI_MTP_VERIFY=pair|tile selects the arm and fails closed; pair is the default; sub-16-slot budgets fall back to tile. The pair schedule beat tile in every measurement (4-bit pass 151.3 -> 143.5 ms, 8-bit ~352 -> 340.7) and emitted output byte-identical to scalar decode in all nine interleaved A/Bs across both quantizations and both arms. B3 failed its promotion gate, measured rather than assumed: the published- protocol scenario class at 83.7% acceptance and 1.837 emitted tokens per pass still lost 47.6% against scalar. The floor analysis in the plan explains why no acceptance rate can rescue width-2 MTP here: the verify floor is ~1.6x a token (the second row's experts do not share weight reads, the fetch has nothing to hide behind, and Track S made the scalar denominator 30% cheaper), while the benefit is capped at 2.0. MTP remains off by default. The pair schedule stays because it is strictly better wherever MTP runs, and any future drafter work starts from an honest verify. 781 tests pass serially, tools/lint.sh is clean, the release build is warning-free, and tools/golden-baseline.sh --check 4 is byte-identical. Benchmarks: benchmark/nvmai_mtp_phases.py, nvmai_mtp_b3_qualification.py. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The 8-bit function-scenario qualification completes the promotion record for both quantizations: 86.0% acceptance, 1.860 emitted per pass, -7.8% against scalar, output byte-identical. Together with the 4-bit row (-47.6% at 83.7%) the two bracket the mechanism: MTP is closest to break-even exactly where decode is slowest, and loses by half where decode is fast. The gate verdict is unchanged; MTP remains off by default. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Splits the prefill attention instrumentation by layer kind, which settles the question the ANE plan never asked: this architecture is 10 full-attention plus 30 Gated-DeltaNet layers, and on a real 6,103-token 4-bit prefill the ANE-expressible full-attention layers cost 84.3 s of 133.2 s (63.3%) while the inexpressible GDN scans cost 13.3 s (10%). The quadratic term makes the offloadable share dominant and growing. benchmark/nvmai_ane_attention_probe.py builds the complete full-attention block in MIL -- packed QKV with output gate, per-head q/k RMS norms, NeoX-subdim RoPE, GQA SDPA against KV history, sigmoid gate, O projection -- and matches a float32 reference at fp16-noise level. Two findings that shape the integration: the fused scaled_dot_product_attention op produces NaN/inf on this M3's ANE from sequence length 2048 even at score std 0.25, while decomposed matmul+softmax+matmul is clean and slightly faster; and the apparent long-sequence fp16 softmax error (8-12% at seq 6144) is an artifact of uniform random attention -- with realistically peaked scores it is 0.0002. benchmark/nvmai_ane_realweight_rehearsal.py dequantizes the actual int4 affine weights of all ten full-attention layers out of the installed .gturbo, bakes each into a per-layer Core ML program, and replays the exact 20 layer-chunk sequence of the 6,103-token prefill: 3.15 s on the ANE against the GPU's measured 84.3 s -- 26.7x on the offloadable block, a projected 133.2 -> 52.1 s (2.56x) end-to-end, worst per-layer rel err 0.0101, zero NaN/inf. What remains is Swift integration (exporter, Core ML path in executePrefillChunk, K/V quantized into the existing cache, opt-in switch), not research. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Routes the prefill attention block of every full-attention layer through a Core ML sidecar on the Neural Engine, opt-in via NVMAI_PREFILL_ANE=on (off by default, fails closed on unknown values, fails at load with the export command when the sidecar is missing). GDN layers, the MoE, the .gturbo format, the KV cache, the server API, and all of decode are untouched. tools/export_ane_prefill.py dequantizes the model's int4 affine attention weights to fp16 and emits one multifunction .mlpackage per full-attention layer (~53 MB after cross-function weight dedup); functions h0..h12288 differ only in KV-history length, which is always chunk-aligned. The graph uses decomposed attention because the fused scaled_dot_product_attention op produces NaN/inf on this M3's ANE from sequence length 2048, and takes the causal mask as an input because a constant mask gets const-folded to 64 MB per function. The runtime feeds the post-input-norm chunk and a token-major fp16 shadow of earlier chunks' K/V, receives the attention output plus cache-layout K/V, and quantizes into the cache through the existing copyPrefillKVToCache, so decode sees an ordinary cache. Two integration hazards were found and fixed by measurement. Short single-chunk prompts pad to 4,096 tokens and cost more on the ANE than the whole GPU prefill, so only full chunks and continuation chunks are eligible. And each loaded Core ML function pins an E5RT inference arena (~1 GB for the h4096 variant): keeping twenty resident pressured the 8 GiB expert slot cache out of RAM (server RSS 2.85 GB, decode 1.9 tok/s at 200 ms/token of expert-I/O waits), so at most one model is resident at a time and it is released before decode -- the same request then decodes at full speed. Qualification (interleaved gpu/ane/ane/gpu, fresh server per run, 4-bit, 6,103-token prompt, greedy, cache off): GPU prefill 132.90 s -> ANE 57.52 s (2.31x, spreads +/-0.1%) decode after prefill 8.70 vs 8.68 tok/s -- unchanged Output is not byte-identical across arms by construction (fp16 reduction order, ~1% per-layer deviation vs fp32); each arm is internally deterministic and the golden baseline runs with the switch off and is byte-identical. 785 tests pass serially; lint and the warning-free release build are clean. Details: docs/v4.5-ane-prefill.md. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The v4.6 medians on the published 512-token continuous-generation protocol, measured at this commit's runtime: 4-bit 22.53 tok/s (+37.0% over v4.1), 8-bit 9.40 tok/s (+7.4%), with wall times 25.59 s and 59.48 s. Adds the tiled Top-K sampler and the experimental opt-in ANE prefill to the performance feature list. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… weights Three contained follow-ups from the v4.6 optimization inventory. The softcap+softmax front-end was still the pathology the Top-K half already lost: two full-vocabulary passes in one 256-thread threadgroup. It now runs as a tiled reduction -- 4,096-entry tiles to (max, sum-of-exp) pairs, a merge, then a device-wide normalize -- selected by the same NVMAI_SAMPLER_PATH control so an A/B still measures both halves. A test pins it against the single-threadgroup form at the production vocabulary and at tile-boundary sizes; worst elementwise difference is below 1e-5. ANE prefill now preloads the next covered layer's Core ML model as soon as the current prediction returns, so the ~0.5 s load overlaps the MoE stage the caller is about to run on the GPU instead of serializing ahead of the next prediction. At most one preload is outstanding and it only becomes resident at the moment the previous model is dropped, so the one-arena rule that keeps the expert cache in RAM is unchanged. MLModel is not Sendable, so the handoff goes through a documented single-consumer box. The sidecar now records the manifest digest of the model_weights.bin it was exported from, and the runtime refuses a mismatch. A sidecar built from different weights computes plausible-looking but wrong attention and nothing downstream would catch it; the manifest is itself receipt-bound, so its digest is a trustworthy identity. Absent a digest the runtime says the binding is unverified rather than silently trusting it. Also generalizes the exporter to 8-bit source weights (attention is 1 byte per weight there, not a nibble), which the 4-bit-only unpack would have silently misread. 787 tests pass serially; lint and the warning-free release build are clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Item 7 (8-bit expert I/O, 30.9% of the token and the largest apparent decode target left) is closed by one measurement: 8-bit expert reads run at 3.39 GB/s and 4-bit at 3.61 GB/s, against a measured device ceiling of about 3.2-3.9 GB/s for expert-sized random reads. The storage path is already saturated, which retroactively explains the twenty v4.2/v4.3 scheduling experiments -- event synchronization, submission timing, pool layout, GPU residency classification, Metal I/O, predictive prefetch. They were not engineering failures; they were aimed at a device with no idle capacity to schedule. Hit rate is likewise closed (87.70% against a 93.6% policy ceiling, and the gap is purchasable only with RAM that large budgets already prove counterproductive). The one remaining lever is fewer bytes per miss, which is a mixed-precision format decision (8-bit attention with 4-bit routed experts is already expressible in manifest.quant) rather than a runtime change, and the document states its +24% as a model, not a measurement. Item 8 (3-bit weights) fails both of its gates, answered from the installed weights with no runtime code. Quality: affine round-trip error on real attention and shared-expert tensors is a median 10.56x the 4-bit error -- weights already at ~2% go to 21-24%. Packing: 3 bits does not divide a 32-bit word, so the nominal 25% saving is 22% via the cross-word extraction path that costs extra ALU per weight, or 11% word-padded; the precedent for non-power-of-two packing here is 6-bit, measured at 46.8 GB/s against 60 and withdrawn. No repacker, kernel, or receipt work was written. Two methodology fixes, both prompted by an invalid sweep that is discarded rather than reported. The benchmark prompt was built from README.md and AGENTS.md, so editing those files silently moved it from 6,103 to 6,593 tokens between sweeps and, attention being quadratic, made the arms non-comparable; it is now pinned and self-contained. And the preflight refused only a busy GPU, which did not catch a machine running a browser at 117% CPU with swap in use -- the worst-affected run reported an expert cache hit rate of 0.277 against a normal 0.877, the signature of the slot cache being evicted, which reads exactly like a code regression. It now also refuses on other processes' CPU, free memory, and swap. Lint and the release build are clean; sampler, softmax, and ANE suites pass. Throughput qualification for items 1, 2, and 6 remains outstanding and the document says so. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is a pointer to the Pummelchen/NVMAI fork; the full code, wiki, and benchmark suite live there.
Improvements over the original repo (measured on M3 24 GB, 4-bit Qwen 3.6 35B-A3B):
Estimated tok/s increase over the original repo: ~+55-65% (compounded decode-rate wins; the original predates the Qwen 3.6 port, so the baseline is the fork's pre-optimization state on the same model and hardware).