apr run: inference fails on tied-embedding .apr models (lm_head 0-byte placeholder not tied to embed_tokens)

[noahgift]

Summary

apr run fails at the final lm_head matmul on tied-embedding .apr models with:

error: Inference failed: Invalid shape: matmul weight has EMPTY data buffer
(in_dim=896, out_dim=151936, qtype=0); likely a MoE per-expert tensor was
registered with len-0 data — see aprender#1789

The error's MoE hypothesis (#1789, closed, was Qwen3-30B-MoE) is a red herring — this model is not MoE. The real cause is tied word embeddings.

Repro

apr run /home/noah/models/qwen2.5-coder-0.5b-instruct.apr --prompt "What is 2+2?" --max-tokens 16
# fails on --no-gpu, --backend cpu, AND wgpu — backend-independent

Diagnosis

apr tensors shows the lm_head is a 0-byte placeholder, while embed_tokens holds the real data:

│ lm_head.weight            │ [151936, 896] │ f32  │ 0 B      │   <-- empty
│ model.embed_tokens.weight │ [151936, 896] │ bf16 │ 259.7 MB │   <-- real data

Qwen2.5-0.5B uses tie_word_embeddings=true: lm_head == embed_tokens (transposed for the output projection). The .apr file correctly stores lm_head as a shape-only placeholder, but the loader registers the empty lm_head for the output matmul instead of tying it to embed_tokens.weight.

apr inspect / apr qa PASS (checksum valid, 291 tensors pass PMAT-235 contracts) — the file is well-formed; the loader is at fault.

Scope

• Fails: tied-embedding models (small Qwen: 0.5B; any model with a 0-byte lm_head placeholder).
• Works: models with a real/untied lm_head — e.g. qwen2.5-coder-1.5b-instruct-q4k publishes "4" correctly.

Regression status

NOT a v0.60.0 regression — the 0.57.0 dev build (/mnt/nvme-raid0/targets/aprender/release/apr) fails identically. Pre-existing (≥0.57.0). Surfaced by post-release dogfooding of the v0.60.0 crates.io binary.

Proposed fix

In the quantized-model loader (OwnedQuantizedModel::from_apr / the output-projection wiring): when lm_head.weight has a 0-byte data buffer AND embed_tokens.weight exists with a matching [vocab, hidden] shape, tie the output projection to embed_tokens.weight (transposed) rather than registering the empty buffer. Add a falsification test that apr run on a tied-embedding fixture produces a non-empty decode (RED before fix, GREEN after), per the contract-ratchet doctrine.

Also: fix the misleading error message to name the tied-embedding case, not just MoE #1789.

[noahgift]

Root cause confirmed + precise fix site

The tie-fix already exists in the F32 GPU adapter but was never applied to the quantized loader — that's the whole bug.

Working path (crates/aprender-serve/src/gpu/adapters/apr.rs:108, PMAT-788):

// source lm_head_f32() so tied-embedding models (empty lm_head_weight)
// upload the byte-identical token_embedding instead of an empty buffer
let apr_lm_head = apr.lm_head_f32().to_vec();

Broken path (crates/aprender-serve/src/gguf/loader_apr_quantized.rs:270):

let lm_head_weight = apr_load_quantized_tensor(
    apr, data, data_offset,
    &["lm_head.weight", "output.weight"],   // 0-byte for tied models → empty tensor
    hidden_dim, vocab_size, transpose,
)?;
// no fallback → the empty buffer reaches matmul_fused.rs and panics

Fix plan (contract-first, per aprender-serve/CLAUDE.md)

1. Contract first (CB-1400): add/extend a provable contract for "tied-embedding lm_head resolves to token_embedding" before touching code.
2. In from_apr (loader_apr_quantized.rs): after loading lm_head_weight, if its data buffer is empty AND token_embedding has the matching [vocab, hidden] shape, build the lm_head from token_embedding (mirroring lm_head_f32()), honoring the row-major LAYOUT-002 mandate + PMAT-216 transpose convention (LmHeadWeight vs LmHeadWeightTransposed).
3. Falsifier (RED→GREEN): apr run <tied-embedding fixture> --prompt "What is 2+2?" must produce a non-empty decode. RED on current main (EMPTY data buffer), GREEN after. Use qwen2.5-coder-0.5b-instruct.apr as the fixture class.
4. GPU parity (PMAT-216): CPU L2 ≈ GPU L2 on a tied model; cross-verify on ≥1 other silicon per the cross-silicon rule.
5. Fix the misleading error message in matmul_fused.rs (drop the hard-coded MoE apr serve: matmul_fused.rs:211 panics with 'index out of bounds: len 0' on Qwen3-Coder-30B-MoE F32 weight #1789 attribution; name the tied-embedding case).

Scoped via post-v0.60.0 dogfooding. Not release-blocking (pre-existing ≥0.57.0).