Skip to content

Disable token timestamps for distilled models detected by config, not name#45

Open
gsdali wants to merge 1 commit into
argmaxinc:mainfrom
gsdali:fix/token-timestamps-distilled-by-config
Open

Disable token timestamps for distilled models detected by config, not name#45
gsdali wants to merge 1 commit into
argmaxinc:mainfrom
gsdali:fix/token-timestamps-distilled-by-config

Conversation

@gsdali

@gsdali gsdali commented May 28, 2026

Copy link
Copy Markdown

Problem

scripts/generate_model.py only disables token-level timestamps for models whose version string contains distil. Some distilled models are not named distil-* yet still ship the original (e.g. large-v3) generation_config, whose alignment_heads reference decoder layers that no longer exist after distillation. Generating token timestamps then crashes with an IndexError in compute_alignment_heads_attention_weights.

Repro: whisperkit-generate-model --model-version kotoba-tech/kotoba-whisper-bilingual-v1.0 — the model has 2 decoder layers but ships large-v3 alignment_heads up to layer 25; the WhisperTextDecoder step fails.

Fix

In addition to the existing name-based check, detect the condition from the config: if any alignment_heads entry references a layer >= decoder_layers, disable token timestamps. Falls back gracefully (warns and continues) if the config can't be read.

Evidence

This change was applied when converting kotoba-tech/kotoba-whisper-bilingual-v1.0 (whisperkittools 0.4.2, coremltools 9.0, torch 2.5.0). With it, the guard fires and the conversion completes:

INFO:scripts.generate_model:Disabling token-level timestamps: alignment_heads reference decoder layers beyond decoder_layers=2
INFO:argmaxtools.test_utils:torch2coreml PSNR=57.4   # TextDecoder
INFO:argmaxtools.test_utils:torch2coreml PSNR=53.9   # AudioEncoder
INFO:argmaxtools.test_utils:torch2coreml PSNR=59.8   # MelSpectrogram

Without the change the same run fails at the decoder step with the IndexError above. I have not re-run the full suite against other models; the change only widens an existing disable condition and is a no-op for models without oversized alignment_heads.

🤖 Generated with Claude Code

… name

The existing guard only disables token-level timestamps for models whose
version string contains 'distil'. Some distilled models are not named 'distil-*'
but still ship the original (e.g. large-v3) generation_config, whose
alignment_heads reference decoder layers that no longer exist after
distillation. Generating token timestamps then crashes with an IndexError in
compute_alignment_heads_attention_weights.

Detect this case from the config — any alignment_heads entry referencing a
layer >= decoder_layers — and disable token timestamps, in addition to the
existing name-based check.

Repro: kotoba-tech/kotoba-whisper-bilingual-v1.0 (2 decoder layers, but ships
large-v3 alignment_heads up to layer 25) fails the WhisperTextDecoder step
without this.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant