fix(TextDecoder): don't early-exit the decode loop on predictions sampled during promptTokens prefill#497
Open
yangzichao wants to merge 1 commit into
Conversation
…pled during promptTokens prefill With DecodingOptions.promptTokens, the initial prompt becomes [<|startofprev|>, ...prompt..., <|startoftranscript|>, ...] and decodeText force-feeds every one of those tokens through the main loop. The model's sampled prediction at each forced step is discarded (the forced token wins), but it still drove the early-exit checks: 1. If the model's top prediction at ANY mid-prompt position is <|endoftext|>, sampleResult.completed ended the segment before a single audio token was decoded -> empty transcript. 2. isFirstToken pointed at prefilledIndex (the <|startofprev|> step), so firstTokenLogProbThreshold was evaluated against a forced-prompt placeholder prediction instead of the first real token. On whisper-large-v3-v20240930_turbo this made ANY non-empty promptTokens (even a single content token) return an empty result, while the same audio decoded fine without a prompt (argmaxinc#372). Whether the model predicts EOT at those positions is model-dependent, which is why some models appeared unaffected. Fix: when promptTokens are present, ignore sampleResult.completed during prefill steps and evaluate the first-token check at the first genuinely sampled position (the last prefill index), matching the reference implementation's sample_begin semantics. The no-prompt path is untouched. Verified against the real 632MB turbo model on recordings that previously reproduced the empty decode: with this change, no-prompt / 19-token prompt / single-token prompt all return correct transcripts.
This was referenced Jul 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Fixes #372.
With
DecodingOptions.promptTokens, the initial prompt becomes[<|startofprev|>, ...prompt..., <|startoftranscript|>, ...]anddecodeTextforce-feeds every one of those tokens through the main loop. The model's sampled prediction at each forced step is discarded (the forced token wins), but it still drives the early-exit checks:<|endoftext|>,sampleResult.completedends the segment before a single audio token is decoded → empty transcript.isFirstTokenpoints atprefilledIndex(the<|startofprev|>step), sofirstTokenLogProbThresholdis evaluated against a forced-prompt placeholder prediction instead of the first real token — this is the spuriousFallback #1 (firstTokenLogProbThreshold)reported in Using promptTokens causes the Transcription to return empty result. #372.Whether a model predicts EOT at those positions is model-dependent, which is why
large-v3-v20240930_turboandbase.enreproduce whilebaseappears fine — but the checks shouldn't be consulting forced-prefill predictions on any model. The reference implementation feeds the prompt without sampling checks and only starts them atsample_begin.Fix
When
promptTokensare present, ignoresampleResult.completedduring prefill steps and evaluate the first-token check at the first genuinely sampled position (the last prefill index). The no-prompt path is untouched — checks keep their existing indices, so default behavior is unchanged.Verification
Standalone harness against the real
openai_whisper-large-v3-v20240930_turbo_632MBCoreML model on recordings that reproduced the empty decode:With the fix, the prompt also demonstrably conditions the output (a punctuated prompt yields punctuated transcripts), i.e. the channel now works as intended rather than merely not crashing.