Support incremental file loading#507
Open
a2they wants to merge 1 commit into
Open
Conversation
--------- Co-authored-by: Andrey Leonov <aleonov@gmail.com> Co-authored-by: ZachNagengast <znagengast@gmail.com>
There was a problem hiding this comment.
Pull request overview
This PR adds an incremental (bounded-memory) audio file loading mode to WhisperKit so long-form transcription can stream audio from disk in chunks, while keeping full-file loading as the default. It also renames/configures audio input settings via a new AudioInputOptions type and wires the new loading mode through the library API, CLI, and documentation.
Changes:
- Add incremental file streaming in
AudioProcessor.loadFileIncrementallywith explicit backpressure (IncrementalLoadController) and VAD-consistent chunk boundaries. - Introduce
AudioInputOptions(renamingAudioInputConfigvia deprecated typealias) withAudioLoadingModeto choose.fullFilevs.incremental(...), and thread this throughWhisperKit.transcribe*APIs. - Add new unit tests for incremental loading behavior + update CLI flags and README documentation.
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| Tests/WhisperKitTests/UnitTests.swift | Updates tests to use per-call AudioInputOptions instead of config-level AudioInputConfig. |
| Tests/WhisperKitTests/TestUtils.swift | Updates channel-mode helper type and adds an async timeout helper used by new tests. |
| Tests/WhisperKitTests/AudioProcessorTests.swift | Adds coverage for incremental chunking invariants, parity with full-file chunking, backpressure, termination behavior, and multi-file concurrency/order. |
| Sources/WhisperKit/Core/WhisperKit.swift | Adds per-call audioInputOptions, incremental transcription paths, and stream-based transcription helper; updates segment offset handling. |
| Sources/WhisperKit/Core/Configurations.swift | Deprecates config-level audio input config and stores AudioInputOptions in a backing property. |
| Sources/WhisperKit/Core/Audio/VoiceActivityDetector.swift | Marks base VAD type as @unchecked Sendable with documentation about subclass requirements. |
| Sources/WhisperKit/Core/Audio/AudioProcessor+IncrementalLoading.swift | Implements incremental audio streaming with VAD-based boundaries and bounded buffering/backpressure. |
| Sources/WhisperKit/Core/Audio/AudioProcessor.swift | Renames input config type, adds AudioLoadingMode, and fixes short-read behavior in the 16kHz mono fast path via readFully. |
| Sources/ArgmaxCLI/TranscribeCLIUtils.swift | Adds helper to derive AudioInputOptions from CLI args, defaulting to full-file loading. |
| Sources/ArgmaxCLI/TranscribeCLIArguments.swift | Adds CLI flags/options for incremental loading and tuning chunk duration/buffer size. |
| Sources/ArgmaxCLI/TranscribeCLI.swift | Validates incremental-loading args and passes audioInputOptions into transcription. |
| README.md | Documents the new incremental loading option and updates the quick example. |
| Examples/WhisperAX/.../Package.resolved | Updates Swift Argument Parser pin used by the example workspace. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
988
to
992
| open func transcribe( | ||
| audioArray: [Float], | ||
| audioArrayOffset: Int = 0, | ||
| decodeOptions: DecodingOptions? = nil, | ||
| callback: TranscriptionCallback? = nil, |
Comment on lines
+845
to
849
| let seekTimeOffset = Float(seekOffsets[windowId]) / Float(WhisperKit.sampleRate) | ||
| let adjustedSegments = segments.map { | ||
| TranscriptionUtilities.updateSegmentTimings(segment: $0, seekTime: seekTimeOffset) | ||
| } | ||
| segmentDiscoveryCallback?(adjustedSegments) |
Comment on lines
+1209
to
+1222
| let seekTimeOffset = Float(chunk.audioChunk.seekOffsetIndex) / Float(WhisperKit.sampleRate) | ||
| let adjustedResults = chunkResults.map { result in | ||
| let adjustedSegments = result.segments.map { segment in | ||
| TranscriptionUtilities.updateSegmentTimings(segment: segment, seekTime: seekTimeOffset) | ||
| } | ||
|
|
||
| return TranscriptionResult( | ||
| text: result.text, | ||
| segments: adjustedSegments, | ||
| language: result.language, | ||
| timings: result.timings, | ||
| seekTime: seekTimeOffset + (result.seekTime ?? 0) | ||
| ) | ||
| } |
ZachNagengast
approved these changes
Jul 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add incremental audio file loading so long-form transcription reads files in bounded-memory chunks rather than loading the entire file into memory. Full-file loading stays the default, so existing behavior is unchanged.
Incremental file loading
AudioProcessor.loadFileIncrementallystreams a file as anAsyncThrowingStreamwith backpressure — at mostchunkBufferSizechunks are queued before the producer suspends (IncrementalLoadController).VADAudioChunker+ internalEnergyVAD, carrying partially-decided chunks across staging buffers, so incremental output matches a single full-file load exactly; time-based cutting is the fallback.concurrentWorkerCount.AudioInputOptionsAudioInputConfig→AudioInputOptions(old name kept as a deprecated typealias); it bundleschannelModeandaudioLoadingMode.AudioInputOptions.AudioLoadingMode:.fullFile(default) and.incremental(chunkDuration:chunkBufferSize:), plus an.incrementalshorthand using the default duration/buffer size.transcribe/transcribeWithResultstakeaudioInputOptions: AudioInputOptions? = nil, right after the audio path and abovedecodeOptions.WhisperKitConfig.audioInputConfigis deprecated — passaudioInputOptionsper call instead.