feat(transcription): OpenAI Realtime transcription service - #643
Open
felipeavilis wants to merge 25 commits into
Open
feat(transcription): OpenAI Realtime transcription service#643felipeavilis wants to merge 25 commits into
felipeavilis wants to merge 25 commits into
Conversation
added 25 commits
June 15, 2026 17:09
Implements TranscriptionService backed by the OpenAI Realtime API (wss://api.openai.com/v1/realtime). One WebSocket session per participant; audio sent as base64-encoded PCM16 16kHz mono via input_audio_buffer.append events. Session configured in transcription-only mode (modalities: text, turn_detection: null). Partial results from transcription.delta events, finals from transcription.completed. Exponential-backoff retry on connect failure. Configure via sip-communicator.properties: org.jitsi.jigasi.transcription.customService=org.jitsi.jigasi.transcription.OpenAIRealtimeTranscriptionService org.jitsi.jigasi.transcription.openai.apiKey=sk-... org.jitsi.jigasi.transcription.openai.model=gpt-4o-realtime-preview-2024-12-17 org.jitsi.jigasi.transcription.openai.websocketUrl=wss://api.openai.com/v1/realtime
…-realtime-whisper
- Fix session.update structure: use session.type=transcription and audio.input.format/transcription nesting per GA spec - Declare audio rate as 16000 (Jigasi PCM pipeline is fixed at 16kHz) - Add input_audio_buffer.commit every 1s (required in manual turn_detection mode) - Add transcriptionDelay config param (minimal/low/medium/high/xhigh)
OpenAI Realtime API enforces a minimum sample rate of 24000 Hz. Add PCMAudioSilence24kCaptureDevice and PCMAudioSilence24kMediaDevice to produce 24kHz PCM; OpenAIRealtimeTranscriptionService now uses this device instead of the 16kHz one used by other services.
…nscription Replace dialog session WebSocket approach with the correct two-step transcription session protocol: 1. POST /v1/realtime/transcription_sessions with API key to obtain a short-lived ephemeral client_secret token 2. Connect WebSocket to /v1/realtime?model=gpt-realtime-whisper authenticated with the ephemeral token (not the API key directly) This removes the broken session.update with session.type="transcription" which caused "Passing a transcription session update to a realtime session is not allowed" errors. Transcription sessions require a dedicated REST endpoint; dialog model sessions do not support session.type="transcription". Also removes MODEL_CONFIG / DEFAULT_MODEL (gpt-realtime-2) and the java.net.* wildcard import that caused WebSocket annotation ambiguity.
/v1/realtime/transcription_sessions → /v1/realtime/transcription/sessions
…r transcription Replace the broken REST-first approach (which hit invalid endpoints) with the correct protocol per OpenAI documentation: - Connect WebSocket to wss://api.openai.com/v1/realtime?model=gpt-realtime-2 authenticated directly with the API key - On connect, send session.update with session.type="transcription" and audio.input.transcription.model="gpt-realtime-whisper" Splits session model (URL param, gpt-realtime-2) from transcription model (session.update body, gpt-realtime-whisper) via separate config keys: openai.sessionModel (default: gpt-realtime-2) openai.transcriptionModel (default: gpt-realtime-whisper)
gpt-realtime-whisper must be passed as input_audio_transcription.model inside a standard realtime session (model=gpt-realtime-2 in URL). Setting session.type="transcription" is rejected by dialog session models. Use input_audio_transcription.model directly per API error guidance.
Confirmed via wscat: session.update requires session.type field. Value must be "realtime" (not "transcription") for dialog session models. Transcription is enabled via audio.input.transcription.model.
… at startup On isConfiguredProperly(), fetch GET /v1/models with the configured API key and log all realtime models available on the account. Warn clearly if the configured sessionModel or transcriptionModel is not in the list. Also improve the invalid_model WebSocket error with a specific log message pointing to the config keys to fix.
Keep only the improved invalid_model error message pointing to the config keys to check.
…odel) Set fatalError flag on invalid_api_key and invalid_model errors so the retry loop stops immediately instead of retrying three times uselessly. Log a clear message pointing to the config key to fix in each case.
…io on closed channel - connected flag is now set in onConnect (WebSocket callback) instead of connectInternal, eliminating the race where connected=true was set before a fatal error message arrived and closed the session - sendAudio checks session.isOpen() in addition to connected flag, dropping frames silently when the channel is closed (avoids log spam on auth errors)
- Add AUTHENTICATION_FAILED to TranscriptionListener.FailureReason - TranscriptionGatewaySession.failed() now sends a human-readable chat message to the MUC before stopping, for both AUTHENTICATION_FAILED and RESOURCES_EXHAUSTED reasons - OpenAIRealtimeTranscriptionService.onError() detects invalid API key errors and calls listener.failed(AUTHENTICATION_FAILED)
When an API auth error arrives during WebSocket handshake, the error may fire before the JVB conference is established and the Transcriber enters TRANSCRIBING state. The previous code silently dropped the failure in this case, preventing the room message from reaching participants. Now stop() handles NOT_STARTED + non-null reason by transitioning to FINISHED and notifying all listeners, so TranscriptionGatewaySession can send the error message to the room and stop the conference.
…riber state" This reverts commit cc2d04e.
…icator.properties Replace deprecated openai.model with sessionModel and transcriptionModel. Fix audio format description: 24kHz (not 16kHz).
…les turn detection
Member
|
@felipeavilis Have you looked at opus-transcriber-proxy? Here are the instructions that we will make part of the handbook soon: https://github.com/jitsi/handbook/pull/645/changes |
Author
|
No. I'll take a look. Thanks for the tip. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
OpenAIRealtimeTranscriptionService, a new transcription backend that uses the OpenAI Realtime API via WebSocket.PCMAudioSilence24kMediaDevice) — required by the OpenAI Realtime API minimum sample rateserver_vadactive by default — API detects speech automatically, no manual buffer commit neededconversation.item.input_audio_transcription.delta, finals via.completedinvalid_api_key,invalid_model) abort the retry loop immediatelyProtocol
WebSocket:
wss://api.openai.com/v1/realtime?model=gpt-realtime-2Authorization:
Bearer <apiKey>On
session.created, sendsession.update:session.type:realtimesession.audio.input.transcription.model:gpt-realtime-whispersession.audio.input.transcription.language: BCP-47 subtag (e.g.pt,en)Audio frames sent as
input_audio_buffer.append(base64 PCM16 24kHz mono).New files
transcription/openai/OpenAIRealtimeClient.javatranscription/openai/OpenAIRealtimeClientListener.javatranscription/OpenAIRealtimeTranscriptionService.javaTranscriptionServiceimplementationtranscription/PCMAudioSilence24kCaptureDevice.javatranscription/PCMAudioSilence24kMediaDevice.javaChanges to existing files
TranscriptionListener.java: addsAUTHENTICATION_FAILEDtoFailureReasonenum (additive, no impact on existing services)TranscriptionGatewaySession.java:failed()now sends a human-readable message to the room before stopping — benefits all services includingGoogleCloudTranscriptionServicewhich already usesRESOURCES_EXHAUSTEDjigasi-home/sip-communicator.properties: adds commented-out config block for OpenAI RealtimeConfiguration
Test plan