Bodhi Realtime Agent Framework

Appearance

bodhi-realtime-agent / OpenAIRealtimeTransport

Class: OpenAIRealtimeTransport ​

Defined in: src/transport/openai-realtime-transport.ts:286

LLMTransport implementation for the OpenAI Realtime API.

Uses the openai SDK's WebSocket transport (OpenAIRealtimeWS) for bidirectional audio streaming with function calling support.

Key differences from Gemini:

  • In-place session updates (no reconnect for agent transfers)
  • Streamed function call arguments (accumulated before dispatch)
  • Client-managed interruption (truncate + cancel)
  • 24kHz audio (vs Gemini's 16kHz)
  • Explicit response.create required after tool results

Implements ​

Constructors ​

Constructor ​

new OpenAIRealtimeTransport(config): OpenAIRealtimeTransport

Defined in: src/transport/openai-realtime-transport.ts:437

Parameters ​

config ​

OpenAIRealtimeConfig

Returns ​

OpenAIRealtimeTransport

Properties ​

onAudioOutput? ​

optional onAudioOutput?: (base64Data) => void

Defined in: src/transport/openai-realtime-transport.ts:321

Parameters ​

base64Data ​

string

Returns ​

void

Implementation of ​

LLMTransport.onAudioOutput

onCacheBust? ​

optional onCacheBust?: (reason) => void

Defined in: src/transport/openai-realtime-transport.ts:344

Fires immediately before the framework emits a session.update that changes instructions or tools — i.e. before a guaranteed full prompt-cache bust on the next response. Pure telemetry.

Parameters ​

reason ​

"instructions_changed" | "tools_changed"

Returns ​

void

Implementation of ​

LLMTransport.onCacheBust

onClose? ​

optional onClose?: (code?, reason?) => void

Defined in: src/transport/openai-realtime-transport.ts:330

Parameters ​

code? ​

number

reason? ​

string

Returns ​

void

Implementation of ​

LLMTransport.onClose

onError? ​

optional onError?: (error) => void

Defined in: src/transport/openai-realtime-transport.ts:329

Parameters ​

error ​

LLMTransportError

Returns ​

void

Implementation of ​

LLMTransport.onError

onFirstAudioChunk? ​

optional onFirstAudioChunk?: () => void

Defined in: src/transport/openai-realtime-transport.ts:332

Fires once per model response, on the FIRST audio chunk emitted to the client. Distinct from onModelTurnStart (which fires as a response begins, including tool-only turns) — this marks the moment audio actually begins, the anchor for TTS-first-audio / stop-to-first-audio latency. Transports reset their per-response "audio started" flag when a new response begins.

Returns ​

void

Implementation of ​

LLMTransport.onFirstAudioChunk

onGoAway? ​

optional onGoAway?: (timeLeft) => void

Defined in: src/transport/openai-realtime-transport.ts:333

Parameters ​

timeLeft ​

string

Returns ​

void

Implementation of ​

LLMTransport.onGoAway

onGroundingMetadata? ​

optional onGroundingMetadata?: (metadata) => void

Defined in: src/transport/openai-realtime-transport.ts:335

Parameters ​

metadata ​

Record<string, unknown>

Returns ​

void

Implementation of ​

LLMTransport.onGroundingMetadata

onInputTranscription? ​

optional onInputTranscription?: (text) => void

Defined in: src/transport/openai-realtime-transport.ts:326

User-speech transcription from the transport's own recognizer.

Parameters ​

text ​

string

An incremental delta, not the whole utterance restated. Gemini Live streams many small deltas per utterance (often mid-word); OpenAI and Qwen fire once per utterance from conversation.item.input_audio_transcription.completed, which is the degenerate one-delta case. Consumers must therefore accumulate across calls and reset at each utterance boundary — treating a call as the complete transcript collapses the utterance to its last fragment.

Returns ​

void

Implementation of ​

LLMTransport.onInputTranscription

onInterrupted? ​

optional onInterrupted?: () => void

Defined in: src/transport/openai-realtime-transport.ts:325

Returns ​

void

Implementation of ​

LLMTransport.onInterrupted

onModelTurnStart? ​

optional onModelTurnStart?: () => void

Defined in: src/transport/openai-realtime-transport.ts:331

Fires when the model begins a response (audio, text, tool call). Used by VoiceSession to trigger STT provider commit.

Returns ​

void

Implementation of ​

LLMTransport.onModelTurnStart

onOutputTranscription? ​

optional onOutputTranscription?: (text) => void

Defined in: src/transport/openai-realtime-transport.ts:327

Parameters ​

text ​

string

Returns ​

void

Implementation of ​

LLMTransport.onOutputTranscription

onRealtimeLLMUsage? ​

optional onRealtimeLLMUsage?: (usage) => void

Defined in: src/transport/openai-realtime-transport.ts:340

Optional: fires when the provider reports token or duration usage for billing/observability.

Parameters ​

usage ​

RealtimeLLMUsageEvent

Returns ​

void

Implementation of ​

LLMTransport.onRealtimeLLMUsage

onReasoningDone? ​

optional onReasoningDone?: (info) => void

Defined in: src/transport/openai-realtime-transport.ts:342

Fires when the model's reasoning step completes for the current response, before any audio/text answer is emitted. durationMs is the wall-clock time the reasoning step took; reasoningTokens is the count if the provider exposes it (otherwise undefined).

Parameters ​

info ​
durationMs ​

number

reasoningTokens? ​

number

Returns ​

void

Implementation of ​

LLMTransport.onReasoningDone

onReasoningStart? ​

optional onReasoningStart?: () => void

Defined in: src/transport/openai-realtime-transport.ts:341

Fires when the model begins emitting its hidden reasoning trace for the current response. Useful for latency observability.

Returns ​

void

Implementation of ​

LLMTransport.onReasoningStart

onReasoningSummary? ​

optional onReasoningSummary?: (text) => void

Defined in: src/transport/openai-realtime-transport.ts:343

Fires with a streamed chunk of the optional reasoning summary text (when the model was configured to emit one). Surface this to ops telemetry only — never to end users.

Parameters ​

text ​

string

Returns ​

void

Implementation of ​

LLMTransport.onReasoningSummary

onResumptionUpdate? ​

optional onResumptionUpdate?: (handle, resumable) => void

Defined in: src/transport/openai-realtime-transport.ts:334

Parameters ​

handle ​

string

resumable ​

boolean

Returns ​

void

Implementation of ​

LLMTransport.onResumptionUpdate

onSessionReady? ​

optional onSessionReady?: (sessionId) => void

Defined in: src/transport/openai-realtime-transport.ts:328

Parameters ​

sessionId ​

string

Returns ​

void

Implementation of ​

LLMTransport.onSessionReady

onSpeechStarted? ​

optional onSpeechStarted?: () => void

Defined in: src/transport/openai-realtime-transport.ts:338

Fires when the transport detects user speech via VAD. Used for TTS-level barge-in when the LLM is idle but TTS is still playing. OpenAI: wired to input_audio_buffer.speech_started. Gemini: may require custom VAD signal — needs empirical testing.

Returns ​

void

Implementation of ​

LLMTransport.onSpeechStarted

onTextDone? ​

optional onTextDone?: () => void

Defined in: src/transport/openai-realtime-transport.ts:337

Fires when the model's text response is complete for this turn. Signals that all text for the current response has been delivered. Ordering contract: fires after all onTextOutput, before onTurnComplete.

Returns ​

void

Implementation of ​

LLMTransport.onTextDone

onTextOutput? ​

optional onTextOutput?: (text) => void

Defined in: src/transport/openai-realtime-transport.ts:336

Fires when the model produces text output (text-mode responses). Only active when responseModality is 'text' (i.e., external TTS in use).

Parameters ​

text ​

string

Incremental text chunk (may be partial word/sentence)

Returns ​

void

Implementation of ​

LLMTransport.onTextOutput

onToolCall? ​

optional onToolCall?: (calls) => void

Defined in: src/transport/openai-realtime-transport.ts:322

Parameters ​

calls ​

TransportToolCall[]

Returns ​

void

Implementation of ​

LLMTransport.onToolCall

onToolCallCancel? ​

optional onToolCallCancel?: (ids) => void

Defined in: src/transport/openai-realtime-transport.ts:323

Parameters ​

ids ​

string[]

Returns ​

void

Implementation of ​

LLMTransport.onToolCallCancel

onTurnComplete? ​

optional onTurnComplete?: () => void

Defined in: src/transport/openai-realtime-transport.ts:324

Returns ​

void

Implementation of ​

LLMTransport.onTurnComplete

onUserSpeechStopped? ​

optional onUserSpeechStopped?: () => void

Defined in: src/transport/openai-realtime-transport.ts:339

Fires when the provider's server VAD detects end of user speech. OpenAI/Qwen: wired to input_audio_buffer.speech_stopped. Gemini Live has no equivalent wire event (callback never fires there — the framework falls back to client-VAD timing). The latency anchor derived from this is callback receipt time: biased late by the provider's silence window + network hop (a bounded lower-bound bias — see the observability design, investment-hai-metrics-observability.md §11).

Returns ​

void

Implementation of ​

LLMTransport.onUserSpeechStopped

Accessors ​

audioFormat ​

Get Signature ​

get audioFormat(): AudioFormatSpec

Defined in: src/transport/openai-realtime-transport.ts:300

Returns ​

AudioFormatSpec

Implementation of ​

LLMTransport.audioFormat

capabilities ​

Get Signature ​

get capabilities(): TransportCapabilities

Defined in: src/transport/openai-realtime-transport.ts:292

Static capabilities — read before connecting, used for orchestrator branching.

Returns ​

TransportCapabilities

Static capabilities — read before connecting, used for orchestrator branching.

Implementation of ​

LLMTransport.capabilities

isConnected ​

Get Signature ​

get isConnected(): boolean

Defined in: src/transport/openai-realtime-transport.ts:510

Returns ​

boolean

Implementation of ​

LLMTransport.isConnected

Methods ​

cancelResponse() ​

cancelResponse(opts?): Promise<void>

Defined in: src/transport/openai-realtime-transport.ts:668

Cancel the in-flight response — wire-only actuation. See LLMTransport.cancelResponse JSDoc on the interface for the contract.

Returns Promise<void>. Never rejects — transient send failures are caught and logged internally.

Parameters ​

opts? ​

CancelResponseOptions

Returns ​

Promise<void>

Implementation of ​

LLMTransport.cancelResponse

clearAudio() ​

clearAudio(): void

Defined in: src/transport/openai-realtime-transport.ts:647

Returns ​

void

Implementation of ​

LLMTransport.clearAudio

clearInputAudio() ​

clearInputAudio(): void

Defined in: src/transport/openai-realtime-transport.ts:652

Clear the provider's pending input-audio buffer (OpenAI Realtime: input_audio_buffer.clear). Optional — Gemini and other transports without a server-side append-then-commit input buffer omit this. Called by VoiceSession at grace-window arming to discard any pre-arming echo residue; safe to call when the buffer is empty.

Returns ​

void

Implementation of ​

LLMTransport.clearInputAudio

commitAudio() ​

commitAudio(): void

Defined in: src/transport/openai-realtime-transport.ts:642

Returns ​

void

Implementation of ​

LLMTransport.commitAudio

connect() ​

connect(transportConfig?): Promise<void>

Defined in: src/transport/openai-realtime-transport.ts:516

Parameters ​

transportConfig? ​

LLMTransportConfig

Returns ​

Promise<void>

Implementation of ​

LLMTransport.connect

disconnect() ​

disconnect(): Promise<void>

Defined in: src/transport/openai-realtime-transport.ts:580

Returns ​

Promise<void>

Implementation of ​

LLMTransport.disconnect

quiesce() ​

quiesce(): Promise<void>

Defined in: src/transport/openai-realtime-transport.ts:787

Pause the transport without disconnecting. Used by VoiceSession to enter transcription mode without tearing the WS down.

Implementation:

  1. If a response is in flight, send response.cancel so the model stops generating.
  2. Set _suppressAudio so any onAudioOutput deltas already in flight are dropped (existing flag re-used).

Idempotent: calling quiesce() while already quiesced is a no-op.

Returns ​

Promise<void>

Implementation of ​

LLMTransport.quiesce

reconnect() ​

reconnect(state?): Promise<void>

Defined in: src/transport/openai-realtime-transport.ts:603

Parameters ​

state? ​

ReconnectState

Returns ​

Promise<void>

Implementation of ​

LLMTransport.reconnect

replayHistory() ​

replayHistory(items): void

Defined in: src/transport/openai-realtime-transport.ts:1892

Replay a conversation into the live session: the host's initial history, and the conversation of a replacement connection.

Parameters ​

items ​

readonly ReplayItem[]

Returns ​

void

Implementation of ​

LLMTransport.replayHistory

sendAudio() ​

sendAudio(base64Data): void

Defined in: src/transport/openai-realtime-transport.ts:637

Parameters ​

base64Data ​

string

Returns ​

void

Implementation of ​

LLMTransport.sendAudio

sendContent() ​

sendContent(turns, turnComplete?): void

Defined in: src/transport/openai-realtime-transport.ts:952

Parameters ​

turns ​

ContentTurn[]

turnComplete? ​

boolean = true

Returns ​

void

Implementation of ​

LLMTransport.sendContent

sendFile() ​

sendFile(base64Data, mimeType): void

Defined in: src/transport/openai-realtime-transport.ts:988

Parameters ​

base64Data ​

string

mimeType ​

string

Returns ​

void

Implementation of ​

LLMTransport.sendFile

sendToolResult() ​

sendToolResult(result): void

Defined in: src/transport/openai-realtime-transport.ts:1010

Parameters ​

result ​

TransportToolResult

Returns ​

void

Implementation of ​

LLMTransport.sendToolResult

transferSession() ​

transferSession(config, _state?): Promise<void>

Defined in: src/transport/openai-realtime-transport.ts:900

Parameters ​

config ​

SessionUpdate

_state? ​

ReconnectState

Returns ​

Promise<void>

Implementation of ​

LLMTransport.transferSession

triggerGeneration() ​

triggerGeneration(instructions?, overrides?): void

Defined in: src/transport/openai-realtime-transport.ts:1081

Trigger a model response.

Parameters ​

instructions? ​

string

Optional one-off instruction (passed as response.create.response.instructions on OpenAI — does not mutate the session prefix, so it is cache-safe for subsequent turns).

overrides? ​

Optional per-response overrides. reasoning.effort bumps the dial for this Response only (resets to session default on the next turn). Only honored by transports advertising capabilities.reasoningEffort.

reasoning? ​

{ effort: ReasoningEffort; }

reasoning.effort ​

ReasoningEffort

Returns ​

void

Implementation of ​

LLMTransport.triggerGeneration

unquiesce() ​

unquiesce(): Promise<void>

Defined in: src/transport/openai-realtime-transport.ts:801

Resume normal operation. Idempotent. Drains any when_idle tool results that accumulated while quiesced — those were deferred so flushPendingWhenIdle wouldn't fire response.create during dictation mode.

Returns ​

Promise<void>

Implementation of ​

LLMTransport.unquiesce

updateSession() ​

updateSession(config): Promise<void>

Defined in: src/transport/openai-realtime-transport.ts:809

Apply a session update. Pre-connect: state-only mutation; coalesces with prior pre-connect calls and resolves immediately. The merged config is sent in the single session.update issued at connect time. Post-connect: serialized via the transport's internal FIFO queue; each call produces one wire session.update and awaits its ack. A custom transport may instead apply the update synchronously and return void.

Parameters ​

config ​

SessionUpdate

Returns ​

Promise<void>

Implementation of ​

LLMTransport.updateSession

Built with VitePress