Bodhi Realtime Agent Framework

Appearance

bodhi-realtime-agent / VoiceSessionOptions

Interface: VoiceSessionOptions ​

Defined in: src/core/voice-session.ts:313

The options of a VoiceSession apart from its Gemini credential.

Extended by ​

Properties ​

agents ​

agents: MainAgent[]

Defined in: src/core/voice-session.ts:319

All agents available in this session.

artifactRegistry? ​

optional artifactRegistry?: object

Defined in: src/core/voice-session.ts:689

Optional per-session artifact registry for cross-tool binary sharing (images, documents).

dispose() ​

dispose(): void

Returns ​

void

store() ​

store(base64, mimeType, description, source?, fileName?): string

Parameters ​
base64 ​

string

mimeType ​

string

description ​

string

source? ​

string

fileName? ​

string

Returns ​

string

artifactStore? ​

optional artifactStore?: ArtifactStore

Defined in: src/core/voice-session.ts:637

When provided, agents/tools can persist artifacts (images, docs, etc.) via session.workspace.saveArtifact().

backgroundAgents? ​

optional backgroundAgents?: BackgroundAgent[]

Defined in: src/core/voice-session.ts:705

User-defined BackgroundAgent instances. Hosted by BackgroundAgentHostActor; each agent's onStart fires once on the first session.connected envelope. Actor-mode only — ignored in legacy mode (the legacy queue has no equivalent host).

behaviors? ​

optional behaviors?: BehaviorCategory[]

Defined in: src/core/voice-session.ts:591

Behavior categories for dynamic runtime tuning (speech speed, verbosity, etc.).

clientAudioInputRate? ​

optional clientAudioInputRate?: number

Defined in: src/core/voice-session.ts:558

Sample rate of inbound client PCM (what handleAudioFromClient receives). When omitted, defaults to transport.audioFormat.inputSampleRate — which matches what the framework instructs clients to send (browser RTC, voice WS clients are told to align to the transport's input rate, and Twilio's G.711 decode lands at the transport's rate too). Override only if your client genuinely sends a different rate and you've taken responsibility for the resample upstream.

clientAudioVad? ​

optional clientAudioVad?: ClientAudioVadConfig

Defined in: src/core/voice-session.ts:561

Tuning for the client-side energy-VAD barge-in. See ClientAudioVadConfig. Omitted fields fall back to defaults suited to a headphones setup.

clientMedia? ​

optional clientMedia?: ClientMediaProfile

Defined in: src/core/voice-session.ts:428

Client media plane profile (websocket PCM+JSON, or direct_rtc split plane: JSON on WS, RTC audio later). Defaults to WebSocket when omitted. See createClientChannel.

clientSender? ​

optional clientSender?: SessionClientSender

Defined in: src/core/voice-session.ts:363

Sender for all output to the client. The server owns the socket and feeds input via feedAudioFromClient / feedJsonFromClient and notifyClientConnected / notifyClientDisconnected.

compressionConfig? ​

optional compressionConfig?: GeminiCompressionConfig

Defined in: src/core/voice-session.ts:474

Context-window compression (Gemini only). Thresholds are in tokens; an unset one is omitted so the server default applies (trigger at 80% of the model limit, target half of it), and {} enables compression with both defaults.

conversationHistoryStore? ​

optional conversationHistoryStore?: ConversationHistoryStore

Defined in: src/core/voice-session.ts:603

When provided, conversation items are persisted at turn boundaries and on session close. For fan-out to multiple stores (e.g. JSON queryable + markdown human-readable), use conversationHistoryStores instead — both fields can be combined; this singular field is resolved first so existing read routing is preserved.

conversationHistoryStores? ​

optional conversationHistoryStores?: ConversationHistoryStore[]

Defined in: src/core/voice-session.ts:610

When provided alongside or in place of conversationHistoryStore, writes fan out to every store via MultiplexConversationHistoryStore; reads delegate to the first configured store (singular before plural). Use this to run a queryable store (JSON / Supabase) alongside a write-only artifact store like MarkdownConversationHistoryStore. Sole-store configurations (e.g. markdown alone) are valid; read methods will throw if your app calls them and the first store does not support reads.

divergenceCorrection? ​

optional divergenceCorrection?: boolean

Defined in: src/core/voice-session.ts:538

With shadowSttProvider set, answer a meaningful divergence with a spoken self-correction: the in-flight answer is interrupted and the model is told what the user actually said. The shadow result arrives after the answer has started, so the start of the wrong answer is still heard. Only a result for the still-current turn corrects; a stale one, transcription mode or the synthetic-output hold send nothing. Default false (observation only).

drainPostSession? ​

optional drainPostSession?: boolean

Defined in: src/core/voice-session.ts:635

Drain mode: await the post-session run before close() resolves (serverless hosts).

echoGuard? ​

optional echoGuard?: EchoGuardConfig

Defined in: src/core/voice-session.ts:550

Acoustic echo suppression at the audio-ingestion chokepoint: inbound client audio whose energy envelope correlates with recently played native model audio (speakerphone loopback) is dropped before the client VAD, the model and STT hear it, which stops the model transcribing its own voice as phantom user commands. OPT-IN: pass { enabled: true } to activate (double-talk on strong-echo paths can drop overlapped user speech, a deliberate per-deployment choice); the environment variable BODHI_ECHO_GUARD=0 hard-disables it. The reference is the native model audio only: ttsProvider output is not fed, and combining the two logs a warning at construction. Suppressed frames are counted in getDiagnostics().echoSuppressed.

functionBehavior? ​

optional functionBehavior?: FunctionBehavior

Defined in: src/core/voice-session.ts:479

Gemini Live default function-call behavior for every tool that does not set its own (see ToolDefinition.behavior). Unset means 'BLOCKING' on every model, so an inline tool's call and the answer spoken from it stay one turn; it is never left to the model's own default, which differs between Live models.

geminiModel? ​

optional geminiModel?: string

Defined in: src/core/voice-session.ts:465

Gemini Live model name (default: DEFAULT_GEMINI_LIVE_MODEL, "gemini-3.8-live").

greetingInterruptGraceMs? ​

optional greetingInterruptGraceMs?: number

Defined in: src/core/voice-session.ts:718

Override the transport-recommended greeting interrupt grace window. When the session's first assistant audio chunk arrives, the framework suppresses user-driven interrupts and drops outbound mic frames for this many ms so browser AEC can converge before echo events count as barge-in. When omitted, falls back to transport.capabilities.greetingInterruptGraceMs (OpenAI Realtime defaults to 1000 ms when framework-owns-interrupt; Gemini defaults to 0 ms). Clamped to [0, 5000]. A session resolving > 0 against a transport that does not advertise frameworkOwnsInterrupt (or does not implement cancelResponse) is downgraded to 0 at connect time with a warning log. Phone sessions should pass 0 explicitly (no browser AEC; dropping caller audio would silence real speech).

greetingInterruptible? ​

optional greetingInterruptible?: boolean

Defined in: src/core/voice-session.ts:728

When false, the greeting is uninterruptible end-to-end: from the greeting send until the greeting turn finalizes (post-playback where the playback protocol is active, else the fallback/estimate timer), interrupts are suppressed and outbound mic frames are dropped — user speech during the greeting is discarded, not queued. Works on every transport (unlike the grace window it needs no frameworkOwnsInterrupt or cancelResponse: withholding mic frames prevents server-side VAD barge-in too). Independent of greetingInterruptGraceMs, which still covers post-greeting AEC convergence. Default true.

historyResumeMode? ​

optional historyResumeMode?: "copy" | "attach"

Defined in: src/core/voice-session.ts:624

Resume persistence policy (default 'copy'):

  • 'copy': prior items are re-flushed into THIS session's record (checkpoint kept at 0); the writer uses createSession (host typically supplies a fresh sessionId).
  • 'attach': prior items are treated as already persisted (checkpoint advanced, not re-flushed); the writer uses ensureSession + reactivateSession and appends only new turns (host typically reuses the prior sessionId). Requires a store supporting ensureSession.

hooks? ​

optional hooks?: FrameworkHooks

Defined in: src/core/voice-session.ts:325

Lifecycle hooks for observability.

host? ​

optional host?: string

Defined in: src/core/voice-session.ts:432

Host for the local client WebSocket server (legacy/local mode).

initialAgent ​

initialAgent: string

Defined in: src/core/voice-session.ts:321

Name of the agent to activate on start.

initialAnalytics? ​

optional initialAnalytics?: SessionAnalytics

Defined in: src/core/voice-session.ts:627

Resume: prior aggregate analytics (the source SessionRecord.analytics) so the close report reflects the full record (incl. totalTokens, which items cannot reconstruct). Optional.

initialHistory? ​

optional initialHistory?: ConversationItem[]

Defined in: src/core/voice-session.ts:616

Resume: prior conversation turns to load into ConversationContext at start() and replay into the transport's model. Supplied by the host (which owns fetch + authorization); the full timeline for attach, or any subset for copy. See historyResumeMode.

inputAudioTranscription? ​

optional inputAudioTranscription?: boolean

Defined in: src/core/voice-session.ts:483

Enable server-side transcription of user audio input (default: true). Has no effect when sttProvider is set (built-in is disabled automatically). Use false to disable all input transcription for privacy or cost control.

listenTimeoutMs? ​

optional listenTimeoutMs?: number

Defined in: src/core/voice-session.ts:434

Listen timeout for local client WebSocket server startup (legacy/local mode).

log? ​

optional log?: (message) => void

Defined in: src/core/voice-session.ts:351

Optional diagnostic logger. Defaults to console.log.

Parameters ​

message ​

string

Returns ​

void

mediaResolution? ​

optional mediaResolution?: GeminiMediaResolution

Defined in: src/core/voice-session.ts:508

Session-wide media token cost for image/video input (Gemini only; MEDIA_RESOLUTION_LOW = 64 tokens per frame). Applies to every realtime-input image on the session; realtime input has no per-send override. Omitted → server default.

memory? ​

optional memory?: object

Defined in: src/core/voice-session.ts:593

Enable memory distillation. Extracts durable user facts from conversation and persists them.

store ​

store: MemoryStore

Where to persist extracted facts.

turnFrequency? ​

optional turnFrequency?: number

Extract every N turns (default: 5).

model ​

model: LanguageModelV1

Defined in: src/core/voice-session.ts:467

Vercel AI SDK model for subagent text generation.

nativePlaybackGating? ​

optional nativePlaybackGating?: boolean

Defined in: src/core/voice-session.ts:576

Rollout switch for native-audio playback-end gating (the OpenAI native path). Default false. When true, and the session is on the native audio path with a generation-gated transport and playbackStateProtocol active, native turn completion is gated on playback end.

nowMs? ​

optional nowMs?: () => number

Defined in: src/core/voice-session.ts:358

Injectable time source (milliseconds) for all metric/latency duration math. Defaults to Date.now. Shared with ClientVadDetector so session-side and VAD-side timestamps come from one clock (never subtract across clocks). Tests inject a fake clock to make latency assertions deterministic.

Returns ​

number

onClientCommand? ​

optional onClientCommand?: (message) => void

Defined in: src/core/voice-session.ts:382

Client protocol frames the built-in handlers do not recognize, such as a host's own retry command. Built-in types always run first and are never forwarded; a malformed built-in is dropped with a log line and reaches neither this hook nor onClientJson. Fires after onClientJson when both are set. Frames that arrive before the attach bootstrap has sent session.config are queued (up to 64) and dispatched after it. A throw from this hook or onClientJson is reported through hooks.onError and does not stop the other hook, the remaining queued frames or the attach.

Parameters ​

message ​

Record<string, unknown>

Returns ​

void

onClientConnected? ​

optional onClientConnected?: () => void

Defined in: src/core/voice-session.ts:391

A real client attached: a connection on the local client WebSocket server (never a probe or verifier), or notifyClientConnected() with a host-owned channel. Runs synchronously once clientConnected is true and before the behavior catalog, session.config and any greeting, so the host can resend durable state first. A throw is reported through hooks.onError and does not stop the attach.

Returns ​

void

onClientDisconnected? ​

optional onClientDisconnected?: () => void

Defined in: src/core/voice-session.ts:396

The real client detached. Runs at the end of the detach handling, once clientConnected is false. A throw is reported through hooks.onError.

Returns ​

void

onClientJson? ​

optional onClientJson?: (message) => void

Defined in: src/core/voice-session.ts:371

Supported seam for handling custom inbound JSON message types. Fired ONLY for an unrecognized type — it does not override built-in types (behavior.set, ui.response, file_upload, text_input, playback.ended, RTC signaling), and a malformed built-in type is dropped (not forwarded here). Without it wired, inbound JSON behavior is unchanged.

Parameters ​

message ​

Record<string, unknown>

Returns ​

void

onConnectionLifecycle? ​

optional onConnectionLifecycle?: (event) => void

Defined in: src/core/voice-session.ts:343

Connection-lifecycle facts: attempt / setup-ok / setup-failed / attempt-close / generation-close, correlated by connectAttemptId. handleSupplied on the attempt is what lets a consumer track resumed lineages without inferring them from log lines. Fires only on transports that report lifecycle (the Gemini Live transport); on a transport that declares neither this nor onUsageMetadata, a warning is logged at construction. With a credentialProvider, an event's failure may hold provider-supplied text and errors: see ConnectFailure before logging it.

Parameters ​

event ​

ConnectionLifecycleEvent

Returns ​

void

onTranscriptionDivergence? ​

optional onTranscriptionDivergence?: (liveText, shadowText, turnId?) => void

Defined in: src/core/voice-session.ts:530

Called when the shadow transcription disagrees with the built-in one (normalized comparison; a much shorter fragment of the other side counts as streaming truncation, not a mishearing). turnId is the turn the shadow result belongs to.

Parameters ​

liveText ​

string

shadowText ​

string

turnId? ​

number

Returns ​

void

onUsageMetadata? ​

optional onUsageMetadata?: (usage) => void

Defined in: src/core/voice-session.ts:349

Server-reported token accounting, the provider's raw payload, once per message that carries it. promptTokenCount is the standing prompt size — the signal for context growth. Fires alongside (not instead of) the normalized hooks.onRealtimeLLMUsage / realtime.usage, and only on transports that report it (the Gemini Live transport).

Parameters ​

usage ​

TransportUsageMetadata

Returns ​

void

onVerifierConnected? ​

optional onVerifierConnected?: () => void

Defined in: src/core/voice-session.ts:445

A verification-role (?verify=1) connection attached to the local client WebSocket server. Narrow hook for embedders (e.g. to wake the upstream); real-client connect handling (greeting, session.config, attachment accounting) never runs for it. Local server only.

Returns ​

void

onVerifierDisconnected? ​

optional onVerifierDisconnected?: () => void

Defined in: src/core/voice-session.ts:448

The verification-role connection detached (clean close or preemption by an arriving real client). Local server only.

Returns ​

void

orchestrationMode? ​

optional orchestrationMode?: "legacy" | "actor"

Defined in: src/core/voice-session.ts:653

Orchestration engine for tool routing/subagent lifecycle (default: legacy).

outputInterceptor? ​

optional outputInterceptor?: AssistantOutputInterceptor

Defined in: src/core/voice-session.ts:334

Host hooks between the provider's native assistant output and the session: screen or hold transcript chunks, drop audio chunks, and release held text before each transcript flush. See AssistantOutputInterceptor. Only native audio output passes through transcript and audio; text-mode and external-TTS output bypass them.

playbackStateProtocol? ​

optional playbackStateProtocol?: "disabled" | "audio_done"

Defined in: src/core/voice-session.ts:571

Playback-state protocol mode for this session's client surface. 'audio_done' enables the audio.done / playback.ended handshake; 'disabled' (default) keeps the estimate-only fallback. Effective participation additionally requires the client sender to support it.

port? ​

optional port?: number

Defined in: src/core/voice-session.ts:430

Port for the local client WebSocket server (legacy/local mode).

postSessionPipeline? ​

optional postSessionPipeline?: PostSessionPipeline

Defined in: src/core/voice-session.ts:633

Optional process-scoped post-session pipeline. When provided, the session registers a snapshot builder and closeWithReason dispatches the pipeline once on close. Absent → no post-session processing (unchanged behavior).

probeState? ​

optional probeState?: () => object

Defined in: src/core/voice-session.ts:440

Supplies the JSON state frame sent to ?probe=1 health-probe connections on the local client WebSocket server. When absent, probes are upgraded and closed (code 1000) without a frame. Probe sockets never attach as the client and never run connect/disconnect handling. Local server only: with clientSender the host owns the socket and must implement probes itself.

Returns ​

object

realtimeInputConfig? ​

optional realtimeInputConfig?: false | GeminiRealtimeInputConfig

Defined in: src/core/voice-session.ts:495

Gemini Live realtime input/VAD tuning. Applied only on the built-in Gemini transport path. When omitted (and vadConfig is not set), the framework applies DEFAULT_GEMINI_REALTIME_INPUT_CONFIG (END_SENSITIVITY_HIGH, silenceDurationMs=500). User-supplied fields deep-merge over the default at the automaticActivityDetection level. false opts out of the default: no realtimeInputConfig is sent at all, so the server's own VAD settings apply. Mutually exclusive with vadConfig. Has no effect when an external transport is injected via config.transport — that transport owns its own VAD config.

reattachContextReplay? ​

optional reattachContextReplay?: boolean

Defined in: src/core/voice-session.ts:423

With reattachGreeting: 'until-first-turn', a client that attaches to an ACTIVE session after a completed turn gets the last ten user and assistant messages (150 characters each) injected as quiet context, which requests no response. Suppressed while synthetic output is held. Default false.

reattachGreeting? ​

optional reattachGreeting?: "per-client" | "until-first-turn"

Defined in: src/core/voice-session.ts:416

Greeting policy for a client that attaches to an ACTIVE session.

  • 'per-client' (default): each newly attached client is greeted once.
  • 'until-first-turn': a client is greeted only while no turn has completed; a client that attaches later is not greeted (see reattachContextReplay).

responseModality? ​

optional responseModality?: "text" | "audio"

Defined in: src/core/voice-session.ts:648

Response modality. Default 'audio' (LLM-native speech). Set to 'text' to run the model in TEXT mode WITHOUT a TTS provider — the assistant's text is surfaced via the normal transcript events and no audio is produced. Used by text-only consumers (e.g. the Agent Composer CLI). When ttsProvider is set, text mode is implied regardless of this field. Requires a transport advertising textResponseModality.

responseWatchdogMs? ​

optional responseWatchdogMs?: number

Defined in: src/core/voice-session.ts:451

Model-silence watchdog (ms) after the user's turn ends. If the model emits nothing for this long, force a reconnect. Default 5000; <= 0 disables.

sessionId ​

sessionId: string

Defined in: src/core/voice-session.ts:315

Unique session identifier.

sessionMetadata? ​

optional sessionMetadata?: Record<string, unknown>

Defined in: src/core/voice-session.ts:612

Application metadata persisted alongside the session record (e.g. channel, surface, callSid).

shadowSttProvider? ​

optional shadowSttProvider?: STTProvider

Defined in: src/core/voice-session.ts:525

Observation-only second transcription. Unlike sttProvider it does not replace the transport's built-in transcription: the provider hears the same client audio, its per-turn transcript is compared with what the model itself heard, and a divergence is logged and reported through onTranscriptionDivergence. What the model hears, answers and records is unchanged (unless divergenceCorrection is on). A model mishearing is otherwise invisible, since its transcript and its answer agree.

Ignored (with a log line) when sttProvider is set, since there is no built-in transcription to compare against. Must be a distinct instance from whisperProvider; sharing one throws a ValidationError at construction. Receives no audio in transcription mode.

speechConfig? ​

optional speechConfig?: object

Defined in: src/core/voice-session.ts:469

Voice configuration for Gemini's speech output.

voiceName? ​

optional voiceName?: string

sttProvider? ​

optional sttProvider?: STTProvider

Defined in: src/core/voice-session.ts:512

External STT provider for user input transcription. When set, transport built-in transcription is automatically disabled. When omitted, the transport's built-in transcription is used.

subagentConfigs? ​

optional subagentConfigs?: Record<string, SubagentConfig>

Defined in: src/core/voice-session.ts:323

Background subagent configs keyed by tool name.

suppressClientAutoActions? ​

optional suppressClientAutoActions?: () => boolean

Defined in: src/core/voice-session.ts:408

Host gate for the automatic actions of a client attach, read once per attach after the behavior catalog and session.config are sent (and when the first setup completes with a client already attached). While it returns true the client is configured but nothing else happens: no greeting, no context replay, and no redial of a session parked in UPSTREAM_LOST. Under upstreamLossPolicy: 'hold' the reconnector reads it too: while it returns true the host owns recovery, so a lost provider connection parks the session at once instead of reconnecting on its own. Both reads log a throw from the gate and treat it as false.

Returns ​

boolean

transcriptionMode? ​

optional transcriptionMode?: TranscriptionMode

Defined in: src/core/voice-session.ts:585

Initial transcription mode for the session. Default 'agent'.

  • 'agent' (default): mic audio flows to transport; the agent responds. Existing behaviour.
  • 'transcription': mic audio flows to whisperProvider and the agent transport is quiesced. Transcripts accumulate in the dictation buffer; injection back into the agent is explicit (see injectDictationBuffer and the built-in inject_dictation tool — design §3.5).

transport? ​

optional transport?: LLMTransport

Defined in: src/core/voice-session.ts:651

Pre-constructed LLM transport. If provided, apiKey/geminiModel/speechConfig/compressionConfig/ mediaResolution/realtimeInputConfig/vadConfig are ignored.

ttsPlaybackFallbackMarginMs? ​

optional ttsPlaybackFallbackMarginMs?: number

Defined in: src/core/voice-session.ts:566

Fallback margin (ms) added to the estimated client playback end before the server force-completes a TTS turn that received no client playback signal. Default 1500; below the 500 ms floor is clamped up; invalid values fall back to the default.

ttsProvider? ​

optional ttsProvider?: TTSProvider

Defined in: src/core/voice-session.ts:642

External TTS provider for speech synthesis (actor-mode only). When set, LLM is configured for text-mode responses. When omitted, LLM-native audio generation is used (default). Requires orchestrationMode: 'actor'. Ignored in legacy mode.

upstreamLossPolicy? ​

optional upstreamLossPolicy?: "close" | "hold"

Defined in: src/core/voice-session.ts:676

What happens when the provider connection is lost for good.

  • 'close' (default): today's behavior. Once automatic reconnection is exhausted (attempt budget spent, no resumption handle, or a failed or timed-out attempt) the session closes with reconnect_failed, and a failed first dial in start() closes it with connect_failed.
  • 'hold': the session parks in UPSTREAM_LOST instead. Nothing is finalized (no session.close, no onSessionEnd, no post-session run), the client listener stays up, and session.upstreamLost is published. An external STT provider is stopped while parked, as during a reconnect, and started again when a redial activates the session. A failed first dial still rejects start() but leaves the session parked; close() finalizes a parked session as usual. Only recoverUpstream() redials a parked session, which a client attach also calls unless suppressClientAutoActions returns true; parkUpstream() parks it on purpose.

'hold' also enables host recovery (recoverUpstream(), parkUpstream(), see getRecoveryCapabilities()), which 'close' rejects with a SessionError. Both orchestration modes support it.

upstreamRecovery? ​

optional upstreamRecovery?: UpstreamRecoveryOptions

Defined in: src/core/voice-session.ts:687

Decide when to redial a lost upstream inside the session: a redial ladder with backoff, a fatal-close backoff, replacement of a dial stuck in CONNECTING, a backstop health tick and an idle park. See upstream-recovery-policy.ts. Requires upstreamLossPolicy: 'hold' and a transport with the recovery primitives (the Gemini transport); otherwise a ValidationError at construction. Works in both orchestration modes. Omitted, the host decides when to call recoverUpstream() and parkUpstream() itself.

userId ​

userId: string

Defined in: src/core/voice-session.ts:317

User identifier (used for memory storage and history).

vadConfig? ​

optional vadConfig?: GeminiVadConfig

Defined in: src/core/voice-session.ts:503

Gemini automatic-VAD tuning, sent verbatim as realtimeInputConfig.automaticActivityDetection with no framework default merged in (built-in Gemini path only). Supplying it together with realtimeInputConfig (including false) throws a ValidationError at construction.

watchdogReplayRecovery? ​

optional watchdogReplayRecovery?: boolean

Defined in: src/core/voice-session.ts:456

Watchdog-stall recovery via retained-utterance replay. When true, the last routed user utterance is retained (bounded, memory-only) and a response-watchdog stall replays it — in-place first, then once more after a reconnect. Default false (ships dark until live-validated).

whisperProvider? ​

optional whisperProvider?: STTProvider

Defined in: src/core/voice-session.ts:589

STT provider used in 'transcription' mode. Must be a distinct instance from sttProvider — sharing entangles the two lifecycles. Typically an OpenAIRealtimeWhisperSTTProvider.

Built with VitePress