Bodhi Realtime Agent Framework

Appearance

bodhi-realtime-agent / VoiceSession

Class: VoiceSession ​

Defined in: src/core/voice-session.ts:755

Top-level integration hub that wires all framework components together.

Manages the full lifecycle of a real-time voice session:

  • Audio fast-path: Client audio → LLM (and back) without touching the EventBus.
  • Tool routing: Inline tools execute synchronously; background tools hand off to subagents.
  • Agent transfers: Intercepts transfer_to_agent tool calls and delegates to AgentRouter.
  • Reconnection: Handles GoAway signals and unexpected disconnects via session resumption.
  • Conversation tracking: Transcriptions populate ConversationContext automatically.

Example ​

ts
const session = new VoiceSession({
  sessionId: 'session_1',
  userId: 'user_1',
  apiKey: process.env.GOOGLE_API_KEY,
  agents: [mainAgent, expertAgent],
  initialAgent: 'main',
  port: 9900,
  model: google('gemini-2.5-flash'),
});
await session.start();

Constructors ​

Constructor ​

new VoiceSession(config): VoiceSession

Defined in: src/core/voice-session.ts:1046

Parameters ​

config ​

VoiceSessionConfig | VoiceSessionProviderConfig

Returns ​

VoiceSession

Properties ​

conversationContext ​

readonly conversationContext: ConversationContext

Defined in: src/core/voice-session.ts:764

eventBus ​

readonly eventBus: EventBus

Defined in: src/core/voice-session.ts:762

hooks ​

readonly hooks: HooksManager

Defined in: src/core/voice-session.ts:765

sessionManager ​

readonly sessionManager: SessionManager

Defined in: src/core/voice-session.ts:763

RECONNECT_DEADLINE_MS ​

readonly static RECONNECT_DEADLINE_MS: 30000 = DEFAULT_RECONNECT_DEADLINE_MS

Defined in: src/core/voice-session.ts:760

Max wait for one reconnect attempt before giving up and closing the session with reconnect_failed. Without this deadline, a reconnect whose dial or setup never settles (an ECONNRESET on the in-flight WebSocket dial) leaves the session in RECONNECTING forever.

Accessors ​

clientConnected ​

Get Signature ​

get clientConnected(): boolean

Defined in: src/core/voice-session.ts:4494

Whether a real client is attached: a connection on the local client WebSocket server (never a probe or verifier), or between notifyClientConnected() and notifyClientDisconnected() with a host-owned channel.

Returns ​

boolean

workspace ​

Get Signature ​

get workspace(): object

Defined in: src/core/voice-session.ts:3131

Workspace API for persisting artifacts (images, videos, docs, etc.) produced by agents/tools. When no artifactStore is configured, saveArtifact returns null without persisting.

Returns ​

object

saveArtifact() ​

saveArtifact(params): Promise<ArtifactRef | null>

Parameters ​
params ​

SaveArtifactParams

Returns ​

Promise<ArtifactRef | null>

Methods ​

clearDictationBuffer() ​

clearDictationBuffer(): void

Defined in: src/core/voice-session.ts:4852

Discard buffered dictation without injecting it.

Returns ​

void

clearResumption() ​

clearResumption(): boolean

Defined in: src/core/voice-session.ts:4595

Clear the resumption handle the session and the transport hold, so the next dial opens a fresh server session. Returns true when the session held a handle. The connection itself is untouched.

Returns ​

boolean

close() ​

close(reason?): Promise<void>

Defined in: src/core/voice-session.ts:3267

Parameters ​

reason? ​

SessionEndReason = 'normal'

Returns ​

Promise<void>

closeClientConnection() ​

closeClientConnection(code?, reason?): boolean

Defined in: src/core/voice-session.ts:4516

Close the attached client connection with the given WebSocket close code and reason (for example 4000, 'goodbye'). The listener keeps accepting connections and the session stays up; the socket's close then runs the usual disconnect handling. Returns false when no socket was attached, and always with clientSender or direct_rtc, where the host owns the socket and closes it itself.

Parameters ​

code? ​

number

reason? ​

string

Returns ​

boolean

feedAudioFromClient() ​

feedAudioFromClient(data): void

Defined in: src/core/voice-session.ts:4435

Feed client audio into the session (LLM + STT). Used when the server owns the socket (multi-user).

Parameters ​

data ​

Buffer

Returns ​

void

feedJsonFromClient() ​

feedJsonFromClient(message): void

Defined in: src/core/voice-session.ts:4476

Feed client JSON (text_input, file_upload, etc.) into the session. Used when the server owns the socket.

Parameters ​

message ​

Record<string, unknown>

Returns ​

void

getClientSocketHealth() ​

getClientSocketHealth(): ClientSocketHealth | null

Defined in: src/core/voice-session.ts:4504

readyState and bufferedAmount of the attached client socket, for example to watch outbound backpressure. null when no socket is attached, and always with clientSender or direct_rtc, where the host owns the socket.

Returns ​

ClientSocketHealth | null

getDiagnostics() ​

getDiagnostics(): VoiceSessionDiagnostics

Defined in: src/core/voice-session.ts:4533

Point-in-time send-path diagnostics; safe to sample on any tick.

upstream/transportGeneration are null on transports that do not report diagnostics (injected fakes, OpenAI, Qwen) — null means unobserved, never zero. echoSuppressed is session-owned (suppressed frames never reach the transport counters): the echo guard's suppressed count, 0 without one.

Returns ​

VoiceSessionDiagnostics

getDictationBuffer() ​

getDictationBuffer(): string

Defined in: src/core/voice-session.ts:4847

Snapshot of dictated text since the last clear, joined with spaces. Useful for built-in tools (e.g. inject_dictation_as_user_message) and for ops surfaces that want to preview the buffer.

Returns ​

string

getRecoveryCapabilities() ​

getRecoveryCapabilities(): RecoveryCapabilities

Defined in: src/core/voice-session.ts:4554

What this session's recovery surface supports. RECOVERY_CAPABILITIES (everything) for a session with upstreamLossPolicy: 'hold' on a transport with abortIncumbent, currentDialGen and currentTransportGeneration (the Gemini transport). Otherwise, under policy 'close' or on another transport, recoverUpstream, reconnectBoundary and syntheticHold are false, turnStartPublication is true and transportGenerations reports the transport's generation counters. Gate host recovery on this.

Returns ​

RecoveryCapabilities

getSessionId() ​

getSessionId(): string

Defined in: src/core/voice-session.ts:4521

Session ID for logging and multi-user association.

Returns ​

string

getTranscriptionMode() ​

getTranscriptionMode(): TranscriptionMode

Defined in: src/core/voice-session.ts:4833

Public stable mode. Transient starting_* / stopping_* states are collapsed to the closest stable mode so callers never observe them.

Returns ​

TranscriptionMode

guardedTriggerGeneration() ​

guardedTriggerGeneration(instructions?, overrides?): void

Defined in: src/core/voice-session.ts:5169

Defensive wrapper around triggerGeneration. Throws if invoked while not in agent mode — surfaces preset bugs loudly in tests rather than silently leaking audio into a dictation flow. Triggers nothing (logged) while the synthetic-output hold is engaged.

Parameters ​

instructions? ​

string

overrides? ​
reasoning? ​

{ effort: ReasoningEffort; }

reasoning.effort ​

ReasoningEffort

Returns ​

void

injectDictationBuffer() ​

injectDictationBuffer(): Promise<void>

Defined in: src/core/voice-session.ts:4867

Inject the dictation buffer as a user message into the agent's conversation, then clear it. No-ops if the buffer is empty or the current mode is not 'agent'. Mirrors the existing text-input path: writes to the transport AND records the user turn in ConversationContext (so history/memory/subagent context see it).

Returns ​

Promise<void>

injectText() ​

injectText(input, opts): Promise<boolean>

Defined in: src/core/voice-session.ts:4926

Inject host-generated text into the model's conversation: live input the model responds to, or quiet context (see InjectTextOptions).

Resolves true once the text was handed to the transport (in live mode, a send the transport accepts into a send buffer counts as handed over) and false when nothing was sent: empty text, the transport not connected, transcription mode, or a transport send that failed (logged). It never rejects for a transport failure, so callers can fire and forget it.

Unlike injectTranscript, the text is not recorded in the ConversationContext and an in-flight response is not cancelled.

Parameters ​

input ​

string | ContentTurn[]

opts ​

InjectTextOptions

Returns ​

Promise<boolean>

injectTranscript() ​

injectTranscript(text): Promise<void>

Defined in: src/core/voice-session.ts:4898

Lower-level: inject an arbitrary user message. Serialized via the shared direct-input FIFO so back-to-back calls don't race the cancel-then-create sequence.

Parameters ​

text ​

string

Returns ​

Promise<void>

isSyntheticHoldActive() ​

isSyntheticHoldActive(): boolean

Defined in: src/core/voice-session.ts:5099

Whether the synthetic-output hold is engaged: after a recoverUpstream({ holdSyntheticUntilFreshSpeech: true }), until the user is heard again (input transcription, an external STT final, a provider interruption, or typed or injected text; microphone audio alone does not count). While it is, framework-generated output (greeting, directive reinforcement, notifications, injected context) is held.

Returns ​

boolean

notifyBackground() ​

notifyBackground(text, options?): void

Defined in: src/core/voice-session.ts:2874

Queue a short spoken update for the user. Delivered immediately when possible, otherwise after the current turn.

options.label is widened from the original 'SUBAGENT UPDATE' | 'SUBAGENT QUESTION' union to allow user-defined labels (e.g. 'TIME REMINDER'). Non-breaking: the two literal strings still type-check. NotificationActor normalizes the label on ingest in actor mode (uppercase + sanitize to [A-Z0-9 _-], max 32 chars, fallback to 'SYSTEM' if empty after sanitize).

Parameters ​

text ​

string

options? ​
label? ​

string & object | KnownNotificationLabel

priority? ​

"high" | "normal"

Returns ​

void

notifyClientConnected() ​

notifyClientConnected(): void

Defined in: src/core/voice-session.ts:4481

Notify the session that the client connected. Used when the server owns the socket (multi-user).

Returns ​

void

notifyClientDisconnected() ​

notifyClientDisconnected(): void

Defined in: src/core/voice-session.ts:4486

Notify the session that the client disconnected. Used when the server owns the socket (multi-user).

Returns ​

void

observeAudioInput() ​

observeAudioInput(observer): () => void

Defined in: src/core/voice-session.ts:4679

Observe each inbound client audio frame as it enters the session, before routing or gating, from the local client WebSocket, the direct RTC audio plane and feedAudioFromClient(). A throwing observer is reported through hooks.onError and does not stop the frame. Returns a function that removes the observer.

Parameters ​

observer ​

AudioInputObserver

Returns ​

() => void

observeAudioOutput() ​

observeAudioOutput(observer): () => void

Defined in: src/core/voice-session.ts:4665

Observe each chunk of native assistant audio as PCM, with its turn id, right before it is sent to the client. Audio from an external TTS provider is not observed. A throwing observer is reported through hooks.onError and does not stop delivery. Returns a function that removes the observer.

Parameters ​

observer ​

AudioOutputObserver

Returns ​

() => void

onDictationTranscript() ​

onDictationTranscript(listener): () => void

Defined in: src/core/voice-session.ts:4858

Observe transcription-mode transcripts without taking over the whisper provider's callbacks. Finals are buffered first; returns an unsubscribe.

Parameters ​

listener ​

(event) => void

Returns ​

() => void

parkUpstream() ​

parkUpstream(reason): Promise<void>

Defined in: src/core/voice-session.ts:4610

Take the provider connection down on purpose, for example when no client has been attached for a while: cancels automatic recovery, parks the session in UPSTREAM_LOST without finalizing it (publishing session.upstreamLost with reason 'host-parked' and reason as its detail), then disconnects the transport. Nothing redials it until recoverUpstream(), which a client attach also calls unless suppressClientAutoActions returns true. Rejects with SessionError under upstreamLossPolicy: 'close', while closing, and outside ACTIVE, RECONNECTING and UPSTREAM_LOST.

Parameters ​

reason ​

string

Returns ​

Promise<void>

prewarmTranscriptionMode() ​

prewarmTranscriptionMode(): Promise<void>

Defined in: src/core/voice-session.ts:5143

Pre-start the whisper session without flipping audio routing. Useful for masking the ~150–500 ms whisper-start latency on the first flip.

Returns ​

Promise<void>

publishSystemNotification() ​

publishSystemNotification(text): void

Defined in: src/core/voice-session.ts:2904

Parameters ​

text ​

string

Returns ​

void

recoverUpstream() ​

recoverUpstream(args): RecoverUpstreamResult

Defined in: src/core/voice-session.ts:4586

Abandon the current provider connection and dial a fresh one, without closing the session. Allowed from CONNECTING, ACTIVE, RECONNECTING (taking over an automatic reconnect) and UPSTREAM_LOST; the session goes to RECONNECTING at once and to ACTIVE when the replacement is set up. From CONNECTING it replaces the first dial of start(), still pending, whose late outcome then settles only start()'s promise, or, when the recovery comes before start() dials, start() dials nothing and resolves. Throws SessionError when getRecoveryCapabilities().recoverUpstream is false, while closing, or from any other state.

Before returning it finalizes the active turn as interrupted, aborts the incumbent connection, publishes session.reset and one session.reconnectBoundary, and clears the resumption handle, so the redial resumes nothing. Tool results and external-STT captures from the abandoned connection are dropped. No greeting is sent on activation; with skipContextInjection: false the recent conversation is injected as quiet context. Notifications are held during the dial and one is delivered on activation when the model is idle.

Single-flight: a call while a recovery is in flight, including one made from an event subscriber during this call, returns that recovery's result. activated rejects when the dial or another recovery step fails, which reports a recover-upstream error and parks the session in UPSTREAM_LOST, or when the session closes or parkUpstream() runs first.

Parameters ​

args ​

RecoverUpstreamArgs

Returns ​

RecoverUpstreamResult

registerTools() ​

registerTools(tools): Promise<void>

Defined in: src/core/voice-session.ts:4700

Add tools at runtime: each becomes executable at once and is merged by name into the declared tool list, which is sent with transport.updateSession({ tools }) (applied in place where the transport supports it, on the next connect on Gemini). A tool missing from the active agent definition executes inline with immediate scheduling. An agent transfer declares the new agent's tools instead.

If updateSession rejects, the declared list is restored and the rejection is passed on; the new executors stay registered. Rejects with SessionError with orchestrationMode: 'actor'.

Parameters ​

tools ​

ToolDefinition[]

Returns ​

Promise<void>

replaceTools() ​

replaceTools(tools): Promise<void>

Defined in: src/core/voice-session.ts:4719

Replace the declared tool list with exactly tools, sent with transport.updateSession({ tools }), for example to restrict what the model may call. Only the declarations change: every tool registered so far stays executable. A later replaceTools() with the full list restores the hidden tools.

If updateSession rejects, the declared list is restored and the rejection is passed on. Rejects with SessionError with orchestrationMode: 'actor'.

Parameters ​

tools ​

ToolDefinition[]

Returns ​

Promise<void>

resetConversationContext() ​

resetConversationContext(reason): object

Defined in: src/core/voice-session.ts:4638

Start the in-memory conversation over without ending the session, and return how many items were dropped. Everything up to now is persisted first, in this order: buffered transcripts are flushed into the context; every pending user message is sealed with the text it already holds (its fallback transcript, since a reset cannot wait for an external STT result); the history writer persists the unflushed items once; then the items, the checkpoint and the pending ids are cleared, and the retained user audio and the reconnect replay state are dropped. reason is only logged; no event is published.

The history stores keep the pre-reset items. Later history flushes, the next memory extraction, the close report's items, the post-session snapshot and the context replayed on a reconnect or transfer cover only items added after the reset. With memory configured, an extraction still in flight during the reset can mark items added after the reset as processed when it completes. The memory distiller and the history writer share one checkpoint, so the distiller then skips those items and the history writer never appends them to the history stores; only the close report's items list them. A warning is logged on every reset while memory is set.

Parameters ​

reason ​

string

Returns ​

object

cleared ​

cleared: number

sendJsonToClient() ​

sendJsonToClient<T>(message): void

Defined in: src/core/voice-session.ts:4882

Send an arbitrary JSON message to the connected client over the client transport (WebSocket / RTC data channel). Used by apps that want to surface custom progress or UI state — e.g. an example pushing Whisper transcript fragments to a web UI during transcription mode.

Accepts core frames, registered ClientProtocolServerExtensions, or an application HostClientFrame whose type is not a core frame type.

Safe to call any time after start(); no-op when no client is connected.

Type Parameters ​

T ​

T extends string

Parameters ​

message ​

AnyServerToClientMessage | HostClientFrame<T>

Returns ​

void

sendRealtimeMedia() ​

sendRealtimeMedia(base64Data, mimeType): boolean

Defined in: src/core/voice-session.ts:4939

Send an image or audio clip (base64) to the model as realtime input through transport.sendFile, e.g. a camera or screen frame. Nothing is recorded in the conversation history. Returns false, sending nothing, in transcription mode or when the transport is not connected, and false when the transport send throws (the error is logged, never rethrown); otherwise true, and the transport decides which MIME types it carries (the Gemini transport takes image/* and audio/*).

Parameters ​

base64Data ​

string

mimeType ​

string

Returns ​

boolean

setTranscriptionMode() ​

setTranscriptionMode(mode): Promise<void>

Defined in: src/core/voice-session.ts:5151

Switch between 'agent' and 'transcription'. Idempotent. Serialised with transferSession() via the SessionMutationQueue — concurrent callers queue rather than race. The mode-flip mechanics live in the DictationController; the serialization stays here.

Parameters ​

mode ​

TranscriptionMode

Returns ​

Promise<void>

start() ​

start(): Promise<void>

Defined in: src/core/voice-session.ts:2942

Start the client WebSocket server and connect to the LLM transport.

When recoverUpstream() runs while the session is CONNECTING, it replaces the first dial and owns the session from then on: start() neither closes, parks nor transitions the session, and runs no post-connect step. If the recovery came before start() began its dial (called the transport's connect()), from a session.stateChange subscriber of the CONNECTING transition or during the text-mode session update a pre-constructed transport gets before it connects, nothing is dialed and the returned promise resolves. Otherwise it settles with the stranded dial's own settlement: it rejects with that dial's error, or resolves if the dial completes late. For a dial stranded before setup completed that can take up to the transport's connect deadline. Await the recovery's activated to learn when the session is ready.

Returns ​

Promise<void>

transfer() ​

transfer(toAgent): Promise<void>

Defined in: src/core/voice-session.ts:3336

Transfer the active session to a different agent (reconnects with new config).

Parameters ​

toAgent ​

string

Returns ​

Promise<void>

tryPublishSystemNotification() ​

tryPublishSystemNotification(text): boolean

Defined in: src/core/voice-session.ts:2919

Observable-acceptance variant of publishSystemNotification: returns false when the session cannot accept the notification — already CLOSED, or actor mode without a live runtime — so claim/ack delivery protocols (e.g. Sutando recovery notices) can requeue instead of silently losing the message. true means enqueued; spoken delivery remains at-most-once by design (queues are in-memory).

Parameters ​

text ​

string

Returns ​

boolean

updateInstructions() ​

updateInstructions(instructions): Promise<void>

Defined in: src/core/voice-session.ts:4731

Replace the system instructions, sent with transport.updateSession({ instructions }) (applied in place where the transport supports it, on the next connect on Gemini). An agent transfer applies the new agent's instructions instead. Rejects with SessionError with orchestrationMode: 'actor'.

Parameters ​

instructions ​

string

Returns ​

Promise<void>

Built with VitePress