Skip to content

OpenAI GPT Live

Status: Current | Last reviewed: 2026-10-07

Tools can bind CallerTurn to receive JSON containing the latest genuine caller caption (text), the spoken question at caller speech start (replyTo), and timestampMs. The publisher stamps this before acknowledging Live caption delivery. It is independent of database transcript finalization and excludes synthetic application turns. The binding overrides any identically named session variable or metadata entry; empty startup evidence cannot establish consent. It conveys evidence, not an automatic authorization decision. MCP whole-value bindings resolve against the live context at tool invocation, just like authored embedded tokens. A startup snapshot previously left caller evidence empty in Macau. Resolved caller text is treated as data: braces or $var: text inside it cannot trigger another variable substitution.

For a scoped startup investigation, OPENAI_LIVE_STARTUP_CAPTURE_AGENTS accepts comma-separated agent IDs. During authenticated simulation attempts only, it records the sent native session model/instructions/history/audio settings and the opening instruction in private evaluation evidence. It excludes transport headers, API keys, caller audio, backend configuration and later instructions. Unset it after diagnosis. It is off by default and never captures real calls. Captures describe the request; an accepted instruction is not proof of speech.

Startup diagnostics distinguish session_start_sent, input readiness, greeting acceptance and first caption; continuous silent output is not evidence that the caller heard the greeting. Upstream on-start workflows can overlap welcome audio using voice startup preparation.

A session that does not acknowledge its start within three seconds can reconnect once, before any caller audio has been sent, output/captions received, or backend work begun. Caller input stays in the existing bounded startup buffer and is drained in order after the replacement acknowledges. The first opening is retained even when that unsent input contains microphone noise. A late acknowledgment from the discarded socket cannot release the buffer. Readiness or call teardown cancels the watchdog; pending application work, private pause and the existing one-restart limit prevent replay. This shares the limit with accepted-but-silent greeting recovery and never reruns lifecycle workflows.

Responses client backends retain later system/developer updates in their conversation position. Only the leading instruction block becomes the Responses instructions field. A supervisor completion or payment receipt must follow the earlier pending status in input; hoisting it to startup can leave the backend answering from the stale status. Caller text keeps its original role, including text that impersonates an advisor.

Tool-input repair preserves the caller's original answer in the transcript. The follow-up repair state is a tagged application notice, not a fabricated caller utterance. Live delivers it through the backend application path with a fixed voice redirect; its text cannot become caller consent or extracted identity. Providers that echo the notice also retain the existing synthetic-turn filter. The actual retry still re-enters the registered tool's authorization and binding gates.

Client delegation can opt into delegation_on_user_turn. Each completed native caller caption starts the existing serialized client worker, even if Live has already spoken or never requested a delegation. A fixed instruction in the initial session configuration reserves task answers for the backend and defers to the authored backchannel policy. A policy requiring silence or complete responses does not permit separate waiting acknowledgements. When the authored policy allows acknowledgements, automatic dispatch caps them at one per completed request; delegation itself never requires a waiting phrase. Identifiers, private capture and closing requests remain silent. Native and automatic dispatch wait for caption delivery and claim the same input revision once; independent application receipts remain separate jobs. Application-owned outputs use delegation_id: null. This does not select or replay tools, bypass capability gates, or introduce a separate transcription/TTS pipeline. Private capture, deep pause, disconnect and accepted closing suppress automatic dispatch. The default is off; enabling it adds backend work to every spoken turn and requires fresh audio/latency qualification.

Automatic caller-turn jobs and application jobs retain the caller generation they consumed. Application jobs capture it when queued, before waiting for the worker. A submitted screen response advances the generation just like a spoken turn; context-only appends (run_llm=False) do not. If a newer response arrives while backend work is running, its older answer goes to the thinking channel instead of forcing the previous question over the caller. The business work is allowed to finish; its result remains available to the backend handling the newer turn. This does not change native barge-in detection or cancel writes. Regression coverage in test_openai_live_service.py crosses native/application jobs with voice/screen interruptions, including a job whose worker starts after a newer response.

Do not append this reservation when the first caller caption arrives. An October 6 Vanguard investigation replayed the same human recording with and without that mid-speech instruction: both 20 ms and 200 ms streams changed the caller's words when the instruction was injected, while both unchanged and startup-policy streams retained the request. The native pipeline recording matched the input with 0.9998 correlation, and Genesys reported no connector drops; changing codecs or microphone gain would not address that control-message boundary. Per-turn dispatch, caption settlement and tool authorization still run normally. Explicit application messages retain their existing instruction path.

A closing delivered inside an existing voice turn can also be acknowledged by its complete final sentence. Captions count only after the backend closing result starts. This avoids replaying an entire recap because numbers were spoken as words or contractions differed. Pending backend work and transport audio must still settle; the check does not grade the recap's factual content.

A new application user response (such as a confirmation click or submitted photo) redirects GPT-Live away from its earlier spoken question while the backend processes that input. The redirect uses fixed platform wording in session.instructions.append; the submitted content remains user data in quiet context and the backend. Context-only appends do not interrupt speech. This does not cancel backend work or establish a successful action receipt.

Client delegation can opt into delegation_reinforce_result. With automatic caller-turn dispatch, the fixed reminder is installed in the initial session instructions. Sending it again immediately before commentary could restart the previous question and disrupt caller speech. Other client configurations send the reminder before each final, non-reasoning answer. It asks Live to retain all facts, qualifications, action state and the next question. The answer itself stays on the commentary channel; no backend or caller content becomes an instruction. Progress and reasoning do not trigger this reminder. It defaults off, is ignored in managed Responses mode, and requires audio qualification: it is guidance, not a guarantee of verbatim delivery.

Client backends can configure delegation_silent_response as an uppercase control token (1–80 characters, letters, digits and underscores). An exact, final, non-reasoning response containing that token becomes a nonempty silent completion on session.thinking.append. Keeping the update nonempty prevents the SDK's spoken no-output fallback. Tool-only jobs that emit no text also use this silent completion when the setting is enabled, so the SDK's empty-result acknowledgement cannot become patient-facing speech. Failed jobs retain their error handling. Ordinary text, progress, reasoning and accepted closing acknowledgements are not replaced. The setting is opt-in; use it only for conversational steps explicitly assigned to the voice prompt. It does not mute native audio or remove backchannels. Backend business actions still execute and report their normal results. A silent completion associated with an older caller turn is discarded after a new caller turn starts, so its continuation cue cannot prompt a response to the wrong turn.

A native voice turn can begin with listening feedback and then deliver the backend's closing without opening another turn. The end handler also recognizes the complete backend closing in post-request assistant captions, ignoring punctuation and case. It still waits for native audio and playback to finish. Earlier speech, caller captions, partial closings and unrelated verification instructions do not satisfy this check. This avoids repeating an already delivered farewell through the recovery path; nonmatching delivery retains the existing bounded recovery.

GPT Live is an additional speech-to-speech service, selected with provider: "openai", type: "s2s", service: "GPT Live API". Existing Realtime agents retain their service selection. The implementation requires Pipecat 1.9 and runs through the existing universal pipeline and WorkerRunner.

Client delegation passes the same conversation tenant through the service registry to its text backend. OpenAI Chat Completions and Responses wrappers must inherit RuntimeContextServiceWrapper; otherwise the registry omits that context and a selected tenant integration can appear unavailable.

After native captions settle, only one client delegation may consume that caller-input revision. Overlapping delegations for the same correction receive a quiet duplicate acknowledgment; they do not enqueue an empty-transcript backend continuation that could mistake a readback for fresh consent. A new spoken turn remains distinct even if it repeats the same words. Application events can still run their own backend jobs without another caller turn.

The runtime retains input that arrives while the first native connection starts. The initial buffer allows eight seconds plus elapsed server-observed setup time since gRPC ServerHello, rounded up and capped at five minutes of setup (308 seconds total). This includes audio accumulated while a Conversation Start workflow runs before pipeline construction. The timestamp is internal runtime state, not caller-supplied metadata. Recovery keeps its eight-second bound. The upstream adapter discarded startup input; the Vanguard investigation observed one to two seconds lost during initialization. The Genesys 4463 call on October 2 also exposed a longer lifecycle backlog overflowing the former fixed eight-second initial bound and permanently suppressing the greeting. On session.started, TrueServe drains it in arrival order before sending the opening. New input cannot overtake that drain. Initial microphone energy must not consume the first greeting: it may be background noise rather than a caller request. The preserved input still reaches Live before the opening instruction. Overflow produces a permanent startup error.

The October 6 agent 4463 incident exposed this distinction: low-level microphone noise crossed the startup energy threshold, marked the opening as already sent, and left the agent silent without ever sending an instruction. The initial connection now always retains its enabled greeting. The recovery path still suppresses a repeated opening when input activity arrives during reconnection. Regression coverage includes silence, quiet noise and louder input, preserving every buffered frame before exactly one initial greeting.

The opening also requires the first nonempty input-audio event after resampling. Transport readiness alone does not establish that audio is flowing to Live. Audio continues while the opening is acknowledged; the runtime adds neither synthetic silence nor a fixed priming delay. Captions still come from Live when native playback is disabled and a downstream ElevenLabs stage supplies the audible voice.

If an accepted opening remains silent for five seconds, the runtime may replace the native connection once, before either party speaks or any backend/application work starts. It no longer repeats the instruction and waits another five seconds on the same silent connection. Meaningful input/output audio or transcripts, pending backend/application work, pause and disconnect retain their recovery fences. A missing acknowledgment does not trigger a reset. The application conversation remains the same. The reset has an eight-second deadline and audio buffer, retains input in arrival order, and suppresses events from the abandoned connection. If the caller starts speaking during reconnection, their audio takes precedence over the unheard greeting. A disconnect cancels recovery; a failed reset or buffer overflow remains a permanent startup error. Simulation traces retain the reset and its latency in the original attempt. This never reruns business tools or resets a live exchange.

Startup diagnostics include elapsed milliseconds, outgoing input-event counts, input readiness, greeting acceptance and the first caption. Silent native audio packets are not proof of an audible greeting. Controlled Vanguard tests of agent 4451 on October 1 still reproduced occasional first-connection silence with identical initial/replacement configurations after the ordering correction alone. Retaining the initial input then produced four consecutive first-connection greetings in the scoped probe. This is evidence for an initialization defect, not a provider latency guarantee. The shorter recovery additionally removes a known five-second penalty. Measure first non-silent TTS PCM and client playback separately from caption delivery.

After activation, four unnamed-greeting checks on Vanguard delivered first non-silent ElevenLabs PCM in 4.8–7.5 seconds from starting the client session, without recovery. A separate matched-caller billing conversation still needed one reset and delivered its named greeting at about 12 seconds; its bill lookup, spoken result and closing completed. The input-drop defect is confirmed, but the remaining accepted-but-silent startup is unresolved. These small, different scenarios do not establish a provider failure rate or a general latency SLA.

Configuration

The service definition is the source for the inspector, canonical configuration schema, and generated authoring reference. Configuration is stored on the owning stage in agent_versions.config; no new database columns or credential store are used.

{
  "id": "support",
  "type": "s2s",
  "provider": "openai",
  "service": "GPT Live API",
  "integration_id": "openai-integration-uuid",
  "config": {
    "model": "gpt-live-1",
    "voice": "marin",
    "delegation_mode": "responses",
    "delegation_model": "gpt-5.6-terra",
    "delegation_reasoning_effort": "low",
    "delegation_parallel_tool_calls": true,
    "delegation_tool_choice": "auto",
    "multimodality_enabled": true,
    "greeting_enabled": false
  }
}

The frontend voice model and backend reasoning model have separate prompts. conversation_instructions controls the voice model's style and delegation behavior. The fully resolved Behavior prompt, including variable substitution, topic selection, and enabled capability instructions, goes to the backend. delegation_instructions appends additional backend guidance. Leaving voice instructions blank uses the structured default in apps/ai-manager/lib/service-definitions/openai-live-prompts.json, shared by the runtime, editor and generation bundle. It includes Backchannel, Interruption and Delegation policies. Explicit saved prompts are preserved. The shared default follows the documented calm delivery, moderate backchannel and interruption wording, with task-answer/cancellation delegation guidance retained. The backend starter is offered for empty prompts and supplied to authoring tools as a structure to adapt, not automatically inserted into existing agents.

Editor ownership

The canvas derives one backend LLM node and a return connection for every Live stage in both delegation modes. They are display-only: they never enter the pipeline array, output lanes, layout tree or frame-flow computation. Clicking the backend edits its owning Live stage through the normal draft/collaboration path. The node ID suffix ::delegation is an inspector address, not a runtime stage ID. Client mode labels the canvas card and inspector Reasoning; the mode selector continues to say Client Delegation. In both modes, the backend card displays the shared subagent/tool/rule/guard summary. The Live voice card suppresses that summary and retains its audio icon, since those behaviors are edited in the backend.

  • Voice → Behavior: greeting first, then the shared Prompt editor for conversation_instructions, followed by channel capabilities.
  • Voice → Properties: voice model/voice/integration, delegation mode, audio settings, and an explicitly explained custom Live endpoint under Advanced connection.
  • Backend → Behavior: the existing resolved business prompt, procedures, tools, and optional delegation_instructions. Copy full prompt includes that suffix.
  • Backend → Properties: backend model/settings, progress options and both timeouts. Responses locks the provider to OpenAI and shows only Live-supported fields. Client Delegation uses Provider → Service → Integration Profile → Model, matching ordinary LLM nodes. Both modes use the shared integration-scoped searchable model picker, displayed above provider settings rather than inside a collapsed group. Client integration defaults use the same tenant-scoped default-selection helper; provider changes clear the prior provider's credentials. Inherited OpenAI Responses displays the voice integration without materializing a nested backend just by opening the node. Choosing a profile creates the explicit backend configuration. Nested provider settings take precedence and their effective inherited values remain visible. Prompts and tools stay on the owner.

The voice Prompt uses the normal CodeMirror/template editor with variable and If / Else completion. prompt/startup_prompt.py renders #when/:else and Mustache values with the same precedence as greetings: system, tenant, template defaults, call context, session metadata and session variables. This happens at service creation; no variable-change subscription rewrites the voice instructions. The normal prompt-variable inventory includes conversation_instructions. Voice TOOL, SUBAGENT and PROCEDURE references describe existing backend work. Their dropdown has no tool-add, subagent-create, run, set or procedure-definition actions. prompt/live_voice_prompt.py resolves names and public descriptions, without schemas, subagent instructions, procedure bodies or action execution. Unknown references explicitly render as unavailable. Run/set statements, component references and procedure blocks still belong in backend instructions and are rejected in the voice field.

The editor adds a read-only generated list beneath Backend tools: (or at the end when the heading is absent). live-backend-list.ts renders it as CodeMirror chrome, outside document text, undo history and collaboration writes. Its sources are the existing capability-aware tool list and enabled subagent/named-procedure metadata. The runtime independently inserts the list from the actual registered tools at session.start, after capability registration. Built-in callable descriptions come from the generated runtime contracts, so payment, verification and consult operations each explain their own use. MCP reference payloads retain each filtered operation's public description; saved/inline tools and registry capabilities supply their authored descriptions. Category hints such as “built-in” or “Payments capability” are not usage descriptions. Descriptions are whitespace-normalized without a character cutoff, preserving conditions later in the text. Missing tool descriptions are identified explicitly for the author. Disabled subagents and procedures are excluded. Metadata is inserted after conditional/variable rendering, so template-like text in a description cannot execute or interpolate secrets. Changing backend configuration updates the editor list and the next session's instructions; it does not append instructions during a running call.

Backend Main and subagent prompts retain PromptResolver/InstructionRefresher execution, including variable updates, conditions, registered actions and scoped capability guidance in both modes. Smart subagents use the selected backend model and replace its active business prompt through the existing topic path. The voice's startup prompt stays separate; scripted subagents retain their explicit handoff/reconnect behavior.

The client backend runs in a separate pipeline. Its observer relays A2UI render and multimodal-enable frames into the live pipeline alongside tool activity. The ordinary publisher then delivers the components and, during a saved test, requests the configured simulated UI action. A tool's “rendered” return string alone is not evidence that a component reached the caller. The relay copies frames only at the backend source, preventing a second render when they pass the backend's assistant aggregator; private backend text remains private.

delegation_context_max_exchanges is a client-only integer, 0–100, default 0. Zero retains full backend history. A positive value keeps that many previous user-led backend exchanges before each new sequential job, preserving the complete tool calls/results within retained exchanges and all system/developer guidance. The current request is always added. This is a count of delegation/application exchanges, not spoken turns or tokens. It does not summarize older state, alter the voice model's context or tune managed Responses context. Reduced windows can lose earlier task details; use full history by default. Native delegations still wait for settled input, and application/capability delivery keeps its existing path.

Field Runtime behavior
model, voice Live model and voice; defaults gpt-live-1, marin.
delegation_mode responses (managed OpenAI backend) or client (Pipecat worker).
delegation_model Independent backend model; defaults gpt-5.6-terra.
delegation_reasoning_effort, delegation_reasoning_summary Optional backend reasoning settings; supported values depend on the model.
delegation_max_output_tokens Optional backend cap, minimum 16, including reasoning tokens.
delegation_service_tier OpenAI backend tier: auto, default, flex, priority, ultrafast, fast_tier_temp_pilot; requires model/project support.
delegation_tool_choice auto, required, none, or function with delegation_tool_name.
delegation_parallel_tool_calls Whether managed Responses may request independent tools together.
delegation_web_search Managed OpenAI web search alongside registered function tools.
delegation_text Managed Responses { "verbosity": "low" \| "medium" \| "high" }; omitted/null uses provider defaults. Live does not expose text.format.
delegation_backend Optional client LLM stage, including provider, service, integration and provider configuration.
delegation_context_max_exchanges Previous client backend exchanges to retain; 0 means full history.
delegation_timeout_secs Client worker timeout; default 120 seconds.
delegation_speak_progress Client progress speech; default false. Reasoning output remains unspoken.
delegation_reinforce_result Fixed complete-answer reminder; installed at startup with automatic caller dispatch, otherwise before final client results. Default false.
delegation_silent_response Optional exact final control token for a silent client-backend completion. Preserves normal answers, actions and accepted closings.
tool_progress_messages_enabled Speak authored per-tool progress_message text using private TTS; default false in both delegation modes. Independent of client-backend progress.
tool_timeout_secs Default registered-function timeout; default 120 seconds, per-tool overrides remain authoritative.
base_url Optional Live-compatible WSS endpoint; defaults to OpenAI's /v1/live/sessions.

The inspector's voice-model dropdown offers gpt-live-1. Its voice dropdown contains all 22 built-in names in the Live create-session reference, including the twelve additional voices documented in the session guide. marin remains the default; existing cedar configurations remain valid. Pipecat's current voice field accepts names, not the API's custom-voice {id} object.

delegation_model uses /api/models/openai with the stage's integration and documented text-model fallbacks. Live/audio/transcription/image families are excluded from that backend catalog. dynamicOptions.autoSelect: false preserves the service default and explicitly saved choices when options load; saved IDs remain visible even if absent from the fetched list. Model/voice IDs remain open strings in the generated schemas.

OpenAILiveTextSettings renders the nested delegation_text.verbosity as a visual selector. Choosing the model default removes that key and removes an empty delegation_text object. Editing verbosity preserves other stored keys; unsupported imported keys are identified and removed only by an explicit user action. Managed-mode preflight and canonical schema validation reject unsupported text fields, including format, before they reach OpenAI. Backend text detail does not configure spoken delivery.

Client delegation defaults to Pipecat's Responses service and inherits the voice stage's integration. It uses the same model, reasoning, output limit, tool-choice, parallel-tool and text-verbosity settings as managed Responses delegation. An explicitly configured backend uses its own integration and provider credential resolution:

{
  "delegation_mode": "client",
  "delegation_backend": {
    "type": "llm",
    "provider": "openai",
    "service": "Responses API",
    "integration_id": "backend-integration-uuid",
    "config": {
      "model": "gpt-5.6-luna",
      "reasoning_effort": "low"
    }
  }
}

The selected backend must implement Pipecat LLMService. Its model and provider settings take precedence over the convenience delegation fields. The resolved business prompt remains authoritative; use additional backend instructions to extend it. Preflight also checks an explicit client backend's service mapping, configuration, and credentials.

Native client-delegation jobs wait for the current caller caption to finish and reach the publisher before taking the transcript for backend reasoning. The wait is bounded by thirty seconds and the configured delegation timeout; an unsettled input reports an error without starting task work. Application-initiated jobs already have their explicit input and do not require another caller turn. This controls when an issued client delegation starts; GPT Live still chooses when to issue native delegations. Final client answers are retained as backend_output diagnostics separately from spoken captions. The default client backend attaches the standard usage hooks under its own delegation stage and model, preserving tenant and conversation attribution.

Migrating an existing business agent

Copy the authored Behavior prompt and specialist addenda to the backend, together with their procedure sidecars, variables, background assessment configuration, tool bindings, capability settings and supervisor routing. Prompt text alone does not bring those dependencies across. Preserve the target Live integration, voice and backend selection. Keep the system-message representation synchronized with template.inline, and use the normal draft update and editor synchronization path.

Use conversation_instructions for speaking style and delegation. Follow OpenAI's Backchannel policy, Interruption policy and Delegation policy structure. List the actual backend capabilities. Delegate answers to active procedure questions, consent, corrections, refusals and session-ending requests, as well as initial service requests. Short answers such as “yes” still need the backend's current question and task state. The voice layer must wait for verified results before announcing a completed action.

Use greeting_message to specify the greeting language, what to say, and that Live should begin immediately then pause to listen. Variable substitution and conditional branches still apply. After session.started, the adapter sends the opening once with session.instructions.append and delegation_id: null. It consumes the opening before the upstream service can send it as commentary. Keep input audio running, including silence, during startup.

For a controlled comparison, copy the saved tests and remap their golden IDs to the target agent while retaining caller personas, expected outcomes, fixtures, turn limits and general goals. Run tests automatically selects synthesized caller audio for Live, exercising the actual voice model and orchestrator. Check the captured target, tool outcomes, spoken transcript and observed session end. An automatic pass alone does not establish that the requested task or farewell was delivered correctly. Preserve first attempts and any retries separately.

Protocol and capability boundaries

The adapter is openai_live_services.py.

  • Tools: managed Responses uses the stage's LLMContext tools and registered callbacks. Client workers share the callback registry by identity, so later consent, argument binding, input healing, PII, and payment-gate wrappers are retained. Worker tool frames are forwarded to the existing publisher and rule engine; their context continuation callback runs only on the backend. Per-tool progress announcements are off by default for Live. Enabling tool_progress_messages_enabled restores authored tool_logic.*.progress_message speech through the private prompt voice. The policy belongs to the owning service so another stage can still announce its own tools. Tool execution, gate checks, argument bindings, result captures, payment scripts, and secure data-capture prompts do not depend on this setting.
  • Tool continuation: Live requires every pending function output before a response.create. Application input waits for active backend work. Results with run_llm=False suppress automatic continuation, preserving asynchronous tools, payment capture, and guided-flow handback. Responding platform updates request a backend continuation, including updates arriving after a tool has intentionally paused. Managed activity spans provider delegation-ID changes; completing one delegation does not retire other active work. Evaluation events use a stable activity_id for this aggregate, while client jobs keep their independent delegation lifecycles.
  • Application input: A2UI actions and typed chat are delivered explicitly to the delegation backend. Plain context mutation does not send messages into an active Live session. Responding system/developer instructions use session.instructions.append; assistant text intended for speech uses commentary, and nonresponding context uses thinking. Caller/UI text remains caller data, never instructions. Each append is chunked below the API's 500-token limit. None of these channels is a secret store. External capability and advisor notices send a fixed voice redirect to await the backend's current answer when a response is requested. Their full business update goes only to the backend; nonresponding external updates stay silent. This prevents the voice and backend from independently asking the same supervisor question or announcing different versions of a decision. Attested capability updates with an operation ID and terminal status fence queued pending updates for that same operation, including late deliveries after reconnect. Unidentified updates and required function outputs are preserved; speech already sent cannot be retracted by this fence.
  • Multimodal: uploaded images and text go together to the backend. The latest camera snapshot is sampled on delegation. In managed mode, an already-running response can require a continuation to inspect that snapshot. The backend must support vision. The voice model itself does not accept images.
  • Payments and secure capture: existing input/output gates and private TTS prompts remain in charge. The adapter does not queue images or typed input while private capture is active. Sensitive values remain in the existing capture/vault path; the model receives the masked outcome. The derived OpenAI private TTS keeps the main voice only if GPT-4o TTS supports it. Live-only voices such as gleam use marin for private prompts; config.aux.handoff_tts can override that voice. The Live voice stays unchanged.
  • Guided flows: tsf_deep_pause closes the Live connection while a scripted subagent owns the call. Handback is queued before reconnect and delivered after session.started. Images received while paused are discarded.
  • Prompts and topics: instruction refresh updates the backend's instructions only when the resolved prompt changes. Speculative refreshes after bot turns must not append business rules to the voice engine: session.instructions.append can interrupt speech and replay the opening script. Explicit topic/capability notices still use the application-input path; the short voice prompt remains separate. Realtime session.update payloads are never used for Live. Procedure checklists and tool observations stay with the backend and critic. Earlier quiet voice hints prompted an independent task response: testing captured correct backend dates alongside invented spoken dates, and complete supervisor questions alongside incomplete spoken ones. The voice now receives the backend answer without those duplicate task prompts. Simulations retain bounded procedure_brief and tool_observation diagnostics without sending them to the voice or counting them as spoken evidence. Tool projections omit private fields and distinguish a successful displayed selection from search alternatives. Required function results still reach the backend in both delegation modes. Rendering remains distinct from confirmation.
  • Transcripts and recordings: native caller transcriptions are copied downstream for the publisher, consent evidence, and procedure monitor. Local transcript turns close after two seconds without a fragment. The upstream 0.8-second interval split permission questions, explicit consent and spelled surnames in Vanguard tests; a trailing fragment such as “text” could replace the caller's complete permission in the capability evidence. This assembly interval usually fits within the 2.5-second consent grace period. If delegation arrives while the caller is still speaking, authentication and payment callbacks allow up to 15 seconds for that active answer to finalize. They stop waiting as soon as recorded evidence arrives; interims cannot grant permission, and a missing final answer still fails closed. It changes transcript finalization, not Live's audio turn detection, and does not infer consent from an incomplete answer. Extraction-bound tools wait up to three seconds for the native caller caption to close and reach the publisher. The publisher acknowledges it only after notifying extraction; the usual extraction-settle and tool gates then run. Timing out leaves those gates closed and asks the backend to wait. It neither forces a caller turn to end nor cancels caption delivery or extraction. A Live tool refused specifically because input processing is pending schedules one coalesced recovery wait, bounded to thirty seconds. After the latest caption reaches the publisher and extraction settles, it queues a backend continuation through the normal serialized path. It does not replay the tool, infer consent or bypass any gate. A normal tool call that already consumes the settled input cancels the extra continuation; disconnect cancels the wait. Its independent activity signal keeps the simulator from answering an interim waiting phrase while this recovery is pending. A separate unanswered-turn guard covers silence without any attempted tool. It tracks the native user-turn opening, so an answer delivered before the final caption still counts. After eight seconds without an assistant turn, it waits up to thirty more seconds for speech, delegation or input recovery to become idle and for caption delivery. It then queues one backend reasoning continuation to answer or clarify the actual caller turn. It never replays a tool. A newer caller turn or an actual assistant turn cancels an uncommitted wait; private capture and disconnect suppress recovery. A committed queue send finishes through normal serialization. Simulation evidence records provider_recovery when used. Judge-gated tools also wait for native caption delivery when they have no extracted bindings, so a just-spoken confirmation reaches the procedure monitor. Simulations separately record delegated assistant text as backend_output, bounded to 6,000 characters with a truncation flag. These events exclude reasoning and do not replace the actual spoken transcript. External instructions also emit application_message after delivery to the backend, preserving source, capability attestation and causal order. Queued, stale and failed sends do not count as delivery. This makes supervisor clarification questions visible to evaluation without treating an instruction as a completed action. The procedure monitor receives the same delivered instructions in its bounded evidence ledger, including outside simulations. capability_attested identifies internal capability results; it is not an authentication verdict on messages from the advisor channel. Caller audio continues downstream for the recorder, subject to existing suppression settings.
  • Speech timing: both transports disable local VAD for Live. Its aggregators use ExternalUserTurnStrategies(enable_interruptions=False). Voice isolation and the authored idle policy remain available. Idle prompts use Live's commentary channel for a phrase, or instructions for an explicit redirect.
  • Ending a call: Live's end-session handler waits for its native transcript turn to finish (LLMFullResponseEndFrame) and for pending delegation work. A backchannel can end while the backend is still producing the farewell; that event must not disconnect the session. A native closing turn must begin after backend output for the accepted end request becomes available: a delayed end event from an earlier acknowledgment is ignored even if the backend has just finished. Managed output deltas and client spoken commentary establish that output receipt; receipt alone does not prove that the caller heard it. If an earlier pending instruction finishes and the latest backend closing remains unspoken, the handler waits for quiet playback and replays that verified closing through the Live commentary channel once. It also sends a short instruction to deliver the outcome without another question or action. This recovery never re-executes the backend or its tools and emits provider_recovery with reason closing_ack_not_delivered in simulations. A closing that continues an already-open turn uses the bounded fallback. Managed function continuations and client delegation tasks remain pending until they finish. Helper-TTS stops and transport silence events must not truncate the farewell. The usual five-second fallback extends to thirty seconds while a native turn or backend response is active, or the post-tool closing has not begun. Closing audio received within the last five seconds extends that deadline up to an absolute two-minute limit. This allows a long financial recap to finish without leaving stalled calls open: after thirty seconds, five seconds without closing audio ends the wait at the next timer check. An inferred closing-turn end starts a two-second settling interval; a new speech turn cancels that pending hangup. Closing also requires actual non-silent native speech to have been observed, at least two seconds since that speech, and an idle output transport. Helper TTS and continuous silent audio do not count as a progressing farewell. Transcript grouping alone is not a provider response-completion signal.
  • Startup: both gRPC and LiveKit prime the context even with greetings off. An enabled greeting is a one-shot session.instructions.append. The request is retained outside conversation history until provider startup, including when transport readiness arrives before context binding or after session configuration. An already-started session receives it immediately; reconnect does not replay it. Live emits 24 kHz audio. Startup diagnostics distinguish the sent greeting, its matching provider acknowledgment, first input audio, first output audio and disconnect. Acknowledgment alone does not prove speech. Simulation provider_startup events retain frame counts and acceptance state without greeting text, audio or caller data. These traces diagnose a frozen session timeline separately from a delayed spoken greeting.

The orchestrator owns the primary Live WebSocket and uses it for both audio and control events. OpenAI's separate sideband connection is useful when the primary audio connection is WebRTC or SIP; it is unnecessary for this server-owned WebSocket transport. Keep one executor for each tool call. An instruction acknowledgment indicates context injection, not corrected speech or completed playback. Blocking speech before the caller hears it would require a separate audio-buffering and approval policy; these changes do not provide that guarantee.

When authoring the delegation prompt, distinguish confirmed facts, the current question awaiting a caller answer, and an external action awaiting its result. Require the backend to reconcile new answers and callbacks with completed work before returning its next caller-facing question or verified result. Avoid progress prefaces such as “let me check” before verification and repeated requests for completed verification, photos or payment. This is prompt guidance, not a pre-playback enforcement mechanism.

Keep conditional procedure prose and its background completion criteria aligned. A supervisor clarification is not a final decision. An approval needs separate permission for the actual approved amount; rejection or expiry must follow its actual offered next step. A payment refusal or choice to settle with the office must not be graded as missing authorization that the agent should keep pursuing. Test these branches separately, including delayed and duplicate responses.

Changing caller or reviewer policy changes the measurement. Establish a fresh baseline with the agent frozen before applying a prompt candidate, retain the original grades, and report qualification results with their policy versions, scenario mix, first-attempt outcomes and execution/review errors. See Agent evaluation workers for causal caption evidence and independently checked caller stopping decisions.

Usage and limits

Live's cumulative session seconds are recorded under speech_engine; repeated usage snapshots are not added together. Reconnected sessions add their own duration. If provider duration is unavailable, the existing runtime-duration fallback is identified as measured usage.

Managed backend responses are recorded separately as llm, using the actual backend model, response ID, token counts, cache counts, and service tier. Client backends use their existing service telemetry. Their stage ID is <voice-stage-id>:delegation. Forecasts include both session time and a backend text-token estimate using the configured workload assumptions.

The public price seed includes GPT Live's published $0.05/minute session rate, billed per second, verified on 2026-09-11. Publishing/importing the price book remains an administrative operation. Backend tokens and hosted tool charges are separate; hosted web-search fees are not forecast by this adapter. Custom model or tier rates still require price-book coverage.

Live owns continuous audio and always produces speech. Realtime-only temperature, VAD tuning, text-only output, and automatic reconnect/fallback settings are not Live configuration options. Changing delegation mode, voice, or model takes effect in a new conversation. Pipecat seeds at most 128 text history entries; image input travels to the backend instead of that startup history. A delegation timeout does not undo tool actions that already completed.

Verification and references

tests/test_openai_live_service.py exercises the wire configuration, startup, tool ordering, private-capture boundaries, prompt refresh, typed/image input, client callback sharing, and usage attribution without calling OpenAI. The existing payment, multimodal, subagent, recording, and Realtime tests remain part of the orchestrator regression suite.

e2e/openai-live-config.spec.ts exercises all 22 voice choices, the independent backend model dropdown, nested text settings, mode switching, and database read-back after autosave. It uses the real editor and a mocked provider catalog, creates and deletes its own agent, and does not open a provider session.

The agent 4343 regression on 2026-09-11 reproduced a roughly 26,000-character business prompt being appended to the voice engine after each assistant turn. The provider heard the caller but repeatedly restarted its greeting. Tests now exercise the real InstructionRefresher with a long opening script in both delegation modes, require unchanged refreshes to send no commands, and verify that changed rules update only the backend. A canceled send remains retryable.

See also