ai-orchestrator

Frame-handled callbacks registered after stage tool wiring inherit that stage's tool_logic. provider_tool_registration applies it after installing the local callback, preserving bound values, captures and availability checks. Without this final pass, end_session.params could disappear from the model's schema while the hangup handler still received unbound arguments.

Status: Current | Last reviewed: 2026-10-15

Customer Authentication and both A2UI tool paths distinguish an already connected customer page from an actual SMS send. They reuse the former without announcing another message. See the shared-page flow.

Responses API stages support opt-in deduplicate_response_messages, preserving output-message boundaries before response guards and collapsing adjacent exact duplicate messages. A suppress guard drops a matching sentence and the rest of that turn without a spoken replacement. This supports silent waiting turns in cascaded speech; other Responses consumers keep their existing behavior.

A bot-playback stop while an LLM response is still open does not finalize the assistant transcript. The publisher retains the same turn identity across that gap and completes it after response generation and playback finish. This prevents a partial payment sentence from becoming a separate transcript row.

ConversationAPIPublisher reserves a caller sequence and timestamp at UserStartedSpeakingFrame. _publish_transcript applies that anchor to all caller publication paths, retaining it across late revisions instead of consuming it in the aggregator-only path. Explicit provider timing wins, and unreserved captions retain their normal fallback. The most recent 1,024 caller anchors are retained to bound long-session memory. Preview sorts by audible start, so delayed S2S caller captions appear before the agent reply.

GPT Live now mirrors caller interims as well as finals downstream as normalized frames. The adapter assigns one UUID per provider-grouped caller utterance and retains its start across every revision. Provider start_ms is mapped to epoch time using the received session.started boundary, marked estimated with audible_start_basis=provider_session_offset; absent provider timing falls back to the first caption receipt. The epoch resets on a new provider session. This identity bypasses the publisher's mutable speech bucket: an agent backchannel or the next caller start cannot reassign a late story caption. Only finals advance backend input revision and relay consent evidence.

ElevenLabs v4's single-stream dialogue adapter releases a short sentence tail after 120 ms without further text following sentence-ending punctuation. It keeps the provider socket open and cancels the timer on more text, interruption or completion. Do not flush every gap between Live's word fragments: that produces unnatural isolated-word delivery. This fixes the buffer wait reproduced near the start of agent 4451's greeting. See v4 streaming.

GPT Live's custom Responses backend forwards delegation_parallel_tool_calls to the provider request. A false value keeps dependent tool calls sequential, including opening supervisor chat before dispatching a review.

The gRPC transport owns each native write independently of its producing task. Cancellation after dispatch does not release the serialization lock until that write finishes; cancellation before dispatch never sends queued stale audio. This prevents overlapping native operations and missing evaluation events after an interruption. Inline Python tool bindings use a forwarding proxy, preserving callbacks and metadata across input-repair retries without copying another proxy's uninitialized state.

The end_session function schema defers to the agent's configured ending policy, including any requirement for caller acceptance of a transfer. The default ending guidance remains available; selecting custom guidance must not be contradicted by the tool's generic description. ElevenLabs Text to Dialogue errors include the provider's bounded explanatory message as well as its code; this improves diagnosis without suppressing protocol failures.

The Pipecat-based real-time pipeline runtime. Every voice or chat session is a pipeline this service builds from agent_versions.config and runs to completion. Form Filling always composes authored custom instructions with its generated field definitions, required markers, and extraction rules before calling the selected structured-output model.

The Allianz annuity demo exercises optional parallel Copilot workers, approved phrase responses, final-transcript form extraction and configured Deepgram endpointing. These paths preserve speaker filters and schema validation. gRPC disconnection does not cancel its own inbound or watchdog task. Message consent distinguishes a send refusal from unrelated negative assumptions such as “no additional withdrawals.”

The GREF Connect journey also checks Copilot startup while Salesforce opens the Case. Redis retains the latest context.ready/context.updated snapshot until the listener registers (30-second expiry, 128 pending conversations maximum). Replay preserves order and disconnect clears pending state; user actions are not buffered. SpeechMetricsProcessor keeps its measurement accumulator separate from Pipecat's native _metrics object so pipeline setup and cleanup remain valid.

Procedure assessment retains bounded, source-labeled evidence and requires valid references before completing eligible judged steps. Tool execution can enforce stored payment receipts and session-scoped idempotency before real or mock dispatch; repaired inputs re-enter those guards. See procedure supervision and execution contracts.

Source

apps/ai-orchestrator

Runtime

Aspect	Value
Language	Python 3.13
Framework	Pipecat 1.9 plus local extensions under apps/ai-orchestrator/lib/pipecat/
Entrypoint	main.py
Wire protocol	gRPC (proto in proto/orchestrator.proto)
Ports	50051 gRPC, 8080 HTTP metrics/health/debug gateway in cluster; Vanguard local uses AI_ORCHESTRATOR_HTTP_PORT=8081
Dev runner	python dev_runner.py (Vanguard)

What it does

OpenAI GPT Live runs as a separate S2S service with either managed Responses delegation or a client LLM worker. It uses the universal tool/capability pipeline, explicit application-message delivery, backend vision, and separate voice/backend usage attribution. See GPT Live.

GPT Live's authored per-tool progress announcements default off in both delegation modes. Stage setting tool_progress_messages_enabled enables them through the private prompt voice; gates, tool results, and required capture scripts remain active.

1. Receives a StartSession gRPC call from a transport adapter (genesys-connector-go, ari-controller, surface-app via conversation-api, etc.).
2. Resolves the agent + version with resolve_agent_version().
3. Hands the resolved agent_versions.config JSONB to the universal pipeline builder.
4. Constructs the Pipecat pipeline (STT, LLM, TTS, tools, transports).
5. Streams audio/text frames in and out.
6. Emits frames to conversation-api for transcript and analytics.
7. For ai-manager test calls with X-Whisker-Mode: live, attaches a Whisker observer on a private loopback port, publishes whisker:session:{conversation_id} metadata to Redis, and serves the signed /whisker-ws/{conversationId} gateway used by the central debugger.

The full builder flow is in universal-pipeline-builder.md.

Inbound telephony adapters pass ANI in gRPC SessionConfig.metadata.ani. Pre-conversation lifecycle workflows read that value directly as their ani input. Conversation persistence, tracing, and MCP session context use the same caller-identity resolver, whose precedence is caller display name, explicit caller id, ANI, extension, then from; omitting ANI here makes the workflow lookup succeed while the stored conversation incorrectly has no caller phone. The synchronous lifecycle outputs are merged before greeting harvesting. Lifecycle trigger inputs carry _lifecycle_hook=on_start or on_complete, persisted with workflow execution history. Thinking uses that marker to show context at both boundaries, even when the same workflow handles both, and continues reading end-workflow results after the call finishes. See Thinking history. Greeting messages evaluate Agent Logic {{#when ...}} blocks and interpolate variables using the same initial precedence as the main prompt: system and tenant values, stage template defaults, call context, lifecycle/session metadata, then mutable session variables. This ensures a lifecycle result such as FirstName controls the first greeting instead of exposing raw conditional directives to a realtime provider.

The harvested greeting (runtime_context["message"]) is always the literal authored words with role="system" — it is the text to speak, not a loose instruction like "please greet the caller". OpenAI Realtime's connect handler (grpc_provider._on_client_connected) must therefore deliver it as an explicit Begin the conversation by saying exactly: "…" directive; passing a system-role greeting through raw made the provider treat the configured greeting as vague context and improvise ("I'm ready to continue from here") instead of speaking it, on every fresh call. When greeting is enabled but the message is empty, an empty append primes the session and the model greets from its system prompt; when greeting is disabled the session is primed with no greeting text.

Pipecat 1.4 runtime note: provider adapters create a PipelineWorker and run it via a WorkerRunner (add_workers(...); run()). Do not add new PipelineTask or PipelineRunner imports. Runtime context is passed as PipelineWorker(..., app_resources=runtime_context) so ADK/MCP/tool handlers can read shared session resources through pipeline_worker.app_resources.

Turn-taking, interruption, idle detection, and voice isolation are resolved through the node-owned config.turn_control block on conversational stages. The resolver keeps older agents compatible by reading legacy config.interruption, config.user_idle, settings.audio.enable_vad, and provider-native turn detection fields. auto keeps provider-native behavior for realtime/S2S services and Pipecat local behavior for cascaded speech/chat. Krisp VIVA voice isolation is enabled implicitly for every voice agent and can be explicitly set to Off in Agent Studio; VAD, end-of-turn, and interruption prediction remain independent opt-ins. The gRPC and LiveKit transports filter normalized pipeline-rate PCM before VAD/STT, selecting the packaged telephony model at ≤16 kHz and Pro model above 16 kHz. The supplied IP v1.1 model is English-only. The retry budget resets when the floor comes back to the caller, not when the caller finally speaks: user speech and final user transcripts reset it, and so does the end of a genuine agent turn (mark_bot_speaking(False) on a change), because a long tool-backed turn or an injected message used to leave a caller charged for prompts they never ignored. Three exclusions keep escalation honest: the idle prompt's own speech does not refund the retry it spent (_idle_prompt_in_flight), a duplicate stop signal — the output frame and the publisher's relay both arrive for one turn — is not a second handover, and an A2UI render counts as engagement (mark_visual_engagement) rather than a suspension. An idle prompt is never delivered over live speech: _handle_idle_callback re-tests _bot_is_speaking/_user_is_speaking immediately before sending (a bot turn can start in the gap left by _check_and_trigger_idle) and reschedules without spending a retry. The idle window is measured against audio playout, not against the speech frames: BotStoppedSpeakingFrame reports that the pipeline finished sending, and an S2S provider streams a whole turn faster than real time (Nova Sonic delivered a ~15 s answer in 2.1 s), so nothing but the audio's own duration knows when the caller stops listening. The publisher relays every downstream TTSAudioRawFrame/OutputAudioRawFrame duration to mark_bot_audio, which keeps a playout deadline, and both idle gates then wait a full timeout measured from that deadline (_idle_wait_remaining) rather than from the stop frame — deferring only while audio remains is not enough, because a 15 s utterance that took 15 s to play out has nothing left at the instant the timer expires, so the prompt still landed on the last syllable of the question. A barge-in clears the estimate (the transport discards the queued audio), a stale deadline stops mattering once its window has elapsed so an absent caller still escalates, and the final farewell drains the remaining playout before EndFrame so the goodbye is not cut off mid-word. And one prompt must produce one provider turn: when the adapter delivers the instruction itself (IdlePromptAdapter.delivers_instruction_natively, true whenever the provider service exposes send_idle_prompt) the handler skips the system LLMMessagesAppendFrame — Nova Sonic converts an injected system message into its own interactive turn, so doing both fired two turns milliseconds apart and the second interrupted the first, which presents as the agent talking over itself. A failed native send appends the instruction before fallback_llm_append, since that fallback runs the LLM on context alone. A finished caller turn holds the timer for a bounded reply window, because a speech-to-speech provider need not emit turn frames at all: Nova Sonic logs that it emits no UserStartedSpeakingFrame/UserStoppedSpeakingFrame and synthesizes a pair ~3 ms apart only when its own response begins, so the caller's final transcript is the earliest signal the floor changed hands. mark_user_turn_end (called on a final TranscriptionFrame, and relayed by the publisher for providers whose transcripts do not pass through the processor) starts a 20-second reply-owed hold that _idle_wait_remaining folds in beside the playout deadline; the agent's first audio releases it (mark_bot_speaking). Without it the caller heard an idle prompt in the gap between their own sentence and the reply to it — the sequence the user reports as “an idle prompt after I spoke and then the actual reply”. The hold is bounded rather than open-ended so a reply that never arrives still escalates. Agent speech lifecycle is forwarded from the output-side ConversationAPIPublisher to the upstream idle processor before S2S speaking-event suppression, pausing the timer for the whole response; both EndFrame and CancelFrame cancel it so a disconnected call cannot emit retries. Known external work composes named idle suspensions: Customer Authentication pauses after an auth tool returns pending and resumes on its trusted completion injection, while secure keypad capture pauses for the whole payment controller window. Any tool call in flight suspends idle prompting for the whole concurrent batch — the publisher counts started and finished calls and only suspends/resumes on the empty↔non-empty transitions, so two overlapping calls suspend once and resume once rather than the first result re-arming the timer while the second is still running. The suspension carries a TOOL_CALL_SUSPENSION_CAP_SECONDS (90 s) watchdog that force-releases it with a WARNING: a lost resume — a tool whose result frame never arrives, a provider that abandons the call — would otherwise disable idle detection for the rest of the session, which is the one failure mode a suspension must not have. These suspensions reset the retry budget, so neither a browser ceremony, deliberate DTMF entry, nor a slow tool produces “Are you still there?” prompts. The final idle timeout waits for the farewell response before emitting EndFrame downstream for graceful transport teardown. Idle speech is supported through the normal LLM path for cascaded/chat services and native live-session controls for OpenAI Realtime, AWS Nova Sonic, Google ADK Live, Google CES, ElevenLabs Conversational AI, and Deepgram Voice Agent. Maven Voice, NVIDIA PersonaPlex/Moshi, and Salesforce Agentforce Voice have audio-only client transports; the runtime force-disables idle prompting for those services instead of silently advancing the retry counter and ending the call.

Payments on an S2S stage use an in-call secure handoff rather than disconnecting the realtime provider. PaymentModeController replaces caller audio with equal-length silence before the provider and consumes DTMF; PaymentS2SOutputGate suppresses provider speech travelling toward the caller; a private prompt-only TTS derived from the same provider integration speaks the fixed collection script. The builder resolves that private lane from the Phase-1 _payments_policy:<stage-id> stash because ServiceRegistry consumes and removes builtin_tool_configs in place before chain assembly; re-reading the sanitized stage config makes a valid S2S payment look unwired and returns capture_unavailable. The masked outcome then resumes the existing S2S session. This is supported only on a linear pipeline: parallel/swimlane payment graphs are rejected until the secure gate can run before fan-out. See Payments.

Krisp Voice Translation is a separate provider=krisp, type=s2s_translation, service="Voice Translation" pipeline service. Its adapter resolves an encrypted Krisp integration key or KRISP_VT_API_KEY, mints a short-lived session token per flow direction, normalizes audio to mono PCM16 at 16 kHz, and submits exact 20 ms frames. SDK thread callbacks cross an asyncio queue before becoming translated audio, source interim/final transcription, or final translated text frames. Startup retries once after token/transient failures; a runtime failure is fatal to that translation direction so SCV can restore its stable enhanced non-translation path. Vocabulary, dictionary, male/female voice, per-direction cloud BVC, and individual output controls are all authored on the translation stage.

The contracted SDK/model archives are Git LFS assets under krisp/. Docker and Vanguard both run apps/ai-orchestrator/prepare_krisp_runtime.py, which installs the current Linux architecture's SDK 1.11 stable-ABI wheel and extracts models/notices into the runtime. The platform injects one process-global KRISP_VIVA_API_KEY; the SDK needs HTTPS egress to sdkapi.krisp.ai and analytics.krisp.ai. See environment variables and Pipecat's Krisp VIVA guide.

Every Krisp session Pipecat 1.4 creates is called with an argument SDK 1.11 rejects, so all three are wrapped at the session boundary in turn_control.py. Pipecat 1.4.0 and krisp_audio 1.11.0 disagree on the frame type in two different ways, and the float pair is far more damaging than the int16 one:

Session	Pipecat passes	SDK 1.11 demands	Adapter
NcInt16 (voice isolation)	read-only np.frombuffer(bytes(...)) view	writable int16 ndarray	_WritableKrispSessionAdapter via _adapt_krisp_filter_for_sdk_1_11
IpFloat (interruption prediction)	frame.tolist() — a Python list	ndarray[float32, order='C'] + voiceDetected	_Float32KrispSessionAdapter via _adapt_krisp_ip_strategy_for_sdk_1_11
TtFloat (end-of-turn prediction)	frame.tolist()	same, plus botSpeaking	_Float32KrispSessionAdapter via _adapt_krisp_turn_analyzer_for_sdk_1_11

An unadapted float session raises TypeError on every frame, and those two sessions decide user-turn start and stop — so a krisp_viva interruption/turn-detection agent stops noticing that anybody is speaking. What the caller experiences is not an error: the agent "ignores" them, and because the pipeline sees an idle call the idle processor fires "are you still there?" prompts on a call where the caller has been talking continuously, then terminates on max_prompts ([UserIdleSpeechAware] Pushed EndFrame downstream to terminate). The only trace is one ErrorFrame per 10 ms audio frame surfacing as LLMUserAggregator exception / PipelineWorker: Something went wrong, which reads as a pipeline fault rather than a two-argument type mismatch. Both float wrappers shadow the factory (_ensure_session / _create_tt_session) rather than the attribute, because both sessions are created lazily on the first frame at the negotiated sample rate; the IP wrapper is idempotent since _ensure_session runs per frame, and KrispVivaTurn's __init__ preload session is deliberately left unwrapped (it only warms the model and is never processed through). Regressions in tests/test_turn_control.py — including one that drives the real SDK and asserts the unadapted call still raises, so the fix cannot pass while the runtime floods ErrorFrames.

Nova Sonic tool results are deferred until AWS has emitted the inbound TOOL contentEnd; sending the callback result earlier causes AWS to reject the response with Tool Response parsing error. That same error is also raised when the toolResult content is not valid JSON: the base AWSBedrockLLMService._send_tool_result json.dumps() only a dict result and forwards any str/list/None raw, so a tool returning a plain-prose string (e.g. enable_multimodal's guidance result) sent non-JSON content, Nova tore down the S2S stream, and the agent went silent after that tool call. SafeAWSNovaSonicLLMService._send_tool_result (in aws_nova_sonic_services.py) now wraps any non-dict result as {"result": ...} before delegating, so Nova always receives parseable JSON. The same wrapper handles live LLMMessagesAppendFrame injection: user actions become interactive user turns; trusted auth/payment facts remain SYSTEM context and are followed by a fact-free interactive wake-up turn, so Nova responds without downgrading a server-attested grant into a caller assertion. AI Manager's Preview runs the ordinary multimodal handshake with demo_notification delivery — the caller is asked, the share link is minted, and only the carrier is simulated (the message lands in the test phone's tray). It no longer sends X-Suppress-Multimodal-Sms; suppression skipped the consent question and stopped a post-verification card from minting its own second link. See a2ui-rendering.md.

Salesforce Agentforce Voice is an S2S runtime path. The browser stays on the normal TrueServe media path; ai-orchestrator reads the selected Salesforce integration, mints a Salesforce OAuth token, resolves Agentforce connection info from connect/conversation-runtime-proxy when available, and otherwise follows the Mobile SDK shape: SFAP calls use https://api.salesforce.com, the tenant header is core/prod/<OrganizationId>, and any configured Runtime Base URL is retained as apiEndpoint metadata rather than used as the call host. The runtime starts an Agentforce session through /einstein/ai-agent/v1/agents/{agentId}/sessions, using the TrueServe conversation id as externalSessionKey when available, joins Salesforce's LiveKit room through /einstein/ai-agent/v1.1/realtime/sessions/{sessionId}/join, and relays Pipecat audio/transcript frames. Salesforce bearer tokens and LiveKit room tokens remain server-side and must never be emitted in logs or client payloads.

If an Agentforce stage enables end_session_analysis, ai-orchestrator adds a local RemoteEndSessionAnalyzerProcessor after the Agentforce service. It buffers finalized transcript turns, calls the configured normal LLM provider/service/model, and only ends the TrueServe session when structured output returns should_end=true above the configured confidence threshold. The structured payload mirrors Google CES endSession.metadata; the shared flattener converts reason, session_escalated, and params into the standard _session_end_params map used by transports and lifecycle workflows.

Maven AGI is supported as both a chat-agent LLM stage and a third-party voice-agent S2S stage. The runtime reads the selected mavenagi integration profile, installs the public mavenagi Python package from the locked dependency set, and calls AsyncMavenAGI.conversation.ask_stream for chat turns. Maven text chunks become LLMTextFrames; rich bot payloads are mapped to existing A2UI render frames so conversation-api continues to publish UI_RENDER. For voice, ai-orchestrator creates a Maven websocket session token, bridges Pipecat InputAudioRawFrame chunks to /v1/voice/conversations, and converts Maven audio/transcript events back into the pipeline.

Assistant Agent stages are supported as preview chat-agent stages separate from ordinary LLMs. ai-orchestrator registers type="assistant_agent" for Google Antigravity, OpenAI Codex, and Anthropic Claude Code, then adapts them onto Pipecat's LLM frame path so existing text, reasoning, tool, and event surfaces keep working. Antigravity streams turns to antigravity-runner; Codex and Claude Code stream turns to workspace-agent-runner. Local file/command tools never run inside ai-orchestrator. TrueServe tools remain registered through the normal register_function path; when a runner emits a host_tool_call, ai-orchestrator executes the existing callback and POSTs the result back to the runner.

Before Assistant Agent tool schemas are resolved, ai-orchestrator expands the node's skill_ids and inline_skills. Library skills are loaded from agent_skills, inline skills are read from the stage config, and both contribute tool_refs to the same TrueServe tool resolution path used by the normal ToolSelector. Skill-local inline_tools are merged into the stage's effective inline_tools before the normal inline Python/manual API resolvers run. The compiled skill payload is sent to the selected runner, which materializes SKILL.md packages. builtin:end_session remains lifecycle-only and is not treated as a business skill. Native CLI-side MCP config for Codex/Claude is deferred until a session-scoped TrueServe MCP gateway is available; Antigravity MCP-backed skills continue through agent-host / the existing host bridge.

Manual API tools (type_id='openapi', config.subtype='manual') run in the voice/S2S runtime through inline_api_tools.py, the same executor async-agent uses for text. Their config.auth is resolved by _resolve_inline_api_auth: inline tool_secrets for saved rows, otherwise the referenced integration's stored credential. Salesforce integrations store no bearer token — they hold instance_url/client_id/client_secret (or JWT material) and a short-lived token is minted per call. When the referenced integration's type_id is salesforce, the resolver mints a bearer via SalesforceAuthProvider.get_token() (client-credentials or JWT, cached per integration) and sets Authorization: Bearer …, mirroring ai-manager's resolveIntegrationAuth and async-agent's _resolve_api_auth. Without this a Salesforce-backed API tool on a voice agent sent no Authorization header and Salesforce rejected the call as requiring an API token.

Knowledge tools are MCP-backed search tools resolved per tenant. The session tenant's Knowledge settings (tenant_settings key knowledge, managed in ai-manager under Settings → Tenant Settings → Knowledge) supply the base URL and API token via the shared libs/knowledge resolver (resolve_knowledge_connection in tool_config_resolver.py, short-TTL cached so an admin's edit lands without a restart). resolve_mcp_tool_configs takes a tenant_id, reads tool_configs.type_id='knowledge', derives the MCP server URL as {tenant base_url}/mcp/{source_slug} (a stale stored server_url is ignored), injects the Bearer token server-side, and exposes only the fixed search method. Inline Knowledge tools are resolved from config.inline_tools[] by resolve_inline_knowledge_tool_configs(raw_config, tenant_id=…): the config carries the slug + tenant, and the callback re-resolves the live connection at call time. When the tenant has Knowledge disabled or incompletely configured (missing base URL or token), shared Knowledge rows and inline Knowledge definitions are skipped even if an agent version still references them. There is no env fallback.

Customer Authentication stages (a stage carrying builtin:customer_authentication in tools[]) register three LLM tools — authenticate_customer, verify_otp_code, get_customer_auth_status — via customer_auth_tool_helper.py and append the CustomerAuthToolHandler observer after the LLM service to server-attest KBA tool results. All policy (idempotence, assurance composition, lockout) lives in conversation-api; the orchestrator only relays with the surface-internal HMAC stamp. See customer-authentication.md.

One runtime-only rule does live here: consent to be messaged. Both authenticate_customer (a bearer link or a code) and the A2UI multimodal handshake (a share link) put something on the caller's phone, so they share one consent record per conversation (MultimodalState.message_consent in a2ui_tool.py). Whichever capability needs it first asks — authenticate_customer takes a consent argument and returns consent_required before any HTTP call, so a refusal mints no challenge — and the other sends silently. A completed texted verification (passkey, otp_sms, otp_email) counts as consent, so a visual card afterwards sends its own second link and the tool result tells the model to explain it rather than ask again. Text channels auto-grant because the caller is already using the destination surface; emulator test mode still asks and only simulates the carrier through demo_notification. Missing caller-turn evidence allows one consent_required retry, then fails closed and terminal instead of looping. Every read of conversation-api's auth state fails open to "unknown" = not granted; the authority never moves.

ConversationAPIPublisher imports the generated client stored inside ai-orchestrator. Do not add apps/conversation-api to its Python import path: each service generates the same wire schema against its own pinned gRPC runtime, and importing the other service's newer client can prevent the publisher from starting. That failure removes both live transcript delivery and the local caller-turn evidence used by the consent gate.

Shared Python tools (tool_configs.type_id='python_sandboxed' or legacy python) are resolved from UUID refs and adapted into the same callback path as agent-local inline Python definitions. In Vanguard/dev mode, MCP URLs pointing at async-agent:8000 or async-agent:8080 are rewritten to localhost so saved tool rows copied from cluster configs still work from the devcontainer.

Pipeline construction

File	Role
lib/pipecat/pipeline/builders/universal_builder.py	Top-level builder; dispatches on subtype. The only entry point — the extracted phases below are imported back into it
lib/pipecat/pipeline/builders/universal/	Six phases lifted out of that file, move-only: normalization, config_translation, payments, post_llm, tool_wiring (tool phases 1–2), llm_context (phase 3). See the module table
lib/pipecat/pipeline/builders/translation_builder.py	Bilingual / interpretation pipelines
lib/pipecat/pipeline/pipeline_config_provider.py	Reads agent_versions, resolves overrides
lib/pipecat/pipeline/service_registry.py	Maps provider/service strings to Python classes

LLM stages with structured_output_enabled are routed to SuggestionLLMProcessor for the simple legacy suggestion types (response_suggestion, knowledge_assist, article, faq, entity_extraction). The builder uses the authored prompt in this order: explicit system_prompt override, template.inline, then system messages. Legacy knowledge-source/RAG prefetch config is not part of this path; knowledge lookup should be exposed as a normal tool.

Transcript Enhancement composes authored custom instructions with its generated enhancement-mode and domain-vocabulary rules; an override does not discard the glossary contract.

Smart Reply is authored as a dedicated processor service (service="Smart Reply Processor", runtime service SmartReplyProcessor). It uses the same suggestion processor implementation internally, but its config is flat and processor-specific. The manager UI shows three sections: Assessment, Knowledge Source, and Suggestions. The processor runs a multi-step workflow:

1. A planner LLM inspects the configured recent utterance window and decides whether the latest customer turn contains an actionable request or question.
2. If actionable, the planner emits query, a canonical lookup question, plus optional structured parameters. If not actionable, the workflow stops without emitting a suggestion. canonical_query remains an output alias for older consumers.
3. The configured single Knowledge Source tool is invoked through the same register_function callbacks used by normal Pipecat LLM services. The default argument mapping is {"query": "{{query}}"} and can map additional planner parameters.
4. A Suggestions LLM receives the recent conversation, planner output, tool result, and normalized citations. Its system prompt can merge {{knowledge}}, the raw normalized tool result. It emits responses[] with text, confidence, and citation_ids.

Smart Reply citations are carried through existing suggestion metadata: SuggestionEntry.metadata.entries["responses"] and ["citations"] are JSON encoded, while response_text remains populated with the first response for older clients.

Dependencies

Service	Purpose
Postgres	agent_versions, integrations, session state
Redis	Pipeline-level caches (CES session map, etc.)
agent-host (:8003)	MCP tool execution
conversation-api	Transcript + event sink
async-agent	Workflow side-effects from voice tools
Provider APIs	OpenAI, Anthropic, Deepgram, ElevenLabs, Maven AGI, etc. (per agent config)
Salesforce Agentforce + LiveKit	Agentforce Voice session start through the Agent API, realtime room join through the SFAP host, and server-side audio bridge
Maven AGI	Chat streaming, voice websocket token minting, and server-side audio bridge
Krisp Voice Translation	Short-lived directional sessions for translated audio and source/target text; optional server-side BVC
Knowledge Sources API	MCP search endpoint for feature-flagged Knowledge tools
antigravity-runner (:8011)	Sandboxed Google Antigravity local workspace tools and streaming chat bridge
workspace-agent-runner (:8012)	Sandboxed Codex CLI and Claude Code Assistant Agent bridge

Multi-agent and event projections

Subagents are still opt-in per agent config. subagents.execution="sequential" uses registered LLM callbacks, "parallel" uses SubagentToolHandler, and "session_switch" remains the speech-to-speech transfer model. New in the Pipecat 1.3 upgrade, subagents.execution="workers" installs PipecatWorkerSubagentAdapter and registers the subagent tools against the same in-process WorkerRunner. Distributed worker processes are not part of this runtime path yet.

Turn events are normalized by TurnEventBridge, which attaches to Pipecat's TurnTrackingObserver when it exists and publishes the same conversation-api TURN_STARTED / TURN_ENDED events as before. The EventSanitizer still owns false-start, deferred-turn-ended, and interruption filtering.

AG-UI is an event projection, not a replacement for A2UI. When an agent enables settings.agui.enabled, sanitized events are projected to agui:{conversation_id} and conversation-api streams them at GET /api/v1/conversations/{conversation_id}/events?format=agui. A2UI render payloads become AG-UI state snapshots/deltas under /a2ui/surfaces/...; the TrueServe web client continues to consume the normal conversation SSE/A2UI path.

RTVI is limited to a local probe route in this upgrade. With AI_ORCHESTRATOR_RTVI_PROTOTYPE=1, GET /experimental/rtvi on the ai-orchestrator HTTP port returns the installed Pipecat RTVIProcessor, RTVIObserver, and UIWorker signatures. It does not start a production client session.

Whisker live debugger

Live Whisker debugging is enabled only for internal ai-manager test sessions. ai-manager sends X-Whisker-Mode: live when an agent's team-wide Debug toggle is on. Each session gets its own loopback backend port so overlapping users and agents do not contend for one fixed Whisker port.

The runtime writes a Redis hash at whisker:session:{conversation_id} with a 24-hour TTL:

Field	Purpose
conversation_id	Conversation/session id used by ai-manager and the gateway route
agent_id	Base or versioned agent id for display/joining
tenant_id	Tenant boundary checked before proxying
mode	live
status	listening while the session is active
backend_ws_url	Informational ws://127.0.0.1:<port> loopback address of the in-pod observer (the in-pod relay reads it; the gateway does not)
created_at	Milliseconds since epoch

The public browser connection is /whisker-ws/{conversationId}?token=... on ai-orchestrator's HTTP port. The token is minted by ai-manager with WHISKER_GATEWAY_TOKEN_SECRET, scoped to tenant/user/conversation, and expires quickly (15-minute TTL, TOKEN_TTL_SECONDS in whisker-debug.ts). An expired token is rejected before the WebSocket upgrade with 403 (_verify_gateway_token returns None), which the browser surfaces only as a bare WebSocket … failed. Because the token is short-lived, the ai-manager debugger UI (app/debug/whisker/page.tsx) re-mints it on every 3-second sessions poll and always connects with the freshest token — the session-merge prefers the newly-minted ws_url over the first-seen one, and the viewer reads the live token from a ref at connect time (never a value frozen at mount) so a session viewed for longer than the TTL, or a socket that drops and reconnects, does not 403 on a stale token. An open socket is never re-checked, so the token only matters at (re)connect. The gateway validates the token and verifies the Redis tenant metadata (rejecting non-listening sessions with 409).

Frames are relayed through Redis pub/sub, not a pod-local socket. An in-pod relay task connects to the observer's loopback server as its single allowed client and republishes every serialized message verbatim to Redis channel whisker:stream:{conversation_id}; the first message (the pipeline graph) is cached at whisker:initial:{conversation_id} for viewers that join mid-session. The gateway subscribes to that channel and streams to the browser. This is required for correctness at scale: ai-orchestrator runs multiple replicas (KEDA / hpa.minReplicas: 3), and the browser's /whisker-ws request round-robins across pods via the ClusterIP Service — a pod-local backend_ws_url was only reachable from the one pod running the call, so any request hitting a different pod showed zero frames for every agent. Redis pub/sub decouples the gateway from pod identity (no NetworkPolicy/port changes needed) and naturally fans out to multiple viewers. The browser is receive-only, so the relay is one-directional. The Redis entries (whisker:session:* + whisker:initial:*) are deleted on live-session teardown.

Diagnosing a stuck debugger — the gateway logs every outcome, and its Redis reads are timeout-bounded. whisker_gateway_handler (whisker.py) logs one line per connection attempt ([WhiskerGateway] connect attempt for <conv>) and one per outcome — 403 invalid/expired token, 404 no session in Redis, 404 tenant mismatch, 409 not live (mode=…, status=…), or 101 upgrading + subscribing stream. Grep the orchestrator pods for WhiskerGateway and <conversation_id> to see exactly which branch fired; note the debugged call runs on a gRPC stream ([Whisker] live-mode observer attached … → ws://127.0.0.1:<port> in that pod's log) while the browser's HTTP /whisker-ws request can land on any replica, so pull logs across all pods, not just the one that ran the call. A browser WebSocket stuck at (pending) with no gateway log line at all means the request never reached a handler — look at the Gateway/HTTPRoute (/whisker-ws → ai-orchestrator:8080, must precede the catch-all /), the ALB target group for the orchestrator's HTTP port, and any WAF rule that could hold the long ?token= query. On KMS/EKS the decisive check is kubectl -n trueserve get targetgroupbindings.elbv2.k8s.aws -o custom-columns=NAME:.metadata.name,SVC:.spec.serviceRef.name,PORT:.spec.serviceRef.port — every public-l7 backend must appear. ai-orchestrator's 8080 is a secondary additionalPort on a Service whose primary port is gRPC (50051); the AWS Load Balancer Controller reads Service.spec.ports[].appProtocol to build the target group and silently skips a secondary HTTP port on a gRPC-primary Service that lacks it — the HTTPRoute still reports ResolvedRefs: True, but no TargetGroupBinding exists, so the ALB has no backend and every /whisker-ws upgrade hangs at (pending) (curl to the route times out with no status). The fix is appProtocol: http on the http-metrics/8080 port — set it in charts/trueserve-stack/values.yaml under ai-orchestrator.service.additionalPorts, which is the block that actually renders on KMS/Kind: the parent stack declares its own ai-orchestrator.service.additionalPorts that REPLACES the subchart's list wholesale (Helm parent values win, lists don't merge), so setting it only in charts/ai-orchestrator/values.yaml is silently shadowed for every real deploy (verify through helm template charts/trueserve-stack -f <env>, never the standalone subchart). Both are emitted by base.service in charts/base-app/templates/_helpers.tpl. On sync the controller builds the TG (default health check GET / → health_handler → 200, no TargetGroupConfiguration needed). The gateway's own Redis clients set socket_connect_timeout/socket_timeout (WHISKER_REDIS_TIMEOUT_SECONDS), so a Redis endpoint that never completes its handshake (e.g. a plaintext client against a TLS-required ElastiCache) now fails the read fast and returns 500 with a logged reason, instead of hanging the upgrade forever as a silent (pending).

Local call readiness

Local call readiness requires both the orchestrator HTTP endpoint and Media Proxy. Overmind's environment loader defaults AI_ORCHESTRATOR_HTTP_PORT to 8099 and ORCHESTRATOR_HTTP_URL to that localhost port, preserving explicit overrides. Without the URL, AI Manager falls back to the cluster hostname and disables Preview even when the local runtime is healthy. After changing these defaults, restart AI Manager so its readiness requests use the corrected address.

If Overmind reports the orchestrator running but its ports are closed, inspect the child process logs: the development watcher can survive a main.py import failure. Both A2UI dispatch paths require normalize_a2ui_operation_envelopes; retain its shared definition when merging. Sync dependencies with uv sync --frozen --inexact in apps/ai-orchestrator (preserves locally installed Krisp wheels), then restart the orchestrator and Media Proxy. Verify an agent's /preflight result, not only process status. The September 28 repair restores this helper and aligns the local Pipecat installation with the committed 1.9 lockfile.

Vanguard

Started by scripts/start-ai-orchestrator.sh under Overmind.

overmind restart ai-orchestrator
overmind connect ai-orchestrator

Required env (.env.local):

• POSTGRES_* pointing at the local docker postgres.
• REDIS_URL=redis://localhost:6379/0.
• MASTER_ENCRYPTION_KEY matches ai-manager's.
• ASYNC_AGENT_URL=localhost:8000.
• CONVERSATION_API_GRPC_ENDPOINT=localhost:1234.
• KRISP_VT_API_KEY for Krisp Voice Translation; local setup falls back to the packaged Krisp key file without exposing it to the browser.
• Knowledge tools need no env vars — enable Knowledge for the test tenant in ai-manager under Settings → Tenant Settings → Knowledge.
• WHISKER_GATEWAY_TOKEN_SECRET matches ai-manager's when using the live Whisker debugger.

See host-orchestrator-env memory note for the full list.

Agentforce Voice uses the tenant's saved Salesforce integration rather than environment-scoped credentials. JWT bearer auth is used when username, org_url, and private_key are present; otherwise the runtime uses Salesforce client credentials with instance_url, client_id, and client_secret. If Salesforce does not expose connect/conversation-runtime-proxy for the org, ai-orchestrator queries Organization.Id, converts a 15-character Id to the 18-character form, and sends x-sfdc-tenant-id: core/prod/<OrganizationId> while using the SDK default SFAP host https://api.salesforce.com. Salesforce setup may label agentforce_sfap_url as Runtime Base URL; that value maps to the SDK's apiEndpoint field when it is a bot-runtime-api...sfdcfc.net URL. The ForceConfig Endpoint should match the integration's Salesforce org URL. These values affect server-side session setup only; they do not bypass the Salesforce session or change the browser audio path.

Maven AGI normally uses the selected mavenagi integration profile. For local smoke tests only, the wrapper can fall back to MAVENAGI_ORGANIZATION_ID, MAVENAGI_APP_ID, MAVENAGI_APP_SECRET, and optional MAVENAGI_BASE_URL; stage config still needs an agent_id. Voice audio defaults to PCM_16000, and telephony pcmu/8 kHz pipelines map to Maven MULAW_8000.

Google Antigravity normally uses the selected Google integration or GOOGLE_API_KEY / GEMINI_API_KEY, plus ANTIGRAVITY_RUNNER_URL for the internal runner. Codex uses the selected OpenAI integration or CODEX_API_KEY / OPENAI_API_KEY. Claude Code accepts native Anthropic integrations, AWS integrations for Bedrock (bedrock_api_key or standard AWS credentials), or Google integrations for Vertex AI; ai-orchestrator resolves the selected integration into runner auth_context metadata and the runner maps it to Claude Code environment variables. Codex and Claude Code both use WORKSPACE_AGENT_RUNNER_URL for the internal runner. Local dev can set runner *_FAKE=true values for deterministic stream tests without provider credentials or CLI binaries.

Production

Helm chart at charts/ai-orchestrator. Deployed as a Deployment with HPA on CPU and concurrent sessions. Sticky on session id is not required — each session is one gRPC stream that lasts the call. For hosted Whisker debugging, route ai-manager's /whisker-ws path to the ai-orchestrator service's HTTP port 8080 so browser WebSockets reach the gateway instead of Next.js.

Tests

cd apps/ai-orchestrator && .venv/bin/python -m pytest -q

pyproject.toml sets asyncio_mode = "auto" and testpaths = ["tests"]. There is no pytest-timeout plugin, so a test that awaits a real socket hangs the run rather than failing.

Three rules the suite depends on:

• tests/conftest.py must import conversation.api_pb2 eagerly, before any test puts apps/ai-orchestrator/lib on sys.path. Two directories are named conversation: the generated protos at <app>/conversation/ (namespace-style, no __init__.py) and the regular package <app>/lib/conversation/ (whose __init__ imports conversation.api_pb2). A regular package short-circuits import resolution, so once a module does sys.path.insert(0, "<app>/lib") — tests/test_turn_taking.py does — conversation resolves to lib/conversation and conversation.api_pb2 disappears. The failure is order-dependent: the file passes when run alone and errors at collection in a full run.
• The dialogflow stubs in conftest.py are conditional on the real library being absent. google-cloud-dialogflow is installed in the orchestrator venv, and stubbing it with a bare types.ModuleType makes google.cloud.dialogflow_v2beta1 a non-package, so agent_assist.py's ...v2beta1.services import fails.
• tests/test_turn_taking.py is a manual integration script gated behind RUN_TURN_TAKING_TEST=1. It opens a live OpenAI Realtime session and replays a captured *_tx-pcm.raw recording. pytest collects it only because of the test_ prefix; without the guard it fails at get_env_credential("OPENAI_API_KEY"), which raises on a missing credential rather than returning empty. See tests/README_TURN_TAKING_TEST.md.

Common pitfalls

• Never use getattr(self, "pipeline_worker", None). FrameProcessor.pipeline_worker is a property that raises a bare Exception ("… pipeline worker is still not set.") when the worker isn't attached, and getattr only swallows AttributeError — so the raise propagates into the caller's except Exception and a tool call returns {"error": "…pipeline worker is still not set."} instead of running. The worker is genuinely optional at these sites (it only supplies app_resources), so use lib/pipecat/utilities/pipeline_worker.py — optional_pipeline_worker(self) / optional_app_resources(self). This silently broke every host-tool call in google_antigravity, workspace_agent, and google/adk.py.
• An inline Redis client must prefer REDIS_URL, never a composed plaintext redis:// — the latter hangs forever against a TLS-required ElastiCache. The orchestrator's Redis is REDIS_URL, which in KMS AWS EKS is a rediss:// endpoint (TransitEncryptionMode: required + AuthTokenEnabled: true). A client built ad-hoc from REDIS_HOST/REDIS_PORT/REDIS_AUTH speaks plaintext redis://, never completes the TLS handshake, and — because redis.asyncio.from_url(...) has no default socket timeout — the await blocks indefinitely. The worst case was tool_wiring.py's A2UI session-policy publish inside should_enable_a2ui_tool: on a multimodal agent (Nova Sonic, direct WebRTC) the await _r.set("mcp:session_policy:…") hung mid-build, the pipeline never finished assembling, and the caller heard total silence — no greeting at all, while the same agent with multimodal OFF (no A2UI block, no Redis write) greeted fine and rome (Overmind, plaintext Redis, no TLS) worked either way. The tell: the multimodal build stalls right after the should_enable_a2ui_tool log with no Published a2ui session policy / Failed to publish / Registered A2UI / greeting following (a hang, so the except → logger.warning never fired). Fix: os.environ.get("REDIS_URL", <composed plaintext fallback>) + socket_connect_timeout=5 (plus socket_timeout=5 on one-shot ops; not on long-lived pubsub.listen() loops — a read timeout tears the idle subscription down). Same fix in ces.py (el-mapping write) and elevenlabs/agent_service.py (_subscribe_ui_actions, connect-timeout only). The fallback keeps plaintext environments byte-identical; the timeout only turns an infinite hang into the pre-existing caught-warning path. agent-host (the policy consumer) was never affected — it already builds its shared client from REDIS_URL.
• A FrameSerializer subclass that doesn't call super().__init__() loses the RTVI filter silently, and the loss is invisible until it reaches a customer. FrameSerializer.__init__ is what sets self._params, and should_ignore_frame() reads self._params.ignore_rtvi_messages — so without the super call that method raises AttributeError rather than filtering, and the guard every upstream serializer relies on (twilio, plivo, genesys, protobuf, vonage, exotel all call it in their OutputTransportMessage* branch) is not merely unused but unusable. PipelineWorker installs an RTVIObserver by default (enable_rtvi: bool = True), so its MetricsMessage reaches every transport's send_message() whether or not anything speaks RTVI. In GrpcFrameSerializer that meant {"label": "rtvi-ai", "type": "metrics", "data": {}} went out as a TextEvent, and — because grpc.py's catch-all else left kind unset, which proto3 decodes as 0 = KIND_TRANSCRIPT — Salesforce recorded it as the caller's first utterance on every translated call. Any branch that builds a TextEvent must set kind explicitly; omitting it is an assertion, not a default. grpc.py is currently the only FrameSerializer subclass in the tree.
• Never case-fold a language string before matching it against Language — and never let one wrapper hand-roll the coercion. pipecat.transcriptions.language.Language is a StrEnum whose values are case-sensitive BCP-47 ("en-US") and whose names are underscored (EN_US), so a region-qualified code misses on both halves of the usual lower()/upper() pair: Language("en-us") raises ValueError and Language["EN-US"] raises KeyError. Three wrappers each hand-rolled that idiom and each failed differently — Deepgram raised (killing the session at build time, before any conversations row existed, so every route 404s and the agent silently never greets), AWS Transcribe silently fell back to Language.EN for all 21 of its region-qualified options (a stage authored ja-JP transcribed as English, the only trace one warning), and both ElevenLabs wrappers dropped the language entirely. Use lib/pipecat/utilities/language_utils.py — coerce_language(value) matches the original string (exact value → exact name → case-insensitive value → dashes-as-underscores → bare subtag) and returns None rather than raising, because an unrecognised language is not a reason to refuse to build a pipeline. auto/multi are authoring sentinels, not languages (auto is Nova-3's own default): is_auto_language() recognises them and the caller must omit the key so the provider detects per utterance. Two properties make the regression test non-vacuous — it drives the real service-definition JSONs (a fixture of 'en' passes on the broken code, since 'en' is the one form the old pair happened to match) and it asserts resolved.value == value, which is what catches a silent widening like Transcribe's. Note the service definitions key params as a dict, not a list.
• A wrapper with no validate_config gets neither key stripping nor per-service validation — so preflight structurally cannot catch its config errors. _get_wrapper_supported_keys returns None unless the wrapper defines its own validate_config, and ServiceRegistry.validate_service_config (the gate preflight.py shares with create_service) then has nothing to check. This is why a copilot agent whose Deepgram stage carried language: "en-US" preflighted {"ok": true} while the live build raised on that exact value: DeepgramSTTServiceWrapper has no validate_config, so the only thing that could reject it was create() itself, which runs after preflight stops. Corollary in the other direction: because create_service raises on validate_service_config errors, a validator stricter than the wrapper's own create() blocks the live build too, not just preflight.
• ElevenLabs Text to Dialogue is an Eleven v3-only TTS service, not an alias for the older TTS WebSocket. The editor's Text to Dialogue choice maps to ElevenLabsTextToDialogueTTSService, whose persistent /v1/text-to-dialogue/multi-stream-input connection registers the selected voice per Pipecat turn and sends incremental inputs[]. Turn completion and interruption both close the context, which flushes the provider buffer and gives barge-in an explicit cancellation boundary; context keepalives prevent the fixed 20-second TTD timeout. Translation pipelines instead pin one context for their continuous TextFrame stream and rotate it after interruption so fragments cannot exhaust the five-context limit. The wrapper rejects non-v3 models and non-PCM output during preflight. See the universal builder protocol note.
• loguru has no exc_info — logger.error(msg, exc_info=True) lies twice. loguru's kwargs feed str.format, so the traceback is never emitted, and a message containing braces (every provider API error embeds a dict repr) raises KeyError inside the logging call — a real 404 … 'This is not a chat model' was logged as Extraction error: "'error'", which reads as a bug in our own JSON handling. Use logger.opt(exception=True).error(...). The copilot chain is guarded by tests/test_logging_exc_info.py because those processors' only report of a failed provider call is that log line.
• A stale dev_runner.py from a previous Overmind run will hold port 50051. pkill -f dev_runner.py if a restart loops.
• Voice agents fail to start with no error if the agent's config.flows.incoming.pipeline is empty. See flow-editor-empty-incoming-flow-loop.
• If Agentforce Voice fails before Salesforce receives audio, confirm the Salesforce integration can query Organization.Id; without runtime-proxy or an org id, ai-orchestrator cannot build the SDK-style tenant header. User-facing errors must not include Salesforce bearer tokens or LiveKit room tokens.
• If Agentforce returns a normal Agent API greeting but realtime join fails with 404 / fault filter abort, verify the join URL includes /join, the call host is SFAP (https://api.salesforce.com or a returned sfapUrl), the tenant header is correct, and the selected Agentforce agent is voice-enabled for the configured Salesforce voice or mobile channel.
• If Maven AGI Chat emits a form, chart, object, OAuth button, or follow-up metadata, it should appear as an A2UI surface through the normal multimodal UI_RENDER path. A missing surface usually means the A2UI handler/publisher path is disabled, not that Maven needs a new conversation event type.
• Adding a new STT/LLM/TTS provider requires edits in seven places. See the checklist in universal-pipeline-builder.md.
• Copilot-capable STT services: Deepgram Nova-3/Flux, Amazon Transcribe (aws_transcribe_services.py, per-channel subclass), OpenAI GPT Live Transcribe (openai_stt_services.py), ElevenLabs Scribe Realtime (elevenlabs_stt_services.py), and Google Agent Assist (via the 3rd-Party engine). Per-channel speaker attribution is tiered per provider — see copilots.md. The Realtime STT wrappers default to provider-side VAD because copilots run with local VAD off — the pipecat local-VAD defaults would never commit a segment. The VAD-segmented tier (Whisper, non-realtime Scribe) is deliberately unwrapped: single-speaker attribution breaks copilot runs_on filters.

See also

• Universal pipeline builder
• Voice request lifecycle
• agent-host
• media-proxy

Agent evaluation

Automatic cascaded agents accept the STT service's recommended turn strategies. Flux's proposed turn stop must reach the assistant aggregator before tool results can trigger another response. Explicit turn-control choices retain priority. Regression checks cover both paths in test_llm_context_build.py. The gRPC disconnect path closes on the control type, cancels helper workers, and emits final evidence without requiring a client half-close.

Performance tests use durable jobs, pinned runtime configurations and shared mock state. See agent evaluation and worker operations for this service’s role, admission limits, authenticated stream ownership and recovery.