Universal pipeline builder

Status: Current | Last reviewed: 2026-10-07

How an agent's JSON config in agent_versions.config becomes a running Pipecat pipeline. This is the most-touched code path in the system. Read it before you change anything in ai-manager or ai-orchestrator.

The shared Responses adapter extracts only leading system/developer messages into instructions. Later trusted messages remain ordered input items after the conversation or tool result they update. This preserves the timing of supervisor decisions and asynchronous receipts in voice and workflow calls.

What it is

Two halves of the same schema:

• The UI side in apps/ai-manager lets a user drag stages onto a canvas and edit their config. The result is JSON.
• The runtime side in apps/ai-orchestrator reads that JSON and instantiates the corresponding Pipecat pipeline.

The contract between them is the TypeScript types in apps/ai-manager/lib/pipeline-types.ts. The Python side does not import these types but mirrors their shape; the schema is the source of truth.

The shape of a pipeline config

A pipeline has:

• A type (voice, chat, translation, async, workflow).
• A subtype that picks the builder (linear, s2s, parallel-agent, branch, data-branch, translation, translation-test).
• A stages array — STT, LLM, TTS, tools, and so on.
• Per-stage provider, service, and config values.
• Optional cross-cutting blocks: transfer (handoff), lifecycle (on-start / on-complete workflow triggers), prompts, and node-owned tool config such as ServiceStage.config.tools, ServiceStage.config.builtin_tool_configs, and Google GECX-only ServiceStage.config.ces_integration (legacy key name).

Read the canonical types: pipeline-types.ts.

UI side

FlowEditor

apps/ai-manager/components/FlowEditor.tsx is the canvas. It uses ReactFlow with custom node types: PipelineNode, ParallelGroupNode, StartNode, EndNode, BranchNode.

Note: inspector edits must not rebuild the canvas. The invariant is that ServiceConfigForm patches leaf-level config on existing nodes; only structural changes (adding/removing stages) trigger a rebuild. Violating this causes focus loss while typing. See the agent-memory entry on flow-editor-focus-loss for context.

The structural diff is keyed on pipelineStructureSignature — a hash of stage IDs and edges. If that hash matches, only leaf config updates.

ServiceConfigForm

Provider changes use getConfigForProviderChange to reset provider-specific parameters while retaining authored behavior. Preserve both config.topics (current in-conversation subagents, including guided scripts and transitions) and config.subagents (separate-call definitions), plus logic, tool_logic, and procedure; otherwise the provider reset silently deletes subagents or their routing rules.

apps/ai-manager/components/ServiceConfigForm.tsx renders the inspector for a selected node. It pulls per-service form schema from lib/service-definitions/ — the canonical registry of providers and services.

Note: the older provider-metadata.ts is marked @deprecated, but it still drives some dropdowns (provider/service pickers). Adding a new service-definition file is not enough to make it appear in the UI; you also have to extend PROVIDER_OPTIONS in provider-metadata.ts. See "Adding a new service provider" below.

Turn Control

OpenAI GPT Live (openai / s2s / GPT Live API) owns continuous speech and interruptions. Its inspector hides local VAD/interruption controls, and both transports disable local VAD even if a migrated stage retained those settings. Voice isolation and idle prompts remain available. See GPT Live for delegation configuration and the preserved tool, payment, multimodal, and guided-flow paths.

Conversational stages (llm, assistant_agent, chat, text_llm, s2s, s2s_llm, realtime) own turn-taking behavior through ServiceStage.config.turn_control:

{
  "turn_control": {
    "interruption": { "enabled": true, "strategy": "auto" },
    "turn_detection": { "strategy": "auto", "vad_provider": "auto" },
    "idle": { "enabled": false, "timeout_seconds": 15, "max_prompts": 3 },
    "audio_enhancement": { "voice_isolation": "off" }
  }
}

auto means provider-native turn control for S2S/realtime services, Pipecat local turn control for cascaded speech and chat, and never Krisp unless the author explicitly selects krisp_viva. For OpenAI Realtime, turn_control.interruption.enabled is mapped into the native Realtime interrupt_response turn-detection flag, so barge-in is controlled by the same Behavior > Turn Control switch as other conversational stages. The runtime still reads legacy config.interruption, config.user_idle, provider-native turn_detection, and settings.audio.enable_vad; the inspector writes only the canonical turn_control block and clears the two legacy behavior keys on edit.

Idle UX vs the transport inactivity watchdog (two different things). The idle block above drives the graceful "are you still there?" flow: when enabled, the UserIdleSpeechAwareProcessor (user_idle_handler.py) prompts up to max_prompts times at timeout_seconds apart, then ends the session with an EndFrame. It is opt-in for text_chat (the universal builder force-disables it unless idle.enabled is true). The adapter targets the concrete service even when an IVR or pipeline wrapper sits around it. Cascaded/chat services run the normal LLM append path; OpenAI Realtime uses response.create; AWS Nova Sonic, Google ADK Live, Google CES, and ElevenLabs Conversational AI send provider-native text turns; Deepgram Voice Agent speaks the exact phrase through InjectAgentMessage. Maven Voice, NVIDIA PersonaPlex/Moshi, and Salesforce Agentforce Voice do not expose a supported outbound text/assistant-message control on their live audio transports, so the builder force-disables the idle handler for them and records idle_prompt_disabled_reason rather than counting silent retries and hanging up. A new S2S provider must implement send_idle_prompt(prompt, instruction=...) before idle handling is enabled. Separately, the gRPC transport has an inactivity watchdog (grpc.py) that force-disconnects a wedged/half-closed stream (e.g. media-proxy after a SIP hangup whose half-close never propagates). The watchdog timeout is now threaded through GRPCParams.inactivity_timeout_seconds and set by grpc_provider.py per modality: voice keeps 120 s; text chat uses timeout_seconds × (max_prompts + 1) + 60 s when the idle handler is enabled (so the graceful handler always fires first), or a 30-minute silent backstop when it is not (idle UX is opt-in, so an unattended chat just stays open until the stream closes). The old hardcoded 120 s watchdog was silently dropping live text chats mid-conversation with no warning — don't reintroduce a fixed text-chat watchdog shorter than a full idle cycle.

Krisp VIVA is integrated as an optional adapter layer in lib/pipecat/pipeline/turn_control.py. Voice isolation attaches a Pipecat audio_in_filter; VAD/turn/interruption selection builds the matching Pipecat VIVA analyzer or user-turn strategies only when the node explicitly selects Krisp. Model paths and API keys come from a krisp_viva integration profile or KRISP_VIVA_* environment fallback variables.

External Consult (warm transfer) is a node capability

External Consult (product name; internal id still builtin:warm_transfer) is a Capabilities-section item on a voice agent's LLM/S2S stage — the same per-node pattern as Multimodal UI, next to it in NodeCapabilitiesAndBehaviors, not a selectable Tools entry (appendNodeBuiltIns no longer offers it) and not a cross-cutting Operations panel. Its rich config (provider, hold/post-merge mode, destinations with per-destination outbound agent + inline brief overrides + routing rules) lives at ServiceStage.config.builtin_tool_configs.warm_transfer (WarmTransferBuiltinConfig). The editor is the shared ExternalConsultForm.tsx inside ExternalConsultCapability.tsx (inline variant for native voice, gecx variant for GECX); it's registered (with a voice-only stageTypes gate) in node-builtins.ts. The outbound (consult) agent that dials out is a separate voice agent with the Transfer Agent capability (its own config provides the B-leg model); the form's "Create External Consult agent" link opens the New Agent wizard preselected to the built-in External Consult Agent template (tpl-outbound-transfer-agent, id unchanged). The consult dial target is the Genesys integration's Routing Number (consult_routing_number), injected at build time by resolve_consult_dial_numbers. See warm-handoff.md.

Runtime side

universal_builder.py

apps/ai-orchestrator/lib/pipecat/pipeline/builders/universal_builder.py is the entry point. It:

1. Receives a config blob (already resolved via resolve_agent_version()).
2. Reads pipeline.subtype and dispatches to a subtype builder.
3. Wires in cross-cutting tools: consult tool helper (warm-handoff), end-session tool helper, DTMF tool helper, A2UI tool handler, MCP manager.
4. Returns a Pipecat Pipeline ready to run.

Phase 1 builds the ToolsSchema incrementally, and every contributor must MERGE into it — never assign over it. The inline Python/API/Knowledge schema-adds run first, then the MCP fetch, then the built-ins (workflow tool, A2UI, auth, payments, topics). All three MCP branches used to reassign tools with their own result, deleting every schema added above them — while phase 2 still registered all the callbacks, so the build log, preflight, and the registration count all looked healthy and the model was simply never told the earlier tools existed (agent 3032's Nova session opened with 9 of 21 tools; the FHIR lookups were among the missing, and the agent improvised in silence). merge_tools_schemas in tool_wiring.py is the one combiner (first-writer-wins on a name collision), pinned by test_tool_wiring_mcp_merge.py. When debugging "the model won't call a tool that's plainly wired", check the Nova context dump's TOOLS: N tool(s) block (/tmp/nova_sonic_debug.log) — it is the ground truth for what the model was told.

MCP callback ownership is per client, even though the model receives one merged schema. MCPManager.get_tools_schema_only retains each client's tool names after filtering. register_tools_with_service intersects the final model schema with those names and preserves existing handlers. A client without discovery metadata registers nothing. This prevents an MCP server from claiming a local API tool or overwriting another server's callback. Phase 2 uses the names actually registered when applying MCP call wrappers. API/Python callbacks are registered later, so inspecting the service registry alone cannot establish MCP ownership. The saved-tool and inline-tool cases in test_mcp_callback_ownership.py exercise both wiring phases and execute a mocked API callback, including when MCP discovery fails.

config.tools means two different things depending on when you read it. Authored, it is a list of string refs (["builtin:warm_transfer", "<tool-uuid>", …]); _resolve_tool_schemas then replaces the key in place with the resolved ToolsSchema and writes the config back onto the service, so every later reader sees an object where the list was. Anything that keys on the authored selection — the builtin:* stage gates in node_builtin_config.py, the {{TOOL:Name}} PromptResolver — must read the _authored_tool_refs sidecar the overwrite leaves behind, via authored_tool_refs(stage_config). Reading config.tools directly makes a gate order-dependent, and it fails silently: the built-in reads as un-selected and the code drops into whatever legacy fallback it has. See warm-handoff.md for the worked example.

Providers wrap that pipeline in a Pipecat 1.3 PipelineWorker, not the older PipelineTask. Runners must use await runner.add_workers(worker) followed by await runner.run(). PipelineParams no longer receives allow_interruptions; interruption behavior is preserved through runtime_context["allow_interruptions"] and the universal turn/user-aggregator strategy path.

Where the code lives: the universal/ package

The builder was 7,252 lines in one file. Six phases have been lifted into lib/pipecat/pipeline/builders/universal/, leaving 3,311 in universal_builder.py. universal_builder.py is still the only entry point — every module is imported back into it, and the class keeps a thin delegating method wherever a call site outside the moved block exists. Start there, not here, when tracing a build.

Module	Owns	Reached through
normalization.py	{{template}} rendering, normalize_pipeline_v2, internal-component injection, harvest_greeting	_normalize_pipeline_v2, _harvest_greeting
config_translation.py	Per-provider config translation: build_service_specific_config, translate_service_config, provider/model mapping, remote end-session analysis config	_build_service_specific_config, _translate_service_config
payments.py	The PCI handoff pieces: controller creation + insertion, handoff owner resolution, the private prompt-only TTS	_get_or_create_payment_controller, _insert_payment_controller
post_llm.py	Everything inserted after the LLM: IVR navigator wrap, CES bindings, speculative speech, TTS gate + cacher, transcript sync gate, interruption gate, think filter, publishers	the _insert_* / _maybe_* methods
tool_wiring.py	Tool wiring phases 1 and 2 — resolve_tool_schemas (schemas, MCP/A2A/inline/subagent resolution) and register_tool_callbacks (callbacks, post-service)	_resolve_tool_schemas, _register_tool_callbacks
llm_context.py	Tool wiring phase 3 — build_llm_context: prompt resolution, variable substitution, legacy MCP fallback, tool instructions, aggregator + context creation	_build_llm_context

The three tool phases are ordered, and the order is not obvious from the call sites. Phase 1 must run before the service is created, because OpenAI Realtime needs its tool list in the session properties it opens with; phase 2 must run after, because it registers callbacks on the service object. Phase 3 then reads phase 1's _ToolWiringContext rather than re-deriving from the stage config — which is why the end_session branches in llm_context.py are guarded on phase1_end_session_decided is None. That guard is the legacy path for a build where phase 1 did not run; dropping it silently re-derives the flag and disables the tool on every agent phase 1 had already decided for. _assemble_services enforces the sequence, and swapping phases 1 and 2 does not fail loudly — a Realtime session simply opens with no tools and the model answers "I can't do that" for every one of them.

Two groups are deliberately still in universal_builder.py: the parallel/branch builders and chain assembly (_insert_input_chain, _insert_output_chain, _build_linear_pipeline). They implement the parallel-group-loses-the-input-chain defect below, whose fix reshapes exactly that seam, and the recorder-placement comments there carry the S2S caller-track invariant. Moving them first would have to be undone.

Every extraction is move-only: behaviour, ordering, log lines and error messages are unchanged, and each function that reads builder state takes builder as an explicit first parameter (the self → builder rename is the whole diff). Each module's docstring records why its phase sits where it does; read that before changing a call order.

AS-296: API verification replaced by Microsoft search

QA investigation on 2026-09-21 found agent 4076 ("chat mcp test") with the defect's exact prompt, saved verify_identity API tool, and MS_Learn MCP tool. Historical QA conversation events on 2026-09-03 recorded microsoft_docs_search with query="test" (conversation c0bbf271-57cb-498a-ad50-12ecab55332b) and query="verify_identity" (b7b7cb48-8a55-42a9-86a5-4f0a08557739).

The corresponding QA Loki logs at 09:02:04–09:02:59 UTC first report adding one API schema, then create the LLM context with only microsoft_docs_search, microsoft_code_sample_search, microsoft_docs_fetch, and end_session. The missing API declaration matches the schema-overwrite defect described above. The model selected a search tool; these records do not show an API invocation being renamed to a search invocation. Prompt text cannot make a missing function declaration callable.

The QA image observed through Prometheus on 2026-09-21 was ai-orchestrator:f503402d8c2d4498ff5e6cc7eabe44d30e9e67b0, which already contains the schema-merge correction. A separate draft, 4141 ("AS-296 QA reproduction"), copied agent 4076's prompt and tools. Its session 12775d5a-df03-4319-a682-a93aa93b03a0 advertised verify_identity, microsoft_docs_search, and end_session and asked for verification digits. After explicit approval for the existing external verification endpoint, session 5578e2bb-0664-4a33-9a71-32213ab96c0c completed the live tool-routing test at 12:44 UTC: synthetic input 1234 produced exactly one verify_identity call with ssn_last4="1234", HTTP 200, and authenticated: true. Subsequent inputs test and What can you help me with today? produced no additional tool calls; there were no Microsoft tool calls anywhere in the session. The model repeated its verification acknowledgment on those follow-ups, so this result establishes tool routing rather than general response quality. The test session was closed and its persisted history remains available. This validates the existing QA image; the additional callback ownership change has not been deployed to official QA. The targeted local regression suite passed 31 tests. The callback ownership correction adds protection beyond the existing merge fix; it must not be described as the proven cause of those historical model selections.

Rome live retest on 2026-09-21 used new saved tools verify_identity (4e7b0773-df40-4395-9285-d63c75921891) and MS_Learn (f96c00c6-b340-4c62-823a-8a50f7782458), copied from the QA definitions, and draft agent 4348 ("AS-296 Rome defect retest"). The prompt was checked for exact equality with QA agent 4076; the model remained Gemini gemini-3.5-flash at temperature 0.1 using Rome's integration. The verification endpoint and MCP search filter were unchanged.

Conversation bf7e1e63-8528-4112-833e-04c636ba2d72 passed the same live tool-routing sequence: the agent asked for digits, synthetic 1234 invoked verify_identity once with the required argument, and the API returned HTTP 200 with authenticated: true. Follow-ups test and What can you help me with today? caused no further tool calls and no Microsoft searches. As in QA, the model repeated its verification acknowledgment on follow-ups. Rome's Overmind logs at 12:49:14 UTC confirmed the new code was running: MCP registered tools: ['microsoft_docs_search'], while the model context retained both that tool and verify_identity. This live run therefore also validates the callback ownership fix locally. The initial history read briefly returned 404 before persistence completed; continuing the same session succeeded. The live session was closed without deleting its history, and the draft and tools remain available for manual retesting.

For retesting, use synthetic digits, confirm the required ssn_last4 schema reaches the model, check that the verification call reaches only the API handler, and confirm ordinary follow-up turns make no Microsoft calls. Also test MCP discovery failure and multiple MCP servers with distinct tools. Prompt restrictions guide model behavior; a tool that must be unavailable should be removed from the node's selected tools or governed by an explicit tool gate. Do not infer an executable allowlist by parsing prose.

Subtype builders

Each lives in lib/pipecat/pipeline/builders/:

Subtype	File	What it builds
linear	linear_builder.py	The default: STT → LLM → TTS chain
s2s	s2s_builder.py	Speech-to-speech (OpenAI Realtime, Gemini Live), no separate STT/TTS
parallel-agent	parallel_agent_builder.py	Multiple sub-agents running concurrently with a router
branch	branch_builder.py	Conditional routing across stage subgraphs
data-branch	data_branch_builder.py	Branch on data-driven conditions during the call
translation	translation_builder.py	Real-time bilingual pipeline; see realtime-translation.md
translation-test	translation_test_builder.py	Translation harness for offline testing

The base class and shared utilities are in base_builder.py and binding_utils.py.

pipeline_config_provider.py

apps/ai-orchestrator/lib/pipecat/pipeline/pipeline_config_provider.py fetches and validates the config from the database. It calls resolve_agent_version() (or resolve_agent_version_weighted() for A/B-enabled agents).

It also applies in-memory migrations — older configs that lack newer fields get defaults filled in, so the runtime never sees a malformed config.

Config-shape guards must stay permissive — a rejected config builds nothing. After resolving the row, db_pipeline_store.fetch_pipeline_config_from_db runs the config through the typing_guards.py discriminators (is_pipeline_entry_config / is_branch_pipeline_config / is_universal_pipeline_config). If none match it logs Pipeline … has unexpected shape; skipping and returns None → the session never starts, no conversations row is created, and every downstream conversation route (/sse/..., GET /conversations/{id}, POST .../responses) 404s while the agent silently never greets. The universal guard is discriminated by a pipeline LIST alone; settings is optional (every builder read uses schema.get("settings", {})). This regressed because ai-manager's default text_chat config carries settings: {}, and the collab Y.Doc round-trip used to drop empty branch maps, so an autosave silently stripped the settings key. Two fixes, defense-in-depth (both 2026-06-03): (1) the guard no longer requires settings; (2) the collab converters now always emit settings even when empty (collab-gateway pipeline-shape.ts + its ai-manager mirror, with a round-trip test in pipeline-shape.test.ts) so the stored config stays byte-stable across autosaves — lifecycle/output_lanes stay conditional since nothing requires them. Don't re-add a settings-present requirement to the universal guard, and don't make settings conditional in the converters' read side again.

service_registry.py

apps/ai-orchestrator/lib/pipecat/pipeline/service_registry.py instantiates a service implementation given a (provider, service, config) triple. This is where "OpenAI / gpt-4o" or "Deepgram / nova-2" becomes a concrete Pipecat processor.

OpenAI text LLM stages have two non-realtime service labels:

• Responses API maps to OpenAIResponsesLLMService and calls /v1/responses. This is the default for newly-added non-realtime OpenAI LLM stages and is the only OpenAI LLM service that exposes hosted Responses tools. The runtime does not translate legacy max_tokens into Responses max_output_tokens; set config.max_output_tokens only when you intentionally want a hard cap. Set config.reasoning_summary to auto when the tester/debug surface should render OpenAI's visible reasoning summary through the same variant="reasoning" transcript lane used by extracted <think> blocks.
• Streaming Completions maps to OpenAILLMService and keeps the Pipecat Chat Completions streaming path. Existing agents/workflows should stay here until they are explicitly migrated.

In-memory migration keeps older OpenAI LLM stages without a service field on Streaming Completions so publishing old configs does not silently move them onto /v1/responses.

For spoken Responses stages, config.deduplicate_response_messages=true preserves assistant output-message boundaries with blank lines and collapses adjacent identical complete messages within one API response. This prevents an answer and a following control token from becoming one sentence before response guards run. Distinct messages, repeated words inside a message and repeats across separate turns remain intact. The option defaults off; shared Responses normalization and workflow output parsing are unchanged. A suppress response guard can then keep a separate silence token out of TTS. The Responses reasoning selector includes none for models that support it. This is an explicit model choice, not a fallback for models that reject that effort value; qualification must cover tool use and response quality.

Responses API: an assistant tool-call turn must round-trip as a function_call item. Pipecat stores a tool-calling assistant turn in chat format ({"role":"assistant","tool_calls":[{"id","type":"function","function":{name,arguments}}]}), but the Responses API needs each call replayed as a standalone function_call item that appears before its function_call_output, with a matching call_id. messages_to_responses_input in libs/reasoning/openai_responses.py converts these. If the assistant-tool_calls branch is missing, the call is dropped and the next request fails with 400 No tool call found for function call output with call_id … — the tool ran (e.g. an A2UI surface rendered), but the branch then errors and the broken context poisons later turns. Symptom looks like "asked for a UI component, got no working response." Don't remove that conversion branch.

Salesforce Agentforce Voice maps (provider="salesforce", type="s2s", service="Agentforce Voice") to SalesforceAgentforceVoiceService. Unlike local STT→LLM→TTS providers, the processor is a server-side bridge: it starts the Agentforce session with the public Agent API agent-scoped start path first, falls back to the Mobile SDK-compatible session request if needed, joins Salesforce's LiveKit room through the SFAP host with /einstein/ai-agent/v1.1/realtime/sessions/{sessionId}/join, publishes incoming InputAudioRawFrame chunks, and emits remote agent audio/transcript frames back into the same TrueServe pipeline. The browser never receives Salesforce OAuth tokens or LiveKit room tokens. The Agentforce externalSessionKey is the TrueServe conversation id when one exists, so Salesforce session diagnostics can be correlated back to the TrueServe session.

Agentforce Voice can opt into config.end_session_analysis, a TrueServe-side transcript analyzer for remote agents that cannot call the built-in end_session tool. When enabled, universal_builder.py inserts RemoteEndSessionAnalyzerProcessor immediately after the Agentforce service. The processor watches final transcript frames, calls the selected normal LLM service/model with structured output, expects GECX-compatible metadata.reason, metadata.session_escalated, and metadata.params, then emits a standard EndFrame with _session_end_params. GECX native endSession and this analyzer use the same metadata flattener so workflows, SIP/Genesys output variables, and conversation.end see the same flat params shape.

Maven AGI maps (provider="mavenagi", type="llm", service="Chat Agent") to MavenAGIChatService and (provider="mavenagi", type="s2s", service="Voice Agent") to MavenAGIVoiceService. ai-manager stores org/app credentials on a mavenagi integration profile and stores the selected Maven agent_id on the stage. The chat path calls AsyncMavenAGI.conversation.ask_stream, emits text as LLMTextFrame, and converts Maven rich payloads (action / actionForm, chart, object, oauthButton, follow-up metadata, and diagnostics for unknown variants) into A2UIRenderFrame so conversation-api and surface-app keep using the existing UI_RENDER event. The voice path mints a voice.session_token(type="websocket"), connects server-side to /v1/voice/conversations, sends Pipecat microphone chunks as Maven websocket audio chunks, and emits Maven agent audio/transcripts back into the pipeline. Audio format injection converts the pipeline wire format to Maven's PCM_16000, PCM_8000, PCM_44100, or MULAW_8000.

Subagents

settings.subagents / top-level subagents support four execution modes. Since 2026-07-31 every non-session-switch mode dispatches through one PipecatWorkerSubagentAdapter (worker_adapter.py) registered as per-tool LLM function callbacks; the modes differ only in labeling, not wiring, because pipecat's LLMService already runs a multi-call batch concurrently (run_in_parallel=True default, grouped so the LLM re-triggers once per batch):

Mode	Runtime path
sequential (default)	Adapter dispatch via registered function callbacks
parallel	Same callbacks — batch concurrency is pipecat's own (the old SubagentToolHandler frame interceptor was REMOVED: it imported FunctionCallsFinishedFrame, which does not exist in pipecat 1.4, so it crashed the build for every parallel-mode agent)
session_switch	S2S agent-transfer graph with transferAgents
workers	Same adapter path (the former pilot label; kept for config compat)

Session-switch transfers are default-all. The builder flattens every enabled, named agent in the authored sub-agent hierarchy into one live session graph. The implicit main agent can transfer to all of them; each specialist can transfer to main and every other named specialist except itself. disabled_downstream_agents is the explicit deny-list applied per specialist. The stored downstream_agents array is retained as a compatibility snapshot, but the current runtime derives the effective list from the graph plus opt-outs so agents added later become transferrable automatically.

Inside the adapter, dispatch is per-spec dual-path. For each inline subagent whose provider SDK is installed (openai, incl. Responses API; google; anthropic/amazon when their SDKs ship), llm_service_factory.py builds a real per-subagent pipecat LLMService (credentials via get_env_credential, integration row → env) and a SubagentWorker — a bus-only BaseWorker with a @job(name="run") handler that runs one-shot run_inference per invocation. ref subagents (config resolved from the DB per call), unsupported providers, and factory failures fall back to the legacy SubagentExecutor (the hand-rolled OpenAI SDK path) with identical result contracts. The adapter lives at runtime_context["pipecat_worker_subagents"]; providers register the workers on the session's WorkerRunner via register_on_runner(runner, parent=pipeline_task) — the parent argument is load-bearing: workers attach as CHILDREN of the session PipelineWorker (a root bus-only worker never finishes on its own and would block the runner's auto-end, wedging session teardown), and the parent doubles as the bus-job requester. A SubagentWorker always responds COMPLETED carrying either {output} or {error, status} — never FAILED status, which would trip cancel_on_error and kill sibling jobs in a concurrent batch. Keep swimlane audio on ParallelPipeline + TurnAwareMixer + TurnGate; Pipecat workers coordinate agent calls and do not replace audio mixing or turn-gating.

context_strategy is now live (it was plumbed but never passed — every subagent ran task_only). The tool callback hands the parent's live LLMContext (FunctionCallParams.context) to the adapter, and context_strategies.py slices it: task_only (nothing), last_n_turns (trailing 6 user/assistant messages), full (whole user/assistant history). System/tool messages and non-text content are always stripped; S2S services that keep history provider-side may supply no context, in which case every strategy degrades to task_only.

Note: a Phase-12 nested subagent pipeline is NOT recursively built — the executor extracts the first LLM stage and runs it as the same stateless text call; other stages are ignored with a build-time warning. See the pipecat-native alignment backlog.

A subagent prompt resolves {{var}}/{{ns.key}} (session/template variables) the same way the main stage does — and this must happen in BOTH subagent paths, which are separate. The manager's variable scope collects {{...}} tokens even inside subagents, so the tester's Template Variables panel shows and fills them; the runtime has to resolve them in whichever subagent path the agent actually uses:

• session_switch (S2S: OpenAI Realtime / etc.) — a subagent's prompt is its subagent.instructions string, sent verbatim in the session.update when the agent switches to it. UniversalPipelineBuilder now renders {{var}} in that string via _render_runtime_template(..., self._subagent_session_variables()) before building SessionSwitchAgentConfig. Before this, a session-switch subagent spoke the literal {{car}}/{{feature}} text while the main agent's prompt (which goes through PromptResolver) resolved fine — the most common "subagent variables don't work" report, because S2S agents default to this mode.
• sequential / parallel / workers (adapter dispatch) — _substitute_templates (subagent_executor.py) runs two passes on each system message: first {{var}}/{{ns.key}} against the session variables (threaded into both the legacy executor and the native SubagentWorker via build_subagent_messages), then single-brace {param} against the LLM's tool-call arguments. The double-brace pass must run first, or a {{car}} leaks its inner {car} into the arg pass.

Both draw on the same session_metadata + session_variables merge the parent stage resolves (filled by the tester's Template Variables panel via the X-Template-Variables header, or by call context). Unknown tokens stay literal.

Subagent visibility uses the existing conversation-api event channel. UniversalPipelineBuilder stores the active ConversationAPIPublisher on runtime_context["conversation_api_publisher"] and exposes runtime_context["subagent_event_emitter"]. Sequential, parallel, worker, and session-switch paths emit PIPELINE_EVENT entries with source="subagent", frame_type="SubagentInvocation" or "SessionSwitch", and details_json.kind values such as subagent_invoked, subagent_completed, subagent_failed, agent_active_changed, and agent_transfer_failed. Each event includes a distinct details_json.turn_id (subagent:{invocation_id}:{kind}) so the event sanitizer does not collapse lifecycle phases. Payloads include safe identifiers, execution mode, status, duration, and active-agent names only; prompts, raw instructions, and large tool results stay out of these pipeline events.

Built-in tools

A service definition's tools field can list builtInTools — entries the picker shows as builtin:<id> selections (e.g. builtin:end_session), distinct from configured-tool UUIDs. Configurable built-ins store rich config under ServiceStage.config.builtin_tool_configs[<unprefixed id>]. Most built-ins bypass configured-tool UUID resolution and each needs its own runtime wiring. builtin:inject_message_to_conversation is the deliberate exception: resolve_builtin_mcp_tool_configs adapts it to one filtered internal agent-host MCP operation without creating a tenant-visible tool_configs row. Currently wired:

Built-in	Wired where
builtin:end_session	end_session_tool_helper.py (all LLM/s2s providers). Always on since 2026-08-28 — should_enable_end_session_tool no longer reads the authored tool list (an agent with no way to hang up strands the caller and runs a CCaaS leg to the platform timeout); the ONE carve-out is CES S2S, which ends its own session natively. The editor renders the picker row checked-and-locked (MANDATORY_BUILTIN_IDS in ai-manager's node-builtins.ts) without healing config.tools, and only the injected guidance honours a per-stage auto_injections override. The session_switch main-agent suppression is unchanged.
builtin:message_injection	Receiving capability, not a callable Tool. node_builtin_config.py synthesizes the policy; universal_builder.py passes it to ExternalInjectionProcessor; conversation-api uses the same synthesized policy for /inject authorization. Voice/chat, singleton owner.
builtin:inject_message_to_conversation	Callable sender for voice/chat agent stages. tool_config_resolver.py creates a virtual agent-host MCP config filtered to only inject_message_to_conversation, binds source_conversation_id from runtime, and optionally binds target_conversation_id from builtin_tool_configs. Workflow Tool nodes use the unprefixed inject_message_to_conversation built-in registered by async-agent and call the same conversation-api endpoint directly. The legacy inject_message_to_voice MCP name remains only as a compatibility alias.
builtin:ivr_navigation	node_builtin_config.py reads the current service config; universal_builder.py wraps only that owning LLM/S2S service in IVRNavigator. Singleton owner.
builtin:warm_transfer	node_builtin_config.py warm_transfer_config_for_stage_config builds the legacy-shaped transfer dict from builtin_tool_configs.warm_transfer; universal_builder.py prefers it over the legacy agent-level config.transfer and wires the origin consult tools (consult_tool_helper.py). Voice-only, singleton owner. Per-destination inline briefing_script/ivr_navigation_goal ride the dialer brief (start_consult → write_brief).
builtin:transfer_agent	Marks the consult leg (Agent B). node_builtin_config.py transfer_agent_config_for_stage_config detects it on the stage; universal_builder.py then auto-injects the consult-side tools (add_consult_participant_tools: send_to_origin/request_merge/abort_consult) + the consult-participant prompt (overridable via builtin_tool_configs.transfer_agent.prompt_override). Also how the origin dropdown discovers transfer agents. Voice-only, singleton. Replaces the dropped agents.role_hint column; legacy config.transfer.role='consult' still works as a fallback.
builtin:send_dtmf	universal_builder.py. Accepts a single digit OR a sequence (digits='4815162342#', separators stripped, 250 ms inter-digit pacing, max 32) — sequences cover "enter your account number" prompts in one call. Legacy digit arg still accepted.
builtin:signal_dialer_ready / _failed	dialer_signal_tool_helper.py (warm-transfer dialer agents)
builtin:code_interpreter	aws_bedrock_inline_agent_services.py — Bedrock Inline Agent only

Google GECX node integration is also node-owned. Core GECX service params stay flat on the stage config (project_id, app_id, deployment_id, location, app_version, input_variables, tools). TrueServe integration extras live under the legacy key config.ces_integration: mcp_url_catalogs, warm_transfer_profile, tool_dispatch, and child_agents. The GECX MCP URL route derives its URL from the first GECX node, not from retired top-level agent config.

Don't offer a provider "hosted" tool as a built-in unless its bridge exists. OpenAI hosted tools are wired only for OpenAI / Responses API LLM stages: web_search, file_search, code_interpreter, and image_generation are passed through to /v1/responses, with response metadata normalized back into TrueServe output fields. They are intentionally not offered on Streaming Completions or Realtime stages. A hard constraint compounds this for voice: the OpenAI Realtime tool union only accepts function and mcp types (openai/types/realtime/realtime_tools_config_union_param.py) — web_search is a Responses tool and cannot be passed through to a Realtime session at all. Google hosted tools still require their own runtime bridge before they should be shown.

Inline tools (agent-local Python + API + Knowledge)

Agent-local inline tools live inside a stage's config.inline_tools[] and are selected from config.tools as inline:<local-id>. They are intentionally not tool_configs rows, so they are not visible in the shared Tools area and cannot be reused outside that agent stage. Three kinds share the array, discriminated by type_id: python_sandboxed (inline Python), openapi with config.subtype == "manual" (inline API), and knowledge (inline Knowledge source search). The single inline_tools array rides inside the already-allowed config key, so the collab-gateway Yjs shape persists it without a per-key allow-list entry.

ai-manager validation enforces that contract before save/import: config.tools must contain string refs only, and every inline:<local-id> ref must have a matching object in the same stage's config.inline_tools[] with id == "<local-id>". Bare inline ids are invalid; the picker will surface them as unresolved rows, and the runtime resolvers skip unresolved refs because there is no schema or callback to register.

Inline Python. Runtime wiring is in inline_python_tools.py. resolve_inline_python_tool_configs resolves the matching definitions from stage config, add_inline_python_tool_schemas appends their FunctionSchema entries before service construction, and register_inline_python_tools registers callbacks with the real LLM service beside MCP / saved Python / workflow tools. Callback execution posts to code-executor with no integration grants and an entrypoint of run (or legacy execute). The editor-facing form encourages type_id: "python_sandboxed" with config.source defining async def run(args, context), where args is the tool call payload and context.vars is read/write; code-executor returns a vars_patch that's merged into runtime_context["session_variables"]. Older generated drafts that used type_id: "python" plus config.code/def handler(...) are adapted to the run(args, context) contract when loaded, and the editor saves them back in the current config.source shape. The inline Python form can run the same source through ai-manager's inline test endpoint before it has a shared tool_configs id. Saved shared Python tools referenced by UUID are also adapted into this shape at runtime, so python_sandboxed rows and agent-local inline definitions share one schema/callback path.

Saved Python UUID resolution only sends canonical UUIDs to the tool_configs query. Built-ins, inline refs, and legacy/non-UUID tool strings are ignored by this resolver and left for their own paths; otherwise one non-UUID value in the list would make Postgres reject the entire uuid[] cast and hide the saved Python tool from the LLM schema.

Inline API. The inline counterpart of a manual-subtype API tool, wired the same way in inline_api_tools.py (resolve_inline_api_tool_configs / add_inline_api_tool_schemas / register_inline_api_tools, all wired in universal_builder.py beside the Python ones). Each callback makes a single SSRF-guarded HTTP request: {var} placeholders in the URL/headers/body resolve from the LLM args first, then the session-variable context (build_variable_context → conversation_id, ani, dnis, tenant_id, …); an optional response_path (JSONPath-lite) trims the response. Because an inline tool has no tool_id, the inline tool_secrets path is unavailable — auth is integration-backed only (none/integration/oauth2, credential read via get_credential_by_integration_id).

Saved manual API. The standalone, tool_configs-backed manual-subtype API tool (referenced from config.tools by its UUID) resolves through resolve_saved_api_tool_configs in the same file, which fetches the row via _fetch_tool_configs, filters to type_id='openapi' + config.subtype=='manual' (the spec subtype and no-subtype legacy rows are skipped — they go to CES, not a single HTTP call), and folds the result into _inline_api_tool_configs so add_inline_api_tool_schemas / register_inline_api_tools wire it exactly like an inline one. This mirrors resolve_saved_python_tool_configs and is why a saved manual API tool now works in voice/S2S pipelines (e.g. Google ADK Live), not just async-agent (text) — before, the voice path had no resolver for openapi rows (the MCP resolver skips them), so the tool was never declared to the live session and the agent hung when told to call it. Because the saved row has a tool_id, it additionally supports inline tool_secrets (auth.source == "inline", resolved via _fetch_tool_secrets + _build_inline_auth_headers) on top of integration-backed auth. The same rows are still ALSO loaded by async-agent's db_loader._make_manual_api_handler for the text path; there is no double-registration in the orchestrator because the MCP/CES saved-tool resolvers all reject openapi.

Inline Knowledge. Inline Knowledge tools are created from the agent/workflow tool picker and store only non-secret Knowledge Source metadata (source_id, source_slug, source_type, public source config, schedule, and default_tool_method: "search") in config.inline_tools[] — never the connection: no server_url, no transport, no token. Runtime wiring is in inline_knowledge_tools.py. resolve_inline_knowledge_tool_configs(raw_config, tenant_id=…) resolves the session tenant's Knowledge settings (tenant_settings key knowledge, set in ai-manager under Settings → Tenant Settings → Knowledge) and returns configs carrying the slug + tenant; the callback re-resolves that live connection at call time, exposes the authored local function name to the LLM, and calls {base_url}/mcp/{source_slug} with the fixed method search and the tenant's Bearer token. When the tenant has Knowledge disabled or incompletely configured, inline Knowledge definitions are skipped. There is no env fallback.

All inline: kinds are filtered out of DB tool resolution (the tool_id_strings list excludes INLINE_TOOL_PREFIX), so they never hit the MCP resolver.

Tool input self-healing

After live tool callbacks are registered, universal_builder.py wraps them with tool_input_healing.wrap_registered_tools_for_input_healing. The wrapper uses the tool schema for preflight checks and interprets recoverable 400/422-style tool results that identify a caller-fixable field. It returns a structured ToolInputIssue envelope through the normal ToolCallEntry.result_json path with status: "needs_input", issue: "missing" | "invalid", fields[], prompt_hint, original_arguments, and retry metadata.

Caller-facing text uses parameter title/label/description first, then a humanized field name. Secret-like fields are skipped, and auth, SSRF, rate-limit, missing-secret, timeout, and 5xx-style failures stay normal errors. ConversationAPIPublisher stores the pending retry manager from runtime_context; the next chat message or final speech transcript is treated as the answer, patched into the original arguments, retried automatically, and fed back to the LLM as a synthetic context message. Default retry limit is two attempts per tool call.

Assistant Agent skills can define the same inline tool shapes inside a skill's inline_tools[], with selected refs stored in that skill's tool_refs[] as inline:<local-id>. expand_antigravity_skill_config merges those definitions into the stage's effective config.inline_tools[] before the inline Python/API/Knowledge resolvers above run, so skill-local inline tools execute through the same path as normal agent-local inline tools. Reusable library skills persist the definitions on agent_skills.inline_tools; node-local inline skills keep them inside config.inline_skills[].

A2A tool consumption (calling a remote agent)

A saved a2a tool (type_id 'a2a', config {agent_url, agent_card_url?, custom_headers?, supported_skills?, timeout_seconds?}) becomes a provider-agnostic callable tool on any LLM branch — OpenAI, Anthropic, Google, Vertex, Realtime/S2S. This mirrors MCP exactly:

• Resolve — resolve_a2a_tool_configs(tool_ids) in tool_config_resolver.py fetches the 'a2a' rows, resolving auth via the same inline-secret path as MCP (_build_inline_auth_headers) plus a legacy auth_token Bearer, and applying the dev-mode K8s→localhost URL rewrite.
• Fetch schema — a2a_manager.py A2AManager.get_tools_from_configs(...) uses the official a2a-sdk A2ACardResolver to fetch the remote Agent Card (trying /.well-known/agent-card.json, then the legacy /.well-known/agent.json), then maps each card skill to a FunctionSchema (a free-text message param). The open A2AClientHandle (built via ClientFactory(ClientConfig(...)).create(card)) is kept for callbacks — same lifecycle as MCPClient.
• Register — A2AManager.register_tools_with_service(handle, tools, llm_service) registers one llm_service.register_function callback per skill. register_function is the single point that makes it provider-agnostic — every pipecat LLM service consumes it identically. The callback sends an A2A message/send, drains the streaming Task/Message events up to timeout_seconds (default 30), and returns the terminal text (collapsing a long-running streaming task into a synchronous-ish tool result — required for Realtime branches).
• Wiring lives in universal_builder.py: _resolve_tool_schemas resolves + appends A2A tools into the same ToolsSchema; Phase 2 registers the callbacks beside the MCP loop. Handles ride on _ToolWiringContext.a2a_handles.

ADK vs generic split. The Google-ADK text-agent path keeps its native RemoteA2aAgent (adk_text_agent.py, reads inline remote_a2a_agent tool dicts) so it participates in ADK's own planner/runner. The generic a2a_manager path covers every other LLM branch. Don't try to unify them — a saved 'a2a' tool flows through a2a_manager everywhere; ADK branches additionally understand the inline dicts.

a2a-sdk must stay explicitly pinned in both apps/ai-orchestrator/pyproject.toml and apps/conversation-api/pyproject.toml. It also arrives transitively via google-adk, but the generic consume path and the expose server import it directly — a google-adk bump must not be able to drop it.

A2A agent exposure (serving our agents as A2A)

Chat Agents can be flipped on to serve a standards-compliant Agent Card + an A2A message/send endpoint so other agents (ours or third-party) can discover and call them. Voice, translation, and copilot agents are intentionally not exposed as A2A agents; the UI hides the toggle and the PATCH route rejects enabling it. The message endpoint is Bearer API-token-only (agents:manage scope), and the served Agent Card advertises the same requirement through securitySchemes. See conversation-api.md for the served routes. Expose state lives in agents.a2a_enabled / agents.a2a_metadata columns — never agent_versions.config (the collab Yjs allow-list would silently drop it).

Amazon Polly node and auxiliary voice

Amazon → Polly is available as an ordinary aws/tts/Polly node and uses the same form when selected for handoff_tts. Its service definition is services/tts/aws-polly.json, registered in the manager registry, provider metadata, and shared service exports. The existing runtime mapping and AWSPollyTTSServiceWrapper serve both paths.

Amazon integration profiles expose the text_to_speech capability; enable it for Polly. Legacy amazon_web_services profiles include this capability through the source mapping. The integration selector is tenant scoped. /api/voices/polly and /api/tts/polly/preview require ManageAgents, explicitly validate the selected active Amazon TTS integration against tenant/platform scope, and proxy to the orchestrator's bearer-protected /polly-preview. The voice catalog pages through DescribeVoices and supplies language and supported engines. Selecting a voice updates its language and keeps a supported engine; changing region or integration reloads the catalog and cancels stale responses. The preview synthesizes a fixed short sample as MP3. It follows the same engine preference as the runtime (requested engine when supported, otherwise generative/neural/standard/long-form), and the preview route is rate limited. Published forms are read-only.

Polly's runtime credential lookup reads only the explicitly selected integration. Absent keys use the workload IAM chain, without falling through to another database profile. No AWS calls occur merely from listing agents; the voice catalog loads when a Polly form is opened, and synthesis occurs only on Play sample. Verification uses mocked provider calls plus browser form/save tests; live voice quality still depends on the selected integration and regional catalog.

Adding a new service provider

This is where the seven-place duplication bites you. Adding a new LLM, STT, or TTS provider currently requires edits in three ai-manager files plus four ai-orchestrator files.

ai-manager (UI)

1. Service definition: add a JSON-shaped definition in lib/service-definitions/services/. This drives the inspector form fields.
2. Provider metadata: extend PROVIDER_OPTIONS and the relevant service map (LLM_SERVICE_TO_PROCESSOR_LLM, STT_SERVICE_TO_PROCESSOR_STT, etc.) in lib/provider-metadata.ts. Without this, the dropdown does not include your new provider.
3. Service definitions index: register the new file in lib/service-definitions/index.ts.

ai-orchestrator (runtime)

1. Service constants: add an entry in lib/pipecat/pipeline/builders/service_constants.py — the canonical name the runtime reads.
2. Service registry: extend service_registry.py with a builder function that returns the Pipecat processor.
3. Tool config resolver: if the provider has tool/function-calling specifics, extend the tool resolver under lib/pipecat/tools/.
4. Service constants used by tools: cross-reference any tool helpers (consult_tool_helper, dtmf_tool_helper) that key off provider names.

Warning: missing any one of these breaks at a different layer with a different error. Symptom decoder:

• Dropdown missing the provider → step 2.
• Form fields blank in inspector → step 1 or 3.
• No service mapping found for provider=…, type=…, service=… → step 4. get_service_mapping raises on a miss — it doesn't no-op. In a multi-branch (parallel) pipeline this aborts the entire build, so every branch goes dark: no session, no conversations row, no greeting, and the surface app shows no responses at all (it looks like a broken agent, not a broken stage). The agent that surfaces the symptom may not be the one that owns the unmapped service.
• No API key was provided / Invalid parameter 'api_key' from a Google service → step 5. The wrapper instantiated the wrong Pipecat class. Vertex AI is not GoogleLLMService with api_key="" — google-genai rejects an empty key. Use the dedicated GoogleVertexLLMService (pipecat.services.google.vertex.llm), which takes credentials/credentials_path + project_id + location and builds a vertexai=True client. See google_vertex_services.py.
• Tool calls silently drop → step 6 or 7.

This duplication is a known refactor target. Track via service-registration-duplication in agent memory.

Cascaded pipelines forward caller transcripts past the user aggregator

pipecat's LLMUserAggregator folds TranscriptionFrames into the LLM context and pushes nothing downstream. On an S2S pipeline the provider (Nova, OpenAI Realtime) sits downstream of the aggregator and emits its own transcript frames, so the ConversationAPIPublisher and every relay that hangs off it (consent evidence, idle timing, the procedure monitor, entity extraction) see the caller's words. On a cascaded pipeline nothing re-emitted them: agent 4306 stored zero user messages, the customer-auth consent check refused the verification link because it found no caller reply, and the procedure relay never fired — while the LLM answered normally. llm_context.py now swaps in TranscriptForwardingUserAggregator for non-S2S stages, which re-pushes final and interim transcripts downstream (LLM and TTS services ignore them). The InterruptionGate was made playout-aware in the same change: BotStoppedSpeaking means finished sending, so the publisher stamps _bot_playout_end_monotonic from its playback estimate and a barge-in during that tail is a real interruption.

Two more cascaded-only rules live in the same aggregator. (1) It holds a UserStoppedSpeakingFrame for up to 0.4 s until the final transcript that belongs to it arrives: Flux queues the transcript but broadcasts the stop, so the stop won and inference ran on a context with no new caller words — the model repeated its previous question verbatim and the transcript rode the next turn. (2) An out-of-band system append (customer-auth's "the caller has just completed identity verification… acknowledge it briefly") becomes a prefixed user notice, and the Bedrock service rewrites every notice except the newest message to the "already acknowledged" form on each invocation — Bedrock's adapter otherwise turns it into plain user text that the model re-acknowledges on every turn.

The assistant aggregator sits AFTER transport.output(), so the transport must forward text frames

pipecat's design is that the LLM context records what the caller actually heard: the assistant context aggregator (LLMAssistantAggregator, or S2SAssistantAggregator on an S2S branch) is the last processor, after the output transport, and it builds each assistant turn from the word-timestamped TTSTextFrames the transport releases as the audio plays, committing on the LLMFullResponseEndFrame the TTS holds until its audio is done. The gRPC output transport's text-out branch used to return for every TextFrame — written to the wire on a text session, dropped otherwise — so on a gRPC session no assistant turn was ever written to the LLM context. Agent 4306 (Deepgram Flux → Bedrock Claude → ElevenLabs Text to Dialogue) made it visible: the model never saw its own previous sentence, asked the same consent question seven times in a row to a caller answering "Yes. Go ahead." each time, and re-narrated "you're verified" / "your appointment is confirmed" turn after turn. The tell was the per-turn [Bedrock] usage line (no cachePoint sits on the messages, so prompt= counts the whole conversation): it grew by the caller's five to ten tokens per turn and never by the agent's twenty-two. S2S providers commit their spoken text the same way (Nova Sonic defers a TTSTextFrame to each audio contentEnd), so their context had no assistant turns either — which only shows at a session-continuation replay.

GRPCOutputTransport._forward_text_frame_downstream now hands every text frame to BaseOutputTransport.process_frame after the wire decision: a frame with pts waits in the clock queue and is pushed downstream when its word is spoken, anything else goes in audio order; write_transport_frame stays a no-op, so nothing reaches the wire twice, and the wire rules (text_out_enabled, text_out_translation_only) are unchanged. Regressions: test_grpc_output_text_forwarding.py (including a pin on the stock aggregator committing from TTSTextFrames) and the transport cases in test_translation_transcripts.py.

ElevenLabs Text to Dialogue: keepalives, closed contexts, and dying sockets

TTD is a multi-context WebSocket. Three rules the 4306 live tests earned: (1) a context's first message must be its voice-registration init, so flush/close_context are skipped for a context that never registered (a turn whose LLM errored before any text); (2) keepalives address only contexts that are still REGISTERED — a keep_alive for a closed context (still streaming its tail) is answered "Context … is closing and cannot accept more messages" and the server closes the socket, cutting the sentence mid-word (both cut-offs landed on the 10 s keepalive ticks); (3) when the socket dies, _disconnect_websocket finishes every open audio context with an error frame — the base pauses downstream frame processing while a context generates, so a context that can never complete stalls everything behind the TTS (a tool result never reached the assistant aggregator and the agent went silent for 25 s).

Bedrock model ids are coerced to inference profiles

Claude 4.x/5 models are inference-profile-only on Bedrock: list_foundation_models reports inferenceTypesSupported: ["INFERENCE_PROFILE"] and a bare anthropic.claude-sonnet-5 fails every ConverseStream call with "Invocation of model ID … with on-demand throughput isn't supported" — which on a voice agent is total silence, greeting included (agent 4306). Because both the static service-definition options and the live /api/models/bedrock catalog hand out bare ids, AWSBedrockLLMServiceWrapper.create runs coerce_bedrock_model_id: a bare anthropic.* id gains the region's cross-region profile prefix (us. for us-/ca-, eu., apac. for ap-*, global. when the region is unknown); prefixed ids, ARNs and on-demand families (Nova, Llama, Mistral) pass through untouched. The coercion is logged at INFO.

Sampling parameters are generation-specific too, and a rejection fails every turn the same way. Probed live (2026-09-04): Claude Sonnet 5 rejects temperature and top_p outright ("is deprecated for this model"); Sonnet 4.5 / 4.6 and Haiku 4.5 accept either but not both ("cannot both be specified"); older Claude models take both. sanitize_bedrock_inference_params applies that table before the service is built, and TrueServeAWSBedrockLLMService retries a rejected ConverseStream once without the named parameter(s), persisting the drop into its settings — so an unknown future model costs one failed round-trip, not a silent call.

The service is built from Settings (build_bedrock_settings) with enable_prompt_caching for Claude ids: the static system prompt + tool schemas (~8k tokens on a voice agent) become a Bedrock cache read after the first turn. Verified live with a system cachePoint on Sonnet 5 and Haiku 4.5; other families are left uncached because an unsupported cachePoint is a hard 400.

Claude 5.x models think adaptively by default. Bedrock streams that reasoning as reasoningContent deltas, which pipecat's Bedrock service ignores (it handles only text and toolUse), so on a voice agent the thinking is pure dead air: tool-deciding turns of 5.7–18 s billing 361–1265 completion tokens for a bare tool call, while a 5-token probe answers in ~2 s. default_thinking_off disables it (additional_model_request_fields.thinking = disabled) for Claude ≥ 5 unless the stage configured reasoning_effort. Even so, Sonnet 5's floor is ~2 s per round-trip against Haiku 4.5's ~0.8 s — the model choice dominates cascaded voice latency once caching and thinking are handled.

Drift guards

apps/ai-orchestrator/tests/test_service_registration_drift.py checks the joins between those seven places on every PR. It reads the ai-manager sources as text (same pattern as payments-mirror.test.ts reading Python from the other direction) so there is no mirrored copy to forget, and it asserts four things:

Join	What breaks without it
SUPPORTED_PROVIDERS ⊇ every SERVICE_LOOKUP_V2 provider	The provider gate runs before the lookup, so a provider missing from the set kills every row it owns — Unsupported service_provider '…' one line above a correct mapping.
Every mapped service resolves in ServiceRegistry	A _try_import block returns a whole family from one import, so one bad name drops the family with nothing but a suppressed log line.
Every row survives build_service_specific_config	Catches a service_type the PipelineServiceStep union doesn't accept.
Every name the editor can write resolves in SERVICE_LOOKUP_V2 or TRANSLATION_SERVICE_LOOKUP	The user-facing direction: the dropdown offering something the runtime refuses. schema.ts's validateServiceCombination reads PROVIDER_OPTIONS, so such an agent passes validation and then fails to build.

Resolving through either runtime table is the rule, which is why there is no hand-kept list of expected misses: the translation-flow-only services (Krisp Voice Translation, LLM Translation, Phrase Splitter) legitimately live in TRANSLATION_SERVICE_LOOKUP alone.

Two things the guard found on its first run, both of which had been true in every environment for months:

• _load_ocr_providers imported a CustomHTTPOCRWrapper that does not exist, so _try_import swallowed the ImportError and all four working OCR providers were absent from the registry — every ocr stage failed to resolve, evidenced only by a DEBUG line. custom_ocr_http is in ai-manager's RETIRED_INTEGRATION_TYPE_IDS; the dangling name, its lookup row, and its PROVIDER_OPTIONS entry are gone, and _try_import now logs at WARNING.
• The two authoring surfaces speak different service-name vocabularies. The editor writes the persisted name from PROVIDER_OPTIONS ("Nova-3", "TranslationTextBridge"), while the generation MCP hands the model definition.service verbatim ("Nova", "Text Bridge") — compactServiceDefinition. Four definition names matched no row at all, so a generated agent could name a service the builder had never heard of. SERVICE_LOOKUP_V2 now carries an alias row for each definition name next to the persisted one.

Adding a provider that only one surface knows about is now a failing test rather than a runtime discovery.

ElevenLabs TTS and Text to Dialogue are different protocols

ElevenLabs stages expose three deliberately separate service choices. WebSocket API uses the ordinary /v1/text-to-speech/{voice_id}/… protocol for Flash, Turbo, and Multilingual v2 models. HTTP Streaming is the complete-text streaming path. Text to Dialogue is the Eleven v3-only realtime protocol: it registers a voice, sends incremental inputs[], and receives snake_case audio/alignment events.

TrueServe implements Text to Dialogue with ElevenLabs' /v1/text-to-dialogue/multi-stream-input variant. Each Pipecat LLM turn owns one TTD context_id; LLMFullResponseEndFrame sends close_context to flush short buffered text, and interruption closes the same context so stale v3 audio cannot play after barge-in. Translation flows have no LLM turn frames, so they pin one context for the continuous translated-text stream and rotate it after interruption instead of consuming one of ElevenLabs' five context slots per fragment. The WebSocket remains open for the call, with a context-scoped keepalive before ElevenLabs' 20-second inactivity timeout. PCM is required because compressed streaming codecs can cross the provider's turn boundary. The implementation is elevenlabs_text_to_dialogue.py, wired by ElevenLabsTextToDialogueTTSServiceWrapper; protocol tests live in test_elevenlabs_text_to_dialogue.py.

Versioning and publishing

agent_versions carries draft (version 0) and published versions (1+). The runtime reads only published versions in normal flow. ai-manager's "Publish" button calls publish_agent_version(); A/B traffic split uses resolve_agent_version_weighted(). See sql-functions.md.

A draft is editable; a published version is immutable. To roll back, publish a new version that copies an older one's config.

Relating to other processes

• Real-time translation — uses the translation subtype builder.
• Warm hand-off — relies on the transfer block + consult_tool_helper.
• LangGraph construction — async-agent uses a similar but distinct config shape (workflow_versions.config, not agent_versions.config).

Branching behavior (speech swimlane audio mixing)

A pipeline can fan out into two or more parallel S2S branches — e.g. two voice models that both answer the same caller. The config shape is a parallel group: pipeline: [ [[branchA…], [branchB…]] ] (a list-of-lists at one index), optionally wrapped in the swimlane_group envelope with a join_policy.

Audio at the join is reconciled by TurnAwareMixer (apps/ai-orchestrator/lib/pipecat/swimlane/turn_aware_mixer.py), inserted as the first post-parallel processor. Pipecat's stock ParallelPipeline only interleaves frames — it does not PCM-sum overlapping audio — so without the mixer two simultaneous speakers come out garbled. The mixer is now inserted whenever audio: "mix" and there are ≥2 branches (previously it was skipped for mix/none, which is why dual-S2S calls produced no usable audio).

To tell the branches' audio apart (frames carry no swimlane identity), each branch chain gets a SwimlaneAudioTagger at its tail that stamps frame.metadata["swimlane_id"]. The mixer reads that tag, runs a tiny jitter-buffer window (MAX_HOLD frames), and PCM-sums one chunk per lane via pipecat.audio.utils.mix_audio. A single active speaker takes a zero-latency passthrough fast path; a silent peer can't stall the speaker (window flushes after MAX_HOLD).

Join policy (join_policy on the group, or settings.swimlane as the default for bare groups):

Field	Values	Effect
turn_strategy	none (default) / explicit_protocol / barge_in	none = free-flowing overlap (no TurnManager/TurnGate). The others arbitrate one speaker at a time; barge_in also lets the caller preempt (requires a registered cancel adapter per speaking service — openai/s2s and google/s2s are built in).
audio	mix (default) / first_speaker / drop	mix sums every branch; first_speaker passes only the token holder; drop discards branch audio (analysis-only).

UI: the join policy is editable from the Audio Input block inspector (speech_to_speech agents only), alongside the "When the conversation starts" lifecycle workflow picker. It writes settings.swimlane. The orchestrator applies settings.swimlane as the default join policy, and an explicit per-group join_policy envelope overrides it. Tests: apps/ai-orchestrator/tests/test_swimlane_mixer.py.

A parallel group loses the whole input chain — injection, recording, lifecycle

For linear pipelines, the lifecycle executor is constructed with the input context but appended after the service/tool processors and before transport output. End-session handlers emit EndFrame downstream from their own location; an input-only listener misses a normal AI goodbye or transfer. The executor dispatches on_complete once even if transport cancellation follows that frame, retaining its bound session-end parameters.

_insert_input_chain (universal_builder.py) builds three things beyond transport.input(): the ExternalInjectionProcessor, the ConversationRecorderProcessor, and the lifecycle WorkflowExecutor. It is called from _build_linear_pipeline only. _build_parallel_pipeline appends transport.input() itself and never calls the helper, so any config with a parallel group silently loses all three, and the branch is taken purely on has_parallel = len(parallel_branches) > 0 — one list-of-lists at any pipeline index is enough.

The consequences are all silent, and none of them look like a pipeline problem:

• Message injection doesn't exist. POST /conversations/{id}/inject still returns HTTP 200 with delivered: 0 (nobody subscribed to the Redis channel), so the caller's own logs read "sent, nothing listening" and blame the session for having ended. A single-branch version of the same agent works, which makes this look intermittent.
• No call recording. The agent-audio track is never captured (store_audio is decided inside the helper).
• config.lifecycle.on_complete never fires. on_start usually survives by luck — main.py runs it before the build and stashes _pre_conversation_on_start_workflow_id — but nothing else runs on_complete, so a wrap-up workflow attached to a parallel agent just doesn't happen.

Practical rule when authoring: an agent that must receive injected messages (HITL push-back, advisor instructions) or fire an on_complete wrap-up workflow has to be single-branch. Two or more text_chat LLM branches also trip the multi-agent text-turns path (is_text_turns, below), so a chat agent that needs injection cannot be a turn-taking group. Fixing this properly means factoring the recorder / lifecycle / injection construction out of _insert_input_chain and calling it from both builders — the injection registry seeds from assembly._llm_service_index, which a parallel build has per-branch rather than once, so it isn't a one-line move.

First-class session variables (GECX, ElevenLabs, Agentforce, Maven AGI)

Some providers accept host-pushed session variables the remote agent can reference and forward to its own MCP tools / prompt templates: Google GECX (config.input_variables → REST runSession inputs[].variables, the only channel that populates $context.variables), ElevenLabs Agent (config.dynamic_variables → WS init), Salesforce Agentforce Voice (config.input_variables → Agentforce start-session variables[]), and Maven AGI Chat / Voice (config.input_variables → Maven transient/custom data). Both the voice CESLLMService and text CESTextLLMService wrappers use input_variables; Agentforce variables must be API-settable in Salesforce.

Operators author the map in the agent editor — the Session variables / Dynamic variables field on the stage (a variable_mapping param type rendered by VariableMappingEditor.tsx). Each row maps an external variable name → either an optional default value or a $var:NAME runtime binding. Unbound rows are treated as declared session inputs; ai-manager surfaces them in the Preview panel's Template Variables card through collectDeclaredSessionVariableDefaults, and external callers can provide the same names through session metadata / X-Template-Variables. The stored shape is a flat Record<string,string> inside the stage's nested config (preserved opaquely by the collab Y.Doc round-trip — no allow-list at the nested-config level; a regression test guards it in pipeline-shape.test.ts).

At session start, _build_service_specific_config in universal_builder.py resolves input_variables/dynamic_variables via resolve_session_variable_mapping(map, build_variable_context(...)) (binding_utils.py). Explicit $var: bindings resolve exactly as before. Non-binding values are defaults: when runtime/session metadata contains the external variable name, that value wins; otherwise the configured default passes through unchanged. Unknown explicit bindings still pass through unchanged. The wrappers then add their auto-injected conversation_id (+ GECX, Agentforce, and Maven AGI orchestrator_session_id). Agentforce also uses that TrueServe conversation id as the Agent API externalSessionKey and sends every configured input_variables key as a Salesforce $Context.* Agent API variable (Caller_Name → $Context.Caller_Name, ani/dnis → $Context.ANI/$Context.DNIS). Maven AGI receives the stage conversation id as _conversation_id/conversation_id metadata so chat and voice sessions can correlate with TrueServe conversation history.

Binding ids are snake_case (conversation_id, agent_id, tenant_id, ani, dnis, plus currentDateTime and any runtime session_variable) because the GECX/ElevenLabs path resolves through build_variable_context(camel_case=False). This is distinct from the camelCase ids (conversationId…) used by the tool-parameter binding picker, which resolves through a camelCase context. The two pickers source from separate lists in known-variables.ts (SESSION_VARIABLE_BINDINGS vs KNOWN_VARIABLES) — don't conflate them.

Agent tool variable mappings

Configured MCP/workflow/A2A tool parameters can be prefilled by agent-stage Variable Mapping in ToolSelector.tsx. The stored config still uses $var:<name> bindings under parameter_bindings, but the editor displays and accepts the prompt-style form ({{conversationId}}, {{agentId}}, {{tenantId}}, {{currentDateTime}}, {{ani}}, {{dnis}}) through known-variables.ts. Keep display syntax and prompt syntax aligned; $var: is an internal persistence/runtime marker only.

conversationId / conversation_id bindings resolve to the persisted conversations.id UUID (runtime_context.conversation_db_id) whenever a conversation row exists. Provider-native channel identifiers such as a Genesys conversation id remain in runtime_context.conversation_id, session_id, and conversation metadata, but must not be sent to UUID-addressed Conversation API routes or CRM fields used to reopen/inject into a TrueServe conversation. UI test calls generate a UUID before connecting and therefore historically masked this distinction; the binding resolver retains the channel/session fallback only for pre-persistence and non-conversation execution contexts.

The picker separates global tool defaults from node-level mappings. Global mappings are read-only in the agent editor, while node-level mappings are grouped under each enabled operation so authors only see variables for operations the agent can call. Runtime-only system values (conversationId, agentId, tenantId, currentDateTime) are suggested in prompts and mappings but are not user-editable Preview template variables. Caller/session values such as ani and dnis remain user-settable in the test surface.

Turn detection and first-audio latency

_build_llm_context (apps/ai-orchestrator/lib/pipecat/pipeline/builders/universal_builder.py) attaches a UserTurnStrategies to every LLM-like service's aggregator. The choice differs by service type:

• S2S (s2s, OpenAI realtime/s2s_llm) — gets empty start/stop strategies. The upstream model (ADK Live, OpenAI Realtime, AWS Nova Sonic) does its own server-side VAD and turn detection on the raw audio we forward, so a local detector is redundant.
• Cascaded STT→LLM→TTS — keeps pipecat's default strategies (VAD + LocalSmartTurnV3), because it has no server-side turn detection to lean on.

ConversationAPIPublisher does not read Pipecat observer internals directly. It attaches TurnEventBridge, which normalizes TurnTrackingObserver.on_turn_started / on_turn_ended into the existing conversation-api pipeline events and leaves sanitizer filtering intact. S2S and translation paths may still disable Pipecat turn tracking intentionally.

The UserTurnStrategies(start=[], stop=[]) trap. Pipecat 1.2's UserTurnStrategies.__post_init__ treats falsy lists as "unset" (if not self.start:) and overwrites them with the defaults — and default_user_turn_stop_strategies() instantiates LocalSmartTurnV3 in its constructor. So passing start=[], stop=[] does the opposite of what it reads like: it silently loads the smart-turn ONNX model. That model is a ~12 s cold load on the first session after an orchestrator boot (it also import-triggers transformers, which prints the PyTorch was not found banner). For an S2S agent that delay lands before first audio — symptom: a normally-1 s greeting takes ~20 s. To get genuinely empty lists, allocate via UserTurnStrategies.__new__(...) and set .start/.stop yourself, bypassing __post_init__. Do not "simplify" this back to UserTurnStrategies(start=[], stop=[]).

Startup warmup (so even the first cascaded call is fast). The ~12 s is almost entirely the one-time, process-global module import (transformers + onnxruntime); the per-instance ONNX session is ~0.05 s. So serve() in apps/ai-orchestrator/main.py schedules _warm_smart_turn_model() as a background task after server.start() — it loads the model on a thread while the gRPC port and health check are already SERVING. By the time the first cascaded call builds its turn detector, the import is paid and construction is instant. Look for [Warmup] Smart-turn model ready in N.Ns in the orchestrator log on boot. Set AI_ORCHESTRATOR_WARM_SMART_TURN=0 to disable (reverts to lazy load on first use). S2S agents skip the local detector entirely, so the warmup doesn't affect them either way.

Google ADK Live: instruction must be an InstructionProvider, not a string (why {{name}} killed all audio)

A Google-ADK S2S agent (services/google/adk.py ADKLiveLLMService) builds an ADK Agent from the resolved system instruction. ADK's Agent.canonical_instruction runs its own {var} templating (google.adk.utils.instructions_utils.inject_session_state, regex {+[^{}]*}+) — but only when instruction is a plain str; for any unresolved placeholder it raises KeyError: Context variable not found: <name>. That exception kills _recv_loop, so the live model never emits audio (the session connects, the greeting is queued, but nothing comes back — symptom: "tested the speech agent, no audio").

TrueServe prompts routinely carry literal {{name}} / {{summary}} placeholders — resolved upstream at session start, or left literal in a test session with no bindings. ADK's regex matches {{name}}, extracts name, finds it absent from session state, and raises. The fix: pass the instruction as an InstructionProvider callable (Callable[[ReadonlyContext], str]) instead of a string. canonical_instruction returns bypass_state_injection=True for callables and skips the regex entirely, so our placeholders pass through untouched. Don't revert instruction= to the bare string. The greeting (sent as LiveRequestQueue.send_content) never goes through instruction templating, so it's unaffected.

S2S + downstream TTS: overlapping voices are allowed, but warned

An S2S/realtime model (OpenAI Realtime, Google ADK Live, AWS Nova Sonic, Deepgram/Moshi/Salesforce voice agents) speaks natively — voice in, voice out — and can emit the reply as text. When an agent author also drops a TTS node after the S2S node (for example: keep the realtime model as the brain, but layer a premium ElevenLabs/etc. voice), both can produce audio: the model's native PCM stream and the TTS stream may overlap. That can be an intentional authored effect, so the runtime must preserve the graph exactly instead of rewriting stage config.

ai-manager surfaces this as a non-blocking warning, not a build-time mutation. validateUniversalPipelineConfig warns for common accidental overlap patterns: an S2S/realtime stage with native audio enabled followed by downstream TTS, multiple TTS stages in one linear branch, and multiple audio-producing branches in the same parallel/swimlane group. The agent can still be saved, tested, and published. If only the downstream TTS should speak, the author must explicitly disable Enable Voice / config.output_audio / config.voice_enabled on the S2S stage, or narrow the stage's output_frame_types so it no longer emits audio; the orchestrator will not infer that intent.

AWS Nova Sonic supports a text-only operating mode through config.output_audio=false. The service definition preserves that flag when the provider/model picker changes, hides voice selection while voice output is off, and shows a Nova-specific warning because Amazon still processes the call through the bidirectional voice API. At runtime the builder inserts NovaSonicTextOnlyGate after the Nova service when audio is disabled: audio frames from Nova are dropped, but text frames continue downstream so a later TTS node can own the spoken voice. Validation warns when a text-only S2S branch has no downstream TTS, because that agent will answer silently on a voice call.

S2S user-transcript persistence (why the caller's turns show up)

S2S services (OpenAI Realtime, GECX/agent-assist, Google ADK, …) emit the caller's input transcription as a TranscriptionFrame pushed UPSTREAM — see pipecat's services/openai/realtime/llm.py (FrameDirection.UPSTREAM). ConversationAPIPublisher sits downstream of the S2S LLM, so an upstream-traveling frame never reaches it: the agent's speech persists (via the normalizer + LLMFullResponseEndFrame finalize), but the user's never does — the Transcript tab shows only the agent, and conversation_messages has zero speaker='user' rows.

Before pipecat 1.0, _build_s2s_service_chain placed a TranscriptProcessor.user() upstream of the LLM that caught the upstream user frame and re-emitted it downstream as a TranscriptionUpdateFrame. Pipecat removed TranscriptProcessor; commit 164cd736 (2026-05-14) dropped it from the s2s chain without a user-side replacement — silently regressing every S2S provider's user-turn recording (cascaded STT→LLM→TTS was unaffected: its STT pushes TranscriptionFrame downstream already).

The fix is S2SUserTranscriptBridge, inserted immediately upstream of the S2S LLM by _insert_s2s_user_transcript_bridge in _build_s2s_service_chain. It intercepts the upstream final user TranscriptionFrame (skipping bot-tagged user_ids like moshi/ces/bot), mirrors it downstream as a TranscriptionUpdateFrame(role="user") that the publisher's existing TranscriptionUpdateFrame handler persists, and forwards the original upstream frame untouched so the context aggregator still sees it. Bot/agent transcripts are not touched here. Diagnose a recurrence by grepping the orchestrator log: RECEIVED input_audio_transcription.completed present but no Mirroring upstream user transcript / Received TranscriptionUpdateFrame = the bridge isn't placed.

Branching behavior (text turn-taking chat)

A text_chat agent with 2+ parallel LLM branches is a turn-taking group, not an audio swimlane. Configured in Settings → Branching Behavior (settings.text_turns: mode ∈ concurrent_serialized | sequential_shared, order ∈ completion | config, allow_silence). Gated to text_chat + 2 branches in the UI.

Why the audio path doesn't apply. The parallel builder's audio TurnAwareMixer only handles OutputAudioRawFrame; text branches emit LLMTextFrames. Worse, each branch's assistant aggregator consumes its LLMTextFrames (turning them into LLMContextFrame) — so they never reach the join. Turn-taking therefore happens inside each branch, not at the join.

Architecture (text_turn_serializer.py + wiring in _build_parallel_pipeline / _build_branch): - A shared TextTurnCoordinator (one per session) arbitrates the "floor". - A per-branch TextTurnGate is inserted right after the LLM, before the assistant aggregator (the only point the branch's text-response frames still flow). It takes the floor on LLMFullResponseStart, streams/buffers the turn, stamps the branch's agent identity ({id,name,color}, assigned by branch index) onto each frame, and on End releases the floor so the coordinator flushes the next waiting branch's buffered turn (in completion or config order). One agent's full turn emits before the next — never interleaved. - A per-branch ConversationAPIPublisher sits after each gate (the single join publisher can't see branch text). Only branch 0's publisher persists the shared user transcript (suppress_user_transcripts=True on the rest) to avoid duplicates. - Agent identity rides the envelope metadata (agent_id/name/color) → conversation-api merges it into conversation_messages.metadata + the live/replay SSE data.agent → surface-app renders each agent with a distinct avatar (initials on the accent colour), a name label, and an accent-tinted bubble (left border), and never merges two agents into one bubble (agentInitials/agentTint helpers + the message render in app/[locale]/page.tsx).

Modes. - concurrent_serialized: all branches answer in parallel (independent contexts); the gates serialize output. - sequential_shared: after a branch's turn flushes, the coordinator injects it as a system note into every other branch's LLMContext, so a later agent can build on earlier replies. Requires buffering (no live stream) so the full text is captured.

Silence (allow_silence). Each branch's prompt gets a [[SILENT]] sentinel instruction; an agent abstains by replying with only the sentinel, and the gate drops sentinel-only turns. Requires buffering so a token-streamed abstention can be suppressed.

Two text-input fixes the parallel path needs (don't revert). (1) Text branches use ExternalUserTurnStrategies, not the default — the default broadcasts an interruption on UserStartedSpeaking that, fanned out across a ParallelPipeline, cancels the just-started branch LLMs (empty LLMFullResponseStart→End, no text). (2) An InterruptionGate at each text branch head drops the bot-not-speaking interruption. Without these, multi-branch text agents produce no output at all.

Streaming + typing indicator. The floor holder streams its tokens live (the gate only withholds frames briefly while allow_silence can't yet rule out an abstention — until the text stops being a prefix of [[SILENT]]). Non-floor branches buffer and flush in order. surface-app shows a typing indicator (a transient {type:'typing'} timeline item) whenever the last visible message is the user's and the agents haven't started streaming yet.

Per-agent identity is designer-controllable. _derive_branch_identity prefers the LLM stage's config.agent_name / config.agent_color (set in the editor inspector's Agent Identity section — formerly "Multi-Agent Identity"), falling back to the stage label + an auto-assigned palette colour by branch index. surface-app renders each agent as a coloured initials avatar (name shown on hover, not inline).

On a text_chat agent, the editor seeds these fields on create only (when the user first picks an identity-bearing service): the Display name defaults to the stage label, and the accent colour is auto-picked from a color-blind-safe palette (AGENT_IDENTITY_PALETTE in agent-accents.ts, Paul Tol's bright/vibrant sets, mirrored by the orchestrator's _TEXT_AGENT_ACCENTS fallback) so it (a) never collides with another agent's colour in the same flow and (b) complements the agent's Branding when a surface config is selected — pickAgentIdentityColor(usedColors, brandColor) scores candidates for hue harmony (analogous/triadic/complementary) while enforcing a minimum separation from both the brand hue and every already-assigned colour. Seeding never overwrites a value the config or user already set; both fields stay freely editable. Existing stages opened in the inspector are not re-seeded (that would dirty the doc on a mere open).

Single-agent identity (no TextTurnGate). A single-agent chat is a linear pipeline, so no TextTurnGate is inserted and nothing stamps frame.metadata["agent"] — before this, the author's Agent Identity was silently ignored and every reply rendered as the generic "Agent" with the default avatar/colour. _build_linear_pipeline now calls _derive_single_agent_identity(services) and passes the result as default_agent_identity to the publisher. Unlike _derive_branch_identity (which always assigns a palette colour + generic name so branches stay distinct), this returns None unless the author explicitly set agent_name and/or agent_color on the LLM stage — an unconfigured single agent keeps the generic label, so no existing chat starts stamping an identity. ConversationAPIPublisher._publish_transcript uses the live per-turn stamp when present and falls back to _default_agent_identity otherwise (the per-turn stamp is cleared each turn, so a build-time default alone would only survive turn 1 — the fallback is what makes it persist). A multi-agent per-turn stamp always wins. The wire from there (envelope metadata → conversation-api data.agent → surface-app render) is identical to the multi-agent path above.

OpenAI Responses-API branch goes mute? Our OpenAIResponsesLLMService wrapper must stay in sync with pipecat: pass convert_developer_to_user, read the model from self._settings.model (not model_name; _settings is a dataclass with NOT_GIVEN sentinels), and create_response must promote an instructions-only bootstrap turn (the proactive greeting) into a developer input turn — the Responses API rejects an empty input with HTTP 400. See the openai-responses-pipecat13-breakage memory.

Tests: apps/ai-orchestrator/tests/test_text_turn_serializer.py (serialize / order / abstain / silence), apps/ai-manager/lib/collab/pipeline-shape.test.ts (settings round-trip).

Context summarization (long-conversation auto-compaction)

Without this, an LLMContext-based pipeline grows the message list unbounded for the life of a session — rising cost/latency per turn, and eventually a provider-side context_length_exceeded (or Bedrock ValidationException) that kills the agent. settings.context_summarization opts into pipecat's built-in auto-summarization to keep the context within the model's window.

Off by default. Configured in Settings → Context Summarization (a single toggle, stored as settings.context_summarization: true). Hidden in the UI for speech_to_speech agents.

Runtime wiring. _build_summarization_assistant_params (universal_builder.py) reads the setting from runtime_context["_pipeline_config"]["settings"] and, when truthy, builds LLMAssistantAggregatorParams(enable_auto_context_summarization=True, auto_context_summarization_config=LLMAutoContextSummarizationConfig(...)). These params are passed into the single LLMContextAggregatorPair construction site in _build_llm_context, so the toggle reaches every LLMContext aggregator the universal builder makes — cascaded STT→LLM→TTS and each parallel text-branch aggregator.

Why S2S is excluded. Realtime providers (OpenAI Realtime, AWS Nova Sonic, Gemini Live) own their conversation context server-side; the local LLMContext doesn't mirror it, so local summarization would compress the wrong thing. The if not is_s2s: guard at the construction site keeps S2S on bare LLMAssistantAggregatorParams(), and the UI hides the card for speech_to_speech.

Behavior + cost. When the running context crosses max_context_tokens (default 8000) or max_unsummarized_messages (default 20) new turns, pipecat asks the LLM to summarize the older turns into target_context_tokens (default 6000) and keeps min_messages_after_summary (default 4) recent turns verbatim. That summarization is an extra LLM call — negligible for text, but it adds turn latency in voice when it fires, so prefer it for chat and long-running sessions. Token counting is pipecat's ~4-chars/token heuristic (not a real tokenizer), so set max_context_tokens comfortably below the model's true window. The 1.3.0 param is enable_auto_context_summarization (the older enable_context_summarization is deprecated and removed in pipecat 2.0).

Config shape. true uses the defaults; the orchestrator also accepts an object form ({max_context_tokens, target_context_tokens, max_unsummarized_messages, min_messages_after_summary}) for threshold overrides — the current UI only writes the boolean. Rides the collab settings branch (a passthrough Y.Map), so no collab-shape allow-list change was needed.

Recording and compliance settings

settings.recording controls persisted conversation artifacts. Audio-capable agents show a Recording card with suppressed, store_audio, and store_transcripts. Text-only (pipeline_type='text_to_text') agents have no audio artifact and intentionally hide recording.suppressed; their settings card is labeled Compliance and only exposes recording.store_transcripts.

settings.compliance.redact_pii is shown as a dependent transcript option, not as a standalone card. It appears only when the agent config already contains a compliance block and recording.store_transcripts is enabled. Agents without a configured compliance block keep the redaction control hidden.

Audio capture: an S2S service MUST re-push the caller's InputAudioRawFrame downstream

There is one ConversationRecorderProcessor, appended by _insert_output_chain after the TTS/S2S services (post-TTS, pre-transport.output()) — see universal_builder.py. From that single position it captures both tracks: the caller (InputAudioRawFrame that flows through the service from transport.input(), written to caller.wav) and the agent (OutputAudioRawFrame/TTSAudioRawFrame the service pushes downstream, written to agent.wav); it also produces the stereo mixed.wav (L=caller, R=agent) that Activity Log playback uses.

This means an S2S service must forward the caller's InputAudioRawFrame downstream after handing it to the provider, or the recorder never sees caller audio and the recording has agent-only sound. Every stock pipecat S2S service does this by always re-pushing the frame at the end of process_frame (services/openai/realtime/llm.py, aws/nova_sonic/llm.py, google/gemini_live/llm.py). The TrueServe ADKLiveLLMService has a per-frame-type if/elif whose terminal else was the only branch that forwarded — so its InputAudioRawFrame branch swallowed the caller audio and the Google-ADK recording had the agent voice but silent caller (bug fixed 2026-07-20: the InputAudioRawFrame branch now calls await self.push_frame(frame, direction) after send_realtime). When adding a new S2S service or a new consumed input frame type, re-push it downstream unless you're certain nothing downstream (recorder, avatar, transport) needs it.

GoogleS2STService had the identical defect, with a comment asserting it was deliberate (# Don't push InputAudioRawFrame downstream - S2ST consumes it) — but "consumes" is about the provider, not the pipeline: consuming a frame and forwarding it are independent, and everything after the service still needs to see it. Fixed 2026-08-05. The tell is specific and worth recognising, because a translation call otherwise looks completely healthy: both caller.wav recordings are absent and channel 0 of both stereo mixed.wav files is digital zero (rms=0.0 peak=0) while the agent channel is fine. Since translation allocates one session per direction, the symptom appears on both legs at once, which reads as "recording is broken" rather than "this service drops a frame".

A2UI UI actions: an S2S service MUST consume the injected LLMMessagesAppendFrame

When a caller taps a button (or submits a form) on an A2UI surface, ConversationAPIPublisher._inject_user_text turns it into a synthetic user turn and pushes an LLMMessagesAppendFrame(messages=[{"role":"user","content":…}], run_llm=True) UPSTREAM (bracketed by UserStartedSpeakingFrame/UserStoppedSpeakingFrame so idle handling resets). Because the publisher sits downstream of the LLM/S2S service, that frame travels up through the service's process_frame — the service is expected to deliver the text into its live session and trigger a response. Cascaded LLMs get it for free (the LLMUserAggregator appends + fires on run_llm=True); OpenAI Realtime and Google ADK Live each have an explicit LLMMessagesAppendFrame branch.

AWS Nova Sonic did not. The base AWSNovaSonicLLMService.process_frame (services/aws/nova_sonic/llm.py) dispatches only LLMContextFrame / InputAudioRawFrame / InterruptionFrame, and on an already-connected session _handle_context merely re-checks completed tool calls — it never transmits a newly-appended user message. So a Nova Sonic S2S agent rendered a multimodal surface fine, but tapping its buttons did nothing: the click reached the orchestrator ([ConversationAPI] UI Action via Redis: … injecting into pipeline) and was silently dropped, so the agent never acknowledged it. Fixed 2026-07-24: SafeAWSNovaSonicLLMService overrides process_frame to call super() (which still dispatches the base frames and re-pushes downstream) and then, for a connected session, delivers each user-role message via _send_text_event(text, role=NovaRole.USER, interactive=True) — interactive=True prompts an immediate model response (the same mechanism the base class uses for the last user message on connect; current-gen Nova models need no separate assistant-response trigger). When adding a new S2S/realtime provider, make sure its process_frame consumes LLMMessagesAppendFrame too, or its multimodal UI actions will be dead.

Proactive greeting (and why the instruction must not persist)

When a conversational stage has greeting_enabled=true, the agent should open the conversation without waiting for the user. _harvest_greeting (universal_builder.py) records the intent into runtime_context (greeting_enabled, and message when an explicit greeting_message is set), and GRPCProvider._on_client_connected (grpc_provider.py) triggers it on connect.

For cascaded/text pipelines (not OpenAI Realtime, which uses response.create) the greeting is kicked by appending a system-role instruction — either the harvested greeting_message or a default "Please greet the caller…" — to the running LLMContext via LLMMessagesAppendFrame(run_llm=True).

The greeting instruction is one-shot and MUST be removed after the greeting turn, or the agent re-greets every turn. The append goes into the persistent context, and the LLM reads messages lazily at inference (the context frame holds a live reference, not a snapshot), so the instruction can't be removed synchronously — it has to survive until the greeting response is generated. _arm_greeting_instruction_cleanup registers a one-shot _GreetingInstructionCleanupObserver that waits for the first LLMFullResponseEndFrame (the reliable boundary — turn-tracking never ends turn 1 in a text-only chat until session timeout), then removes the exact injected message objects by identity (id(m)) via context.transform_messages and detaches itself. Identity removal is deliberate: tagging the message dict with a marker key would be forwarded verbatim to the chat API (the OpenAI/Anthropic adapters pass standard messages through unchanged) and rejected as an unknown property. Don't reintroduce a persistent greeting instruction without this cleanup, and don't switch the cleanup to a marker-key filter. (S2S/Realtime paths don't append into a reusable text context, so the cleanup is a no-op there.)

Bedrock requires the first message to be user-role — the greeting leaves a leading assistant turn, so the next turn 400s. A direct consequence of the one-shot cleanup above: with a proactive greeting the model emits the greeting as an assistant message, the cleanup strips the injected system instruction, and on the user's next turn the Bedrock-converted messages array leads with [assistant: greeting, …]. Bedrock's Converse API hard-rejects this — ValidationException: A conversation must start with a user message — so every post-greeting turn fails, no text reaches TTS, and the agent goes silent (it greets fine, then never responds). OpenAI / Anthropic-direct tolerate a leading assistant message, so this is Bedrock-specific. We use TrueServeAWSBedrockLLMService (subclass of pipecat's AWSBedrockLLMService, wired in aws_bedrock_services.py) whose _get_llm_invocation_params prepends a minimal placeholder user turn when the array isn't already user-led. Don't revert the wrapper to the bare AWSBedrockLLMService.

Greeting-off S2S: the system prompt must reach the session as a session-level instruction

With the greeting disabled on an S2S stage, the first model turn is produced by server-side VAD when the caller speaks — no response.create from us, and no LLMContextFrame has necessarily reached the provider service yet. For providers whose session config carries the system prompt (OpenAI Realtime session.instructions, ADK Live system_instruction, NVIDIA text_prompt), relying on the LLMContext path alone means the very first response(s) run with no instructions at all — the agent answers like a stock model and ignores its prompt (symptom: "S2S prompts ignored when greeting is off"; with the greeting on, the response.create path happened to sync context first, masking the gap).

Fixed 2026-06-11: _build_service_specific_config extracts the stage's system-role message (the standard UI messages format) into the provider's native session-instruction key before the framework-key strip removes messages — _S2S_INSTRUCTION_KEYS maps nvidia → text_prompt, openai → instructions, google → system_instruction (AWS Nova Sonic intentionally absent: it consumes the LLMContext messages directly). template.inline is promoted the same way for google and openai. The OpenAI Realtime wrapper's SUPPORTED_SESSION_PROPERTIES now includes instructions, so the first session.update (sent on session.created) already carries the prompt. Don't remove instructions from the supported props or narrow the extraction back to nvidia-only.

The OpenAI Realtime provider still needs a context frame at connect time. With a greeting, GRPCProvider._on_client_connected queues LLMMessagesAppendFrame(run_llm=True) to trigger the greeting; with greeting disabled it now queues the same frame with messages=[]. This does not inject a canned greeting. It primes the context/tool path deterministically so the session opens per the authored prompt instead of depending on a later incidental context frame. Don't remove the empty append on the assumption that server-side VAD alone is enough.

Assistant Agent (Codex / Claude Code / Antigravity): the Prompt is promoted into system_instruction

assistant_agent stages don't consume an LLMContext — the workspace-agent-runner runner builds each turn from system_instruction + the latest user text (messages is sent in the payload but never read by the runner), and all three wrappers map only config.get("system_instruction"). The agent editor's Behavior-tab Prompt editor, however, writes to config.template.inline (+ legacy config.messages). Left unwired, anything authored in the Prompt field was silently discarded.

_build_service_specific_config promotes the authored prompt into system_instruction for svc_type == "assistant_agent" — template.inline first, else the system-role messages entry — before the framework-key strip removes messages, and only when no literal system_instruction is already present. This runs after the {{variable}} substitution pass, so the promoted instruction is already variable-resolved.

The legacy fields were removed, not just hidden (we don't preserve legacy paths). One-time migration 006_assistant_agent_prompt_canonical.sql (recursive jsonb walker, idempotent, folded into migrate-city-db.sh) converged every stored assistant_agent stage config: system_instruction/messages → template.inline (Prompt wins if already set), and dropped runner_url, the raw skills object, and messages. sandbox is kept — it is an enforced runtime policy (the runner validates + applies it), not a dead field; it is merely hidden from the editor. In the three assistant_agent service defs: system_instruction/runner_url/sandbox/skills are "hidden": true (Runner URL auto-resolves from WORKSPACE_AGENT_RUNNER_URL; authored Skills compile into the skills object server-side). Model Override is now a per-service enum (curated Claude Code vs Codex options + an "Installed default" empty choice) under Advanced — see the editor-layout note. Don't re-add the legacy fields or narrow the promotion back out of the builder.

LLM provider quirks the wrappers must absorb (reasoning-model temperature, Google import guard)

Two provider-specific traps silently kill a text_chat/cascaded LLM stage at the first turn (the agent connects, the greeting is queued, then nothing renders — a permanent loading spinner — because the greeting turn itself 400s/errors and no text ever reaches the client):

• OpenAI reasoning-class models (gpt-5+, o-series) only accept the default temperature (1). The editor defaults temperature to 0.7 and leaves reasoning_effort empty, so a stage on gpt-5.5 sends temperature=0.7 and OpenAI rejects the request: 400 — 'temperature' does not support 0.7 with this model. Only the default (1) value is supported. openai_services.py strips temperature for any model matched by _is_reasoning_model (^o[1-9] or ^gpt-N with N ≥ 5) before it enters InputParams — independently of reasoning_effort (the earlier strip only fired when reasoning_effort was a non-empty string, which the UI usually leaves blank). model/voice/language are open-string fields by design, so an unsupported combination isn't caught at publish/preflight — only at the live call.
• The Google Gemini wrapper guards on GOOGLE_LLM_AVAILABLE, which must be defined. google_gemini_services.py imports GoogleLLMService under a try/except ImportError that sets GOOGLE_LLM_AVAILABLE. The flag was referenced by create() but never defined, so every Google text-LLM build raised NameError: name 'GOOGLE_LLM_AVAILABLE' is not defined and failed the dynamic build. Don't reference the flag without the guarded import that defines it.

Reasoning blocks: extracted, not spoken (ThinkTokenFilter → reasoning channel)

Reasoning models stream chain-of-thought inline with their reply, wrapped in XML-style tags — <think>…</think> (DeepSeek-R1, QwQ, NVIDIA NIM) or <thinking>…</thinking> (Bedrock Nova, Claude-style prompting). Left unhandled, those tokens flow straight to TTS and get spoken aloud, and they pollute the assistant transcript bubble.

ThinkTokenFilter splits the LLM stream into two channels rather than dropping the reasoning:

• Spoken text is re-emitted as the same concrete frame type that arrived (an LLMTextFrame stays an LLMTextFrame, via copy.copy + .text). This matters: the ConversationAPIPublisher's voice_ai transcript path keys on isinstance(frame, LLMTextFrame) — downgrading to a plain TextFrame would break the spoken transcript.
• Reasoning is re-emitted as LLMThoughtStartFrame(append_to_context=False) → LLMThoughtTextFrame → LLMThoughtEndFrame. LLMThoughtTextFrame is not a TextFrame subclass, so the TTS service ignores it (reasoning is never spoken) and append_to_context=False keeps it out of LLM history.

Invariants:

• It matches multiple tag conventions. <thinking> is not a substring-match of <think> (close tags differ: </thinking> vs </think>), so the filter carries a list of (open, close) pairs and matches whichever opens earliest. Adding only <think> silently misses Nova/Claude output. Tags split across streaming chunks (</thi + nk>) are handled via a partial-suffix buffer; a dangling unclosed block is closed at LLMFullResponseEndFrame.
• Placement is AFTER the LLM and BEFORE the ConversationAPIPublisher — _maybe_insert_think_filter runs just before _insert_publishers in the cascade chain builder. The publisher then sees clean spoken text on the transcript path and the separate LLMThought* frames. (This is the reverse of a naive TTS-only filter, which would sit after the publisher and strip reasoning from everything.)

The reasoning rides the transcript channel as variant="reasoning" — no proto change (the proto variant is a free-form string). The publisher accumulates each LLMThought* block and calls _publish_transcript(variant="reasoning", source="reasoning") with its own turn_id (streamed interim, finalized on LLMThoughtEndFrame). _derive_transcript_variant whitelists reasoning so it survives verbatim. conversation-api (grpc_api.py) persists the terminal reasoning row to conversation_messages (speaker agent, its own variant → no unique-index collision with the spoken turn) and (http_api.py) maps variant=="reasoning" to the SSE event type reasoning (live + on reload via event_transformer.py).

surface-app consumes the reasoning SSE event in sdk.ts (handleReasoningEvent, its own store + subscribeToReasoningEvents, upsert-by-id so streaming interims replace in place), and renders it via ReasoningItem — a muted, italic, dashed-border collapsible aside shown inline with the pipeline-event stream and gated behind the same "show events" toggle (showPipelineDebug). It is never mixed into the spoken assistant bubble.

Enabled by default for cascaded voice; opt out with settings.filter_think_tokens = false.

See also

• Data model
• SQL functions
• ai-manager file layout
• ai-orchestrator service doc
• ai-manager service doc
• Customer Authentication — the builtin:customer_authentication capability's tools + KBA observer wiring
• Pipecat-native alignment — draft analysis of subagents/text turns vs pipecat-native workers