Copilots (Agent Assist)

Status: Stable | Last reviewed: 2026-09-30

Workers can render in the Assist rail or Center workspace, selected by Display in in the editor (layout_region: "rail" | "workspace" | "context"). Composition context retains output for SDK/host subscribers without another visible card. This also applies to form workers with no catalog card: their suggestion metadata carries the context region, so neither the transcript nor the rail displays a generic form suggestion. Form workers preserve the selected region through extraction, completion and applicability updates. The shared Salesforce workspace attaches to the same conversation as the rail and consumes only workspace-placed events. See the Takeda demo for a native-LWC center workspace, sidebar catalog, reviewed-form commands, measured provider latency and dedicated DevSDO app.

Host context and event-driven workers

Each custom/coaching worker accepts input_mode: transcripts | events | both (default transcripts) and optional event_names. An empty event-name list accepts any host event when events are enabled. Speaker filters, transcript debounce and the utterance minimum apply to speech; explicit host events use a bounded per-worker queue and do not fabricate transcript turns. The queue holds 32 pending events and retains 50 events in EventHistory; call teardown cancels pending work.

The authenticated UI-action API accepts delivery: card | stream | both. card preserves existing card handling. stream reaches opted-in workers without injecting customer chat text; both also runs the card path. The publisher consumes the delivery marker in both gRPC and Redis paths, after deduplication. SDK sendCopilotEvent defaults to stream.

context.ready and context.updated replace the HostContext session namespace. The Salesforce Agent Assist wrapper sends the record identity and configured Context Flow outputs after readiness; JSON object/array outputs remain structured. RefreshView sends an updated snapshot. Acknowledgement confirms transport acceptance, not model completion or a Salesforce write. Host snapshots cannot replace trusted root identity or routing variables.

Prompts can reference {{HostContext}}, {{LatestEvent}}, {{EventName}}, {{EventHistory}} and {{ConversationStarted}}. Startup model workers receive conversation.started lifecycle context without invented speech; tool workers with explicit result bindings retain deterministic projection. For a record-dependent opening card, subscribe to context.ready rather than racing the host snapshot with run_on_start.

Form workers can omit catalog cards and emit standard SUGGESTION form events to a native UI. SDK subscribeToEvents and the embed bridge forward raw events before card rendering, including UI render/state events and headless forms. In Salesforce, TrueServeCopilotEvent__c carries record-scoped events and targeted form_context commands. See Salesforce components.

The native Copilot phone tester forwards its template variables with the test request, including the shared fictional scenario when opened from a Studio multi-agent demo. This initializes the rehearsal; it does not bridge audio or transfer context from a separate self-service test call.

Copilot STT assigns a separate utterance ID per speaker before enhancement and suggestion processing. Interims and their final share that ID and the first STT result time; the next final opens a new utterance even when the words repeat. Enhancement carries the original ID instead of updating whichever turn is current when its model returns. The publisher must not use voice-LLM response boundaries for these transcripts: a listening copilot has none. In test replay, conversation-api omits transcript events from the appended -standin voice partner stream, while retaining its other events and its direct standalone view. The copilot's two-channel STT is the displayed transcript source.

A copilot listens to a live human-to-human call and helps the human agent: it suggests answers, coaches, fills a form, translates, and summarizes at transfer or after the call. It never speaks to the customer. Read this page when you touch the copilot editor, CopilotPipelineBuilder, the Agent Assist card catalog, or the rail those cards render in. For the end-user view, see the user guide page.

The distinguishing property is that a copilot is prescriptive, not hand-wired. An author picks behaviours from a list; the runtime derives the processor chain. The editor shows a fixed canvas (Start → Transcription → Gate → one node per type → Translation → Summarization → End) whose nodes open section forms in the shared inspector — but the canvas is a view of the derived chain, not an authored graph: only the type fan-out is addable/removable, node positions are auto-laid, and nothing view-only ever reaches config. Unlike every other agent subtype, config.pipeline is not where the work lives.

The editor also has an engine selector (Custom vs 3rd-Party · Agent Assist, copilot.engine): in agent_assist mode Google Agent Assist replaces the Transcription + Gate + Summarization sentinels with one node (Google owns all three functions), while custom types[] still run on AA's transcripts. Switching to AA stamps bypass_gate: true onto every worker — with no gate step, a gated worker waits forever for a verdict (_gate_allows) and silently emits nothing.

Architecture in one line

config.pipeline holds exactly one real stage (transcription); everything else is planned from the top-level config.copilot branch by a step planner shared between the builder and preflight, and each worker publishes an A2UI card into the Agent Assist rail.

config.pipeline[0]                    config.copilot
  transcription stage                   ├── transcription   → TranscriptEnhancementProcessor  (optional)
  (a real (provider,type,service)       ├── gate            → CopilotGateProcessor            (one, shared)
   triple, resolved normally)           ├── types[]         → one worker EACH, in tandem
                                        └── summarization   → on_transfer / post_call workers

   transcription
        ↓
   [enhancement]          ← rewrites the transcript before anybody reads it
        ↓
   [gate]                 ← ONE "is this actionable?" verdict, fanned out
        ↓
   worker  worker  worker ← flat chain, but CONCURRENT (see below)
        ↓
   ConversationAPIPublisher → A2UI card per worker → rail

The chain is flat but not serial

Every worker is a self-accumulating pass-through processor: the frame flows straight through while the worker fires its own async extraction. So N workers in series run concurrently, which is why this is a flat chain and deliberately not a ParallelPipeline. Adding a worker adds no latency to the ones beside it.

Where the config lives

The branch is top-level config.copilot, beside pipeline — documented field by field in config-jsonb.md and typed as CopilotConfig in pipeline-types.ts. Discrimination is is_copilot_pipeline_config, keyed on copilot.types being a list — an empty list still matches, because a copilot with nothing enabled is a real agent that transcribes and suggests nothing. The legacy hand-wired shape carries the same agent_type: 'agent_assistant' but has no copilot branch, so it keeps building through UniversalPipelineBuilder.

copilot must stay in MISC_ROOT_FIELDS in both copies of pipeline-shape.ts (collab-gateway's and ai-manager's). The Y.Doc shape is an allow-list: an unlisted top-level key is dropped on every autosave, with the change visible in the editor and never reaching the database. See the collab invariant.

The planner is shared with preflight, on purpose

The step configs are built by copilot_steps.py, not by the builder. It is pure and pipecat-free, and both callers plan through it:

Caller	Card schemas resolved over	Then
CopilotPipelineBuilder	asyncpg	ServiceRegistry.create_service per planned step
preflight.py	psycopg2 (sync)	validates the same steps, opens no provider socket

The split exists because of those two database clients, and sharing the planner is what stops the buildability gate from drifting away from the build it predicts. Copilot steps are builder-owned processors, not authored stages, so preflight iterates them through its own _iter_copilot_steps arm rather than letting them reach get_service_mapping. Without that arm both test buttons in the editor go dead for every copilot, silently — the same failure mode that hit cascaded translation agents (see the preflight invariant).

Response schemas and card mappings

types[].card names an A2UI component as <catalog-uri>#<ComponentName>, e.g. t/<tenant>/copilot-cards#CoachingCard. The builder reads that component's a2ui_components.props_schema and hands it to the wrapper verbatim as json_schema, wrapped by card_schema_envelope as {name, strict: true, schema}.

Built-in card workers use the card schema directly, with editable description overrides. Custom Copilot (kind: custom) and an authored post-call summary can instead supply response_schema and card_bindings. Bindings map a card property to response.<field.path>; matching names are the default. utilities/copilot_output.py normalizes custom schemas for strict output (closed objects, optional fields represented as nullable), validates mappings against the destination card schema, and keeps the raw extraction separate from SuggestionFrame.card_props. The publisher uses mapped props for both live cards and the persisted wrap-up envelope. The existing wrap-up modal hosts the selected Agent Assist component; no new modal renderer is required.

Two consequences:

• An unresolvable card ref means the worker is SKIPPED, not downgraded. The wrapper's validate_config requires json_schema for every copilot kind; building without one would publish free text into a card and render blank. Preflight reports this gated card_ref, not structural.
• A tenant editing a card changes its required properties. For built-in workers this also changes the model contract; for mapped custom output it changes mapping validation.

Translation is the exception: it renders inline, gets builtin_schema_for_kind, and never consults a card. It is no longer a types[] kind at all — it lives at copilot.translation (its own section, a peer of summarization, with its own glossary); the runtime dual-reads a pre-037 draft's legacy types[] entry, preferring the section.

Translation defaults to bypassing the actionability gate; an explicit bypass_gate: false retains filtering. It triggers after one final utterance with zero debounce, so consecutive short customer phrases are not discarded by card throttling. Its default context window is 12 utterances; an authored window is preserved, and the prompt translates only the latest utterance. Enhancement variants do not trigger a second translation. Speaker, source utterance ID, and first STT timestamp are captured before the model call. The publisher derives a separate stable translation ID from the worker and source IDs and orders it immediately after the source line.

The translation request separates utterance_to_translate from context_only. Without that boundary, a model can combine consecutive same-speaker STT finals and repeat the previous sentence in the next translation. Context remains available for disambiguation but is excluded from the requested output.

Temporary coaching cards and transcript evidence

Advanced configuration on suggestion workers supports card_ttl_seconds (0–300 seconds; zero or absent disables expiry). Coaching, Suggested Answers and Custom cards can use it. Translation and form filling do not expire this way. The runtime starts the timer when it emits an applicable card. A new applicable turn refreshes the stable surface and cancels the old timer, including when the new advice has identical properties. Transcript/VAD notifications for the same turn are still coalesced. Inapplicable results neither dismiss the temporary card early nor renew it. Expiry emits a native A2UI deleteSurface; End/Cancel and processor cleanup cancel outstanding timers. Persistent cards retain their existing deduplication and applicability behavior.

Approved response entries may set runs_on: customer|agent|both to restrict the matching speaker within the worker's trigger filter. match_mode: utterance requires the complete final transcript segment, normalized for case and punctuation, to match. This prevents fast matching inside a longer negated or quoted request. The default phrase behavior still matches literal word sequences anywhere in the segment; unmatched input follows approved_only.

min_speaker_utterances: {agent: 2} requires two agent segments in this worker's context window before an agent turn triggers extraction. Omitted speakers default to one. This optional threshold precedes both approved matching and model invocation, in addition to the global min_utterances threshold. It counts final STT segments, not measured sentences or audio quality.

The Allianz scenario uses temporary cards and speaker-filtered approved responses for supportive speech clarity reminders. Its model can act on one clearly garbled agent segment, including essential missing words, without customer input or a two-segment threshold. Borderline wording needs corroboration and a new issue in the latest turn. It does not infer actual speed, pronunciation, accent, nationality or employee performance from text.

The gate: one verdict, fanned out

Direct OpenAI workers use the same sampling compatibility as the voice wrapper. Reasoning families omit temperature; ordinary models keep the authored/default value. This applies to the gate, query refinement/finalizer, coaching, summaries, form filling, and enhancement. A rejected gate request fails open; it does not prove that the downstream workers can make successful model calls.

Each worker wants the same answer before spending a call on a large model — does this conversation have real content yet, or is it still pleasantries? Asking per worker means N small-model calls for one answer, so CopilotGateProcessor asks once and broadcasts a CopilotGateFrame that every worker with require_gate: true keys on.

It subclasses SuggestionLLMProcessor because transcript accumulation, the speaker filter, debounce, and the four-provider structured-output path must behave identically; only the schema, the frame type, and the dedup policy differ. Three properties are load-bearing:

• It re-emits an identical verdict rather than deduping. Two consecutive actionable turns asking the same thing are normal and each needs its own permission.
• It can never itself be gated (require_gate forced to False) — nothing upstream produces a verdict, so that would deadlock.
• bypass_gate: true on a worker ignores it entirely. That is how greeting-time coaching still fires on a conversation the gate correctly calls unactionable — and it is the single behaviour that proves fan-out works, because greeting-only audio should produce the coaching card and no suggested answer.

The verdict frame carries the speaker, and each gated worker re-applies its own runs_on to it. This is load-bearing, not diagnostic: for a require_gate: true worker the verdict is its only trigger (the per-utterance runs_on check fires extraction on the ungated path only), so without the stamp runs_on is silently unenforced for every gated worker — the observed symptom was a runs_on: "customer" suggested-answers worker answering the agent's own greeting. The gate itself stays runs_on: "both" (a both worker needs a verdict on either speaker), which is exactly why the filter must live worker-side. An unstamped verdict (speaker="") passes through — an older gate must not silence every speaker-filtered worker.

The upstream gate can emit its verdict before forwarding the source transcript. Workers match the verdict's source utterance ID to their latest buffered utterance, falling back to turn_index when IDs are unavailable. Identity also handles enhancement variants that advance the gate's count without creating another translation. VAD callbacks check the latest buffered speaker against runs_on, since VAD has no speaker of its own.

The gate prompt is per-rail

gate.prompt_override is optional in the schema, but for anything other than suggested answers it is required in practice. The processor falls back to SMART_REPLY_PLANNER_PROMPT, and that default is written for one rail: it asks whether the customer "made an actionable request or asked a question the agent should answer", and explicitly discards greetings, pleasantries, acknowledgements and small talk.

That is exactly right when the worker is drafting a reply and wrong for coaching, next-best-action, form filling or a knowledge nudge — whose most valuable beat is a customer stating a grievance with no request attached. "This is the third shipment that arrived short-dated" is a complaint, not a question, so the default planner returns is_actionable: false, no verdict fans out, and _gate_allows silences every require_gate worker for that turn.

The failure has no error anywhere. The gate logs a decision per turn ([CopilotGate] actionable=false … reason='…small talk'), STT is transcribing both channels, the workers are built and idle, and the transcript reads like a substantive call — so it presents as "the copilot is broken" rather than "the gate disagrees with you about what matters". Nothing validates it either: preflight cannot judge a prompt.

An authored gate prompt should:

• enumerate what counts as actionable for this rail — objections, complaints, pattern claims ("it keeps happening"), stated preferences about how something should happen, and new facts that change the next action, not only questions;
• restrict false to genuine non-content — greetings, thanks, acknowledgements, goodbyes, and the rep's own procedural narration ("let me pull that up");
• say "when in doubt, true" — a card the rep ignores costs nothing; a card that never appears is the failure the rail exists to prevent;
• keep query phrased as a lookup for legacy workers. Suggested Answers now performs its own query refinement with the selected refinement model and prompt.

bypass_gate: true on every worker is the alternative and it works, but it triggers each worker independently and loses the shared actionability decision — the cost the gate exists to avoid. agent_id=4213 (Dispute Resolution Coach) is the worked example in the demo seed.

A tool-bearing type travels as smart_reply

The wrapper routes to SmartReplyWorkflowProcessor (planner → tool → finalizer) only on the exact value suggestion_type: "smart_reply". Every other value lands on the plain SuggestionLLMProcessor, which has no tool path at all. So a coaching/custom type carrying tool_ids, or a suggested-answers type carrying knowledge_tool_ids, keeps its kind in worker_id and telemetry but is emitted as smart_reply. Don't "correct" that to match the kind. copilot_kind preserves the authoring kind: Coaching/Custom bypass the question planner, execute attached tools with normal availability/input/capture bindings, and use their own final prompt. Suggested Answers runs query_refinement → Knowledge tools → answer. The refinement stage has independent provider/model/integration/prompt settings, uses the fast provider default when no model is authored, and requests a fixed {query: string} schema. Its query is available to tool_argument_mapping (default {"query":"{{query}}"}). An empty query skips retrieval and answering.

Add copilot selects the copilot-add-type inspector, rendered by the shared AddNodePanel in the left sidebar. Its list includes Suggested Answers, Coaching, Custom Copilot, and Form Filling; selecting one creates the stable type ID and opens its settings.

Type ids are minted once and never re-derived

uniqueTypeId in copilot-config.ts mints coaching, coaching-2, suggested-answers, form-filling from the kind, at add time — not from the label. The builder derives worker_id and therefore the A2UI surface_id (copilot-<id>) from it, and a surface replaces in place only while its id is stable. Re-deriving an id from a renamed label makes every rename stack a new card instead of updating the old one.

Transcription: every choice needs a real service row

The transcription stage is a normal stage resolved through the normal (provider, type, service) machinery, so every entry in TRANSCRIPTION_CHOICES must have a real SERVICE_LOOKUP_V2 row. get_service_mapping is a bare dict lookup that raises rather than aliasing, so an unbacked choice is a hard build failure at session start — no session, no conversations row, and every conversation route 404s while the agent silently never starts.

Per-channel attribution comes in three tiers

The caller arrives on channel 0 and the agent on channel 1; without per-channel attribution both parties transcribe as one speaker and every runs_on filter downstream is meaningless. Providers fall into three tiers, and the builder (copilot_builder._build_transcription) picks the mechanism per provider:

Tier	Providers	Mechanism
A — native	Deepgram (Nova-3, Flux), Amazon Transcribe	builder forces multichannel: true, channels: 2; a per-channel subclass (DeepgramMultichannelSTTService, AWSTranscribeMultichannelSTTService) stamps a distinct user_id per channel. AWS bakes channel identification into the presigned URL, so its flags must be set before _connect(), and it must iterate all Results (the stock service reads Results[0] only, which drops the second channel entirely). Google Agent Assist splits channels itself inside its own service class.
B — split upstream	OpenAI GPT Live Transcribe, ElevenLabs Scribe Realtime	no provider channel concept — the builder runs one service instance per channel behind build_channel_split_stt (a ParallelPipeline of channel-selector → mono STT → user-id stamp branches, plus a passthrough branch that keeps the original stereo frame flowing for the recorder). Children are created audio_passthrough: false.
(not offered)	OpenAI Whisper, ElevenLabs Scribe (non-realtime)	VAD-segmented (one audio buffer, one speaking flag) — every utterance attributes to ONE speaker, silently breaking every per-side runs_on filter, so these have no SERVICE_LOOKUP_V2 row, no wrapper, and no picker entry. If a future non-copilot use case needs them, wrap them then — and keep them out of TRANSCRIPTION_CHOICES.

Tier B's per-channel instances also need provider-side VAD, because a copilot runs enable_vad: False — pipecat's local-VAD defaults never commit a segment and the service transcribes nothing with entirely healthy logs. ElevenLabs takes CommitStrategy.VAD. OpenAI is harder, and openai_stt_services.py exists for three traps that each present as "no cards on a call that transcribed fine":

1. turn_detection is tri-state and only a dict is transmitted. {"type": "server_vad"} requests server VAD; False sends turn_detection: null, which means local VAD; None omits the key, so behaviour is whatever the model defaults to and the config asserts nothing. The wrapper sends an explicit dict.
2. pipecat's own default model, gpt-realtime-whisper, does not support turn detection at all. It rejects an explicit request, and with the key omitted it streams …transcription.delta forever while …completed never fires. Interim text is not a final, so the gate's min_utterances counter never advances (no cards, and the post-call summariser reports an empty transcript on a call that visibly transcribed both speakers); a cascaded voice agent on the same config hears the caller and never replies. The wrapper switches to gpt-4o-transcribe when segmentation is needed.
3. Server VAD arms an interruption broadcast. OpenAIRealtimeSTTService is the one pipecat STT service that calls broadcast_interruption() on speech-start (should_interrupt, default True), and pipecat answers an interruption by cancelling every processor's process task — so a worker mid-await loses that work, and because asyncio.CancelledError is a BaseException, an except Exception around the await neither recovers nor logs it. A ~5 s structured-output call dies with no error, no result, and no retry, since the debounce was already charged. The wrapper forces it off; barge-in belongs to the transport's VAD analyzer, which knows whether the bot is speaking, and a copilot has no bot output to interrupt anyway.

Channel-index conventions differ per provider and are normalized at each boundary: Deepgram channel_index is 0-based, AWS ChannelId is "ch_0"/"ch_1", Google Agent Assist channel tags are 1-based. A channel beyond the caller/agent pair (a supervisor barging in) maps to user_id="supervisor" on the agent side in every tier — misreading a supervisor's coaching as an actionable customer turn is the expensive direction of that error.

Speaker roles then resolve in the normalizer (deepgram_normalizer.py; the other providers register the same class via generic_stt_normalizer.py), which imports the publisher's BOT_USER_IDS rather than re-declaring it — the normalizer pre-empts the publisher's copy of that decision, so a second literal would be a silent override of it. A new provider needs both a module-level register_normalizer(...) and an entry in load_all_normalizers()'s hard-coded module list — a missing normalizer is silent.

The glossary reaches the STT service

copilot.transcription.glossary.terms is not just prompt material: the builder feeds it to the transcription service as vocabulary biasing — Deepgram keyterm, ElevenLabs keyterms, OpenAI transcription prompt. The editor's glossary is uploadable (merge-then-dedupe on term, never replace) and edited as a term/hint table.

Enhancement prompts and variables

Domain terms only corrects specialized vocabulary and proper names using the transcription glossary, when present, and recent conversation context. The glossary is optional; with no terms, the model relies on context and its own vocabulary knowledge. This mode does not instruct the model to fix grammar or phrasing.

The Transcription inspector's Advanced → Prompt displays the complete generated runtime prompt, including the selected mode and glossary hints, as editable text. Displaying it does not write system_prompt into the config. Editing saves an override; Reset to default removes the override and regenerates from the current settings. A custom prompt replaces the entire generated instruction, so subsequent mode/glossary changes do not rewrite it. Conversation history and the structured response schema are still supplied separately by the processor.

lib/copilot-prompts.ts mirrors the dynamic Python prompt builders; its parity tests execute the actual enhancement, translation, and form functions. Static prompts and output schemas are exported from the runtime into lib/copilot-runtime-defaults.json by python3 apps/ai-manager/scripts/generate-copilot-defaults.py; the Node suite runs that generator with --check. Regenerate the snapshot when the source prompts/schemas change.

Copilot prompt pickers offer SESSION_VARIABLE_BINDINGS (the snake_case runtime IDs, plus currentDateTime), authored copilot.variables, and the declared output_parameters of config.lifecycle.on_start. The picker follows the workflow API's draft redirect for unpublished workflows. Workflow outputs use bare names such as {{customer_name}}; completion-only outputs are not offered for live prompts. Switching or removing the start workflow drops its old output suggestions. Inserting a non-reserved bare token creates the existing $var:<name> binding, resolved against session metadata/variables by the copilot builder; workflow-start outputs are promoted to session metadata before building.

LatestMessage, LatestCustomerMessage, LatestAgentMessage, and ConversationHistory are reserved live variables. utilities/copilot_prompt_context.py derives them at invocation time, with history limited by that node’s context settings. KnowledgeSources is reserved too and populated only for the Suggested Answers finalizer with retrieved data/citations. The builder skips these names during session-start substitution. Transcript values render without HTML escaping; the normal #when evaluator handles conditional prompt sections. Each tool-bearing invocation captures its turn context before awaiting retrieval, so other workers and new turns cannot change its bindings mid-run. The normal tool-logic wrapper uses an invocation variable-store view, while captures still update shared session variables.

SpeechMetrics is a reserved live measurement input. It contains separate agent and customer values derived from original finalized STT word timings. Unsupported providers leave it empty, and samples under five seconds have no WPM value. The same measurements publish to context-only catalogue surfaces so components can update without a model call. See measurement basis and event wiring.

Every prompt exposes its response contract

PromptField in CopilotEditor.tsx supplies editable defaults, reset, variable insertion, and a Response format field editor below the prompt for the gate, all worker kinds, translation, and both summaries as well as enhancement. A generated default is never persisted by mounting or by CodeMirror echoing an external value update. Reset removes the authored override. Worker prompt reads follow runtime precedence (system_prompt before prompt_override); writes clear the other key so a hidden override cannot defeat edits or reset.

The gate uses the complete SMART_REPLY_PLANNER_PROMPT, including the non-actionable result instructions. Suggested Answers with Knowledge uses SMART_REPLY_FINALIZER_PROMPT; Coaching and Custom Copilot retain their own DEFAULT_PROMPTS entry even with tools. Query refinement has its own generated default and fixed output schema. The form prompt preview lists the first configured form's fields; each runtime form worker receives its own live definition, including when the author replaces the default prompt; translation uses copilot_steps.translation_prompt, not the separate live-audio translator's prompt. Card output schemas are read from the selected catalog component's props_schema; a missing card is shown as unresolved, never substituted with a generic output schema.

ResponseFieldsEditor lists existing schema properties, including nested array items and union branches, with immutable names/types/required flags. Edits persist in each prompt-bearing node's response_field_descriptions, keyed by JSON Pointer (for example /properties/corrections/items/properties/reason). utilities/response_fields.py deep-copies the runtime schema and applies only string descriptions to existing properties. Unknown paths are ignored; neither card schemas nor field structure are mutated. Overrides reach OpenAI, Google/Vertex, and Bedrock, including the dynamic form schema. Prompt reset and per-field description reset are independent.

Copilot editors use the scoped completionMode="copilot" menu: variable insertion and If/Else, without procedures, assignment, tool references, or action execution. Coaching and Custom Copilot use copilot_tools, adding the normal Tool library and input settings dialogs without Run/Set/procedure sections. Custom outputs use the existing visual OutputSchemaEditor; built-in outputs keep ResponseFieldsEditor. Other prompt editors retain their full menus. Context windows are labelled in turns.

Suggested Answers passes knowledgeOnly to ToolSelector, filtering saved tools and inline creation to Knowledge. Inline creation saves both the definition and selected reference together. The builder forwards inline definitions and wires their Knowledge callbacks; its saved-tool resolver uses knowledge_only=True, and Python/API registration is skipped for this worker kind. This restriction also applies to older or directly authored configs that contain non-Knowledge references. Publishing rejects enabled Suggested Answers with no Knowledge references, wrong-type tools, inactive/foreign tools, or unresolved inline Knowledge. The planner skips missing-Knowledge workers and preflight reports the missing Knowledge requirement. Coaching/Custom use normal prompt tool authoring; saved and inline Python/API/Knowledge tools are wired, with normal per-tool tool_logic bindings, availability checks, and captures.

Bedrock native structured output

Enhancement, form filling, and both SuggestionLLMProcessor call paths (including gate, Smart Reply planner/finalizer, translation, and summaries) use utilities/bedrock_structured_output.py. It calls Converse with outputConfig.textFormat.type=json_schema and structure.jsonSchema.schema containing the serialized schema. This replaces the Claude-specific InvokeModel body and supports the model families served by Converse. It requires boto3 ≥1.43.0.

Only an explicit model-capability ValidationException triggers the schema-in-prompt fallback. The unsupported-model result is cached within that processor/region for the session; other models still attempt native output. Schema validation errors, bad model IDs, access failures and throttling do not downgrade. Both native and fallback responses are parsed and validated with jsonschema against the original response contract, with remote $ref retrieval disabled. Incomplete, blocked, or tool-use responses are rejected before publication. Usage is recorded even if local validation fails, and spans carry structured_output.mode=native_json_schema|validated_json.

AWS accepts a subset of JSON Schema; unsupported schema features remain errors rather than silently weakening the contract. First-use schema compilation can add latency; the reusable client allows a 300-second read timeout and always sets a 4096-token output cap. Tests use the actual botocore Converse request model with Stubber (no paid model calls), including capability fallback, schema/access errors, truncation, usage and processor wiring.

Audio: channels are speaker attribution — never mix them

A copilot listens to a call already in progress, so its input frame is interleaved multi-channel (ch0 = caller, ch1 = agent, 2+ = a supervisor or later barge-in, matching media-proxy's interleaveStereo), and there is no TTS — the agent's voice arrives on the input frame or not at all. Two consequences:

• The recorder must split the input. ConversationRecorderProcessor's _handle_output_audio never fires on a copilot, so split_input_channels deinterleaves ch0 into the caller track and downmixes ch1..N into the agent track (a third party mixes agent-ward). Without it both voices land on the caller track as interleaved bytes and the agent track stays empty.
• Krisp voice isolation is skipped outright on multi-channel input (voice_isolation_supports_channels in turn_control.py). Every pipecat audio filter is written for mono — KrispVivaFilter reshapes the raw bytes into fixed frames, so on an interleaved buffer adjacent samples are different speakers and the model "cleans" by mixing them: measured on a greeting-only stereo fixture, the silent caller channel went rms 0 → 631 (the agent's words smeared in), Deepgram returned the same transcript on both channels, the caller copy won attribution, and a suggested-answer card fired on a call where the customer had not yet spoken. Skipping is the right trade, not the cheap one — per-channel filtering would need one stateful filter per channel with the count growing mid-call when a supervisor joins, against streams that arrive already channel-separated from the carrier. The skip is a logged decision, not an error.

Two diagnostics rules from getting this wrong: GRPCInputTransport's max_amp log is a per-frame sample taken on the diagnostic cadence, not a running maximum — a low value on one line does not mean the stream is silent; and an E2E audio fixture must round-trip through the real STT provider (synthesized PCM that "looks like speech" can transcribe as nothing at all, making every downstream assertion vacuously pass).

Cards reach the rail over gRPC

A worker's card is published as a UIRenderEntry. Two fields on that message exist for copilots:

string layout_json = 6;   // {region, order, span, title} canvas layout hint
string treatment  = 7;    // quiet|focus|reframe|unfold

layout crosses as a JSON string because protobuf has no bare-dict type; conversation-api's event_transformer.py accepts a dict or the string and relays both only when present. See a2ui-rendering.md.

Client-side, copilot-surfaces.ts partitions render events into the rail and everything else once, at ingest, because uiRenderEvents feeds eight consumers in the surface page. A surface is classified by the last event that carried a layout, not its first — layout is latest-wins per surface, so a hint-less createSurface followed by an updateComponents carrying region: 'rail' is a rail surface, and both of its events must travel together or the rail renders an update for a surface it never created.

Summarization

A summary worker is driven by summary_target, not the gate (require_gate: False, runs_on: "both", logged gate=bypassed): its trigger is an explicit request naming a target, and _request_summary does address matching, not broadcast — a request for on_transfer is ignored by the post_call worker and vice versa, and an empty transcript buffer declines rather than summarizing nothing.

The two targets reach the client differently, and each leg of the path has a reason:

• on_transfer publishes live — the SummaryCard pins into the rail while the call is still up. The request arrives from the CRM as POST /conversations/{id}/responses {"type": "ACTION", "data": {"action": "copilot_summary_request", ...}}, relayed over the publisher's gRPC stream; _request_copilot_summary then pushes CopilotSummaryRequestFrame UPSTREAM, because the publisher is the last processor in the chain and the summary worker sits above it. It is not persisted — a live rail exists to render into, and conversation_events is the replay store, not an activity log. The absence of an on_transfer row there is by design.
• post_call extracts while the pipeline tears down, too late to publish live. _capture_post_call_summary calls the worker's summarize_now directly and awaits it — never via a frame: a CopilotSummaryRequestFrame is a DataFrame, and on the cancel path the queue that would deliver it has already been drained by CancelFrame, so a frame-based request is dropped with no error anywhere. The result persists as a copilot_summary conversation event carrying a render-ready A2UI envelope (built by the same helpers as the live card, so persisted and live are the same bytes); conversation-api replays that row as a UI_RENDER on the historical SSE stream, pinned by layout: {region: "rail", order: -100}. The Salesforce Wrap-Up Summary wrapper just opens the record's conversation read-only and the summary arrives with the replay — see crm-embed-components.md. _persist_copilot_summary's only call site is inside _capture_post_call_summary — keep it that way.

Inline translation: a transcript, not a card

Translation is configured at copilot.translation (enabled flag, target/source language, runs_on, model fields, its own glossary or glossary_ref: 'transcription', and a load-bearing bypass_gate — a gated translator with the gate off waits forever for a verdict). The planner (translation_spec) dual-reads: the section wins; a pre-037 draft's legacy types[] entry with kind: 'translation' is the fallback and is skipped when the section is present, so exactly one translation worker is ever planned.

At runtime it stays an inline kind: card_ref = "" regardless of any authored card, and an empty card_component routes the frame to the plain transcript path instead of drawing a card. Each translated line publishes as its own transcript entry with source: "translation" and its own message_key/turn_id, additively beside the original (source: "agent_assist") line.

Client-side, surface-app deliberately bypasses its either/or language filter for copilots (hasCopilotTranslation in the surface page, detected structurally as some(translation) && some(agent_assist) because share-code sessions never see the agent type): both lines always render inline, and the transcriptSources=agent_assist CRM presets force-keep the translation companion they would otherwise strip. Consequently preferredLanguage is a view toggle ("caller" | "agent") for translation agents, never a copilot target language — the copilot's target lives on the type's own target_language (default "en").

The card catalog

The copilot-cards catalog is platform-owned and created by 029_copilot_cards_catalog.sql with catalog_type='agent_assist'. The row must exist with that type before catalog-builder publishes the source, because POST /publish?bySlug=true only resolves an existing row and createCatalog defaults the type to multimodal — a row created any other way publishes through the non-blocking path, so the strict-schema gate silently doesn't apply. Owning the row in a schema script makes the type a property of the environment rather than of whoever clicked publish first. See catalog-builder.md.

Components live in apps/catalog-builder/catalogs/tenant/copilot-cards/: CoachingCard, SuggestedAnswerCard, FormFillCard, SummaryCard, plus Root (excluded from the picker — it is not a card). It is a baseline catalog, cloned into each tenant by provisionTenantBaselines.

Editor warnings

copilotWarnings is the author-facing check: no types enabled, a type whose card doesn't resolve, suggested answers with no knowledge sources, a transcription stage that names an unresolvable service, and so on. They are advisory. The blocking gates are structural validation (JSON Schema) and preflight buildability — see the two tiers.

Migrating a hand-wired copilot

Copilots predating this shape wired their behaviours as output_lanes.suggestions.branches[].processors[]. 033_copilot_config_migration.sql converts them: it normalizes the transcription stage's service name, rewrites stt_suggestion → stt, derives copilot.types[] from each legacy branch, lifts the enhancement processor into copilot.transcription, populates copilot.gate, and re-cuts the three tpl-copilot-* templates.

Two framing properties matter if you extend it:

• Every statement is scoped to the copilot predicate (subtype = 'copilots' OR config->>'agent_type' = 'agent_assistant'). The legacy service names also live on non-copilot voice_agents versions, and an unscoped normalization silently rewrites them.
• It is a guarded UPDATE throughout and a clean no-op on a fresh boot — no applied init script seeds those agents or templates, so there is no seed file to edit and re-applying over the published versions it converts must be safe. The guard is copilot.types being absent or empty.

output_lanes is left in place. No copilot code path reads it, so it is dead data on a converted config — but it is also the only surviving record of the original hand-wired configuration, which makes the migration reversible.

037_copilot_translation_engine.sql follows the same discipline: it promotes any legacy types[] translation entry into copilot.translation (removing it from types[]) and stamps engine: 'agent_assist' where pipeline[0] is google/Agent Assist. Both are guarded (NOT (copilot ? 'translation') / NOT (copilot ? 'engine')) so re-application is a no-op, and it is in migrate-city-db.sh's Phase C loop.

Agent Assist engine mode (copilot.engine: 'agent_assist')

Google Agent Assist replaces transcription, the gate, and summarization; custom types[] still run on AA's transcripts (AA pushes TranscriptionUpdateFrame with real roles, and SuggestionLLMProcessor has a dedicated branch for that shape — AA is self-normalizing, no registry entry). Facts that shape it:

• engine is invisible to all three config guards — they assert only that copilot is a dict and copilot.types is a list. types must stay an array (even []) in engine mode.
• The gate is off, so every worker must carry bypass_gate: true — the editor stamps it on the engine switch and on every subsequently-added type.
• Session variables reach the Dialogflow virtual agent via Config.initial_virtual_agent_parameters on the first request of every bidi stream. GoogleAgentAssistServiceWrapper is a RuntimeContextServiceWrapper and reads runtime_context["session_variables"] (the copilot builder merges copilot.variables $var: bindings over the transport's session variables before the transcription stage is built). This is the only per-session variable channel the API offers besides ingest_context_references.
• Context injection uses the NESTED Conversation.ContextReference and a map of reference-id → reference on IngestContextReferencesRequest.context_references. The old top-level ContextReference import never existed in the SDK; the ImportError was swallowed by a blanket except, so context injection failed silently for as long as that code was live.
• The conversation_profile_id (+ region, default global) is edited on the Agent Assist canvas node; without a profile the service's validate_config hard-errors and the agent cannot build.

See also

• Universal pipeline builder — the builder every other subtype uses
• A2UI rendering — how a card actually paints
• catalog-builder — publishing and the schema gate
• CRM embed components — the Agent Assist and Wrap-Up surfaces
• config JSONB — the copilot branch as a contract

Applicability and SDK dismissal

Card-producing workers wrap their payload schema with the fixed envelope {applicable:boolean, reason:string, result:object|null}. utilities/copilot_output.py deep-copies the result schema and makes only its root nullable. Required result properties remain required when the result exists. Description overrides for applicable and reason use /envelope/properties/<name>; existing response-field pointers still describe result fields. All providers receive this envelope. Plain transcript translation retains its existing empty-text exit, while the gate uses is_actionable:false and query refinement uses an empty query.

The worker unwraps an applicable result before normal card mappings. An inapplicable result bypasses mappings and produces a SuggestionFrame.not_applicable; post-call capture skips it. ConversationAPIPublisher sends UI_RENDER with message_type=copilotStatus instead of issuing a delete. Form skips include preserve_values:true and retain extracted/confirmed state. An empty refinement query also reports inapplicability without invoking Knowledge.

Both SDK copies consume status events, emit copilot_status, and apply manual (default) or on_not_applicable dismissal, optionally overridden per surface. dismissCopilot(surfaceId) sends only a local deleteSurface. The rail removes that card's container and permits a later component update to recreate it. The Agent Assist UI exposes a Dismiss button; the Embedded SDK exposes acknowledged dismissal commands. Neither dismissal nor inapplicability completes or resets a form. Dismissal is not persisted across UI sessions.

Multiple forms and dynamic updates

types[].catalogs limits the editor's catalog choices. Card-producing types and summary targets choose one card; Form Filling uses a cards palette plus forms[].card, defaulting to the sole available card. The catalog manifest harvests the palette and per-form references. Runtime schema resolution includes every per-form card. forms[].card_bindings maps the standard deterministic form display data (form_label, fields[{label,value,confidence}], missing_fields, note) to a custom card's properties.

The builder creates one independent FormFillingProcessor per configured form. A single form retains the owning worker ID; multiple forms use <copilot-id>--<form-id>. Surface IDs append the instance number. Configured form_id stays stable when a pushed definition changes form_name. repeat_mode defaults to once; repeatable permits new_instance only after completion. completion_mode defaults to agent_confirmed; required_fields completes only with at least one required field, non-empty required values, and no suggestions below confidence 0.5. Confirmed fields replace proposed values with confidence 1. Completion freezes extraction. A new instance resets values/dedup and sets a transcript boundary so old facts cannot fill it again.

All Copilot form workers listen for controls; only allow_dynamic workers accept active_form replacements. The Redis payload contains variables and an optional form_target object/JSON string with both copilot_id and form_id. With multiple workers a target is mandatory; with one worker untargeted legacy commands still work. Replacement clears proposed and confirmed values and refreshes the card. A revision counter discards model results overtaken by a schema update, confirmation, pause, reset, completion, or new instance.

POST /sessions/{id}/variables persists the flat variable map, then _apply_local_session_variables publishes the four form keys to form_context:<id> through publish_to_aliases. This is now alias-aware for phone-call UUID/runtime-session pairs as well as browser sessions. SDK object values are JSON strings on this endpoint. Send target and command together; persisted targets are not implicitly attached to later events. The command acknowledgement means API acceptance, not model completion. These are live commands and should follow connection/transcription startup; runtime state is scoped to the active pipeline.

The concrete Embedded SDK/TypeScript/HTTP examples and controls are in the live forms user guide; the transport contract is in postMessage protocol. Tests cover provider-envelope validation, skipped card mappings, server status-only behavior, per-form worker construction, targeted updates, completion/repetition, in-flight schema races, both SDK copies, bridge acknowledgement/origin checks, and browser persistence.

Greeting before speech

Custom and coaching workers can set initial_response to an approved, non-personal response object. Card at call start in the worker inspector edits it. The runtime validates it against the extraction schema, emits it once after StartFrame, and leaves the transcript empty. It makes no model or tool call. Invalid initial props are skipped. Never put unverified customer or patient facts in this response; use a scoped tool for those facts after the caller identifies the request.

For verified context before speech, Load context at session start (run_on_start: true) runs a custom/coaching worker after StartFrame. With tool_result_bindings, tool output is validated and projected without a model call. Without those bindings, the normal model/tool path receives conversation.started lifecycle context without a synthetic transcript. Null, invalid, ambiguous or failed results produce no facts. End/Cancel cancels the pending lookup. Speech buffered during startup is processed afterward. The native Takeda workspace waits for Salesforce context.ready instead, so the opening transfer card can inspect the linked Case data.