Live copilot ingress and shared page sessions¶

Status: Draft | Last reviewed: 2026-08-12

How to let a Salesforce, ServiceNow, or custom-web component select a copilot, start or attach a live audio source, and drive transcript, Agent Assist, and copilot widgets from one shared conversation. Read this before adding browser media capture to a host component or exposing a copilot media endpoint.

Decision¶

Build one media-owning session controller. Every visible widget subscribes to the controller's conversation events; no transcript, suggestions, sentiment, or copilot-chat widget opens its own media connection.

Use three protocols with separate responsibilities:

Plane	Protocol	Purpose
Control	HTTPS JSON	List permitted copilots, create a logical session, attach context, and end the session
Media	WebRTC with WHIP-compatible signaling	Send low-latency browser audio to TrueServe
Events	Existing conversation-api SSE	Fan transcript, suggestions, insights, status, and tool events out to any number of widgets

Use browser WebRTC only when the browser is the audio source. Treat a combined room microphone as a mixed mono source and use speaker diarization to separate in-person participants. If a Salesforce VoiceCall or another CCaaS call already exists, attach the selected copilot to the server-side call tap instead. A Salesforce-page microphone can hear the room, but it does not give reliable access to the remote party's isolated softphone audio.

Warning: Do not market microphone-only capture as two-party Agent Assist. Use a server-side SCV, Genesys, Five9, Amazon Connect, SIPREC, or TrueServe media tap when speaker roles matter.

Why WebRTC and WHIP¶

WebRTC already gives the browser encrypted Opus media, congestion control, NAT traversal, and ICE/TURN. WHIP standardizes the small HTTP offer/answer and teardown layer needed for one-way media ingest. It matches this use case more closely than a bidirectional voice-call protocol.

Option	Decision
Direct Pion plus WHIP	Use for the first browser-ingest release; it reuses media-proxy and keeps the client small
LiveKit room	Reconsider when a session needs multiple browser participants, SFU routing, or room semantics; the current TrueServe provider is avatar-oriented
WebSocket PCM	Do not use as the primary media path; it recreates codec, congestion, jitter, and backpressure behavior already provided by WebRTC
Janus/SIP	Keep for telephony; a single browser producer does not need a SIP bridge

Existing building blocks¶

Building block	Reuse
surface-app embed display modes	Render the existing SSE stream without knowing how media arrived
CRM embed components	Salesforce and ServiceNow wrappers that frame those display modes, with server-side token minting already solved
Copilot API design	Supplies the `/copilot/*` facade and normalized suggestions contract
conversation-api SSE	Carries transcript, suggestion, custom-insight, tool, and status events
Direct WebRTC handler	Pion peer, Opus transport, ICE, and gRPC bridge
Agent Assist sidecar linking	Routes caller and agent audio into a copilot session for platform-managed calls
Workflow Transcribe node	Already normalizes batch diarization from Deepgram, AWS, Azure, and Google into speaker-labelled segments
`embed_public_clients.allowed_agent_ids`	Limits which agents a host may offer in its selector
Agent Studio for Salesforce	Apex token minting, record binding, context Flow, and App Builder exposure

The current direct-WebRTC implementation is voice-agent shaped, not copilot-ingest shaped:

session.go opens SessionType.AGENT with one input channel and an outbound TTS track.
opus.go decodes stereo Opus and downmixes it to mono.
The first inbound track wins; later audio tracks are discarded.
/api/proxy/webrtc/offer requires an ai-manager user or emulator session and is not an external surface contract.
The current <trueserve-session> owns REST and SSE state, but no MediaStream, RTCPeerConnection, or media-session lifecycle.
The live Deepgram Nova-3 definition does not expose diarization, and its wrapper does not convert word-level speaker tags into speaker-aware transcript frames.

Preserve the existing voice-test route. Add an ingest-specific path instead of weakening its authorization or changing its behavior.

Audio source modes¶

Source mode	Component action	Server action	Speaker quality	Priority
`existing_conversation`	Select a copilot and attach to the current record or external call id	Link a new assist sidecar to the existing media sessions	Best; roles already separated when the connector supplies them	Production default
`browser_dictation`	Prompt for microphone access and publish one track	Run an ingest-only assistant session with diarization off	One known speaker	First browser MVP
`browser_meeting`	Prompt for microphone access and publish the mixed room track	Run streaming diarization and emit participant-labelled utterances	Model-estimated speakers from one combined source	Primary meeting mode
`host_stream`	A custom in-DOM host sets a `MediaStream` property	Accept one or two labeled tracks	Best when the host supplies separate customer and agent tracks	Custom-app phase
`display_audio`	Ask the operator to share a tab/window with audio	Treat as a mixed source	Browser-dependent, mixed, and privacy-sensitive	Unsupported by default

existing_conversation must fail explicitly with 409 source_unavailable if the call transport has no tap. Do not silently fall back to the page microphone.

Mixed-microphone diarization¶

Diarization answers "which anonymous speaker said this?" It does not normally identify a person's real name. Preserve provider labels such as speaker_0 and let the operator map them to names after hearing the first few turns. Voice-reference enrollment is an optional enhancement, not a requirement for the live MVP.

Use a two-pass pipeline for meeting mode:

Pass	Default	Purpose
Live	Deepgram Nova-3 streaming with `diarize_model=latest`	Low-latency captions and copilot input; the current streaming diarizer is v1
Final	Deepgram Nova-3 batch with `diarize_model=latest`	Recompute the meeting with the newer v2 batch diarizer and publish a refined transcript
Known-speaker finalization	`gpt-4o-transcribe-diarize`	Optional final pass when the tenant supplies short reference clips and names for known participants

Azure Conversation Transcription is the strongest existing-cloud alternative for real-time meeting diarization. AWS Transcribe and Google Cloud STT also support streaming diarization, but TrueServe does not currently have live speaker-aware wrappers for them. Evaluate providers on the tenant's actual rooms, accents, speaker counts, and microphones before making one the global default.

For dictation mode, disable diarization and stamp the known operator identity. Running diarization on one expected speaker adds label instability without adding value.

Represent a live diarized utterance with fields that permit correction:

{
  "turn_id": "turn-42",
  "text": "Let's move the launch to Friday.",
  "speaker_id": "speaker_1",
  "speaker_label": "Speaker 2",
  "speaker_confidence": null,
  "is_final": true,
  "revision": 1
}

Live speaker assignments may change as more audio arrives. Feed only finalized utterances into suggestions and other agent actions. The final pass may publish refined_transcript events with a higher revision; clients replace the earlier segment by turn_id instead of appending a duplicate.

Target architecture¶

flowchart LR
    subgraph HOST[Host page]
        control[Live copilot controller]
        bus[Host session bus]
        transcript[Transcript]
        suggestions[Agent Assist]
        chat[Copilot chat]
        other[Sentiment and future widgets]
    end
    subgraph EDGE[TrueServe public APIs]
        catalog[Copilot catalog]
        session[Copilot session control]
        whip[WHIP signaling]
        sse[Conversation SSE]
    end
    subgraph MEDIA[Media runtime]
        proxy[media-proxy ingest or existing call tap]
        grpc[gRPC audio stream]
        orch[ai-orchestrator copilot pipeline]
        redis[(conversation event stream)]
    end

    control --> catalog
    control --> session
    control -->|browser audio only| whip --> proxy
    session -->|existing call| proxy
    proxy --> grpc --> orch --> redis --> sse
    control --> bus
    bus --> transcript
    bus --> suggestions
    bus --> chat
    bus --> other
    sse --> transcript
    sse --> suggestions
    sse --> chat
    sse --> other

The shared value is a safe session descriptor, not audio and not a bearer token:

{
  "sessionKey": "case-assist",
  "conversationId": "7b13d5d8-...",
  "sessionId": "7b13d5d8-...",
  "agentId": "3021",
  "agentName": "Claims Copilot",
  "sourceMode": "browser_meeting",
  "captureProfile": "meeting",
  "state": "live"
}

Each cross-origin widget gets its own short-lived token bound to that conversation. Never put a token on the host message bus.

Component contract¶

Add two framework-free elements and expose them as one host component:

Element	Responsibility
`<trueserve-live-session>`	Headless state machine, selected agent, logical session, optional WebRTC ingest, and registration of the resulting `TrueServeSDKClient`
`<trueserve-copilot-launcher>`	Accessible selector plus Start, Mute, and Stop controls bound to a live session

The Salesforce wrapper is c-true-serve-live-copilot. It composes both elements and remains the single App Builder component an admin places. Custom apps may use the elements separately.

Recommended inputs:

Input	Meaning
`client-id`	Public embed-client/config id whose `allowed_agent_ids` forms the selector catalog
`agent-id`	Optional fixed agent; hides the selector
`source-mode`	`existing_conversation`, `browser_dictation`, `browser_meeting`, or `host_stream`
`expected-speakers`	Optional minimum/maximum hint for meeting diarization
`conversation-id` / `external-id`	Existing call binding for attach mode
`session-key`	Stable page-local name used by sibling widgets
`context`	CRM context copied into the logical session before media starts
`media-stream` property	Custom in-DOM source; never serialized through an iframe bridge

Do not support automatic microphone start. A visible operator gesture must start capture and the component must keep a persistent live/muted indicator on screen.

Recommended outward events:

Event	Detail
`trueserve-session-starting`	`{sessionKey, agentId, sourceMode}`
`trueserve-session-started`	Safe session descriptor
`trueserve-session-state`	Descriptor plus `state` and optional safe error code
`trueserve-session-ended`	Descriptor plus `reason`
`trueserve-session-error`	`{code, message, retryable}` with no credentials or provider secrets

Host fan-out¶

Host	Fan-out mechanism
Same DOM	Composed DOM events between the capture controller and the widgets sharing its page
Salesforce	A `TrueServeLiveSession__c` Lightning Message Channel; wrappers subscribe, mint a conversation-bound token through Apex, then connect their own iframe
ServiceNow	UI Builder action plus the existing origin-checked iframe bridge
Plain iframe	Origin-checked `postMessage` carrying only the safe descriptor

Lightning Message Service belongs outside the surface-app iframe. The iframe emits a session event to its wrapper; the wrapper publishes it to the page. An LMS message does not cross an iframe boundary by itself.

Whichever component owns capture needs allow="microphone; autoplay" on its iframe: microphone access in a cross-origin iframe requires permission delegation. The CRM wrappers use a plain sandboxed iframe, so they can set it. Passive transcript and suggestion components need nothing.

Public API shape¶

Extend the draft Copilot API with the following surface.

Catalog¶

GET /copilot/agents?client_id=<id> returns safe summaries only:

{
  "items": [
    {
      "id": "3021",
      "name": "Claims Copilot",
      "description": "Live claims guidance",
      "capabilities": ["transcription", "suggestions", "query"],
      "audio": {"minChannels": 1, "preferredChannels": 2}
    }
  ]
}

Filter to the caller's tenant, allowed_agent_ids, published versions, and agent_type="agent_assistant". An empty allowlist returns no agents. Do not expose prompts, integrations, provider credentials, draft versions, or full agent_versions.config.

Salesforce calls the catalog through Apex and its Named Credential. After selection, Apex mints the normal agent-scoped embed token. A custom host with a backend follows the same pattern. A browser-only catalog token is a later PKCE enhancement, not required for the MVP.

Logical session¶

POST /copilot/sessions accepts:

{
  "agent_id": "3021",
  "source": {
    "kind": "browser_meeting",
    "tracks": [{"role": "participant", "channels": 1}],
    "diarization": {"enabled": true, "min_speakers": 2, "max_speakers": 6}
  },
  "external_id": "salesforce:0LQ...",
  "context": {"case_id": "500..."},
  "idempotency_key": "c25b2e2d-..."
}

It validates the exact agent capability and tenant, resolves the published version, creates the conversation, applies context, and returns:

{
  "conversation_id": "7b13d5d8-...",
  "session_id": "7b13d5d8-...",
  "state": "created",
  "events_url": "/sse/conversations/7b13d5d8-...",
  "media": {
    "protocol": "whip",
    "endpoint": "/copilot/sessions/7b13d5d8-.../whip",
    "token": "short-lived-session-capability",
    "expires_at": "2026-07-15T12:01:00Z"
  }
}

For existing_conversation, return no media block. The server links the selected copilot to the existing call tap and publishes the same status events.

WebRTC ingest¶

Make /copilot/sessions/{id}/whip conform to the basic WHIP exchange:

POST an SDP offer as application/sdp with the session capability in Authorization.
Return 201 Created, the SDP answer, and a Location for the media resource.
DELETE the returned resource to stop ingest.

The capability authorizes one session, agent, origin, and limited setup window. It is not a tenant API token. Return ephemeral TURN credentials through the WHIP ICE-server relation or the logical-session response; never ship the static coturn credential to a browser.

Keep live output on GET /sse/conversations/{id}. Do not create a second WebSocket or WebRTC data-channel vocabulary for transcript and suggestions.

Media-runtime changes¶

Add an ingest-only Pion session alongside the existing voice-agent session:

Open gRPC with SessionType.ASSISTANT, no outbound TTS track, and published-version-only metadata.
Preserve mono input as one channel and stamp its configured role.
Add a speaker-aware live STT adapter that groups contiguous word-level speaker tags into utterances and emits stable turn_id, speaker_id, finality, and revision fields.
Add diarize_model to the Deepgram service definition and wrapper. Use latest; do not set both the deprecated diarize flag and diarize_model.
Record meeting audio once when policy permits, then submit it to the selected finalizer and publish the result as refined_transcript rather than overwriting audit history in place.
For dual-track input, decode each track independently, align 20 ms PCM frames, interleave customer and agent channels, and advertise num_channels=2 to the orchestrator.
Bound per-track queues and record packet loss, late-frame drops, jitter, queue depth, and role mapping.
Reject an unexpected third track or a track that does not match the declared role map.
Close the peer, gRPC stream, Redis ownership entry, and conversation status from any teardown path.

Do not modify the existing decoder to return stereo for every voice call. Add an explicit ingest decoder/mixer so voice-agent behavior and recordings stay stable.

Store transient ownership and lifecycle in Redis with a TTL instead of an ai-manager process-local map. The durable audit record remains the conversation and its messages. The Redis record contains tenant, agent, conversation, media resource id, owner node, state, and last heartbeat so DELETE and timeout cleanup work across replicas.

Security and compliance¶

Bind catalog access to embed_public_clients.allowed_agent_ids; deny an empty or missing allowlist.
Bind every session and WHIP capability to tenant, agent, origin, conversation, expiry, and a maximum concurrent-session count.
Keep the media-proxy API key server-side. The browser talks only to the authenticated public facade and to ICE candidates negotiated for its session.
Enforce conversation-api allowed origins and Salesforce CSP Trusted URLs.
Use a visible operator gesture, microphone indicator, mute state, and stop control.
Default recording off for ad-hoc browser capture until the tenant supplies a recording/retention policy and the host captures any required consent.
Redact bearer values, SDP ICE passwords, TURN credentials, provider tokens, and raw CRM context from logs.
Treat audio and transcripts as tenant data for retention, deletion, audit, and telemetry routing.
Rate-limit catalog, session creation, and SDP setup separately. Cap session duration and idle time.

Delivery phases¶

Phase 0 — transport and host spike¶

Prove microphone capture in a normal Salesforce iframe on supported Lightning browsers.
Verify that the wrapper's sandboxed iframe delegates microphone access via allow="microphone; autoplay" on supported Lightning browsers.
Adapt a Pion ingest session to the WHIP POST/DELETE exchange.
Prove a mono SessionType.ASSISTANT stream creates one conversation and emits transcript plus one suggestion over existing SSE.
Prove two people on one room microphone produce distinct speaker-labelled final utterances and that a later correction replaces rather than duplicates a turn.

Exit when a Salesforce sandbox can select a fixed copilot, start microphone capture by gesture, see an SSE transcript, and stop with no leaked peer or gRPC session.

Phase 1 — composable microphone MVP¶

Add catalog and logical-session endpoints with allowlist enforcement and idempotency.
Extract the reusable WebRTC state machine from the React-only test hooks into a framework-free client.
Add explicit Dictation and Meeting capture profiles; only Meeting enables diarization.
Add <trueserve-live-session>, <trueserve-copilot-launcher>, and the Salesforce wrapper.
Publish the safe session descriptor over Salesforce LMS and connect existing transcript/suggestions/copilot wrappers to it.
Add one-use media capabilities, ephemeral TURN credentials, Redis ownership, timeouts, and explicit error codes.

Exit when one media stream drives at least three independent widgets without any widget opening another media connection.

Phase 2 — real-call attach and speaker separation¶

Add dynamic attach/detach for TrueServe, SCV, Genesys, Five9, and other transports that expose a tap.
Preserve separate caller and agent channels through media-proxy and the copilot pipeline.
Add host_stream support for custom apps that can provide labeled tracks.
Resolve Salesforce VoiceCall/external ids through the existing record-binding flow and persist the chosen conversation id where page reloads can recover it.

Exit when the same component can attach to a live Salesforce call, shows correct customer/agent transcript roles, and returns source_unavailable for unsupported telephony instead of using ambient audio.

Phase 3 — hardening and additional hosts¶

Exercise ICE/TURN across enterprise NAT, failover, page navigation, utility-bar pop-out, and multi-replica routing.
Add ServiceNow and plain-iframe wrappers with the same safe session event contract.
Load-test concurrent ingest, Redis cleanup, SSE fan-out, and provider quotas.
Add dashboards for setup latency, active sessions, media gaps, transcript latency, suggestion latency, and abnormal teardown.

Acceptance criteria¶

Only explicitly allowed, published copilot agents appear in the selector.
Start always requires an operator gesture in browser-capture mode.
One logical conversation id ties media, transcript, suggestions, query, status, and audit together.
Multiple widgets receive the same ordered SSE events and none duplicates media ingest.
Dictation mode produces one known speaker without unnecessary diarization.
Meeting mode emits anonymous speaker labels, supports operator renaming, and keeps suggestions behind finalized utterances.
Mono mode labels its single declared role honestly; dual-channel mode preserves customer and agent roles.
A final diarization pass revises existing turns by id and never creates duplicate transcript text.
An existing-call attach never falls back to microphone capture.
Stop, page unload, idle timeout, and network failure all release media and gRPC resources.
No long-lived API token, media-proxy key, static TURN credential, or provider token reaches the browser.

Decisions to confirm before implementation¶

Decision	Default
Primary Salesforce use case	Dictation or an in-person meeting through one combined microphone
Selector scope	`embed_public_clients.allowed_agent_ids`, fail closed
Browser media protocol	Audio-only WebRTC with WHIP-compatible signaling
First channel shape	Mono; known identity for dictation and model-derived speakers for meetings
Live meeting diarizer	Deepgram Nova-3 streaming with `diarize_model=latest`
Final transcript	Optional batch rediarization; publish as `refined_transcript`
Salesforce delivery	The CRM wrapper iframe, with `allow="microphone"` added on the capture-owning component only
Recording	Off until tenant policy and consent are configured
Event transport	Existing conversation-api SSE only