multimodal-service

Status: Current | Last reviewed: 2026-10-08

The Python service exposes A2UI rendering tools over MCP. Platform mode writes the existing TrueServe conversation streams. Edge mode adds standalone app sessions, authenticated SSE delivery, and HTTP submission.

For the Salesforce developer handoff, read Agentforce Voice companion forms. The earlier Loop, Hooks, and After Wait probe agents were deleted from ccaas on 2026-09-22. Historical results below remain evidence; use the original demo version 5 for current testing.

Source and runtime

Source: apps/multimodal-service. FastAPI / Python, default port 8080; the devcontainer launcher uses port 3520. MCP SDK 1.27.1 is pinned because SDK 2 changes the FastMCP API. Set MULTIMODAL_PUBLIC_URL to the canonical origin when clients use a hostname other than localhost; MCP host validation includes that configured hostname. This also applies to internal service DNS names in platform deployments.

Endpoint	Purpose
/mcp	Stateless Streamable HTTP MCP, authenticated
/health	Health check
/internal/render	Internal platform render endpoint
/v1/sessions	Edge session creation; operator bearer token
/v1/connections/redeem	Exchange an opaque SMS link for a scoped browser token
/v1/sessions/{id}/events	Session JWT, SSE with replay IDs
/v1/sessions/{id}/ui-action	Session JWT, validated/idempotent app action
/oauth/token	Edge confidential-client OAuth token endpoint
/demo/	Optional mobile-friendly auto-intake example

Pass the application session ID as experience_session_id in tool arguments or X-Trueserve-Session-Id in custom clients. Do not reuse MCP transport session IDs. Edge tools enforce session existence, expiration, and tool_scope.

Render history and app actions use Redis streams. Submissions are retained for late/retried tool calls; deduplication is atomic. The browser receives only a short-lived session token. Operator and MCP credentials remain on servers.

Add a Driver / Add a Vehicle

Enable MULTIMODAL_MODE=edge and MULTIMODAL_AUTO_DEMO_ENABLED=true. New connected experiences select acg-auto, the separate Auto Club Group catalog with ACGAddDriverForm and ACGAddVehicleForm. Sessions explicitly using insurance-auto still render the original AutoChangeForm contract.

The ACG React catalog lives in catalogs/tenant/acg-auto. It owns the field definitions and scoped CSS used by the React components and the standalone adapter. Run apps/multimodal-service/scripts/sync_acg_catalog.py after editing its contract, stylesheet, or logo; --check verifies generated copies. The supplied revised color PDF takes precedence over ACG/sample.md. Mission Gothic is declared with a local font source and Arial fallback because the supplied assets contain no installable font files.

The catalog is independently published in Agent Studio as a multimodal catalog. Its show-component tools use the normal context action contract; the standalone app POST uses data_model as before. This does not add an Agentforce action for live spoken field patches. Completed spoken intake still goes through submit_spoken_auto_change, and does not fill the visible form.

Validated ACG screens: driver on mobile and vehicle on desktop.

MCP action	Behavior
connect_auto_experience	Create/reuse an experience and send its private link using AWS SMS
start_auto_change	Render driver or vehicle intake for the supplied session
await_auto_change	Wait for a submission, or return pending; optional timeout_seconds is 0–20, default 20, with zero for an immediate read
submit_spoken_auto_change	Validate explicitly spoken and confirmed details

The demo collects synthetic data and acknowledges receipt. It does not quote, bind coverage, or update a policy system. Reopening a completed form returns the stored receipt. Create a new session to run the same workflow again.

The app POST wakes an outstanding MCP call. It is not a general push API into an idle Agentforce conversation. The original TrueServe Auto Voice Demo is now on version 5, with optional ANI and internal connection state. It creates its own experience and sends the AWS link after consent; no operator experience ID is needed. Without ANI it asks for a mobile number.

TSAutoMcpStart(waitForForm=false) opens immediately and returns typed status and surfaceId alongside resultJson. The script binds those outputs, selects app or spoken mode, and runs the short-check pattern in app mode: an immediate read before new turns, then at most thirty two-second checks after reasoning. No LLM response is generated between polls. A terminal result reports the validated receipt and clears the mode. Switching Driver/Vehicle reuses one connection, and switching spoken intake back to the app explicitly rearms checks. The default legacy Start call still performs three 20-second waits for old versions. The original version 3's long wait delayed answers to form questions.

The upgrade's live authoring test uses an agent-created no-SMS experience, then switches to app entry and submits both forms using that same session. Supplied ANI was checked with simulated actions; no actual text was sent during these tests. New published Voice sessions ask for a mobile number with ANI absent. See the customer-flow evidence. Checks remain bounded: late submissions are stored but may need another turn. Authenticated Builder/PSTN acceptance and spoken partial-field updates remain outstanding.

The separate TSAutoMcpPoll diagnostic action performs an immediate read using wait_for_submission(timeout_seconds=0). It needs that tool in the synthetic experience's scope and is not enabled in the main demo Agent Script. Checking after each spoken turn can refresh form state; it does not schedule a future check when the customer submits silently. An outbound asynchronous dispatch likewise needs a separate inbound mechanism to trigger the same Voice session. The retired TS_Auto_Voice_Hooks_Probe version 1 used typed status outputs with native before_reasoning / after_reasoning checks and a receipt transition. Live preview and Voice tests caught a rapid driver submission in the after hook. Voice also answered a question while the vehicle form was pending. A delayed vehicle submission was stored but went unread during silence; a later question triggered the before hook and its receipt response. These are request lifecycle hooks, not timers or audio-playback callbacks. The main demo was version 3 when these diagnostic tests ran. See the Salesforce diagnostic notes and hook evidence. The earlier prompt-loop evidence is retained separately. Neither probe establishes the complete Builder experience.

The retired TS_Auto_Voice_MCP_Probe version 3 opened the form immediately and runs the existing Apex MCP wait from after_reasoning. A form-only submission returned the wait and produced a spoken receipt. Caller speech did not shorten the pending 20-second wait; recorded audio answered the field question roughly 18 seconds after its final transcription. Another wait then ran after that turn. The diagnostic was deactivated after testing and is now retired. See the after-wait evidence. This confirms a hook can execute a wait, not that it runs independently of the Voice turn.

Version 6 of this diagnostic adds bounded short waits. TSAutoMcpAwait accepts optional waitSeconds and returns a validated status alongside resultJson. The opening subagent's after_reasoning transitions into a deterministic checking subagent. Its before_reasoning waits two seconds, increments a counter, and transitions directly to the next check or the receipt subagent. It performs at most eight checks per request, with the counter reset on a new request. No LLM response is generated between checks. Version 4 instead reasoned between checks and repeatedly spoke the same field explanation.

In a version 6 Agent API/LiveKit Voice call, the agent answered a field question about four seconds after its final caller transcription, then acknowledged a later silent UI submission about three seconds after HTTP acceptance. Captured audio and Apex traces establish both in the same call. The test uses the Apex MCP adapter; it does not establish direct native MCP cancellation, indefinite idle polling, authenticated Builder browser behavior, or PSTN/SIP acceptance. The main demo was version 3 when these diagnostic tests ran. See the short-wait evidence and setup notes.

The mobile SDK and browser diagnostic investigation identifies public iOS/Android transcript callbacks for a client-owned Voice connection. It has not identified a documented observer-token API for a Builder-owned room. An optional local browser probe observes legacy transcript packets on an existing connection; it is tested against SDK wire fixtures and a Chromium WebRTC loopback, but not yet against an authenticated Builder preview. It does not relay transcripts to MCP or implement interruptible form waits. The production target now explicitly requires native APIs/MCP/prompting over PSTN/SIP. Browser observers and carrier media forks are outside that scope. No documented live caller-transcript subscription or native MCP cancellation on speech has been established for that target. The after-wait tests used published Agent API/LiveKit sessions; PSTN/SIP acceptance remains outstanding.

Google Cloud Run and GECX

The Cloud Run sandbox runs the edge app, Postgres and Redis in one private Cloud Run instance with all three embedded catalogues. One instance stays warm, and replacement resets sessions. The GECX service agent already has Cloud Run invocation permission in the sandbox project. Configure Service Agent ID Token authentication using the GECX setup guide.

MULTIMODAL_GOOGLE_SERVICE_ACCOUNTS is an opt-in, comma-separated identity allowlist for edge MCP calls. The service verifies Google's RS256 signature, issuer, expiry, an exact service/MCP audience, verified email and allowlisted service-account email. It does not trust decoded claims or replace operator/session authentication. Signing-key refresh runs off the SSE event loop. MULTIMODAL_GOOGLE_ID_TOKEN_AUDIENCES supplies additional exact Cloud Run URL aliases. The default empty account allowlist preserves existing bearer/OAuth behavior.

The operator previews the browser through gcloud run services proxy, which adds Google authentication without replacing the session's Authorization header. The GECX launcher supports --google-iam and a local --app-origin. It saves the browser link in a private file. Direct unauthenticated access to the Cloud Run URL is denied.

AWS deployment

deploy/aws-demo.yaml creates Global Accelerator, an HTTPS ALB, a dedicated ECS Fargate service, DNS, IAM and logs. Client IP preservation is disabled on the Accelerator's public ALB endpoint; the ALB permits AWS's published Global Accelerator IPv4 ranges. The prefix list exactly collapses those ranges without adding any addresses; refresh it when AWS ip-ranges.json changes. The example uses us-east-1.

The single demo task includes Postgres and Redis sidecars. Task replacement resets all demo sessions. Use managed persistent stores before adapting this template for production or multiple tasks. Store ports are not exposed by the task security group.

Create a dedicated Secrets Manager JSON secret with independently generated values of at least 32 bytes for api_token, mcp_token, session_secret, oauth_secret, oauth_signing_secret, and db_password. database_url uses that password with the local Postgres sidecar. Supply only the secret ARN to CloudFormation. The example uses client ID sf-auto-demo and scope multimodal:tools.

Validate with cfn-lint, aws cloudformation validate-template, and review a change set. Use an explicit AWS region: the workspace may set AWS_REGION independently of the CLI profile. The template uses the VPC/subnets and ACM certificate supplied as parameters.

For operator-only API testing after deployment:

cd apps/multimodal-service
python examples/create_auto_session.py --output /private/path/auto-session.json

This file contains a private app link and agentforce_input_variables.experienceSessionId. The utility reads the operator secret using the AWS CLI and prints no credentials.

Salesforce and Agent Studio

The example project is examples/salesforce. Use the ccaas CLI alias. Register /mcp with sf agent mcp create, --auth-type OAUTH, the /oauth/token identity-provider URL, client ID sf-auto-demo, scope multimodal:tools, and --client-secret - through a private stdin pipe.

Read the discovered assets and allowlist only the four auto-intake tools. TS_Auto_Voice_Demo is authored in Agent Script and uses small Apex invocable adapters through the OAuth named credential. The start action renders and waits internally; a Voice model did not reliably chain separate render/wait actions in testing. Include all action input/output schema files and grant TS_Auto_MCP_Access to the runtime user. See the deployed example and CLI setup. The dedicated agent leaves existing voice agents and telephone routing separate.

The agent creates the experience through connect_auto_experience. Provide optional input_variables.ANI on Agent Studio's Salesforce Voice node; the wrapper sends $Context.ANI. Without ANI, the agent asks for a mobile number. The returned experience ID is stored in Agent Script state and bound to form actions. Once populated, a condition hides the connection action.

The MCP service implements Agent Studio's share-link/SMS connection pattern: the customer opens /demo/#connect=<opaque code>, exchanges that code for a session token, and subscribes to the same experience's SSE stream. This does not create a second Voice session. The app's Form session connected label reports its SSE connection, not Voice connectivity.

SmsOriginationArn in the CloudFormation template selects an existing registered AWS End User Messaging pool/phone and scopes the runtime role's sms-voice:SendTextMessage permission. Configure MULTIMODAL_SMS_REGION and MULTIMODAL_SMS_ORIGINATION_IDENTITY outside CloudFormation if needed. MULTIMODAL_SMS_MAX_PRICE defaults to 0.10 USD per message part. Request and experience locks suppress duplicate sends; recipient limits allow five new attempts per hour. sms_submitted means provider acceptance, not handset delivery. Failed/ambiguous sends are not automatically retried.

The AWS example also supplies an EC2 container host for unavailable Fargate capacity. FargateTaskCount=0 with DemoHostPrivateIp registers that host behind the same ALB. A template rule prevents two independent stores from serving the same experience. See the deployment runbook for provisioning, health checks, and returning to Fargate.

Validation

Ordinary tests cover platform/edge routing, rendering, and internal auth. The opt-in tests/test_live_edge.py uses a real MCP SDK over HTTP and real Postgres/Redis. It covers OAuth, initialization, discovery, both forms, silent submission while a tool waits, cross-session denial, conflicting retries, late reads, SSE replay, and revocation.

Set MULTIMODAL_TEST_CONFIG to a private JSON file containing base_url, api_token and oauth_secret, then run pytest tests/test_live_edge.py.

The Salesforce example also includes ten Apex tests for MCP request handling, the bounded wait loop, and errors, plus six tests for the separate immediate polling diagnostic, including typed hook outputs and invalid status/phase rejection. All sixteen passed on 2026-09-22. A submitted form after the combined wait window remains available for a later read; it does not automatically resume an idle Voice agent.

Connection tests cover concurrent retries, recipient isolation, expired/revoked links, no-SMS spoken intake, provider failure, and the AWS SDK request contract. Builder testing uses the Agent Script preview with live actions; set ANI in preview variables or supply the mobile number conversationally. Only perform a real SMS test with an authorized destination.

See the integration guide for the app contract and user guide for the demo flow.