Director Agent Assist stages

Status: Current | Last reviewed: 2026-10-12

Use this guide for Salesforce and Headless Scene configuration, manual cards, saved Copilot output, workspace replay and visual cues. Director covers shared persistence, audio generation, the voice stage and MP4 worker operations.

Salesforce and Headless chapters

The chapter's Stage tab selects Voice Agent Call, Salesforce Agent Assist, or Headless Agent Assist. Legacy documents without a stage retain the voice renderer. Desktop chapters store stage.type=agent_assist, app=salesforce|headless and a versioned presentation snapshot in the existing schema-version-1 document. Custom App and standalone Amazon Connect Workspace are not offered in this release.

lib/director/workspace-model.ts owns bounded schemas, presets, record bindings, workspace actions, captured Assist events, annotations and the pure workspaceAt projection. components/director/workspace owns separate Salesforce and Headless presentations over the shared interaction state. Salesforce presets follow the authenticated devsdo Clear Connect and Business Plus references; the pinned assets and source manifest are in public/director. Headless restores the original v8 canvas presentation from apps/headless-app at 17e3dc78^ (v8 port 3c45b053), using its scoped CSS, Figtree/Poppins/Inter fonts, narrow navigation, greeting, canvas/rail slots and persistent conversation composer. It does not render Salesforce record tabs, profile sidebar or Omni-Channel. Private browser session/reference files under .local-data are not application assets.

Both stages configure app label/logo/colors, records, incoming context, components and Assist. Salesforce configures the primary Account tab and record subtabs, optional Voice Call record, B2B/B2C profile/sidebar, ordered incoming-work context, and console sections. Sections select main workspace, sidebar or Assist rail and may be scoped to a subtab. The section types are details, summary, timeline, related records, form, static HTML/CSS and published catalogue component. Existing catalogue versions and the shared A2UI renderer support domain-specific dynamic forms; there is no duplicate Takeda form engine.

Static blocks can select the bundled Inter face with font-family: "Scene Inter", Arial, sans-serif. The parent loads only the fixed /director/fonts/inter.ttf asset and embeds its bytes into the opaque iframe; authored CSS still cannot load URLs or import styles. Preparation is keyed by both HTML and CSS. The bridge waits for font loading and image decoding before reporting readiness, measuring height and publishing cursor targets. A failed font load prevents an incomplete export. Preview and export use this same path.

Allianz typography follows the inspected DevSDO component: Arial in the Case header and Assist heading, Inter in the illustration, and the system stack in Salesforce navigation and the producer profile. See the Allianz Scene reference for sizes.

Headless stores optional stage.headless canvas/composer configuration and per-section canvas={region:hero|main|rail|full,span:1|2} placement. Existing snapshots without those fields use the original shell defaults and project their authored sections into canvas slots. Home/Work selection, navigation expansion, suggested prompts, the conversation panel and the voice tray use saved workspace events. The Headless voice tray presents the existing phone state machine in source-derived Headless call controls; Salesforce retains its native Omni widget. Captured Headless surfaces retain the original layout.region/order/span/title hints, merging later partial hints and restoring placement on reverse seek. An unhinted surface uses the original legacy mapping or main canvas with span 2. Hero surfaces replace the fallback greeting unless a headline is explicitly authored; catalogue titles use the source section-header treatment. Switching applications resets the presentation while keeping records, branding, incoming context, phone state and Assist configuration. Undo restores the layout. The retired Headless application, SSO flow and provider runtime are not restored.

Salesforce headers store explicit ordered highlights and actions arrays. Highlight order remains stable after JSONB persistence, and labels and values are authored independently from record field names. Source presets include Clear Connect Interaction actions and Business Plus Case actions. Sidebar sections support definition fields, numbered summaries and bulleted summaries. A group with tabLabel and groupActive builds a replayable tab set in its region. The Business Plus preset includes Company Details/Other Locations, record-related tabs, Current Case/All Cases and an ordered status_path. Rich activities save their channel, title, actor, recipient, timestamp, status, priority and detail rows; timestamps come from authored values. Group selection is a bounded ui_set event using sectionGroup.<group-id>.

{{account.name}}, {{contact.phone}}, {{case.number}}, {{voiceCall.status}} and related-record bindings resolve against the chapter's records, followed by reusable Details. Workspace events update those records. Form edits remain local to the projected form until a form_submit event applies bound record fields. Subsequent desktop chapters can inherit final fields for matching record IDs. Every seek rebuilds state from the saved snapshots and preceding events. Main-workspace scrolling records a bounded percentage in workspace.scroll. Omni-Channel preserves separate Inbox and Phone scroll positions, including long incoming context. Programmatic restoration is excluded from interaction capture; content resizing reapplies the saved position without creating another moment. Speech, generated audio and Assist replay input resolve record bindings from the authored chapter snapshot. Console edits update the visual state; they do not rewrite a saved spoken line or its audio take.

Omni-Channel is the only Salesforce utility. Its Phone and Inbox share state with incoming offers, arbitrary ordered context rows, acceptance/decline, outbound dialling, the twelve-key pad, mute, hold/resume, consult/transfer, conference, end and wrap-up. The visual phone state machine does not invoke a telephony provider. Salesforce uses the active org's OpenCTI v1 standard call-entry and control templates. During consultation, Swap alternates the held participant. Merge activates both conference legs; each leg can then be held independently. Participant audio follows those same projected leg states. The first desktop chapter starts at zero and ignores the voice-only callOpeningDuration; switching back preserves that setting.

Static blocks render in an opaque-origin iframe with a restrictive CSP and a fixed, nonced action bridge. Resolved HTML is validated and sanitized. Author scripts, embedded documents, external requests and form destinations are blocked. Owned Scene images and branding images are fetched by the authenticated parent from allowlisted application routes and inlined into the sandbox. Image loading must finish before a frame is ready for export. Declared data-scene-field, data-scene-action, data-scene-show and data-scene-hide attributes produce replayable field/action state. data-scene-target exposes workspace.<section-id>.<target> for visual cues; the bridge reports geometry and target picks without executing the picked action.

POST /api/director/[id]/images accepts a bounded 5 MB image upload under ManageAgents. Signatures must match PNG/JPEG/GIF/WebP; SVG is sanitized while preserving static geometry, text and internal symbol references. Images use the same tenant/project-owned assets table with image metadata and appropriate MIME types. Saves and exports validate canonical project image URLs and template-image access. Duplication owns its image bytes; deleting the source Scene cannot break a copied logo. External HTTPS image URLs remain dependent on their source.

Manual catalogue cards and feedback

An Assist agent is optional. Stage → Agent Assist cards → Manual cards adds published components directly to the chapter. Each card pins its catalogue ID, version, name, schema, attributes, data model and declared event metadata. The picker includes tenant/shared agent_assist and multimodal catalogues. Schemas and event hints come from the published version's immutable source_manifest, not editable component rows.

Chapter sections display from the start; kind: component moments display at their scheduled time. New manual moments default to componentMode: append. Explicit replace, and legacy documents without this field, replace earlier moment cards in the same region. Section cards remain independently configured. The schema form and JSON editor share the same bounded attributes.

lib/director/component-interactions.ts owns typed card targets ({type: section|moment, id}), response rules and pure componentStateAt. kind: component_event moments support semantic actions, ordinary button clicks, proxy requests, props/data-model patches, scoped fields, standalone toasts and reset. Captured actions retain the native action name, context, data model, delivery mode, surface ID and label. The clicked DOM label is stored separately: catalogue action labels need not match button text.

Rules match the first exact action name, button label or capability/action pair. Each response specifies success/error, a 0–10-second delay, optional JSON result, shallow props/data-model patches and a toast. Toasts have info/success/warning/error variants, title, message and a 0.5–15-second duration. Duplicate triggers are reported as invalid rather than given ambiguous ordering. These are authored responses, not receipts from a CRM or agent. External calls and conversation writes remain disabled by the Scene transport.

DirectorComponentHost supplies the same projected card state and toast state to preview and export. Capturing native component actions saves semantic payloads; ordinary buttons fall back to scoped click events. Form edits are scoped to their card, so identically labelled fields in different cards do not overwrite each other. Replay invokes the real component's button where available and returns its configured outcome, preserving native error/retry behavior. Stable card instances avoid replaying a just-captured click. Reverse seek and reset rebuild local state. Reset also drops preceding fields, pending responses and toasts for that card. An exact authored DOM-button rule intercepts the original handler, so a simulated Copy result cannot attempt an OS clipboard write or display a conflicting error.

Recorded response timing follows the playhead. Response/toast tails extend the chapter without delaying concurrent dialogue. Unrecorded try-outs use a bounded local preview clock, freeze later scripted events and do not modify the document. Leaving a try-out by seeking discards its temporary feedback, including when returning to the exact frame where the card was tried. All component-event frames are rendered at export frame rate, including Voice Agent Call chapters. Toasts render within the stage: Salesforce uses the captured SLDS Plus treatment; Headless uses its own typography and palette. They never use the editor's global notification system.

Missing card references, events before their card appears and empty toast text produce actionable validation errors before export. Asset ownership and duplication cover initial props, response patches and event payloads; copied moment IDs are remapped only in typed targets.

Generated or captured Agent Assist

The Agent Assist stage can render authored catalogue moments or save actual Copilot output. POST /api/director/[id]/assist supports generate, capture and cancel; GET lists tenant-visible agents, source conversations and recent runs, or retrieves one chapter-bound run by jobId. These controls are under Generate or capture · optional and load only when expanded.

Generate freezes the resolved human dialogue and host context, then uses the existing durable simulation jobs and Copilot transcript replay transport. Only copilot/agent_assistant agents qualify. The run is always sandboxed: configured external tools still require valid mocks, while real Copilot model workers generate the UI. No separate simulated-caller or scoring integration is needed. The server-only captureOnly target flag and purpose=director_capture skip evaluation assertions/scoring; ordinary test runs retain their requirements. CALL_STARTED receives Account, Contact, Case and optional Voice Call records plus authored host context. Narration is excluded. Replay supports up to 30 human lines per chapter.

One active run per tenant/project/chapter is admitted under an advisory lock. The existing worker owns startup, capacity, retries and cancellation fencing. The browser may leave and follow the run later. Partial output is not imported. A completed take contains normalized createSurface, updateComponents, updateDataModel and deleteSurface events, pinned catalogue versions, source conversation ID, agent version/config identity and a canonical input fingerprint. Generated events anchor to their triggering line's end with observed response latency. Existing-conversation capture keeps relative UI-event timing and pins the currently accessible catalogue version; it does not claim to recover an unknown historical bundle.

The browser replays saved events through the shared A2UI components with an inert transport, scoped catalogue CSS and related-surface context. Main-workspace and rail layout hints select the corresponding console region. Component loading and image readiness gate export. Save/reload and seeking do not call the Copilot. Changing the dialogue, records, selected agent or other capture inputs marks the take stale. A result arriving after its inputs changed is rejected instead of silently replacing the current chapter's take.

Narration and visual cues

Caller and Agent represent the customer and human representative in desktop chapters. Additional persona lines default to narration; an explicit call audience supports a consult participant. audioSegmentsForMoment supplies the same call-state/mute/hold/consult gain intervals to browser Web Audio and the PCM export mixer. Silent intervals resume at the correct source sample offset. Narration and sound effects remain separate audience tracks.

Annotation moments support highlight, tooltip, spotlight, cursor and zoom. They use the existing start/end/word anchors and durations in voice and desktop chapters. DirectorAnnotations resolves stable targets through native phone shadow DOM, catalogue documents and the validated static-block geometry bridge. Camera transforms and annotation positions derive from presentation time. An invisible/missing target produces a preview error and prevents incomplete exports. Target picking records the target without triggering its action. The export canvas mounts in a dedicated body portal that hides application chrome and developer overlays, preserving the complete stage rectangle.

Verification and reference coverage

The 2026-10-07 source audit used authenticated Clear Connect and Business Plus Case pages, their rendered DOM and computed styles, native OpenCTI modules and the org's SLDS styles. Reference comparisons use a 1920 × 1128 browser cropped by the 48 px sandbox banner, preserving a 1920 × 1080 image without rescaling. Preset comparisons use the same records and images as their reference; shipped defaults use synthetic records. Reference data and sessions stay private.

Automated comparisons check both element geometry and pixel differences. The twelve dialpad buttons and multiple record/sidebar regions match exactly. This is not a claim of full-canvas pixel equality: residual icon, border and subpixel differences are recorded separately. Connected, consultation and conference layouts were checked against the active OpenCTI v1 source templates; a real-call screenshot baseline is not included.

The rebuilt Headless comparison runs the historical v8 fallback home with the same synthetic identity at 1920 × 1080, DPR 1 and its original 110% zoom, without image registration or resizing. Both collapsed and expanded navigation match all measured geometry. The included frame is 99.99367% byte-exact; the remaining 131 pixels differ by at most 3 RGB levels around the Activity control. Navigation, greeting, rail and composer regions are byte-exact. The new Voice affordance is explicitly excluded. This baseline covers the original empty canvas, not custom components or the added voice tray. Private reports and source hashes live under .local-data/scene-headless-reference-2026-10-07. Actual editor-preview and render-only typography also matches for the headline, subheadline, composer input and card headline, including fonts, sizes, weights, line heights, spacing and colors.

Validation covers database-backed Director tests, authoring and image ownership, Salesforce/Headless replay, native call controls, independent scroll restoration, actual Copilot capture and ElevenLabs human/narration audio. The MP4 regression checks the isolated canvas, chapter fonts, visual cues and decoded audio gates. The chapter-transition regression also checks reverse seeking, exact boundaries, all transition styles, return-to-voice activity isolation and saved authoring. A 9.2-second Salesforce → Headless MP4 verifies the crossfade against decoded endpoint frames and confirms silence between complete participant/narration clips.

Manual-card regressions use real published AppealDraft and StageAssist bundles in both shells: scoped form edits, native event context, confirmation, failure, retry, delayed props/data changes, toast expiry and seek/reload without live writes. Authoring tests cover two appended SuggestedAnswerCard instances and independent responses without selecting an agent. A 4.5-second Voice → Headless MP4 verifies fractional response timing and toast expiry against decoded frames.

See the application API and user guide.

Allianz producer presentation

The allianz preset adapts the existing Salesforce shell to the DCC producer layout. It supplies compact header actions, the five-stage status Path, producer verification, sidebar details and Assist styling. The project supplies business records and catalogue versions. presentation: "modal" sections render above the console and use the existing visibility reducer, including reverse seeking. Cursor fromTarget, moveDuration and clickAt values are projected from the timeline; they do not invoke actions or rely on browser animation clocks. Allianz Assist catalogue sections use the rail's padding without another inset inside the catalogue iframe. The Scene catalogue's type sizes and guidance/card alignment follow the supplied SVGs.