Skip to content

Director authoring and rendering

Status: Current | Last reviewed: 2026-10-20

Director edits Scene agents at /agents/[agentId]/director. Scene agents are an optional deployment feature. Set FEATURE_SCENE_AGENTS=true in ai-manager's runtime environment and restart ai-manager to enable them. The default is off; the parent Agents capability must also be enabled. The server supplies the same flag to Studio's creation menus, repository filters and Home portfolio. Disabled Scenes are omitted from agent lists and direct reads; Director pages, creation, editing, assets and exports return 404. Existing saved data is retained for a later opt-in.

This is a runtime-only flag, independent of FEATURE_ANALYTICS_AGENTS. Public build variables cannot enable either type, so a promoted image defaults off unless the destination environment opts in. Local Vanguard opts in through its gitignored .env.local; checked-in templates remain false.

agents.subtype=scene_agents, pipeline_type=text_to_text and the minimal version envelope agent_type=scene identify authored experiences in the shared Agents library. The transport field is storage compatibility only: Scenes cannot be published to the runtime. New Agent creates Scenes without a voice agent or LLM configuration. Selecting Scene Agent creates the starter as Untitled Scene and navigates directly to the script; there is no setup dialog or landing page. The newAgent=1&type=scene_agent shortcut uses the same action. Creation stays disabled until navigation completes, preventing repeated clicks from making duplicate Scenes. Legacy voice-agent Director links remain supported. It owns authored demos independently of agent versions, live conversations, the collaboration document and Mock Caller.

Personal-assistant journeys add a phone app, IVR keypad, standalone browser and Salesforce Messaging Session. See Personal assistant Scenes for the shared session, audio and editable return starter.

Alder Homeworks Scenes add guided product support, channel discovery, persistent service guidance and generated inspection footage. The native Video moment supports ElevenLabs generation, upload, take comparison, trim, display overlays and reviewed event markers.

Persistence and access

Migration 074_director.sql creates director_projects, director_assets and director_exports. Each has forced tenant RLS and explicit tenant predicates. Child rows use composite tenant/project foreign keys. Project writes require ManageAgents, validate the bounded document and compare the saved revision. POST /api/director/scenes atomically creates the Scene identity, initial version and project. A Scene owns one project; the editor resolves it from the agent identity and opens it automatically. Creation returns /agents/<id>/director. Legacy ?demo= links normalize to this address after resolving ownership; a Scene route cannot select another agent’s project through that query parameter. The Scene back arrow returns to the unfiltered /agents list. It uses the same destination after saving pending changes; it never applies a Scene type filter. The internal project UUID remains in API/asset paths and frozen render URLs. Saves synchronize the identity/version name in the same transaction. Duplication uses sourceAgentRef and copies audio and uploaded images to new project-owned keys, remapping take IDs and image URLs. Legacy promotion uses projectId, locks the row, preserves all assets/exports, and leaves the source voice agent intact; repeated promotion returns its Scene. The document's sourceVersion is provenance; export routes use the current owner. Generic pipeline creation/clone/ZIP export and runtime publishing reject Scenes with directions to the appropriate Director action. Scene deletion cascades through the existing tenant/agent FK; existing object-retention behavior still applies.

Reads and asset downloads require RunAgents. Every API route uses withAuth.

The browser serializes autosaves and retains unsaved edits after failures. The header reports save status; there is no always-visible manual Save control. Retry save appears only after a failed save and uses the same revision-checked queue. Both the Scene toolbar and library action are labeled Export; the recording format remains MP4. Export immediately exits enlarged Preview, selects the Exports tab and shows startup status before waiting on autosave or the export POST. Its returned job is inserted into the list and a toast confirms acceptance; the normal poll continues progress. A ref prevents duplicate submissions during startup. Audio generation returns a stored take; the editor then saves the association. Existing recordings retain a document snapshot and revision. Audio and video objects use the shared assets store under a tenant/project prefix; object keys and provider credentials are not accepted from the browser.

Documents are limited to 2 MB, 40 scenes and 500 moments. Export duration is limited to 600 seconds. Each voice take is bounded to 120 seconds. Assets are checked against their owning project and stored metadata before accepting document updates or rendering.

Guided support and video calls

The Alder Homeworks Scenes demonstrate product visual product tiles, a website support chat with appointment widgets, owner SMS verification, a mobile Safari video portal, and a Salesforce Video Call. The website has customer-facing product/chat content; policy suggestions appear in the representative's Agent Assist rail as rich recommendation and coaching cards. Independent customer and representative video tracks share the preview/export clock. The short front-camera check plays once, while the inspection continues through the brew fault and serial label. Paused customer video retains audio; the floating preview can show Jordan during the return form.

Typed chat and spoken narration are separate moments. The website stage does not render narrator subtitles. Narration can anchor to visual actions so it does not hold navigation. Ari replies can show thinking before delivery. Optional chapter clock/date and conversation-scope settings support a push notification that opens a fresh appointment-reminder conversation.

Script and playback

The Studio toolbar and canvas tokens frame two panels: script and recording preview. Internal scenes are labeled chapters in the UI. Chapter selection, management and inline moment settings share the script panel; there is no fixed chapter sidebar or inspector. Insertion menus splice at a chosen array position. Their scroll height respects both 70dvh and Radix's available viewport space, so the expanded workspace/visual-cue choices stay reachable near screen edges. Scrubbing and playback follow the current chapter. The optional timeline hides unused tracks and includes persona/pause tracks; authored activity is labeled Agent actions, separate from the AI Activity output. Editor CSS excludes [data-director-stage] and its contents so preview and export share the native activity presentation. Playback updates the selected chapter only when its identity changes. Do not dispatch that selection from an effect on every clock tick: repeated no-op updates can trigger React's maximum-update-depth warning during playback. Catalogue renderer inputs retain their references while their resolved JSON values are unchanged; actual card events and record bindings still update them. The shared phone initializer must wait for window.adapter before loading or initializing Janus, including its fallback timer. A slow adapter download must not produce an unrelated adapter is not defined error on Director pages.

Chapters are also the recording's acts: each owns its stage and moments, with no additional hierarchy. Optional scene.transition={type,duration} stores the incoming cut, crossfade, fade-through-black or slide (0–3 seconds). Missing transitions remain cuts; + Add chapter defaults to a 0.6-second crossfade. Markdown imports retain cuts. The compiler inserts a silent interval after the outgoing chapter's complete content and before incoming content. Compiled transitionStart/transitionEnd bound that interval; scene.start remains the content start. The first chapter ignores its incoming transition. Transition time counts toward the 600-second cap.

Optional project closingFadeDuration (0–3 seconds) adds a fade to black after the final chapter, followed by 0.25 seconds of black. Omission or zero preserves the existing ending. closingFadeAt freezes the complete final frame and derives its opacity from timeline time, including backward seeks. Export captures every fade frame even for voice-only Scenes. Details → Ending exposes the duration; the fade and black tail count toward the 600-second limit without trimming content.

activeSceneAt keeps the outgoing chapter selected through the gap. transitionAt supplies exact outgoing-end/incoming-start frames; boundary actions are retained. Two keyed stage layers use timeline-driven easing for playback, backward seeking and MP4 export. Each layer scopes visual cues to its chapter and disables interaction capture during the transition. Crossfades composite the incoming frame over an opaque outgoing frame; fade-through-black uses a black backdrop. Export treats transitions as full-motion frames, including voice-only recordings. Stage → Transition into this chapter and the timeline boundary strip expose timing and a boundary preview without changing moment coordinates. Frozen boundaries apply to projected Scene state and native phone animation. Custom catalogue or static CSS animations with independent clocks are not sought by this compositor; use static presentation for those components at a boundary. Consecutive cuts within the same stage type, desktop app and preset share a mounted layer. A new phone card using the same catalogue version reuses its styled iframe and resets only the card contents. Chapter changes and backward seeks reset local card state while preserving the iframe; animated transitions retain separate outgoing and incoming layers. The browser regression in director-playback-continuity.spec.ts checks continuous playback, console errors, card continuity and seek restoration. Static HTML blocks also retain their iframe and previous prepared document while replacement HTML, images and fonts are prepared. A pending marker still makes capture wait for the new document. Bridge messages carry a document identity so late readiness or action messages from the previous snapshot are ignored. During continuous playback, an authored tap no longer remounts hosted card contents; the host's chapter/seek epoch still resets them when required. Static HTML snapshots also apply their initial field values and visibility flags before first paint. Subsequent field and UI updates travel over the existing bridge without rebuilding the document. Cursor geometry is compared before dispatching a React state update. Returning the previous value inside an updater still queued passive updates on successive playback ticks after a cue, eventually triggering the update-depth warning. The continuity regression includes a cursor followed by ongoing playback.

DirectorTimeline uses pointer capture and an animation-frame drag preview. It preserves the grab offset and a stable seconds-to-pixels scale during a move. The ruler, zoom/Fit controls and edge scrolling support placement beyond the old chapter end. Both moving edges snap to other tracks’ starts/ends within nine screen pixels; Alt or the Snap toggle bypasses snapping. Overlaps stack within a track. Pointer cancellation/Escape discards the preview; a drop is one save/Undo operation. Arrow keys nudge by 0.1 seconds (Shift: one second). The Scene's sidebar inset allows its flex child to shrink (min-width: 0), so wide tracks scroll inside the timeline without pushing the whole canvas sideways.

moveTimelineMoment freezes unrelated unanchored moments at their current positions, preventing the timeline compiler’s sequential cursor from moving them as a side effect of dragging another clip. Existing anchors remain linked to their speech. Moving an anchored moment directly replaces its anchor with a pinned start. Automatic script timing still applies until manually placed. Undo/Redo clamps the playhead when the chapter becomes shorter.

Record phone interactions is enabled on paused components. Capture inserts field/tap/scroll moments at the current chapter-relative time, coalesces repeated field/scroll changes at that time, and preserves distinct taps. Playing or seeking ends capture. Undo retains the state before a captured interaction.

lib/director/model.ts contains the schema, timeline compiler, audio identity and phone projection. Scenes follow one another. Moments normally follow the previous moment, may have a pinned scene-relative start, or may follow a spoken line's start/end/word plus an offset. Audio duration controls ripple timing. Word anchors use generated alignment, with estimated placement before audio exists; missing words produce export-blocking notices.

ScriptLine preserves textarea selection for an Insert detail menu and shows resolved text without altering authored {{key}} tokens. The same control edits messages, activity results and component string properties. Details uses an explicit name/value form with validation; spaces in names become underscores. The compiler rejects missing/empty referenced details and audio generation checks the selected line before contacting ElevenLabs. Take identity includes resolved words, so unused-detail changes retain existing audio.

phoneAt reconstructs state from the timeline on every seek, including cross-scene phone navigation, authentication, visual connection, component, field values, scrolling and taps. Tap replay waits when a button is disabled, including aria-disabled, and observes its enabled state before advancing. This lets earlier photo-selection taps finish their React updates before the upload tap runs, including when seeking directly across all three actions. DirectorStage uses native PhoneFrame, AppShell, MessagesApp, MailApp, BrowserApp, HomeScreen, LockScreen, NotificationBanner and call-state components. Authentication and connection moments show an incoming banner before opening Messages and the browser; the duration compiler is unchanged. Customer auth uses the same side-effect-free PhoneCustomerAuthPresentation as live Preview simulation, with the customer-facing heading Identity Verification. The recording starts at 9:41 local time. PhoneFrame, LockScreen and the message relative-date reference receive that start plus elapsed playback time, so the phone clock advances, pauses and seeks with the recording in preview and export. Message and activity timestamps remain anchored to their moments’ start times. Optional clock props preserve live defaults. NotificationBanner’s optional presentationTimeMs seeks the same native entrance curve without its live dismissal timers. The compact status-bar clock omits AM/PM to fit beside the call island; full message timestamps keep it. PhoneMotion pauses native CSS animations and seeks them using screen-relative time; exports capture native motion at 24 fps instead of reusing static frames. No duplicate SMS/browser markup should be added to Director. Catalog rendering uses the shared A2UI renderer, pinned version namespaces and the phone's stylesheet boundary. Its transport does not invoke backend actions or write conversation state. Phone component moments can opt into componentEntrance: "slide" for a 0.4-second fade and upward arrival, derived from timeline time. Instant display remains the default. Both modes use the same pinned catalogue renderer.

AI Activity uses PreviewBrainPanel's presentational TranscriptTurn, InlineAction and ActionCard components with authored evidence. No observer stream or conversation row is created. Both transcript and action timing follow playback. An optional voice-stage showActivity overrides the recording-wide setting for that chapter; omission inherits it. Stage exposes this as AI Activity in this chapter, including Phone only. Transition layers use their own chapter's setting, keeping the outgoing and incoming phone layouts consistent. Each capability/activity moment can pin its inputs and result open in the recording. Director explicitly sets timestampPrecision: 'minutes' and omits action duration metadata. This affects both inline and split activity layouts, including expanded inputs/results, while preserving live agent Preview preferences. Pinned catalog versions retain the catalog's CSS wrapper and copy only that version's stylesheet into the phone document.

Browser audio uses one Web Audio clock and independently scheduled caller, agent and sound buffers. Pausing cancels all sources. Replaying or seeking uses saved audio; it never calls ElevenLabs.

Salesforce and Headless chapters

Chapters can select Salesforce or Headless Agent Assist alongside Voice Agent Call. They share the Scene timeline, saved assets, visual cues and audio routing. See Agent Assist stages for workspace schemas, reference assets, Omni-Channel state, manual catalogue cards and responses, captured Copilot output, static HTML isolation and annotations. The application API describes image uploads and durable Assist jobs.

Phone typography

Director passes the bundled Scene Inter family to the shared PhoneFrame. The phone's clock, call timer, caller name, Messages and other native chrome use this portable fallback from call-screen-fonts.md, so Linux exports match the editor. Regular and semibold faces preload for phone chapters; the existing export font-readiness gate waits before capturing. Caller names wrap in full. Native phone text is independent of Scene brand fonts and catalog card fonts. See Phone emulator typography for the shared metrics and default system stacks.

Desktop fonts follow the selected application and component. The Allianz preset uses the audited Arial/system/Inter families instead of a universal Salesforce font. Static HTML blocks can use the bundled Inter face through the shared isolated font-loading path, which waits for the embedded font before preview/export capture.

Call direction and opening

Details → Call setup exposes callDirection (inbound or outbound) and callOpeningDuration (0–30 seconds). Direction is relative to the agent: inbound means the customer calls the agent; outbound means the agent calls the customer. The recorded phone always belongs to the customer. Thus outbound uses the shared CallScreenIncoming with Accept/Decline, while inbound uses CallScreenCalling. The opening connects automatically; no live call or transport action occurs.

The schema defaults to inbound with zero opening duration so old documents and exports retain their original timing. New full and blank starters explicitly use 1.5 seconds. Newly authored Scenes should always allow 1–2 seconds before the first spoken line; use an explicit pause for a desktop-first Scene. Selecting another direction enables a three-second opening if its duration was zero. Authors can then choose zero to start connected. A card above the first chapter opens these settings; editing either setting seeks to the opening.

For a recording beginning with Voice Agent Call, the compiler starts chapters after the opening, offsetting sequential, pinned and anchored moments together. Voice take identity is unchanged. phoneAt projects the opening from the saved direction and time, and begins the connected screen’s animation at the answer time. The phone timer excludes the opening and freezes at the ended moment. Explicit phone moments retain their saved screen. The renderer samples the incoming screen’s native pulse throughout ringing. Preview and export use the same projection and frozen settings; no migration or phone-emulator package changes are needed.

Scene branding

The optional document branding field stores a bounded presentation snapshot: template ID/name, primary/secondary/header colors, caller/agent bubble colors, logo and avatar URLs. Old documents without it retain their original appearance; no database migration is needed.

GET /api/director/branding lists active templates in the current tenant under RunAgents, without requiring the ManageTools permission used to edit templates. Selection copies supported presentation values. Names remain separately editable. Save validates a changed snapshot against its active, tenant-owned template; an unchanged existing snapshot remains valid after template edits or deletion. Update from template explicitly refreshes the snapshot. Arbitrary snapshots and image references cannot be attached through the project API.

GET /api/director/branding/image?key=… serves uploaded images only when linked to the caller's tenant through an active template, saved Scene or frozen export. It validates the image key and uses the assets store; remote template URLs are loaded in the browser, never fetched by the server. Immutable uploaded image keys preserve snapshots. External images remain dependent on their source URL.

DirectorStage applies branding inside the phone's browser content and AI Activity. Native phone applications and the shared authentication presentation retain their original styling. Catalog theme variables are passed inside the stylesheet iframe because outer-page CSS does not cross that boundary. Components with fixed colors retain their published styling. The shared TranscriptTurn accepts optional bubble styling; callers that omit it keep the existing Preview presentation.

Voice and sound generation

New Sound effect moments default to Standard phone ring. The optional soundPreset: "phone-ring-v1" selects the versioned two-second recording in public/director/sounds/phone-ring-v1.wav, extracted from the supplied Takeda demo at 02:12–02:14. Its PCM repeats with a 0.5-second gap, with 5 ms boundary fades, for the authored duration (default 4.5 seconds). Preparation saves a normal project-owned immutable take and needs no ElevenLabs integration. Its fingerprint contains the preset version and duration, independent of voices or description. Existing generated sounds keep their source identity. The visual Sound source picker switches between the recording and generated effects; changing duration requires preparing a new take. Use this preset for telephone ringing in future Scene scripts, including Agent Assist incoming-work moments.

Documents may include up to 12 personas, each with stable ID, unique display name, voice settings and showInActivity (default false). A persona speech moment references personaId. Schema validation rejects missing/duplicate identities and reserved speaker names. Old documents default to an empty array; Caller/Agent audio identity remains unchanged. The schema stays version 1 and requires no database migration.

Voices exposes Add persona, per-persona voice/model selection and transcript visibility. Insertion menus and the Speaker field use stable IDs; script import resolves persona names. In-use personas cannot be removed without reassigning or deleting their lines. voiceForMoment is shared by take identity and provider generation, so changing one voice invalidates only that speaker’s takes. Renaming a persona does not invalidate audio.

Additional speech uses the same playback/export mixer and receives its own timeline track. Hidden narration remains audible without being presented as the agent’s activity. Visible personas use TranscriptTurn.speakerLabel; live Preview callers that omit the prop retain their existing labels. Export snapshots preserve cast, voices and transcript choices.

The shared Combobox now uses a native scroll container bounded by available popover height. A max-height-only Radix ScrollArea root had let its viewport grow with the list, preventing scrolling. Keyboard navigation reveals the active option, closing clears remote search, and the Scene picker retains the saved voice label when it is outside current search results. Pointer highlighting follows actual mouse movement, not mouse-enter events: scrolling beneath a stationary pointer must not replace the keyboard selection.

Generation resolves the selected tenant-visible ElevenLabs integration on the server. Voice requests use the timestamp endpoint with eleven_v4 or eleven_v3; generated effects use the sound-generation endpoint. Takes are stored as 24 kHz mono PCM WAV with duration, waveform peaks, word alignment and source identity. The identity includes substituted text, direction and voice settings. Changed input invalidates selection without deleting prior takes.

Generation is explicit, rate limited and time bounded. Stop generating finishes and saves the current take, then stops the batch. A partial batch preserves completed takes. Keys never enter browser state or logs.

MP4 rendering

POST /api/director/[id]/exports saves an immutable snapshot row and starts a Next.js after task. A tenant advisory lock admits one active render at a time. The renderer uses Playwright Chromium to open the same DirectorStage through an authenticated page. The isolated browser receives host-only session cookies; other authorization headers are forwarded only to the configured local app origin. Catalog and font hosts do not receive them.

The render page exposes a seek function after hydration. The renderer seeks on a 24 fps clock and streams lossless PNG frames with backpressure to FFmpeg. Voice-only recordings remain 1440 × 1000; any desktop chapter selects a fixed 1920 × 1080 canvas for the entire recording, centering voice chapters within it. Desktop and active annotation frames are captured individually; unchanged voice-only frames may reuse their previous image. Original voice samples are mixed separately at their timeline offsets, including overlaps and sound volume, then encoded as H.264/AAC MP4. The encoder uses four threads; frames wait for isolated catalog styles, fonts and all branding images before capture. Broken branding images produce a visible preview error and fail the export instead of silently producing an incomplete MP4.

The task updates progress, observes cancellation, has a 45-minute deadline, and cleans up Chromium, FFmpeg and temporary files. Finished MP4s go to the shared asset store. Exports with no heartbeat for five minutes are marked interrupted when the export list is read; the user can retry from saved assets. This first release uses the app process for render execution. Jobs do not survive an application restart, but project, assets and export history do.

Runtime dependencies are playwright-core, Chromium, FFmpeg and fonts. The ai-manager image installs Chromium and font-noto and explicitly copies playwright-core alongside the standalone server.

Setting Default
DIRECTOR_RENDER_ORIGIN http://127.0.0.1:${PORT or 3001}
DIRECTOR_CHROMIUM_PATH System Chromium when installed, otherwise Playwright's browser

Keep the render origin on the same app instance. It is trusted deployment configuration, never a request parameter. For local development, install the matching Playwright Chromium if the browser is not already cached.

Validation

lib/studio/optional-agent-features.test.ts covers default-off behavior with baked public flags, independent opt-ins, the parent Agents switch and route boundaries. e2e/optional-agent-features.spec.ts checks menus, filters, lists and direct links against the running deployment's flags; run it with the same environment as ai-manager. Both disabled-Scene and enabled-both browser runs passed alongside Scene creation/duplication and Analytics preview checks.

lib/director/model.test.ts covers planned duration, ripple/word timing, audio invalidation, backwards seek, overlap and script import. It also verifies missing-detail validation and selective take invalidation. Call setup cases cover legacy defaults, duration bounds, both directions, backward seeking, and offsets for pinned/anchored timing without invalidating audio. e2e/director.spec.ts covers authenticated persistence, conflicting saves, inline insertion, chapter following, saved phone interactions, phone/AI Activity preview, and a full four-minute H.264/AAC export with audio at both ends. With the published ACG demo catalog available, it also checks native styles, field restoration, tap replay and a native-card MP4. The success case in e2e/customer-auth-preview.spec.ts compares native SMS dimensions, typography, colors and layout between a live Preview phone and Director before completing authentication. Rendering fixtures store synthetic audio through the real object store; no test-only application route bypasses authentication. Additional cases cover cursor insertion, resolved text, creating details, branding swaps/reload, image access, forged-snapshot rejection, immediate export feedback from enlarged Preview, minutes-only timestamps and a six-second branded MP4 whose snapshot survives later branding changes. The call-direction browser case saves/reloads settings, checks connected and ended timers, and exports both directions with synthetic audio. It verifies H.264/AAC output, voice starting after the opening, and snapshot isolation while the live Scene’s direction changes. The clock regression plays across a minute boundary, pauses, seeks backward, checks status/lock clocks and message/activity timestamps, and exports a 66-second recording with frames before and after the minute change. Timeline tests cover edge snapping at different zooms, free placement, overlaps, independent positions and anchors. Browser checks exercise captured dragging over other tracks, Escape, Undo, keyboard nudges, positions beyond the old ending and reload. The persona browser case scrolls an 80-voice catalog, selects by keyboard, adds two personas and exports their audio alongside the agent while checking transcript visibility and snapshot isolation.

With the published Surgery Partners Scene catalog, the dependent-button regression seeks across front-photo, back-photo and upload taps together. It verifies that upload waits until enabled and that seeking backward resets the component before replaying the same sequence.

Local validation on 2026-10-07 also generated 20 ElevenLabs v3 voice takes and a notification sound, then exported a 210.16-second walkthrough at 1440 × 1000. Type checking, Director lint, timeline tests, migration replay, tenant RLS/composite-key checks and baseline drift passed. The repository-wide route-auth audit recognized all Director handlers; eight existing findings remained in unrelated Allianz, Surgery Partners and Takeda handlers.

The subsequent Details/branding update passed nine timeline tests, six browser cases and a six-second branded H.264/AAC export. The current walkthrough was re-exported from its saved 201.76-second script with 19 existing speech takes; its final frame includes the closing line and minutes-only activity timestamps. Type checking and targeted lint passed. The latest route audit accepted all Director handlers and reported four unrelated existing Allianz/Takeda findings.

See the Director user guide. The Scene Agent Assist design record records the Salesforce/Headless V1 scope, reference investigation and future Custom App/Amazon Connect direction.