AI generation (Generate with AI)¶

Status: Current | Last reviewed: 2026-10-06

How the feature-flagged "Generate with AI" creation path turns a natural-language request into a draft Agent or Workflow using Claude Code or the managed OpenAI Agents API.

The conversational designer reuses this architecture for live four-type draft creation. Designer sessions have their own protocol, structured output and revision-checked saves; the existing creation/editor experience remains available.

Scenario builds also use the shared scenario generation skill. It packages the Allianz voice/Copilot/supervisor lessons, supplies focused prompt and validation references, and derives unexecuted regression/latency plans through get_scenario_test_plan. The same skill and planning functions are available to external management MCP/A2A and REST clients. Native build gates and target authorization remain with the enclosing generation/designer protocol.

Big picture¶

The diagram below shows the Claude Code execution path. OpenAI uses the same job, scoped MCP tools and callbacks through a function bridge, with environment.type=none. Provider/integration identity is pinned on each job.

ai-manager UI ──POST /api/generations──> ai-manager
  │ placeholder agent/workflow (normal createPipeline / createWorkflow draft)
  │ generation_jobs row + per-job scoped tsf_ token (agents/workflows manage, 2h TTL)
  │ resolves tenant's runner integration, decrypts creds → auth_context
  └──dispatch──> workspace-agent-runner POST /v1/generations
                   seeds workspace (AGENTS.md protocol, docs bundle, .mcp.json)
                   runs `claude -p --output-format stream-json --mcp-config …`
                   CLI works through the generation MCP server in ai-manager
UI polls GET /api/generations/{id}; bell polls /api/notifications

The CLI never touches the database. Everything substantive — discovery, validation, saving the draft, plan/progress/questions, completion — flows through the generation MCP server (/api/mcp/generation), authenticated per-request with the per-job token. The runner only reports process-level signals (started / failed) to /api/generations/{id}/runner-events.

Gating (three independent switches)¶

The option appears in the New Agent wizard / New Workflow dialog only when ALL hold:

Deployment flag NEXT_PUBLIC_FEATURE_AI_GENERATION (default off) — feature-flags.ts. The runtime var FEATURE_AI_GENERATION overrides it when set (isAiGenerationEnabledServer checks it first). On city EC2 dev boxes there is no Helm layer, so the flag is an env var in the box's .env.local; the opt-in in scripts/ec2-dev/city-update.sh (step_env sets it unconditionally) makes it durable across rebuilds on every city box. The flag is only Gate 1 — each box also needs the ai_generation tenant_settings row (Gate 2) and an active runner integration (Gate 3), which are live per-box DB writes, not set by the script.
Tenant setting ai_generation in tenant_settings (Settings → User Experience → AI Generation): {enabled, runner, integration_id}. Implements claude_code and openai_agents; codex/antigravity/opencode CLI authoring options remain listed as coming soon.
The selected integration exists and is active. Claude Code accepts three auth shapes mirroring the Assistant Agent chat path: native anthropic (API key), amazon_web_services (Bedrock bearer/keys), google (Vertex service account). OpenAI Agents API accepts an openai integration. The server rejects inactive, out-of-tenant or mismatched integrations.

Server-side single source of truth: resolveAiGenerationAvailability() in ai-generation-settings.ts; client hook: useAiGenerationAvailability().

The same three gates guard the external management API — a request to POST /api/v1/management/generations (and the /api/mcp/authoring generate tool) runs resolveAiGenerationAvailability(tenantId) for the token's tenant and returns AI generation is not available: <reason> (feature_flag_disabled, tenant_disabled, integration_not_configured, integration_inactive, integration_provider_mismatch, runner_not_implemented) until all three hold. The tenant here is the one behind the tsf_/OAuth token, not the box — enabling the flag alone still yields tenant_disabled until that tenant has an ai_generation row with an active runner integration.

Entry points¶

Workflows — NewWorkflowDialog gains a third top-level mode generate: path → name (+ required prompt textarea).
Agents — NewAgentWizard gains AgentPath 'generate' on the PathStep. Per-type step orders (the two types that historically had no path step gain one only while the feature is available, with reduced Custom|Generate cards):

Type	Without feature	With feature
voice / chat / copilot	type → path → …	same, PathStep + Generate card
translation_agent	type → name	type → path → name
other	type → modality → name	type → modality → path → name

The chosen type/modality pins pipelineType/subtype (type_pins), which the AI must generate within.

Template-first baseline (current models)¶

Agent generation is template-first: the placeholder is a clone of a curated builtin template for the subtype (resolveGenerationBaseline in lib/generation/template-baseline.ts; voice_agents→tpl-voice-openai, chat_agents→tpl-chat-openai, copilots→tpl-copilot-openai-assistant; translation + unknowns fall back to the legacy getDefaultConfig). At clone time it bakes the service-definition defaults into each stage (getConfigWithDefaults), so the current recommended model/voice are written explicitly rather than left to inherit. This fixes two things at once: the AI starts from a known-good structure (it customizes the prompt/tools, not the plumbing), and the model is a real current value the editor dropdown can display — the AI is told via get_current_draft (starting_template_id) + AGENTS.md to KEEP the baseline model unless the user asks otherwise (it must not invent model strings like gpt-realtime-2 that may be stale/unselectable). Service-def model defaults are kept current (OpenAI Realtime → gpt-realtime-2, OpenAI LLM → gpt-4.1, Responses → gpt-5); templates carry model:None and inherit, so refreshing the def default is the single source of truth — no template SQL migration needed.

Data model (migration 192)¶

generation_jobs — status machine queued → planning → generating → (awaiting_input ↔ generating)* → finalizing? → completed|failed|cancelled; carries prompt, type_pins, AI plan, append-only progress, questions (id/question/options/answer), cli_session_id (for claude --resume), api_token_id. RLS tenant-isolated.
user_notifications — minimal persistent per-user notifications (the page-header bell and Home Notifications widget; producers: generation question/complete/failed).
agents.generation_job_id / workflow_configs.generation_job_id — placeholder markers. Columns, never config: the collab Yjs shape allow-list would silently drop config keys. While non-null, the editor renders GenerationStatusPanel instead of the canvas (and the workflow page never joins the collab room), and list pages badge "Generating…". Cleared on completion/cancel; kept on failure so the panel shows the error. Every agent-load query MUST select this column — the editor page reads selectedPipeline.generation_job_id from /api/pipelines/[pipelineId], which uses getPipelineVersion; if that SELECT omits a.generation_job_id the gate never fires, the editor mounts the (default) draft, and collab autosave flushes that generic config back over the AI's result. getPipeline / listPipelines (via buildPipelineSelect) and getPipelineVersion must all carry it.

The generation MCP server¶

/api/mcp/generation is a stateless JSON-RPC (Streamable HTTP, JSON mode) endpoint; tools live in lib/generation-mcp/tools.ts. Tool groups:

Discovery: search_agent_templates/get_agent_template, search_workflow_templates/get_workflow_template, search_prompt_templates/get_prompt_template, search_tools/get_tool (secret-ish config keys stripped), get_builtin_tools (platform contracts, distinct from saved-tool search), list_service_definitions/ get_service_definition, list_workflow_node_types/ get_workflow_node_manifest, list_tenant_integrations (names/providers only).
Validation: validate_agent_config (universal schema + per-stage service-definition checks via getDefinitionByLookupKey + validateServiceConfig; never requires settings — typing-guard parity) plus structural + quality guards that catch the LLM's common mistakes: duplicate stage ids; text_chat with ≠1 llm stage; objects in config.tools (must be string refs — inline tool definitions go in config.inline_tools[]); inline:<id> refs with no matching inline_tools entry; a bare inline id missing the inline: prefix (silently unwired at runtime); empty-shell inline tools (Python with no async def run(args, context)/empty config.source, API with no url) — rejected because a tool the agent can't call is worse than none (the agent-4119 failure); and a prompt-quality heuristic (warnings): when a stage wires tools, the system prompt must have a Tools section, name each wired tool, and include example/sample phrases per the model-family prompting guide. validate_workflow_config (node manifests via validateWorkflow + edge referential checks). complete_generation re-runs the agent/workflow validator as a final gate, so a structurally broken config can't reach "completed". Structured errors the CLI iterates on.
Mutation (draft only): apply_agent_template, save_agent_draft_config, save_workflow_draft_config — validate first, write through updateAgentDraft/updateWorkflowDraft straight to Postgres. Never publish. Because this bypasses the collab Y.Doc, the draft must be reconciled with collab immediately after each successful write and again on completion (see Collab reconciliation below).
Lifecycle: report_plan (also flips queued/planning → generating), report_progress, ask_user (registers questions, flips to awaiting_input, notifies; instructs the CLI to end its run), complete_generation (re-validates the saved draft as a final gate, clears the marker, revokes the token, notifies), fail_generation.

Collab reconciliation (why the AI's save doesn't get clobbered)¶

The MCP save tools write the draft directly to Postgres, but the collab-gateway Y.Doc is the live source of truth for a draft — when an editor connects it hydrates from Redis (or, on a reused agent id, a STALE pre-generation snapshot) and flushes that back, overwriting the AI's result.

Reconciliation closes the gap from both ends, via the gateway's POST /reset/:scope/:roomId endpoint (service-token + x-collab-tenant-id): - On dispatch (POST /api/generations, retry) — resetCollabRoom evicts any cached doc before the run so a reused-id stale snapshot can't interfere. - After successful generation draft writes (apply_agent_template, save_agent_draft_config, save_workflow_draft_config) — re-seed any live room from the just-written DB draft so an already-open editor cannot flush an older Y.Doc over the generated config. - On completion (completeGenerationJob, after the tx commits) — re-seeds the room so the editor that opens post-generation sees the AI's config.

resetRoom (rooms.ts): if a room is live in memory it calls resetFromDb(), which runs hydrateDocFromDb against the live doc — pipelineJsonToYDoc/workflowJsonToYDoc .delete(0, len) each shared collection and rebuild it, so it's an authoritative REPLACE (duplicate/extra Y.Array entries are deleted, not merged — a naive fresh-doc state-vector delta would only add). The pre/post delta is broadcast to connected clients and fanned out to other replicas via pubsub. If no room is live, it just DELs the collab:doc:<scope>:<roomId> Redis key so the next join hydrates fresh from the DB. Best-effort: a gateway failure is logged, not thrown (the DB already holds the AI's config).

Reset works here because a generation write REPLACES whole collections, so the broadcast delta covers everything an open editor is holding. It is not sufficient for a hand-repair of a draft row: a narrow change (deleting one key, fixing one nested value) leaves the rest of the open tab's React state authoritative, and it reconciles the repair away on the next flush. Use POST /purge/:scope/:roomId for those — see presence-and-collab.md.

The runner side (workspace-agent-runner)¶

generation.py adds a generation-job mode beside the untouched /v1/chat/stream chat contract:

POST /v1/generations (202; background task), GET /v1/generations/{id}, POST .../resume, POST .../cancel.
Workspace ag-<job> persists across awaiting_input (unlike chat's ephemeral context manager); deleted on terminal status or by the TTL sweep (GENERATION_WORKSPACE_TTL_SECONDS, default 24h). Seeded with: generation AGENTS.md (the plan→discover→ask→build→validate→save→complete protocol + plain-language ask_user rules), .trueserve/request.json (prompt, target, type_pins), .mcp.json (generation MCP URL + per-job token), and the docs bundle copied from GENERATION_DOCS_DIR.
Invocation: claude -p --output-format stream-json --verbose --append-system-prompt-file AGENTS.md --mcp-config .mcp.json --strict-mcp-config --allowedTools "Read,Glob,Grep,LS,mcp__trueserve__*". No Bash/Write — the MCP tools are the only effectors.
Generator model is pinned, not the CLI default. The CLI command passes no --model; the model comes from ANTHROPIC_MODEL / ANTHROPIC_SMALL_FAST_MODEL in the dispatched auth_context.env (applyGenerationModels in ai-generation-dispatch.ts), which _apply_auth_context copies through to the CLI process. Defaults: main = Claude Opus 4.8 (authoring quality), small/fast = Haiku 4.5 (cheap sub-agent/formatting calls), keyed per provider — Bedrock uses bare inference-profile ids (global.anthropic.claude-opus-4-8, global.anthropic.claude-haiku-4-5-20251001-v1:0), Anthropic-direct uses public ids, Vertex uses Vertex ids. Override per deployment with GENERATION_MODEL_{BEDROCK,ANTHROPIC,VERTEX} / GENERATION_SMALL_MODEL_{…}. Without this pin the generator silently rode whatever the Claude Code CLI version defaulted to.
Activity Log summaries reuse the same gate and small-model pins. The lazy /api/audit-log/{id}/summarize route checks resolveAiGenerationAvailability() before making a model call. When available, it uses the selected runner integration and the provider's GENERATION_SMALL_MODEL_* value to summarize the shaped audit diff; when unavailable or the call fails, it stores a deterministic explanation instead. The summarization prompt defaults in code, can be overridden deployment-wide with AUDIT_LOG_SUMMARY_PROMPT, or tenant-side via tenant_settings.setting_key = 'audit_log_summary' with {"prompt": "..."}.
The runner watches stream-json for the init session id (reported upward for --resume) and for lifecycle tool_use blocks: terminal tool seen → cleanup; ask_user seen → keep workspace, status awaiting_input; CLI exit without either → failed callback with stderr tail. Wall-time (GENERATION_WALL_MS, default 30 min) kills runaway runs.
Q&A resume: answers POST → ai-manager records them on the questions → POST runner /resume → claude --resume <session> in the same workspace with a Q/A transcript prompt. If the session id was never captured, a fresh run still works (protocol + request.json + saved draft carry state).
Concurrency: asyncio semaphore (GENERATION_CONCURRENCY, default 2).
WORKSPACE_AGENT_RUNNER_FAKE=true drives the real MCP endpoint over HTTP (plan → progress → complete) without any CLI — used by tests and smoke checks.

Research and platform knowledge¶

Generate with AI, the editor assistant and Studio share reference discovery in lib/generation-mcp/discovery.ts: existing same-tenant agents/workflows (published version preferred, otherwise an identified draft), capability packs and published multimodal catalogs. References are read-only and legacy credential fields are filtered. Existing templates, tools, integrations and runtime builtin contracts remain available.

get_configuration_schema returns a compact service field index or a focused section with its referenced definitions. It combines shared stage fields such as logic, tool gates and procedures with service-specific settings from the generated agent schema. Use it alongside get_builtin_tools, which describes callable arguments rather than the saved capability settings. Use the agent. section prefix for agent-wide settings, such as agent.transfer (External Consult) or agent.variables.

All three authoring protocols include app/authoring_research.md. It requires relevant reference searches, detailed capability checks, informed functional questions and safe technical defaults. A demo brief remains scoped to the selected target; a Voice design builds one voice agent. Business tasks become procedures, while startup lookup, authentication and ending use their existing configuration and prompt mechanisms. Agent get_current_draft and designer get_design_state return authoring_defaults, including redacted tenant authentication readiness. Protected member information implies native Customer Authentication: built-in identity defaults to on-demand AAL2 passkey with native enrollment and verified tool gates. Existing identity requirements are preserved; OIDC needs a compatible policy rather than an invented assurance profile. Caller consent remains required. Native end_session handles goodbye and human assistance from the main prompt, with params.summary covering the AI interaction. Routine phone ownership, summary fields and authentication options do not become author questions or separate procedures. External Consult is the separate, explicitly requested dial/brief/merge requirement. Example business policy is not inherited.

For demo planning, list_trueserve_library_tools and research_trueserve_library use the fixed Gallery MCP endpoint. Set GENERATION_LIBRARY_TOKEN_FILE on ai-manager to a private file containing the token. For the current Gallery, set GENERATION_LIBRARY_READ_TOOLS=search_gallery,get_content; its catalog omits read-only annotations. The bridge restricts calls to reviewed/annotated reads, validates against the actual remote schema, blocks redirects, bounds response size and pages long results via next_offset. Credentials never enter prompts or the runner. Tool failures and missing configuration do not block local research. Library content is inspiration, with source titles/IDs/links, not instructions, current runtime authority or the author's business policy.

Public research uses OpenAI's bounded search_public_web function or Claude's WebSearch/WebFetch where supported by the provider. Queries should contain only public domain terms; never drafts, customer data, Library excerpts or credentials. Cite actual sources when relying on findings. See OpenAI authoring for the request boundary and existing-session behavior.

GPT Live authoring defaults include the shared voice prompt, backend starter structure and official reference links. The dedicated prompting-guides/openai-live.md explains that the editor's backend branch is a projection: generation must keep business prompts/tools on the owning Live stage and must not add an audio branch. It distinguishes startup-only voice variables/conditions from backend live conditions, actions and subagent execution. The bundle appends starters from apps/ai-manager/lib/service-definitions/openai-live-prompts.json, the same source used by the runtime and editor.

Docs bundle¶

shared/generation-docs — npx tsx build.ts emits dist/ (committed): hand-written guides from src/ (agent/workflow config shape, best practices, per-model-family prompting guides distilled from /prompting) plus generated references (services/<category>.md from the ai-manager service-definition JSONs; workflow-node-reference.md from the node manifests). Re-run the build after changing service definitions or node manifests. The runner copies the bundle into each workspace; the MCP discovery tools remain the authoritative live source.

research-and-planning.md and capability-guide.md provide the research sequence and capability checklist. The build also copies ten detailed capability/runtime guides from the handbook and user guide into handbook/, including authentication, payments, secure capture, External Consult, human supervision, logic, procedures, subagents and visual rendering. Rebuild after changing these source guides. OpenAI sessions can search/read the bundle; Claude reads it from its workspace. Consult only guides relevant to the task.

The bundle also includes builtin-tools-guide.md and the generated builtin-tool-reference.md. The latter uses apps/ai-manager/lib/generation-mcp/builtin-contracts.json, exported without provider imports by python shared/generation-docs/generate_builtin_contracts.py. The exporter resolves unconditional local copies of base schema constants; conditional per-agent requirements, such as a mandatory ending summary, remain runtime configuration. Run that export after changing a runtime builtin schema, then rebuild the bundle; the orchestrator's test_generation_builtin_contracts.py catches drift. get_builtin_tools exposes the same contracts to generation and assistant sessions. In particular, end_session carries reason, session_escalated and custom string params back to the connected system for goodbye or human escalation. It does not require a saved transfer tool. See conversational design.

prepare_mock_tool supplies working sample actions when no real connection is specified. Its data-only specification generates Python and exercises varied inputs, a default response and every named scenario through code-executor. mock-tools-guide.md requires consistent fixtures, simulated-effect wording, stateless limitations and exact test inputs in the final reply. Design repeats these checks at build completion. Deliberately disconnected scaffolds are an explicit opt-in; no real business action or Performance evaluation runs as part of these sample checks.

Security properties¶

Per-job tsf_ token: agents:manage for agent jobs or workflows:manage for workflow jobs, 2h expiry, revoked on every terminal transition; it is the only credential the sandbox holds for TrueServe, and the MCP server binds it to exactly one job (and therefore one tenant + one target).
Runner env scrubbing (_runner_env) drops MASTER_ENCRYPTION_KEY, DATABASE_URL, POSTGRES_PASSWORD; model credentials arrive only via the decrypted auth_context ai-manager builds per dispatch (ai-generation-dispatch.ts).
/api/generations/{id}/runner-events and /api/mcp/generation are self-authenticating (allowlisted in scripts/audit/route-auth-allowlist.yaml) — they verify the per-job token against generation_jobs.api_token_id themselves.

Failure handling¶

Dispatch failure → job failed immediately (route returns 502 with the generation id; placeholder remains an editable draft).
CLI crash / wall-time → runner failed callback → notification.
Cancel → runner subprocess killed, token revoked, marker cleared — the draft keeps whatever was saved and is editable normally.
Late callbacks can't resurrect terminal jobs (setGenerationJobStatus refuses to leave terminal states).
Job stuck at queued (no error, created_at == updated_at) = the runner can't reach ai-manager's callback/MCP base. The dispatch payload's callback.url + mcp.url come from aiManagerInternalUrl() (GENERATION_AI_MANAGER_URL → AI_MANAGER_INTERNAL_URL → http://localhost:PORT). When the runner runs containerized (WORKSPACE_AGENT_RUNNER_USE_DOCKER=true, the norm on EC2 dev boxes), localhost is the runner container itself, so every _post_callback fails — and callbacks are best-effort (logged, never fatal), so the job silently never leaves queued. Set GENERATION_AI_MANAGER_URL=http://host.docker.internal:3001 (the runner is started with --add-host=host.docker.internal:host-gateway); the env template ships this default, but pre-existing city .env.locals may lack it. Diagnose via the runner logs: docker logs tsf-local-workspace-agent-runner | grep callback → All connection attempts failed.