Browser E2E tests¶

Status: Current | Last reviewed: 2026-10-12

The Playwright suite that drives Agent Studio in a real browser. It lives in apps/ai-manager/e2e/ and runs against an already-running stack — it does not boot one. Read this before adding a spec: the auth fixture and the locator rules are both non-obvious, and the app has almost no data-testid attributes to lean on.

For the unit/integration suites and what CI gates on, see Running the tests.

Multimodal stylesheet prerequisites¶

The AS-285 regression spec compiles Surface App's actual catalog-preview CSS with its own PostCSS/Tailwind dependencies. Install both apps/ai-manager and apps/surface-app dependencies with npm ci --no-audit --no-fund before running it. Surface App does not need to be running for this spec. The authoring CI job installs both apps and runs node scripts/generate-phone-catalog-styles.mjs --check from apps/ai-manager before the browser suite to detect stale generated styles.

What it covers today¶

Spec	Proves
`auth.setup.ts`	`POST /api/auth/login` issues a session, resolves an active tenant, and expands permissions
`local-login.spec.ts`	`/login?local=1` accepts valid local credentials, rejects invalid credentials and preserves Viewer permissions; ordinary login still redirects to SSO
`smoke.spec.ts` — agents list	the authenticated shell renders for a session restored from `storageState`
`smoke.spec.ts` — agent editor	the canvas mounts, a stage opens, and `ServiceConfigForm` renders a control — with no uncaught page errors
`agent-lifecycle.spec.ts` — create	the wizard's Template path produces a publishable chat agent and navigates into its editor
… — edit	a prompt edit in the stage inspector drives `AutosaveIndicator` to Saved, and never to Save failing
… — persist	that edit is in `agent_versions.config` — read back from `GET /api/pipelines/<id>`, in `config.template.inline` specifically
… — publish	the structural validator and the orchestrator preflight accept the config, and a version row appears
… — chat	the tester starts a session and the agent answers with actual words (ai-manager → ai-orchestrator → conversation-api → surface-app)
… — transcript	both turns of that exchange come back through the Activity sheet's transcript player
`agent-logic-inspector.spec.ts` — layout	the Behavior tab renders ONE Tools header, ordered prompt → Subagents → Advanced (rules & guards) → Tools
… — no reload	Add subagent / Add rule / Add guard mutate in place — a `window` sentinel survives the clicks, proving no button native-submitted the inspector's `<form>` (shadcn Button has no default `type`)
… — subagents	an added card slugs its id from the name and becomes a Route to a subagent target; Run a tool offers the stage's callable tools as a dropdown
… — quick-create	typing `@newname` in the prompt offers ＋ New subagent — picking it inserts the `{{SUBAGENT:…}}` token AND mints the card in the section
… — if/else	picking the `{{` If/Else skeleton opens the condition builder immediately (Builder tab, seeded row); ＋ New variable… declares `isAuthenticated` → `is_authenticated` on the spot and the NEXT builder's picker lists it (page-level `config.variables` round trip); saving lands the caret in the branch body (the Radix focus-restore race); ＋ ELSE IF adds a branch with its own condition
… — component answer mapping	enables Multimodal UI, opens mapping from a `{{COMPONENT:…}}` pill, creates a variable in context, saves the answer path, and reopens the component to verify that the mapping persisted
`data-tool-wiring.spec.ts`	creates its own disposable catalog on a clean schema-only database, then proves catalog MCP resolution is find-or-create and validates malformed/unknown slugs without relying on demo seeds
… — settings	the agent Settings dialog carries no Variables panel (variables auto-create on prompt reference; the canvas toolbar lists scope)
`capability-lifecycle.spec.ts` — create	the Capabilities tab's mini-form creates a capability (metadata only) and the pro-code workspace scaffolds via editor-link
… — author + publish	workspace files written directly (capability.json + prompt.md — what an author does in the Studio editor), `validate-workspace` passes, `publish-from-workspace` auto-bumps to v2 with the assembled manifest (falls back to API authoring on hosts that can't see the workspace bind mount)
… — enable	a fresh agent's always-present Add capability library lists built-ins (grouped first) AND the authored capability by search; adding renders the card EXPANDED with its schema form (the fold-state seeding this spec's first run caught missing)
… — persist + publish	the draft carries the `capability:` ref + `capability_configs` through collab autosave, and Publish succeeds (publish is where the expander runs, so the toast is also an expansion assertion)
`workflow-lifecycle.spec.ts` — create	the New Workflow dialog produces a draft and lands in the graph editor
… — build	an AI Agent node and an Action node are added from the palette and connected, and autosave reaches Saved
… — persist	both nodes, the edge, and the AI Agent's auto-selected `integration_id` are in `workflow_configs.config` — read back through `GET /api/workflows/<id>`
… — inline API tool	an inline API tool created from the Action node's picker survives a full reload, definition included
… — sandbox	the inline Python editor's Run executes in code-executor's nsjail sandbox and returns a result
… — publish	the graph-wide validators and the publish route accept it, and a version row appears
… — run	one manual run through async-agent reaches a terminal `completed`

The editor half of the smoke spec is deliberately also a deletion guard. The 2026-08-12 dead-code pass cut ~700 lines from FlowEditor.tsx and a hidden "Additional Configuration" block from ServiceConfigForm.tsx; verifying that used to be a manual "open it and look". A removed-but-still-referenced helper throws at module scope or on first render, and React can swallow it into an error boundary that still looks like a working page — so the spec asserts on page.on('pageerror') as well as on what rendered.

Two assertions in the lifecycle spec carry more weight than their line count suggests:

Reading the prompt back out of the API. Autosave runs editor → Y.Doc → collab-gateway → Postgres, and the Y.Doc converters enumerate a fixed allow-list of top-level keys — an unlisted key is dropped on every flush with the editor still showing the change and no error anywhere (see the collab-gateway Yjs-shape invariant in apps/ai-manager/CLAUDE.md). Only a read-back distinguishes saved from displayed.
Publishing at all. Publish runs the structural JSON Schema validation plus the orchestrator's buildability preflight, so "v1 exists" is a statement that the config the wizard produced can actually be built into a pipeline.

The chat and transcript steps need ai-orchestrator, conversation-api, and a working LLM integration on the tenant. They skip loudly when the session can't start, quoting the reason the product gave — a red suite that means "no OpenAI key here" trains people to ignore red suites. A session that starts and then never answers is a real failure and is reported as one.

The sandbox step draws the same line, and drawing it took reading a traceback. SandboxCrashed: sandbox exited rc=255 with no envelope is the message for both "your code died" and "nsjail cannot create a mount namespace on this host", so the spec reads the Test panel's traceback block as well as its alert and skips only on the nsjail signature (initCloneNs, MS_PRIVATE). Any other crash fails. See code-executor — the fix is CODE_EXECUTOR_DOCKER_PRIVILEGED=true, now the dev default, and the reason it hid for so long is that /healthz and /v1/validate both stay green without it.

Three product bugs this spec found¶

Both were silent, both were in the pre-hydration/allow-list machinery around collab, and neither is visible from the editor:

The first inbound Y.Doc update clobbered edits made before the socket connected. Outbound writes stay locked until the doc sees a non-local update, so an edit made while the WebSocket was still connecting lived only in React state — and a brand-new workflow's room is unpopulated ({nodes: [], edges: []}), so applying it deleted that edit. Add a node, add a second one a moment later, and the first is gone with the chip saying Saved. The spec caught it as toHaveCount(2) receiving 1. The tie-break now lives in lib/collab/empty-room.ts — on the first apply only, an unpopulated room loses to a local graph — extracted as a pure module so the rule is stated directly in a unit test that runs in seconds, rather than only reproduced by a browser journey that takes a minute and a whole stack.
The same first-hydrate race also affected populated agent drafts. A template-created agent already has an LLM stage, so the empty-room rule did not fire. On a cold CI server the prompt editor became interactive before sync step 2; typing changed React state while outbound was locked, then the populated room arrived and silently restored the older prompt. Each bridge now snapshots its config when a Y.Doc attaches. If that config changes before the first hydrate, the local config is replayed into the doc before remote state can overwrite it; later remote updates remain authoritative collaborative edits. The pure tie-break tests cover both empty and populated rooms, while the lifecycle spec reads the unique prompt marker back from config.template.inline in Postgres.
inline_tools was absent from the workflow Y.Doc allow-list, so every inline tool authored on a workflow node was dropped on the flush to Postgres while the node kept its config.tool = "inline:<id>" reference. The picker read "1 tool selected", the chip read "Saved", and the reloaded workflow pointed at a tool that was not in the config at all. Both converter copies now share one DECLARED_INTERFACE_ARRAYS list so the three round-trip directions cannot drift again, with round-trip coverage in lib/collab/workflow-shape.test.ts.

Running it¶

The stack has to be up (bash .devcontainer/wake-up.sh) and ai-manager has to be reachable on the base URL.

# once per machine: fetch the browser build this Playwright version pins
cd apps/ai-manager && npx playwright install chromium

# from the repo root
bash .devcontainer/scripts/with-env.sh -- bash -c 'cd apps/ai-manager && npm run test:e2e'

with-env.sh is not optional: the auth setup needs DATABASE_URL to provision its user, and that variable only exists inside the loaded environment.

Script	What it does
`npm run test:e2e`	headless run, list + HTML reporter
`npm run test:e2e:ui`	Playwright's watch UI (pick and re-run individual specs)
`npm run test:e2e:report`	open the HTML report from the last run

Useful flags: --grep "agent editor" to run one spec, --headed to watch it. For a flake check on the lifecycle specs, run the whole suite three times in a loop rather than --repeat-each=3 — each is a single serial journey over module-scoped state, so repeating individual tests inside one process replays step 4 against step 6's agent.

for i in 1 2 3; do
  bash .devcontainer/scripts/with-env.sh -- bash -c 'cd apps/ai-manager && npx playwright test --reporter=list'
done

Run it through nginx, not `localhost:3001`¶

playwright.config.ts defaults baseURL to https://ai-manager.$PUBLIC_FQDN and falls back to http://localhost:3001 only when PUBLIC_FQDN is unset. That default is load-bearing, not cosmetic. Parts of the product exist only through the reverse proxy, and the important one is location /collab/ in trueserve-vanguard.conf.template, which same-origin-proxies the Yjs WebSocket to collab-gateway on :3510.

Point the suite at the Next.js port directly and resolveWsBase() in lib/collab/yjs-client.ts derives ws://localhost:3001/collab/…, which nothing serves. The socket closes before it opens, the Y.Doc is never shared, the gateway never flushes, and every edit is silently discarded — the editor still shows your change, GET /api/pipelines/<id> never gains it, /api/auth/collab-flush-state reports {"dirty":false,"alive":false}, and AutosaveIndicator renders nothing at all (its idle branch returns null), so there is no error to notice. A suite on :3001 therefore cannot test autosave, and reports a healthy green run while not testing it. Set E2E_BASE_URL explicitly if your stack fronts ai-manager somewhere else.

How authentication works¶

There is no auth bypass and no test-only route. fixtures/e2e-user.ts UPSERTs an ordinary users row with a scrypt hash plus an ordinary tenant_memberships row, and then the setup project signs in through the product's own POST /api/auth/login. Everything the login route does — session insert, cookie, tenant resolution, expandRolePermissions — is exercised for real. The suite only skips the OIDC hop, because the login page redirects into Authentik whenever NEXT_PUBLIC_USE_SSO !== 'false' (the dev default) and POST /api/auth/login is the same endpoint the non-SSO form posts to.

The resulting cookie is saved with request.storageState() to e2e/.auth/user.json, and the chromium project loads it via dependencies: ['setup'] — so specs start already signed in and no test spends time on a login form it isn't testing.

Why provision instead of reusing a seeded account. scripts/migrate-rbac.ts seeds admin/admin123, but the dev DB is routinely restored from snapshots where that hash no longer verifies, most real users carry an sso:no-password sentinel that can never authenticate, and CI starts from an empty DB with no seeded human at all. A suite that creates its own account works in all three.

Four rails on that, each because a provisioning fixture is a credential factory:

Local databases only. assertLocalDatabase() refuses any host outside localhost/127.0.0.1/::1/host.docker.internal. Override with E2E_ALLOW_REMOTE_DB=1 if you really mean it — pointed at a shared environment this leaves a standing account nobody remembers creating.
A fresh random password per run unless E2E_PASSWORD is set. Nothing to store, nothing to leak, and the row is useless to whoever finds it later.
Least privilege. Tenant admin, is_platform_admin = false. A spec that needs platform scope fails loudly rather than passing with authority a tenant admin doesn't have.
The hash function is duplicated, not imported. lib/rbac/local-rbac-service.ts pulls in lib/db.ts and the ~540 KB generated config-schema bundle, neither of which belongs in a test runner. The copy cannot drift silently: auth.setup.ts asserts the login succeeds, so changing the app's hashing fails the setup project on the next run with a message that says to fix the fixture, not the app.

Warning: e2e/.auth/user.json holds a live session cookie. It is a credential, not an artifact — it and the test-results//playwright-report/ directories are gitignored, and the traces and videos in them capture agent configs and conversation transcripts.

Configuration decisions worth knowing¶

No webServer block. The golden paths cross service boundaries (ai-manager → ai-orchestrator → conversation-api, workflows → async-agent). A lone next dev booted by Playwright would fail them for reasons that have nothing to do with the code under test, so the suite targets a stack that is already running and says so when it can't reach one.

workers: 1, fullyParallel: false. Not a placeholder. The specs share one tenant's agent list and one dev stack, and collab-gateway holds a room per agent document — two workers editing the same seeded agent produce failures that read as autosave bugs.

No hardcoded agent ids. The editor spec asks GET /api/pipelines for a real row and skips itself when the tenant has none. A hardcoded id fails as "the editor is broken" when it only means the seed changed.

A golden path is one serial journey on one page, and it cleans up after itself. agent-lifecycle.spec.ts uses describe.configure({ mode: 'serial' }) plus a single shared Page created in beforeAll, because a fresh page per step would re-mount the chat iframe (dropping the conversation) and would hide the class of bug the spec exists to catch — state that looks right until something else touches it. Names are tagged with a per-run token so a leftover from a crashed run is never picked up, and afterAll deletes the agent it created even when the run failed part-way: the smoke spec opens agents[0], so a tenant slowly filling with half-built E2E agents makes every later run slower and its failures harder to read.

Writing locators for this app¶

ai-manager has essentially no data-testid attributes, so prefer, in order: roles and accessible names (getByRole('button', { name: 'Edit' })), visible text, then a library's stable classnames (.react-flow, .react-flow__node are React Flow's public contract). Add a data-testid only where a surface has no stable hook of its own, and say in a comment why.

Five exist today. Four are in ai-manager: node-inspector and node-inspector-title on NodeInspectorPanel.tsx — the surface every stage edit goes through, with no role/text handle that survives across stage types — stage-prompt-editor on ServiceConfigForm.tsx, and workflow-instructions-editor on WorkflowLLMConfig.tsx. The fifth, message-text, is on surface-app's components/ui/message.tsx. The last three exist because of traps below: a CodeMirror instance has no accessible name, and an AI Agent node's inspector holds more than one editor.

Traps¶

Canvas nodes are not React Flow-selectable. nodesDraggable={false} and the editor puts explicit controls on each node, so .react-flow__node.selected never appears. Open a stage by clicking the node's own Edit button.
The Properties/Behavior tab row is conditional. It renders only when stageHasBehaviorTab(stage) is true, and the first node on a canvas is usually a transport (Text Input) that has no behavior surface. Asserting the tabs exist fails on a perfectly healthy editor — so walk the nodes and take the first one whose Behavior tab appears, rather than assuming an index.
The inspector title is stage_name; the node's heading is the stage label. They are different fields and routinely differ ("Text Input" vs "Input"), so never assert one echoes the other.
fill() does not work on CodeMirror. It sets textContent directly, CodeMirror's document state never hears about it, and the change looks applied while saving nothing. Click into .cm-content and type through page.keyboard.
The stage inspector renders two CodeMirror editors. Greeting (config.greeting_message) and prompt (config.template.inline), in that DOM order, whenever the service supports a greeting. A positional .cm-content locator writes to the greeting — and the test still passes, having verified the wrong field. Use getByTestId('stage-prompt-editor'), and assert persistence on config.template.inline by name so a mis-aimed write fails loudly.
A pre-existing Saved chip is not an acknowledgement of the edit you just made. A newly-created agent can still show the wizard/bootstrap flush as Saved during the prompt bridge's 300 ms outbound debounce. Waiting only for visible Saved can therefore pass before the prompt transaction exists and race the next Postgres read. Immediately after typing, first require the old Saved state to disappear, then wait for a new Saved acknowledgement. The authoritative API read still polls for the exact marker: the indicator and GET are separate requests, and a cold Next dev server can finish them out of order while compiling routes.
The chat tester is a cross-origin iframe. surface-app has its own vhost, so nothing inside iframe[title="Chat Testing"] is visible to page.locator — use page.frameLocator.
surface-app's composer is Lexical, not a textarea. There is no value, so fill() finds nothing to fill; click the [contenteditable="true"] and type. The send control is getByRole('button', { name: /Send reply/i }).
A chat agent greets on connect, so one .is-assistant bubble exists before the agent has heard anything. Count the assistant turns before sending and wait for the count to grow.
A message bubble contains its own metadata, so a bubble-level "not empty" assertion is satisfied by the speaker • time line — it passes for the silent-agent failure it exists to catch (a truncated tool argument makes the model emit no text at all). Assert on [data-testid="message-text"], and remember a streaming bubble mounts empty and fills in, so retry rather than checking once.
The Activity sheet's conversation list already contains your probe text in its preview column, so getByText(PROBE) passes without the transcript ever opening. Assert the player's own Transcript (…) heading first; ConversationHistoryView swaps the list out for the player, so after that the text can only be coming from the transcript.

Traps in the workflow editor¶

React Flow's minimap covers the bottom-right of the canvas, and a node's centre lands under it. node.click() times out with react-flow__minimap-svg … intercepts pointer events. Click a corner the user can actually reach — click({ position: { x: 32, y: 32 } }) — rather than reaching for force: true, which clicks through an overlay a real user cannot.
A workflow node opens its inspector on a plain click (no per-node Edit button, unlike the agent canvas), and the inspector's heading is Edit Node: <label>.
ToolSelector's trigger is a <Button> with an explicit role="combobox" and no accessible name. Two ways to miss it, both presenting as "the tool picker never rendered": getByRole('button') doesn't match because the role attribute overrides the tag, and getByRole('combobox', { name: /Select tools/ }) doesn't match either — combobox does not take its accessible name from contents, and the label is an unlabelled nested <span>, so the element has no accessible name at all. Use getByRole('combobox').filter({ hasText: … }).
The code node is unreachable from the palette. It's deprecated in favour of inline tools and absent from WORKFLOW_NODE_PALETTE, so the workflow Tool node is the only workflow-side surface for authoring one — that's where this spec exercises the sandbox. (Palette visibility is that array, not the manifest's decorative "palette" key.)
PythonSourceEditor seeds a working async def run(args, context), so Run works with nothing typed. Take that path: CodeMirror auto-indents and auto-closes brackets, so typing Python through page.keyboard is a way to write a test that fails on its own syntax. The debounced Syntax OK is a real round trip to code-executor's /v1/validate, which also confirms the service is reachable before asking it to execute anything.
Both Test-panel outcomes render as an Alert. Waiting on the success text alone turns a sandbox error into a timeout that says nothing; wait for whichever alert arrives, then read it — and read the pre blocks too, since the message alone can't distinguish a host problem from a code problem.
The Action node accepts one tool, so a spec that creates a second inline tool to test something else must Cancel rather than Save, or it deselects the tool the previous step asserted on.
integration_id is a node-level field, deliberately outside config. A read-back that asserts on node.config.integration_id fails against a perfectly saved workflow. It's the mirror image of the runtime bug that invariant exists for: a copy in config doesn't error, the runtime just ignores it and calls the provider with the host's ambient credentials.
Workflow autosave is collab-only — there is no explicit Save — and collab-gateway owns a ~10 s Postgres debounce, so a read-back straight after Saved can still miss. Wait for the indicator, then poll the API.
The tester panel has its own "Saved test inputs" heading, which a loose /^Saved / matches; and a run's SummaryField label="Completed" collides with the status badge of the same word. Scope both, or assert on the status element specifically.
Edge insert buttons render through React Flow's EdgeLabelRenderer portal, so they are not DOM descendants of the edge and cannot be scoped to one. Reach them by position or by driving the palette instead.

Environment variables¶

Variable	Default	Purpose
`E2E_BASE_URL`	`https://ai-manager.$PUBLIC_FQDN`	the origin to drive; must be the nginx one, see above
`PUBLIC_FQDN`	—	supplied by `with-env.sh`; only used to derive the default `E2E_BASE_URL` (absent → `http://localhost:3001`)
`DATABASE_URL`	— (required)	the DB the test user is provisioned in; supplied by `with-env.sh`
`E2E_USERNAME`	`e2e-runner`	the provisioned account
`E2E_PASSWORD`	random per run	set it to keep the account usable by hand
`E2E_TENANT_SLUG`	`trueserve`	which tenant to grant membership in
`E2E_ALLOW_REMOTE_DB`	unset	`1` disables the local-database guard
`E2E_STACK`	`full`	`minimal` = ai-manager + Postgres + collab-gateway only; the execution steps skip. See below
`E2E_SEED_STUB_INTEGRATION`	unset	`1` gives a bare tenant one placeholder `llm` integration so the auto-select assertions have something to select
`NEXT_PUBLIC_COLLAB_WS_URL`	unset	read by the app, not the suite; overrides the derived `/collab` path so a proxy-less origin can still autosave
`CI`	unset	enables `forbidOnly`, one retry, and the `github` reporter

In CI: the authoring half only¶

The suite runs on every pull request as the e2e job in test.yml. It runs half of itself, and the half it runs is a deliberate line rather than a budget.

What the specs cover splits cleanly in two:

Authoring — wizard → editor → Y.Doc → collab-gateway → Postgres → validate → publish. Three processes; fits in a GitHub-hosted runner.
Execution — chatting with the agent, reading the transcript back, running a workflow, running Python in nsjail. Needs ai-orchestrator, conversation-api, async-agent, code-executor, and a provider credential with money behind it.

E2E_STACK declares which half is present. minimal skips the four execution steps immediately; full (the default, so a local run is unaffected) runs everything.

Why that is a declaration and not a probe. The execution steps already notice a broken environment from the error text the product shows them, and that stays — it answers "the service should be here and isn't". But the chat step waits for Start Chat to become enabled, and that button stays disabled while the orchestrator preflight can't answer. With no orchestrator there is no error text to match: the step burns its 60 s timeout and then fails. A red run meaning "this runner has no ai-orchestrator" is worse than no run, because the next red one gets ignored too. So the profile is stated up front and e2e/fixtures/stack.ts names the missing services in the skip reason.

The runner's topology, and why it isn't nginx¶

Postgres (pgvector) + Redis as service containers, the schema baseline applied with the same loop as the conversation-api-unit job, then collab-gateway under tsx and ai-manager under next dev.

/collab/ is an nginx location, not a Next.js route, so a runner that skips the proxy hits the silent-autosave-loss trap described above — and would report green while never testing autosave. Two topologies avoid it, and the job takes the second:

Run nginx in the runner and front both services with it. Faithful, but it copies the Vanguard vhost template into CI where it would drift unnoticed.
Set NEXT_PUBLIC_COLLAB_WS_URL=ws://localhost:3510 so the client dials collab-gateway directly. Cookies are not port-scoped and ws:/http: count as the same site, so the ai_manager_session cookie still rides the WebSocket handshake and the gateway's /api/auth/collab-verify callback still authenticates it.

Option 2 was verified by hand before the job was written: the gateway accepts the upgrade on :3510 from a localhost:3001 session and answers with hello, a presence roster, and a Yjs sync frame.

Three smaller choices in that job, each load-bearing:

next dev, not next build && next start. A production build costs several minutes here, and app/api/auth/login/route.ts sets the session cookie secure: NODE_ENV === 'production' — which a plain-http runner silently drops, so the whole suite would fail on a login that returned 200. The cost is on-demand compilation, which is why the start step warms /agents, /workflows, and both editor routes before Playwright starts.
Redis is not optional. apps/collab-gateway/src/redis.ts throws at import without REDIS_URL, and the gateway is the thing that makes autosave testable at all.
The baseline step asserts the template count. The suite's first step creates an agent from a builtin template, and a database without templates fails that as a wizard timeout with nothing pointing at the seed. Reference data is invisible to the drift guard (it dumps --schema-only), so the count is checked where it can be read — see migration conventions.

The nightly-against-Vanguard fallback was rejected, for reasons rather than taste. There are no self-hosted runners and no schedule: trigger anywhere in .github/workflows, that environment's Postgres is not publicly reachable, and e2e/fixtures/e2e-user.ts refuses a non-local DATABASE_URL unless E2E_ALLOW_REMOTE_DB=1 — because it UPSERTs a user with a known password, which is not something to leave standing in a shared environment.

What a green CI run does and does not prove¶

It proves: the wizard's template path, the editor canvas and stage inspector, prompt and graph autosave reaching Postgres through the real Y.Doc converters, inline-tool definitions surviving a reload, the structural validator, and publish.

It does not prove anything about a live session — no agent replies, no transcript, no workflow run, no sandboxed Python. Those four steps report as skipped with the services they need named, and a full stack is the only place they run. Run the suite locally against Vanguard before merging anything that touches the runtime, not just before touching the editor.

One consequence worth knowing: a fresh CI database has zero non-template agents, so the smoke spec's editor test would skip itself in exactly the environment where its deletion guard matters most. It now instantiates a builtin template through the wizard's own POST /api/templates/<id>/instantiate when the tenant is empty, and deletes it in a finally. That branch never runs locally, where the tenant already has agents — CI is its only exercise.