Running the tests¶

Status: Current | Last reviewed: 2026-08-14

Every command needed to run a service's test suite locally, and what CI runs on each pull request. Read this before adding a test file — the suites discover their files in four different ways, one of them silently drops a file that nothing lists, and one silently skips a test that isn't marked.

What runs where¶

All ten Tier-1 services, plus the route-auth audit and a partial browser gate:

Suite	Runner	Command (from the app directory)
ai-manager	`node:test` + `tsx`	`npm test`
surface-app	vitest	`npm test`
ari-controller-node	vitest	`npm test`
collab-gateway	`node:test` + `tsx`	`npm test`
ai-orchestrator	pytest	`.venv/bin/python -m pytest tests/ -q`
async-agent	pytest	`.venv/bin/python -m pytest tests/ -q`
agent-host	pytest	`.venv/bin/python -m pytest tests/ -q`
payment-vault	pytest	`.venv/bin/python -m pytest tests/ -q`
code-executor	pytest	`.venv/bin/python -m pytest tests/ -q`
conversation-api	plain-assert scripts	`for f in test_*.py; do .venv/bin/python "$f"; done` (see Tests that need Postgres)
route auth (ai-manager)	audit script	`npm run audit:routes`

CI runs all of them on every pull request via test.yml. There is no paths: filter, deliberately — see Why CI has no path filter.

Five of these were ungated until 2026-08-12: ari-controller-node, payment-vault, agent-host, code-executor, and collab-gateway (which had no tests at all). Four had real suites — 68 and 360 tests among them — that never ran on a PR. One of the four, code-executor, had been 13-of-25 red for an unknown length of time. If you add a Tier-1 service, add its job in the same PR; a suite nobody runs is documentation.

Adding a test file¶

ai-manager discovers tests by glob, so a new *.test.ts under app/, components/, or lib/ runs with no further action:

"test": "node --import tsx --test \"app/**/*.test.ts\" \"components/**/*.test.ts\" \"components/**/*.test.tsx\" \"lib/**/*.test.ts\" \"lib/**/*.test.js\""

The globs are scoped to those three directories rather than a bare **/*.test.ts because node's glob does not exclude node_modules the way its default test-file discovery does — a repo-wide pattern matches 263 dependency test files and the run dies in somebody else's fixtures.

Warning: This replaced a hand-maintained list of 62 file paths. When it was a list, 14 tracked test files were missing from it and had never run in any suite (72 passing tests, including all four components/workflow-editor/ files). If you ever reintroduce an explicit list, you reintroduce that failure mode — and it is invisible, because the suite reports success.

collab-gateway uses the same runner with its own glob (src/**/*.test.ts), and tsconfig.build.json excludes test files so they don't ship in the image. Note that importing a module there can pull in redis.ts, which throws at import time when REDIS_URL is unset and otherwise opens a connection — so a unit test that reaches it dies on a bare checkout, which is what CI is. authz.test.ts gets away with importing authz.ts only because that file's Scope and SocketIdentity imports are import type, erasing the transitive presence.ts → redis.ts edge at compile time. Keep them type-only, and reach for a pure module rather than a value import when a gate needs testing. The rest discover by convention: vitest and pytest find their own files, and the conversation-api loop globs test_*.py.

apps/conversation-api/test.py is not a test despite the name — it is a manual Redis stream-writing scratchpad. The CI loop skips it by name.

agent-host: an unmarked `async def test_` is SKIPPED, not failed¶

agent-host has no pytest.ini or [tool.pytest.ini_options], so pytest-asyncio runs in its default strict mode: an async def test_… with no marker is collected and silently skipped. A suite of skips and a suite of passes have the same exit code.

Two ways out, both in use there: a _sync wrapper that calls asyncio.run (what the older modules do), or pytestmark = pytest.mark.asyncio at module scope (what tests/test_mcp_round_trip.py does, because its MCP client is an async context manager wrapping the whole test body). Either is fine — but read the collected count, not the colour, the first time you run a new async file.

Python test dependencies¶

pytest is generally not in pyproject.toml/uv.lock or requirements.txt, because the runtime lockfiles are resolved for the container images and a test runner has no business in a production image. It lives in a separate pinned file per app — with one deliberate exception:

# ai-orchestrator (uv, python 3.13)
uv sync --frozen --no-dev
uv pip install -r requirements-test.txt

# async-agent (pip, python 3.12)
pip install -r requirements.txt -r requirements-test.txt

# payment-vault (pip, python 3.11)
pip install -r requirements.txt -r requirements-test.txt

# code-executor (pip, python 3.12) — the shared runtime is separate because the
# sandbox WORKER imports tsf_runtime at module scope
pip install -r requirements.txt -r requirements-test.txt
pip install -e ../../shared/python-runtime

# agent-host (pip, python 3.11) — pytest + pytest-asyncio are IN requirements.txt
pip install -r requirements.txt

Run pytest through the venv's python (.venv/bin/python -m pytest), not uv run pytest: uv run re-syncs the environment against the lockfile first, and pytest is deliberately not in that lockfile, so it uninstalls the runner it was asked to run.

agent-host is the exception to the split-file rule: its test pins are already in requirements.txt. Leave them there — moving them would rebuild its image for no coverage gain — but know that its CI job is therefore a bare pip install -r requirements.txt.

Not pip install -e ".[dev]" for payment-vault. That was the documented way to get its test runner and it has never worked: payment-vault is a flat-layout application with two top-level packages, so setuptools refuses to auto-discover and the install fails outright. The pins moved to requirements-test.txt on 2026-08-12. If you find that incantation in an older note, it is wrong, not stale.

code-executor's first run pays ~15 s building its base venv. tests/conftest.py builds it with --system-site-packages so the sandbox worker inherits the current environment instead of resolving the much heavier requirements-base-venv.txt — which is why the editable shared/python-runtime install above is load-bearing rather than a convenience.

conversation-api has no pytest at all. Its suites are plain-assert files with a __main__ runner that exits non-zero on failure. Don't copy pytest into it, and don't copy its runner into the other apps.

Tests that need Postgres¶

Some conversation-api tests execute the real SQL under test rather than a paraphrase — the payments node policy, the payments outcome UPDATEs, and the customer-auth node policy. With no reachable DATABASE_URL they skip loudly rather than passing quietly:

31 payments policy tests passed; SKIPPED 14 SQL tests: no reachable DATABASE_URL (those execute the real query against Postgres)
1 record_outcome tests passed; SKIPPED 16 DB tests: no reachable DATABASE_URL (these execute the real conditional UPDATEs against Postgres)
SKIPPED all 9 customer-auth node-policy tests: no reachable DATABASE_URL (these execute the real query against Postgres)

Those three lines are 39 tests that did not run, and they cover the queries that decide what may be charged and what assurance a caller has reached. Each builds its own scratch rows inside a transaction it rolls back, so pointing them at the dev database is safe:

cd apps/conversation-api
bash ../../.devcontainer/scripts/with-env.sh .venv/bin/python test_payments_node_policy.py
# all 45 payments policy tests passed

Use with-env.sh <command>, which execs under the loaded environment. Don't try to read the URL out of with-env.sh --summary — the summary masks credentials.

CI runs them against a pgvector/pgvector:pg15 service container loaded from the core init scripts, so they are a real gate there. Demo-only scripts live under seed-demo/ and are outside that core glob. One core script is skipped, for a reason worth knowing:

Skipped	Why
`099_normalize_user.sql`	Strips `SUPERUSER` from `tsf_app` for production RLS fidelity. db-init-local.sh skips it by default and schema-drift.yml excludes it, so applying it in CI would test the suite under RLS rules no developer runs it under.

Unlike db-init-local.sh — which pipes every psql through || true — the CI loader runs with ON_ERROR_STOP and no swallowing, so a broken migration fails the job instead of leaving a half-applied schema that surfaces later as a confusing test failure.

Why CI has no path filter¶

The neighbouring workflows (typecheck.yml, schema-drift.yml) filter on paths:. The test workflow does not, because its highest-value tests are cross-app drift guards that read another service's source as text:

payments-mirror.test.ts reads ai-orchestrator's and conversation-api's Python to prove the payments vocabularies agree across four processes.
workflow-code-export/langgraph mirrors async-agent's vendored kernel.
collab-gateway/src/shape/shape.test.ts reads both copies of the Y.Doc shape converters — the gateway's and ai-manager's — and asserts their ten allow-lists match. A field added to one copy round-trips fine in the browser and is deleted on the next flush, so this guard has to run on a commit that touches only the other app.
collab-gateway/src/authz.test.ts reads ai-manager's permissions.ts and asserts the gateway's copy of the retired-name compatibility map, and the five current permission names, still agree. A consolidation done in ai-manager alone left the gateway denying every non-platform-admin catalog write for six weeks, silently — so this guard exists to fire on the ai-manager commit, not on a gateway one.
ai-manager/lib/auth-permissions-audit.test.ts walks apps/ai-manager, apps/collab-gateway/src, deploy, docs, scripts, and userguide for retired permission names, allow-listing the handful of compatibility maps. It scans two apps and two doc trees, so no single path filter covers it.

A filter on apps/ai-manager/** would skip the ai-manager job on exactly the commit that edits universal_builder.py and breaks the mirror. The guard would then fire on an unrelated change weeks later, pointing at the wrong author — which is worse than not having it. Every job here is minutes, not tens of minutes.

Suites that are not unit tests¶

npm run test:integration-api, test:management-api, test:a2a-e2e, and test:a2a-e2e:local in ai-manager reach live services and are not part of the CI gate. The A2A pair needs a running ai-manager plus a TRUESERVE_A2A_TOKEN (an OAuth access token or a tenant API token), so they belong in a manual verification pass, not on a PR.

The Playwright browser suite (npm run test:e2e in ai-manager) is a partial gate. The e2e job stands up Postgres, Redis, collab-gateway, and ai-manager and runs the authoring half — wizard, editor, autosave, validate, publish. The four steps that need a live session (chat, transcript, workflow run, sandboxed Python) declare themselves absent through E2E_STACK=minimal and skip, so a full local run is still required before merging anything that touches the runtime. See Browser E2E tests.