Operating agent evaluation workers

Status: Current | Last reviewed: 2026-10-20

Vanguard testing workspace workers

The additive testing workspace uses three additional processes in .devcontainer/Procfile.testing: testing-conversations, testing-assessments and testing-preparation. Start them independently of the main stack:

overmind start -D -d /workspaces/trueserve-framework \
  -f .devcontainer/Procfile.testing -s .overmind-testing.sock -N -r all
overmind status -s .overmind-testing.sock
overmind echo -s .overmind-testing.sock

Start only one auxiliary pool per city. It runs separately so adopting the pilot does not replace or restart the main process supervisor. The main Procfile's legacy simulation workers remain necessary. They exclude entity_model=2; the new conversation publisher/consumer handles only that model and uses the separate agent_testing_conversations queue.

Apply migrations 076 and 077 before starting this pool. Reload the changed legacy simulation workers and the shared orchestrator runtime when no active conversations are running. The new pool's authoring role uses agent_test_preparations; it must not claim legacy suite preparation. Review workers retain original assessments and independent leases. To reload the pilot after its worker code changes:

overmind restart -s .overmind-testing.sock \
  testing-conversations testing-assessments testing-preparation

Keep the auxiliary socket running for review. It needs to be started again after a host/devcontainer reboot; it is not part of the main stack's automatic startup.

Shared execution behavior

Before a qualification batch, audit every local Overmind worker pool, including alternate sockets. Restart participating conversation and review workers after their code changes; old pools can consume new jobs from the same broker. Keep mixed-runtime pilots and failed attempts in the record. Do not edit watched orchestrator Python while conversations are running.

Copilot normalization 2026-09-29-copilot-human-input preserves replay speakers as human evidence, without duplicating representative captions as AI speech. Caller policy 2026-09-30-completion-and-closing requires the stop verifier to supply the customer's exact next utterance rather than narrating the persona. The service goal may already be met while a required customer closing remains to be spoken; that is a valid continuation. Stopping still requires literal evidence from the visible conversation. These versions are stored with evidence and reviews.

Automatic tests for cascaded STT/LLM/TTS profiles use synthetic audio so the microphone, provider turn signals and playback path participate in the test. Explicit text diagnostics remain a separate transport choice. Copilot replay continues to exclude speech recognition.

Copilot replay can include startup_events in its saved scenario. The authenticated runtime snapshot supplies the event names, delivery modes and JSON data. The publisher sends them once through its normal host-event handler after workers register, before the startup checkpoint. This exercises context and event filters without inventing transcript turns. Limits are 12 events and 32,000 serialized characters; browser/LMS interaction remains a separate check.

Backend activity can carry a stable activity_id when the event describes an aggregate, such as GPT Live's managed continuation chain. The stream worker keys these snapshots independently from per-delegation client jobs. A provider delegation-ID change must not leave the previous ID pending forever, and one aggregate becoming idle must not clear another job. Tool work, audible playback and the normal reply quiet window still gate the next synthetic caller turn. Restart the conversation worker after changing stream-transport code; changing the orchestrator alone does not update that worker.

The publisher reserves eligible jobs with one autocommitted database statement before sending broker messages. It does not hold job locks while waiting for broker confirmation: a fast consumer can claim immediately. If publication fails, the reservation expires after 30 seconds and the queued job is published again; normal attempt fencing handles duplicate deliveries. A PostgreSQL integration test exercises a consumer claim before publisher confirmation and recovery after lost publication.

Supported GPT-4.1, GPT-4o, GPT-5.4 and GPT-5.6 reviewers use a strict response schema keyed by every requested goal and golden ID. The stored review retains its existing array format. Older/custom model IDs retain JSON mode and required-goal validation. Missing, truncated, refused or malformed assessments remain review errors after one bounded review-only retry. GPT-5.4 and GPT-5.6 requests use Responses with medium reasoning and a 12,000-token output budget; the other models retain Chat Completions. Only a completed assistant output message is parsed, never reasoning or a refused/partial response. Selecting a model still requires appropriate integration access. Token usage is recorded for both APIs.

Numbered evaluation events finish their serialized send even if a speech interruption cancels their producer while waiting for the shared gRPC writer. This prevents a cancellation from leaving an evidence-sequence gap. Actual send errors still reach the producer, and the worker still rejects missing evidence. GPT Live backend_output events contain only delegated assistant output text (bounded to 6,000 characters, with a truncation flag), separately from spoken transcripts. They support diagnosis and are not scored as customer-facing speech. GPT Live also records application_message when an external instruction reaches the backend, with its source and capability attestation. Scoring includes these as unspoken application instructions at the observed sequence. This preserves supervisor questions before their answers; queue acceptance, discarded progress and failed sends never count as delivery. A requested action or claimed outcome still needs independent tool or runtime evidence. Normalization version 2026-09-23-application-origin names the flag capability_attested: it describes internal capability results, not whether an advisor-channel message was authenticated. Historical evidence and grades remain unchanged. The supervisor clarification coordinator requires a spoken question followed by a new input sent by the simulated caller and a caption containing speech. Ambient [breath] transcripts and late revisions of an earlier caller answer cannot advance the decision phase. This establishes a real exchange; the reviewer still evaluates whether the question was complete and the answer adequate. GPT Live's procedure_brief and tool_observation events are diagnostics only. They are not extra voice prompts or evidence that the caller heard their text.

Tool executions are explicitly unspoken evidence. Reviewers must distinguish them from actual assistant dialogue and inspect the observed disconnect event. They must honor each goal's output location: a summary required at params.summary is assessed in that tool argument, without inventing a requirement to read it aloud. Routing to a specialist is assessed separately from protected record access, verification or payment. Actual unauthorized attempts and false or incomplete summaries still fail. New simulation reviews record details.judge_prompt_version; historical reviews retain their original results. When calibrating the reviewer, preserve those results and check both valid conversations and negative controls, such as a fabricated one-sentence summary or access without verification. A change in the reviewer prompt is not evidence that the agent improved. The reviewer presentation omits the top-level instructions field in tool results: these embedded control prompts and examples are not executed outcomes. Actual arguments, status, errors, record data (including nested fields named instructions) and every spoken word remain. Archived evidence is unchanged. New reviews also record details.evidence_normalization_version. Validate presentation changes with both positive and negative examples; preserve the original grades when privately recalibrating historical conversations.

An accepted end request and its surrounding caption fragments are presented as one closing response, retaining the original tool arguments and every spoken word. Grouping stops at caller input (including input without a caption), another tool, or disconnect. Two real goodbyes remain in the combined text and must still fail a one-farewell requirement. Cancellation fails the deterministic termination check even if a terminal status was emitted. A claim of repeated farewells must cite both actual spoken utterances, excluding golden examples, caller speech and tool metadata. This instruction reduces ambiguity but does not make a model grade authoritative: archive and inspect contradictory assessments rather than counting a favorable re-grade as an agent improvement.

The 2026-09-23-causal-captions normalizer places each final caption prefix where it was first observed, splitting at intervening caller, tool or delivered runtime events. A growing caption cannot move a later approval statement back before the approval. Corrected recognition hypotheses are replaced by the final caption. Tool evidence records both the request sequence and the latest observed result sequence, and appears at the latter. This represents evidence arrival; it does not establish exact audio playback timing.

The 2026-09-23-actor-aware-requirements reviewer receives selected scenario requirements separately from observed platform checks. Direct approval does not require clarification. Strict-schema reviews require at least one literal actual-evidence quote for every issue, including minor/info findings; missing or invented quotes trigger the existing single review-only retry, then a review error. Quote validation establishes that the evidence exists, not that the model interpreted it correctly. Preserve positive and deliberately faulty calibration examples alongside qualification runs. Normalizer 2026-09-23-utterance-segments also supplies an utterance identifier, part number/count and first-observed sequence for each spoken segment. The reviewer can reconstruct continuous wording across tool rows without moving later claims before their supporting results. Distinct repeated requests and two goodbyes inside one merged closing row remain visible. Conditional goals apply to actual preceding caller intent; later requests cannot authorize an earlier action. Each strict goal assessment states its conditional applicability before its score. The stored optional applicability explanation supplements existing goal evidence; it does not change goals, pass thresholds or historical assessments. When an expected outcome begins with an exact copy of the pinned caller goal, the review labels that passage as the caller's task and retains every remaining requirement. Customer instructions such as “say you need no further help” must not become words the assistant is required to say. Nonmatching outcomes keep their original presentation. Evidence quotes may have one matching pair of quotation delimiters only if the complete enclosed text matches the cited turn.

Vanguard runs the Athens Performance authoring/review surface with durable simulation workers. Apply 044_agent_sim_batches.sql, 064_agent_evaluation.sql 065_agent_test_cases.sql and 066_capability_test_events.sql to an existing database through the normal migration workflow. Fresh databases use the updated baseline. Do not restore the baseline over a populated database.

Local roles are registered in .devcontainer/Procfile:

bash .devcontainer/scripts/start-simulation-worker.sh conversations
bash .devcontainer/scripts/start-simulation-worker.sh reviews

Each command is a long-running process. The conversation role needs Postgres, RabbitMQ (BROKER_URL, falling back to RABBITMQ_URL), access to the orchestrator gRPC endpoint and an active tenant OpenAI integration for auxiliary model calls. The review role uses the same database and integration, independently of chat workers. Credentials are resolved with tenant access checks and cached in memory for at most 60 seconds; simulation snapshots and wake-up messages contain no provider credentials. Internal nested workflow jobs carry a short-lived attempt capability outside model-visible context.

GPT Live backend activity remains active while function results or their continuation are pending. A completed response that requests a tool is not the end of the delegated task. The caller worker uses this signal to wait for the actual result instead of reacting to an intermediate pause.

The caller checks for newer agent output both after deciding what to say and after speech synthesis. A supervisor question or secure-UI result arriving during synthesis discards that unsent sentence, refreshes caller history and asks for a new decision. Discarded speech consumes no caller turn and supplies no consent or closing evidence. Ten consecutive refreshes still fail the run; this does not extend the response timeout or change the judge's criteria.

Setting	Default	Purpose
SIMULATION_WORKER_ROLE	conversations	conversations or reviews
SIMULATION_WORKER_CONCURRENCY	20	Outstanding conversation jobs per process
SIMULATION_WORKER_AUDIO_CONCURRENCY	4	Audio/transcript sessions per process
SIMULATION_GLOBAL_ACTIVE	100	Global admission fallback when no table override exists
SIMULATION_ORCHESTRATOR_ADDRESSES	ORCHESTRATOR_ADDRESS or localhost:50051	Comma-separated endpoints; new streams rotate between them
SIMULATION_CALLER_MODEL	gpt-4o-mini	Simulated caller
SIMULATION_CALLER_VERIFIER_MODEL	gpt-4.1	Independent check of proposed caller stopping decisions
SIMULATION_JUDGE_MODEL	gpt-4o-mini	Judge
SIMULATION_TTS_MODEL / SIMULATION_TTS_VOICE	gpt-4o-mini-tts / marin	Synthetic caller audio
SIMULATION_AUTHOR_MODEL	gpt-4o-mini	Instruction-based tool mock drafting
ORCHESTRATOR_MAX_SESSIONS	1000	Per-orchestrator stream ceiling
ORCHESTRATOR_DRAIN_FILE	/tmp/ai-orchestrator.draining	Presence rejects new streams while existing streams finish

agent_sim_capacity overrides shared limits by scope: global, tenant:<uuid> or integration:<uuid>. Each row sets active_limit, starting_limit and starts_per_second. Without a row, global limits are 100/20/20 and tenant/integration limits are 20/5/5. Batch concurrency is another upper bound. Increase limits only with enough workers, orchestrators, database capacity and provider quota. A request for 1,000 simultaneous conversations does not override these limits.

For example, an operator can configure a tested tenant allocation:

INSERT INTO agent_sim_capacity(scope,tenant_id,active_limit,starting_limit,starts_per_second)
VALUES ('tenant:<tenant-uuid>','<tenant-uuid>',100,10,10)
ON CONFLICT(scope) DO UPDATE SET active_limit=excluded.active_limit,
  starting_limit=excluded.starting_limit, starts_per_second=excluded.starts_per_second;

Admission uses a short advisory-lock transaction. It never holds that lock over provider startup or conversation I/O. RabbitMQ prefetch bounds local work. Shared admission and per-process audio slots serve different purposes; grow the worker count as well as shared limits. Existing gRPC streams stay with their owner when endpoints are added. Configure process/container supervision to restart workers after broker connection loss.

Leases last 60 seconds and renew every 15 seconds. Recovery runs every 15 seconds. A lost worker is fenced; a fresh attempt gets a new conversation and fresh fixture state, with at most two attempts by default. Startup, caller, reply and speech requests have deadlines; conversations have a 15-minute limit. Cancellation revokes active ownership, which the runtime checks periodically. Shutdown stops intake, gives active work 30 seconds, then aborts streams so leases can recover. Queued rows republish when broker delivery is lost.

Before sending a generated caller reply, the worker checks whether the agent spoke again while the caller model was running (for example, after a photo or payment event). It adds that completed speech to caller history and regenerates the reply, including any decision to finish. Discarded drafts do not count as conversation turns. caller_context_refreshes records these updates in attempt evidence. Ten consecutive refreshes are allowed; continued agent output beyond that budget fails explicitly instead of generating caller drafts indefinitely. An observed session end discards the pending caller reply.

Caller policy 2026-09-23-persona-facts records proposed decisions and their reasons in evidence.caller_decisions, with evidence.caller_prompt_version. Only proposed stops receive one additional low-temperature verification request. It sees persona facts and caller-visible conversation, without fixture outcomes, tool evidence or grading rules. It may continue unfinished intake, accept completed work or a valid refusal/withdrawal, or return uncertainty. A stop requires a literal supporting conversation quote; uncertainty, malformed output or provider failure becomes a caller error, never automatic completion. Each model request has a 45-second deadline. Selected and stale/discarded decisions are recorded, with at most 500 diagnostic entries per attempt. This adds one model call per proposed stop, not per normal customer reply. Normal caller turns answer the current question from supplied persona facts, including requests to repeat known details. Emotional tone does not justify inventing forgotten facts or new refusals. Explicit persona refusals and genuinely unavailable facts remain valid. Calibrate the configured caller model on both cooperative answers and refusal cases when changing this policy.

Audio input policy 2026-09-23-spoken-iso-dates renders standalone valid 1900–2099 ISO calendar dates in English caller speech with month, ordinal day and year words. For example, 1990-02-14 is spoken as “February fourteenth, nineteen ninety,” avoiding numeric-hyphen pronunciation that changed a birth year in a live test. Other locales, dates embedded in identifiers/URLs/timestamps, and invalid dates remain unchanged. intended_utterances preserves the original text; speech_utterances records the TTS input and audio_normalization_version identifies the policy. Reviews still use what the agent actually heard and said. No lookup fixture is loosened to accommodate an incorrect birthdate.

Sandbox execution failures classified as provider throttling/unavailability, broken streams, startup timeouts or content-filter rejections receive at most one automatic retry, after 10–15 seconds of jittered delay. A retry uses the same pinned test, model, prompts and fixtures in a new conversation. It does not replay the prior dialogue verbatim: the simulated caller and agent generate new turns. A filter rejection is recorded as a provider failure with an unknown trigger, not proof that the content was inappropriate.

The previous attempt retains its error and evidence, including failure.kind, failure.retry_scheduled and failure.retry_delay_ms. Callbacks and unfinished supervisor work are fenced before the job is queued again. Disconnect still runs after a provider error so the old stream can finalize. Retries obey the same admission limits as first attempts. A second failure is terminal; configuration, harness, user cancellation, ordinary reply timeouts and judged behavior failures are not automatically retried. Non-sandbox provider failures are not automatically replayed.

The batch detail API returns aggregate job/review progress and at most 100 jobs per page (after=<ordinal>). Inspect last_error, attempt status, stop reason, review status, audio lateness and pinned target hash. Provider/harness failures are separate from judged agent quality. Failed scoring can be retried without redialing.

Supervisor-agent runs reserve two active slots per main conversation. Both streams pass the shared startup and starts-per-second gates. Ensure a tenant's active limit permits two slots before enabling this mode. The worker manages child streams and callback polling; starting one does not issue a browser chat request. Cancellation/recovery closes event ownership and cancels pending participants and suspended test workflows.

Supervisor event delivery uses the configured ASYNC_AGENT_URL; trusted conversation injections and platform responses use CONVERSATION_API_URL. Only internal calls carry the attempt capability. The participant and event tables enforce tenant matching to their parent attempt, in addition to RLS. Inspect agent_sim_events.status/payload and participant evidence for a stalled supervisor. A request without a configured scenario is a preflight/setup error.

The Studio Testing editor also needs the updated code-server extension and Studio API. Run implementation tests requires the updated code-executor runtime protocol and nsjail: network_access=none creates a separate network namespace and refuses the subprocess fallback. In local Docker setups where the auth proxy shares code-server's network namespace, restart the auth proxy after restarting code-server.

The saved-test picker uses compact ?view=tests responses. Aggregate counts cover the whole run, while individual statuses and latest agent replies are paginated. The visible run polls every 1.5 seconds, slowing to 5 seconds when hidden. Closing the window stops polling; durable workers continue execution.

Selected cases snapshot their revision, expected outcome and linked golden. Jobs retain case ID/revision and use the case's own turn limit. Scoring compares each conversation with that case's golden and expected outcome plus the general agent goals. It records test_outcome and revision in review details. Passing requires overall score, expected-outcome score and every must-goal score to be at least 0.8, with no major or critical issues. Checking failures remain unscored; they cannot appear as passed.

HealthPlus example qualification

The synthetic Vanguard example is agent 3032. From the repository root, refresh its capability/tool fixtures and case response settings:

bash .devcontainer/scripts/with-env.sh -- bash -c \
  'cd apps/ai-manager && node --import tsx scripts/seed-3032-capability-tests.ts'

This requires the existing local 3032 fixture corpus and tenant. It versions changes, preserves capability prompts/logic and unrelated workspace files, and fixes the refusal persona's consent contradiction. Repeating it without changes must not create new revisions. It is a targeted example-maintenance script, not a general migration.

Run the saved cases from Preview. Qualify the exact snapshot from the resulting batch independently of model behavior:

bash .devcontainer/scripts/with-env.sh -- bash -c \
  'cd apps/ai-orchestrator && .venv/bin/python scripts/check_healthplus_mocks.py <batch-uuid>'

The checker creates a temporary fenced attempt, exercises all unique pinned operations and examples through the real shared dispatcher/Postgres state, then checks retry state, future appointments, confirmation/photo, $10/$15/$25 and fractional payments, no-match responses, and UUID/operation-based Salesforce workflow dispatch. It deletes only its temporary batch and makes no model or external-backend calls. It also executes the pinned startup workflow through async-agent and checks the resolved first/last name, retaining that synthetic workflow trace. This verifies the native Salesforce envelope and workflow output mapping, not just JSON Schema acceptance. It does not evaluate the agent or bypass guards in an actual conversation.

Vanguard's local .env.local sets SIMULATION_JUDGE_MODEL=gpt-4.1 for these saved tests. Calibration against the recorded appointment conversation removed false missing-consent findings from the smaller reviewer while retaining the actual fabricated UI details, duplicate payment record and closing violations. Closing calibration distinguishes one final acknowledgment after end_session from an actual repeated goodbye or speech after runtime termination. Supervisor caller goals explicitly require consent to the relayed amount and a payment outcome; receiving approval alone is not completion. Malformed assessments or missing goal scores get one bounded reviewer retry using the same conversation evidence. Repeated invalid output remains unscored. The general worker default remains unchanged. Recheck saved evidence after changing reviewer settings; do not redial just to rerun assessment. A model appearing in discovery does not establish that the configured connection can invoke it—verify an actual completion before changing the worker.

Simulation review evidence now includes the observed STATUS_ENDED at its actual position and explicitly reports whether params.summary is a nonempty string. A recap embedded in a malformed property name is not a summary. The deterministic "Agent ended the session" check requires both a successful tool result and a later terminal status; a successful request alone fails. These evidence-format changes do not rewrite stored reviews. When comparing an experiment across formats, preserve the original and record a separate review of the same captured conversation, with source hashes and unchanged goals. Inspect the evidence even after a pass: the judge can still miscount goodbyes or miss malformed arguments.

Inspect harness_errors separately from tool_input_errors, callback evidence, and judged failures. A blocked lookup before authentication or before patient resolution is expected enforcement of the real capability logic. Do not remove those guards or return successful writes for unknown record IDs to turn a test green. Repeated record_copay calls are behavior failures even when the fixture correctly reflects both requests.

Validation commands from apps/ai-manager:

npm run typecheck
npm run test:simulation
# Stop conversation workers, or use an isolated database with migrations applied:
SIMULATION_DB_TEST=1 node --import tsx --test lib/performance/simulation/jobs.integration.test.ts
CAPABILITY_EXECUTOR_TEST=1 node --import tsx --test lib/capability-testing/testing.test.ts

scripts/smoke-supervisor-runtime.ts exercises a disposable real workflow and root agent with supervisor 3026. Opt in with SIMULATION_LIVE_SMOKE=1; choose SUPERVISOR_SMOKE_MODE=agent|decision, SUPERVISOR_SMOKE_OUTCOME=approve|reject|clarify|timeout, and optionally SUPERVISOR_SMOKE_NODE=human_review. It records private evidence, verifies a real session-end tool call, and checks duplicate and post-completion responses. Set SUPERVISOR_SMOKE_PLATFORM=1 with the approval outcome to include real authentication assurance, a simulated photo upload through the UI event listener, and payment settlement for the approved amount. Only the external ceremonies and provider result are simulated; no card is collected or charged. These bounded provider checks do not certify 1,000 concurrent paid sessions.

scripts/smoke-simulation-runtime.ts is opt-in and runs one paid conversation through a disposable draft. Set SIMULATION_LIVE_SMOKE=1, SIMULATION_SMOKE_SOURCE=<draft-agent-id> and optionally SIMULATION_SMOKE_SCORE=1. It replaces business behavior with a small booking fixture and removes its temporary definitions afterward. It records evidence in a private /tmp/evaluation-smoke-*.json file. A controlled 1,000-start test proves startup is not globally serialized; it does not certify 1,000 live provider sessions or speech timing on an undersized host.

Set SIMULATION_SMOKE_QUEUED=1 to exercise RabbitMQ, the outbox, the caller and the independent scoring worker together. Both worker roles must be running. Caller models must support strict JSON-schema responses: the caller explicitly reports pending, met or unresolvable and stops when its original goal is met. A conversation reaching its turn limit is finalized and scored; it is not an infrastructure failure.

The scoring lease is returned as text to preserve PostgreSQL microseconds. Converting it to JavaScript Date before comparing would reject a valid scorer as stale. Nested mock failures are persisted separately from tool results so a model recovering from an error cannot hide an invalid test harness.

Local validation also includes EVALUATION_DB_TEST=1 with the orchestrator's tests/test_evaluation_context.py and the ai-manager browser test e2e/agent-evaluation.spec.ts. Controlled 1,000-start and 1,000-job tests use local services and deterministic provider boundaries. They are not a live provider load certification.

The opt-in async-agent test EVALUATION_HTTP_TEST=1 with tests/test_evaluation_http.py checks the actual HTTP workflow executor. It uses a valid pinned graph with a mock and deliberately invalid mutable live graph, proving that the nested call uses the snapshot. Run it only against the local service after it reports healthy.

Vanguard validation, 2026-09-21

Real ai-orchestrator conversations completed with text cascade (gpt-4o-mini), OpenAI Realtime (gpt-realtime-2), GPT Live (gpt-live-1) and Nova Sonic. The queued worker path completed conversations and independent scoring for text cascade, GPT Live and Nova Sonic. GPT Live used paced synthesized caller PCM; Nova Sonic used native text input with spoken output. Browser checks covered saved-case selection, the draft save gate, reopening active progress, saved result snapshots, case authoring and editing tool mock revisions.

A saved case also passed through the actual Preview Play button, durable queue, ai-orchestrator, checked tool mock and independent scorer. Reopening restored progress; editing its definition and retrying submission retained the original case revision and run. Database checks cover tenant ownership, stale edits and failed expected outcomes despite high overall scores. The case migration passed an isolated clean restore and replay check.

The controlled database and startup checks exercised 1,000 jobs and 1,000 concurrent readiness requests. They did not run 1,000 simultaneous paid model sessions. Other provider adapters still need credentialed smoke runs in their target environments before being treated as live-validated.

Build the standalone worker image from the repository root:

docker build -f apps/ai-manager/Dockerfile.simulation-worker -t tsf-evaluation-worker:local .
docker run --rm --network none tsf-evaluation-worker:local node --import tsx scripts/run-simulation-worker.ts --check

Supply database, broker and provider-integration decryption settings at runtime. Run the same image with SIMULATION_WORKER_ROLE=reviews for scoring. Neither role requires a Next.js web process. A production deployment must configure the same private service connectivity and database TLS trust as ai-manager.

When adding workers to an already-running Vanguard stack without restarting its other processes, the supplemental local Overmind instance can use .devcontainer/local/.evaluation-overmind.sock. Inspect it with overmind status -s .devcontainer/local/.evaluation-overmind.sock. The regular Procfile includes both roles for subsequent full stack starts. Multiple instances coordinate through the same database leases and limits.

The final validation set passed 89 TypeScript tests, two opt-in database integration tests, 126 orchestrator regression tests, the 1,000-start conversation API test group, nested workflow execution checks, and three browser checks including authentication setup. TypeScript checking and feature lint completed without errors; existing warnings remain. The worker container also passed its offline --check startup.

Nested workflow capabilities follow configured internal service origins, including deployment FQDNs. A required workflow hop fails if the capability cannot be forwarded; it must never silently become a run with real external tools. External tool origins never receive the attempt capability.

Supervisor validation also completed approval, rejection, clarification and timeout paths against real workflow checkpoints, including a human-review node. Full-agent approval and clarification ran supervisor 3026, delivered its actual tool-based relay and finalized both streams. Duplicate delivery was consumed once; post-completion callbacks were refused.

The combined platform check completed authentication, photo upload (including a repeated action), the actual supervisor relay, workflow resume, payment settlement for exactly the approved $10, and a successful end_session call. Studio browser validation generated responses, saved a full-agent supervisor scenario and executed authored Python from the Testing editor. The executor's network isolation and the two-slot supervisor admission rule passed focused tests. These checks use disposable root agents; saved case results for agent 3032 remain separate evidence of that agent's authored behavior.

Four simultaneous bounded checks also completed with separate workflow owners: decision approval/rejection and full-agent approval/clarification. An earlier disposable check was refused by the ownership guard before event registration; it did not reproduce in these concurrent checks. Registration reports missing execution, wrong workflow and wrong attempt separately without relaxing the guard.

Repeated APPT-06 runs for agent 3032 draft used the same saved case revision 5 and target configuration. The original content-filter rejection did not recur. The reruns exposed a Nova prompt-start race and missing end_session callback; both were corrected. A real connection failure exercised the automatic retry after 12.720 seconds, retaining its failed attempt and creating a new conversation. The retry-policy tests separately cover content-filter retries and the two-attempt cap.

Final batch a39e32fa-e8d0-41af-a95d-9cdcf26d6051 recorded completed end_session tool calls in both conversations. Neither test passed: one missed its supervisor outcome after an appointment lookup problem; the other completed the supervisor exchange but retained a tool-input schema error. These are authored-agent/test-contract failures, not evidence that APPT-06 is qualified.

The passive test viewer was checked against both the real caller and supervisor records, including normal visual cards, with no execution mutations or failed surface requests. Browser regressions cover incoming messages, disabled visual controls, earlier retry attempts, reload, return/reset/close/reopen and mobile sizing. The shared A2UI package was rebuilt and installed into both frontend apps; type checks, focused lint and both documentation builds passed (existing lint/documentation warnings remain).

Nova’s provider error “The system encountered an unexpected error during processing. Try your request again.” and InternalServerException receive the same one bounded sandbox retry as temporary unavailability. The exact Nova stream rejection “Invalid input request, please fix your input and try again.” is classified separately as provider_rejected and may receive the same one unchanged sandbox retry to test reproducibility. Known configuration/credentials/handler failures, tool/mock failures and completed but failing grades still do not retry. Preserve the first failed attempt and report first-attempt and post-retry results separately.

For Nova simulation failures, provider_error evidence includes at most 96 protocol metadata events (event kind, identifiers, roles, timing, tool-result length and JSON validity). It excludes prompts, tool values, credentials and audio, and does not change outbound messages. Preserve this alongside the failed attempt when investigating provider input rejections.