Workflow startup and recovery

Status: Current | Last reviewed: 2026-10-13

The first workflow expansion package gives direct HTTP and agent-host MCP requests one durable admission path, reserves interactive worker capacity, and measures startup through the first useful node. It retains LangGraph and the existing RabbitMQ topology. No gateway, routing model or team framework is added.

Admission and execution

POST /api/v1/workflows/{id}/execute with execution_mode: "async" commits workflow_executions and a private scheduled_jobs dispatch envelope in one transaction. It immediately publishes a persistent RabbitMQ job with mandatory routing and publisher confirmation. Only confirmed publication marks the dispatch row dispatched. A failed or uncertain publish leaves it pending; the HTTP process's recovery loop retries eligible rows every second.

The response remains HTTP 200, status: "submitted", with an execution_id. This receipt means durable acceptance. It does not assert that a worker has started. Healthy publication is immediate; a broker outage can consume the five-second publish timeout before the receipt returns.

Optional Idempotency-Key deduplicates direct asynchronous submissions within a tenant/workflow. Identical inputs, caller chain and execution overrides join the same execution; changed arguments return 409. Evaluation attempts are isolated from one another. Keys last as long as their execution records.

The admitted root configuration/version is copied into the private dispatch envelope. Resume jobs recover it from that envelope. Nested workflows still resolve through the existing provider; this is not a transitive dependency snapshot. Execution-time authorization and credentials remain separate from the configuration snapshot.

Keep dispatched envelopes for the lifetime of resumable executions. They can contain evaluation capabilities and must not be exposed as public execution metadata. The generic scheduled-job poller excludes these canonical envelopes. Event intake, schedules and voice /prepare retain their existing admission paths; they are not converted to transactional outbox producers by this package.

New asynchronous runs are pending, with null started_at, while setup runs. The first non-start/end/passthrough node wrapper records the start time and publishes the existing started progress event. The Test panel renders an absent start time as Pending.

Capacity and readiness

Setting	Default	Meaning
WORKFLOW_JOB_CONCURRENCY	4	Background workflow slots and queue prefetch
WORKFLOW_INTERACTIVE_CONCURRENCY	2	Additional priority >=5 workflow slots and queue prefetch
WORKFLOW_WORKER_LANE	all	all, interactive, or background queue consumption

RabbitMQ uses independent channels and semaphores for the two job lanes. Normal work cannot occupy reserved interactive slots. Direct HTTP/MCP jobs use priority 5; existing priority producers share that capacity. An all process now permits six full workflow jobs instead of four. This consumes additional resources; the measured gain is not a same-capacity comparison. Per-provider/model limits continue to apply.

For stronger process isolation, run separate instances with WORKFLOW_WORKER_LANE=interactive and background, sharing dependencies. Interactive instances omit scheduler, batch, Salesforce Bulk and fanout pollers. Both roles still host HTTP and dispatch recovery; this package adds the role setting, not a second chart deployment. Keep background capacity running when choosing interactive-only instances.

HTTP startup and consumer startup warm PostgreSQL, Redis and the required Postgres checkpointer. Missing checkpoint support or initialization failure prevents successful startup; execution no longer silently loses durability. Database and Redis first initialization are serialized within the process. Tool loading and graph compilation still occur per run.

Recovery guarantees and limits

Execution leases use random ownership tokens, periodic renewal and compare-and-delete release. Lease loss cancels the local task and requeues the job. A stale owner cannot release a successor's lease.

Duplicate start jobs reuse a stable execution/run. Completed or suspended starts are not restarted. Interrupted active work resumes the saved checkpoint, including cancellation before graph entry. Jobs without an execution ID derive a stable identity from their job ID. Resume job IDs are recorded in private checkpoint state so redelivery of one approval cannot approve a later gate. Broker shutdown cancels tracked workflow tasks and requeues interrupted work.

Cancellation is cooperative. An external write may succeed before its checkpoint commits; a node interrupted mid-call can run again. This is not exactly-once execution or external-system fencing. Business writes still need downstream idempotency and reconciliation. Ordinary application failures retain the existing error/DLQ behavior.

MCP callers

Generic and named agent-host workflow tools call the authenticated HTTP API. They no longer create their own run or write the unused Redis job stream. trigger_workflow returns execution_id; check_workflow_status accepts that receipt or a run ID. Named tools add workflow_idempotency_key; generic tools accept idempotency_key. Both map to the HTTP header.

wait_for_result polls the same durable execution for up to 120 seconds. Timeout or polling failure preserves the receipt. Status and approval forward the validated caller bearer and tenant. Only verified internal/evaluation contexts can use the configured service credential fallback; the downstream API still verifies tenant and scope. Refresh client tool schemas after rollout.

Set ASYNC_AGENT_URL on agent-host; its default is http://async-agent:8000. The local launcher defaults to http://localhost:8000.

Observe startup

GET /api/v1/workflow-runs/{id} accepts either the submitted execution receipt or a specific run UUID and enforces tenant authorization. It returns the nullable run ID/timestamps, status, run metadata and execution SSE path.

Scrape /metrics on async-agent over the internal service network. It exposes trueserve_workflow_startup_seconds{stage,lane} without tenant/execution labels. Startup stages include authentication, durable admission, publication, consumer entry, lease acquisition, run claim, configuration/tools/graph ready, runtime entry and first useful node. Delivery and slot-acquisition timestamps also travel in the worker context. workflow_runs.run_metadata.startup retains the worker's known timestamps, first-node ID and lane.

The ASGI boundary records initial HTTP receipt and receipt of the complete body. Histograms measure elapsed time from complete-body receipt. First-node instrumentation timestamps entry into the node wrapper, before its startup bookkeeping and trace writes; it is not model first token or business output. The benchmark below measures entry into the actual assignment handler after that bookkeeping. Histograms use wall clocks across services, so synchronize hosts and account for clock skew. Scraping is process-local.

API-token authorization, expiry and membership checks still run per request. Only the nonauthoritative last_used write is coalesced in the background, at most once per minute per token per process.

Example p95 query:

histogram_quantile(
  0.95,
  sum by (le) (
    rate(trueserve_workflow_startup_seconds_bucket{
      stage="first_node",lane="interactive"
    }[5m])
  )
)

Inspect pending canonical dispatches with a read-only query:

SELECT count(*) AS pending, min(execute_at) AS oldest
FROM scheduled_jobs
WHERE status = 'pending' AND job_type = 'execution_start'
  AND payload ? '_dispatch_job';

Local performance evidence

The before source was preserved before editing the application. Both versions use the same loopback HTTP harness, real API-token authentication, PostgreSQL, Redis, RabbitMQ and Postgres checkpoints. Synthetic workflows invoke native Assignment; background workflows wait 350 ms without models or business APIs. Ten warmups precede 200 idle requests and 100 mixed-load trials. Each mixed trial fills four background slots before submitting one interactive request. Trials wait for completion; this is not a constant-arrival capacity test.

Measured on 2026-10-13:

Measure	Before p50 / p95 / p99	After p50 / p95 / p99
Idle request to native handler	13.44 / 15.84 / 16.73 ms	14.86 / 17.76 / 19.08 ms
Idle HTTP acceptance	9.17 / 10.86 / 13.40 ms	13.65 / 15.94 / 17.66 ms
Interactive start with four background jobs	351.31 / 372.86 / 656.04 ms	32.17 / 37.19 / 69.22 ms

The mixed-load p95 falls about 90%. Four of 100 final mixed trials still exceed 50 ms; a shorter repeat recorded mixed p95 70.25 ms. The larger run supports the warm local p95 goal, but tail latency remains variable. Idle startup is slightly slower, and HTTP acceptance costs about 5 ms more at p95 because admission now persists recoverable dispatch intent.

Before results and after results include all raw samples, capacity settings and runtime source hashes. The sequential workload, shared devcontainer, no TLS/proxy and no model calls limit interpretation. These are not official ingress, burst-load, cold-start or post-guardrail measurements.

Reproduce in disposable dependencies

Run from the repository root. These fixed test endpoints and credentials are only for isolated fixtures. The schema includes Supabase roles/extensions, so the fixture uses the matching Supabase Postgres image.

docker run -d --name tsf-workflow-package-postgres \
  -p 127.0.0.1:55439:5432 \
  -e POSTGRES_PASSWORD=workflow-test-only -e POSTGRES_DB=workflow_package \
  supabase/postgres:15.8.1.060
docker run -d --name tsf-workflow-package-redis \
  -p 127.0.0.1:56389:6379 redis:7-alpine
docker run -d --name tsf-workflow-package-rabbitmq \
  -p 127.0.0.1:56739:5672 \
  -e RABBITMQ_DEFAULT_USER=workflow_test \
  -e RABBITMQ_DEFAULT_PASS=workflow-test-only rabbitmq:3-management-alpine

Wait for PostgreSQL and RabbitMQ readiness, then load the disposable schema:

docker exec -e PGPASSWORD=workflow-test-only -i tsf-workflow-package-postgres \
  psql -U supabase_admin -d workflow_package -v ON_ERROR_STOP=1 \
  < deploy/init-scripts/000_full_schema.sql
docker exec -e PGPASSWORD=workflow-test-only tsf-workflow-package-postgres \
  psql -U supabase_admin -d workflow_package -v ON_ERROR_STOP=1 \
  -c "CREATE ROLE workflow_runner LOGIN SUPERUSER PASSWORD 'workflow-test-only'"
apps/async-agent/.venv/bin/python \
  apps/async-agent/scripts/benchmark_workflow_startup.py \
  --label after --samples 200 --mixed-samples 100 \
  --output /tmp/workflow-startup-after.json

For a before/after comparison, preserve the earlier async-agent app directory before edits and pass --source /absolute/path/to/apps/async-agent. Its normal repository-relative libs and shared imports must remain available. The benchmark binds only loopback port 18009 and records client-send-to-handler and HTTP-response latency. Do not run two copies simultaneously.

Regression checks

With the fixtures running:

cd apps/async-agent
WORKFLOW_INTEGRATION_TESTS=1 .venv/bin/python -m pytest tests -q
cd ../agent-host
.venv/bin/python -m pytest tests -q

Executed results: async-agent 256 passed, 1 skipped; agent-host 70 passed. The eight opt-in integration cases cover publication recovery, idempotency conflicts, completed duplicate starts, checkpoint recovery after mid-node and pre-graph cancellation, renewed leases, stale ownership, duplicate human approval and authenticated tenant boundaries. MCP tests traverse the real SDK dispatch path with an HTTP transport stub; the separate benchmark exercises real HTTP and broker dependencies.

Critical Python lint and focused workflow-panel ESLint pass. A broader Python check also exposed a pre-existing unreachable JSON argument parser in the OpenAI batch adapter; it was repaired and covered with valid/invalid argument regressions. No model calls were needed. No official environment or external assistant client was exercised.

Remove only these fixtures when finished:

docker rm -fv tsf-workflow-package-postgres \
  tsf-workflow-package-redis tsf-workflow-package-rabbitmq

Before production adoption, measure constant-arrival mixed load at the real ingress with declared resources, rates, payloads and tenants. Include TLS, cold starts, failure rate, p99, queue age and required routing/safety calls. Drain older worker versions before relying on duplicate-resume protection; older binaries do not honor the new lease and checkpoint contracts.

See also

• Workflow expansion assessment
• Async-agent service
• Workflow API
• Official Dev/QA debugging

Native safety-profile regression check

The October 13 native-guard package retains the no-policy startup path. A repeat of this benchmark with guards unconfigured measured idle p95 16.357 ms and mixed-load p95 37.007 ms. The preceding run overlapped development activity and recorded substantial mixed-load outliers; both runs and their limits are in the native safety report. Enabled model checks add their own latency and are not included in these results.