Workflow agent expansion and startup latency¶
Status: Draft | Last reviewed: 2026-10-13
Use this assessment to plan richer background agents, request routing, guardrails, and external assistant integrations while targeting a request-to-worker start below 50 ms. Keep the existing LangGraph runtime, strengthen its execution contract, and measure the complete startup path before replacing ingress.
This assessment records the pre-implementation local environment/vanguard
checkout at base commit 8b7ce59f, including uncommitted work present on
2026-10-13. It does not establish what is deployed in official Dev, QA, or
production. The first implementation package now addresses canonical HTTP/MCP
dispatch, durable admission, startup instrumentation, reserved interactive
capacity, dependency warmup and execution leases. Its local HTTP/broker/database
measurements and recovery evidence are in the
startup and recovery runbook.
Descriptions below of the old Redis MCP path, shared semaphore, early running
state and nonrenewing lock are retained as the original assessment.
Recommended direction¶
- Make the existing workflow execution service the canonical entry point for HTTP, MCP, A2A, and channel adapters.
- Define and instrument “agent running” as entry into the first useful node after required routing and authorization. A submitted response or a database
runningflag is insufficient. - Reserve warm worker capacity, reduce per-request setup, and fix dispatch/recovery gaps before selecting another gateway.
- Add a versioned routing/policy contract with deterministic routes, a small local classifier, and explicit escalation to Decisions or a reasoning model.
- Keep LangGraph responsible for parent workflow state. Attach specialist subworkflows or isolated framework workers with typed inputs, bounded capabilities, deadlines, and verified outputs.
- Expose business outcomes through a narrow workflow tool/task API. Package that API differently for each external assistant or messaging channel.
Kong or APISIX may provide useful API governance and overload protection. Neither removes the application's database calls, broker wait, worker occupancy, tool loading, checkpoints, or inference latency.
What already exists¶
| Proposed capability | Current foundation | Extension required |
|---|---|---|
| Stateful orchestration | LangGraph builder, execution/run history, Postgres checkpoints | Immutable execution versions, reliable recovery, explicit ownership of child tasks |
| Intent routing | Rule router and typed OpenAI Decisions assessment | Shared admission-time classifier interface, local adapters, abstention and deadline policies |
| Specialist agents | Bounded LLM tool loop, structured outputs, tool nodes, subworkflows | Per-worker capability grants, verification contracts and isolated execution where needed |
| Team execution | Parallel, foreach and durable fanout facilities | Durable child identities, merge policies, budgets, cancellation and branch failure semantics |
| Human intervention | Wait, human review and tool-input suspension/resumption | Consistent external task/input/approval mapping |
| Guardrails | Tenant authorization, scoped integrations, PII utilities, sandboxed code execution | Input/retrieval/output safety, injection detection and business-action policy hooks |
| External access | Workflow HTTP API, exposed MCP tools, management MCP/A2A, Teams gateway | Unify dispatch and expose durable business workflow tasks |
Source: async-agent service, graph builder, LLM agent node, router node, and external authoring API.
Keep the Pipecat conversation runtime distinct from background workflow execution. Its voice router offers a related intent/delegation pattern, but its generative classifier has a seconds-scale timeout and falls back to the first route on classification failure. Reuse contracts and evaluation data; do not adopt that fallback for privileged workflow actions.
The actual startup path¶
The direct async execution path currently performs:
HTTP ingress
→ bearer-token database lookup and awaited usage-stamp UPDATE
→ workflow lookup and tenant/scope checks
→ pending execution INSERT
→ RabbitMQ publication
→ delivery and worker concurrency slot
→ Redis execution lock
→ execution claim, run INSERT and phase initialization
→ workflow configuration resolution
→ tenant database tools and inline tools loaded
→ lazy checkpoint initialization and graph compilation
→ progress publication
→ graph invocation, checkpoint work and first useful node
The local events-host configuration already proxies directly to async-agent. Workflow ingress does not inherently need an ai-manager/Next.js hop. This local topology is not proof of official deployment topology.
| Finding | Evidence | Implication |
|---|---|---|
| Authentication has two awaited database operations | auth.py | Coalesce usage stamping; preserve revocation and membership checks when optimizing authentication |
| Direct async response means submitted | http.py | It does not prove execution has started; the current endpoint defaults to HTTP 200, while event intake uses 202 |
| Persistence and publication are separate operations | workflow_dispatch.py | A failure between them can leave a pending execution without a delivered job |
| Normal and priority queues share one semaphore | rabbitmq.py | Long runs can occupy all slots; priority alone does not reserve startup capacity |
| Default workflow job concurrency is four per worker process | config.py | Establish capacity under mixed short and long work before increasing limits |
running is written before setup completes |
consumer.py | Current run timestamps overstate readiness |
| Tools load and graphs compile per run | consumer.py | Avoid paying for every tenant tool when a workflow uses only a few |
| Config cache is short-lived; no compiled graph cache is used here | workflow_config_provider.py | Cache immutable plans with correct version, tenant and invalidation boundaries |
| Checkpointer initializes lazily and can fall back to no durability | checkpointer.py | Warm required dependencies before readiness; reject durable work if durable execution is unavailable |
RabbitMQ uses push consumers, not a periodic job polling loop. Redis progress publications and phase/history initialization also add awaited work. Profile them before deciding which can be combined or deferred without weakening audit or recovery guarantees.
Define the 50 ms objective¶
Use p95 below 50 ms from receipt of a complete request at the TrueServe edge to entry into the first useful workflow node, at a declared request rate, payload range and admitted concurrency. Report p99, failures, rejected requests and cold starts separately. Also record first-byte arrival so large request uploads are visible.
Maintain separate measurements for:
- Durable acceptance: the request and dispatch intent have been persisted.
- Runtime entry: the graph starts executing.
- Selected worker start: required intent/safety checks finish and the chosen specialist begins.
- Model request and first token: model connection, scheduling, prefill and generation.
- Completion or first useful result: the user receives useful work.
An empty START node, an HTTP success response, or setting status=running does not satisfy selected-worker start. When routing or guardrails must precede a worker, their latency belongs in that measure. Starting harmless setup concurrently is useful, but label it as setup.
The initial target is plausible for warm, explicit routes with reserved capacity; it remains unproven here. A target covering arbitrary inference-based routing, cold sandboxes, remote models and unbounded queueing is not supportable from the available evidence. External assistant reasoning and network delivery before the request reaches TrueServe require their own user-experience measure.
Measured local graph overhead¶
The reproducible probe uses the real graph builder with synthetic linear passthrough nodes. Python 3.12.1 and LangGraph 1.2.11 ran 20 warmups and 200 measured sequential iterations per size. Raw results retain percentile values.
| Nodes | Compilation p95 | Invocation to first synthetic handler p95 |
|---|---|---|
| 1 | 0.55 ms | 0.58 ms |
| 10 | 2.52 ms | 0.59 ms |
| 50 | 10.27 ms | 0.71 ms |
This excludes ingress, authentication, database, broker, checkpoints, tools, models and telemetry export. Compilation includes the builder's validation path. The first handler is a no-op, not a business worker. These measurements isolate possible graph overhead; they do not establish ingress latency or predict concurrent production performance.
Run from the repository root:
cd apps/async-agent
.venv/bin/python ../../docs/02-architecture/workflow-start-probe.py
Optimize the measured bottleneck¶
- Add distributed timestamps/spans for edge receipt, authentication, durable admission, publication confirmation, broker delivery, concurrency-slot acquisition, configuration/tools ready, graph ready and first useful node. Propagate one request/execution identity; account for cross-host clock skew.
- Establish a dedicated warm interactive worker pool. Separate it from batch and long-running work, bound prefetch, apply tenant fairness and cap admission. Scale on queue age and occupied slots as well as resource use.
- Warm broker, database, Redis, required checkpoint storage and local inference models before readiness. Run checkpoint schema setup as controlled initialization.
- Combine suitable database work and coalesce token usage stamps. Cache immutable workflow plans and tool schemas, loading only reachable tools. Preserve current credential grants and revocation checks.
- Pin workflow version and policy version at admission. Use transactional dispatch intent/outbox with immediate publisher wakeup and recovery, rather than a coarse polling interval. Add tenant-scoped caller idempotency keys.
- Re-run load tests, then compare gateways if the ingress span is material.
Do not globally share today's compiled graphs without an isolation review: node closures can contain mutable tool-resolution state, and tool registries are mutable. Cache keys need tenant, immutable workflow version/configuration hash, tool schema version and policy version. Bind caller identity, credentials, execution state and resources per invocation.
Kong and APISIX¶
Both projects provide gateways with routing, authentication and traffic-policy facilities (Kong, APISIX). Either can help enforce admission policy or remove duplicated edge responsibilities. Gateway selection is primarily an operational and API-management decision until tracing shows otherwise.
| Option | When to consider it | Performance decision |
|---|---|---|
| Keep current ingress | Existing routing and operations are adequate | Baseline; prioritize application and worker costs |
| Kong | Its policy/plugin and management model fits platform requirements | Test the exact deployed edition, configuration and plugin chain |
| APISIX | Its routing/plugin and operational model fits platform requirements | Test the exact topology and production policy chain |
Compare them as alternatives at the same boundary with equivalent TLS, authentication, authorization, request sizes, keep-alive and logging. An additional proxy hop or remote authorization/model plugin is extra work. Gateway throughput benchmarks do not establish workflow startup latency. A gateway cannot reclaim a worker slot occupied by a long execution.
If ingress accounts for only a small part of the measured request-to-start duration, its removal provides only that much possible improvement. Do not select either vendor based on a presumed sub-50-ms agent-start guarantee.
Routing and guardrails¶
Use a shared, versioned assess_request contract. The decision contains the allowed route identifier, route/policy versions, calibrated confidence, safety flags, and an explicit allow, reject, clarify or escalate result. Resolve tenant and identity from trusted authentication, not model output. A classifier chooses among already authorized capabilities; it cannot grant them.
| Mechanism | Appropriate role | Main tradeoff |
|---|---|---|
| Explicit workflow/tool name and deterministic rules | Typed integrations, known events, simple lookups | Smallest execution path; free text still needs interpretation |
| Local embedding or discriminative classifier | Narrow intent taxonomy and out-of-domain detection | Evaluate inference plus tokenization, language coverage and calibration |
| Prompt Guard | Detect prompt attacks and jailbreak patterns | Not a general content or business authorization policy |
| Llama Guard | Content safety classification | A separate task from intent classification and prompt-injection detection |
| Local generative SLM | Ambiguous intent and richer context | Warm residency helps, but prefill/decoding and queueing remain |
| OpenAI Decisions | Typed choices, predicates and scores | Existing adapter; remote latency and fallback behavior must be measured |
| NeMo Guardrails | Compose input, dialog, retrieval, execution and output policies | Each configured check has its own cost and failure behavior |
Start with explicit routes. For free text, try a narrow classifier with per-route thresholds and an out-of-domain/abstain outcome. Escalate ambiguous cases to Decisions or a reasoning model. A safe read-only lookup can bypass generative orchestration while retaining authorization, policy, audit and result validation.
Local Llama-3-8B is an inference-model candidate, not a complete routing or guardrail design. Compare smaller discriminative models before assigning every request to an 8B generative model. Evaluate multilingual requests, adversarial text, long inputs, concurrency and GPU contention, not just a short warmed-up prompt.
Meta's Prompt Guard 2 22M model card reports 19.3 ms per classification for 22M versus 92.4 ms for 86M on an A100 with 512 tokens. These are vendor experiment values, not TrueServe percentiles or end-to-end results. The card also reports accuracy tradeoffs and a 512-token context window; chunking longer inputs adds work. Even a small guard can consume a substantial part of a 50 ms budget.
Llama Guard 3 classifies content safety; its model card describes adversarial limitations. NeMo Guardrails provides programmable policy integration. Neither replaces deterministic checks on who can access a record or execute an ERP transaction.
Extend the existing Decisions integration¶
The current assessment service already calls /v1/decisions, validates named predicate, choice and score answers, handles refusal, and uses tenant-scoped integration credentials. This is a practical extension point.
Current OpenAI documentation describes Decisions as public beta with gpt-6-luna, and reports relative speed against Responses. It does not promise an absolute sub-50-ms request including network time. Calibrate thresholds using application examples.
The adapter currently constructs a new HTTP client per assessment, allows seconds-scale timeouts, and retries some failures after 0.5 and 1 second. An interactive routing policy needs connection reuse, a total deadline, and explicit timeout/escalation behavior. Independent intent and classification questions can share one Decisions call; dependent questions cannot.
Adding a local provider requires extending both ADAPTERS and validation/integration selection. The current code deliberately restricts Decisions to official OpenAI hosts; changing an integration base URL is insufficient. Preserve that credential boundary. Add deterministic evaluation fixtures: the current evaluation sandbox refuses unmocked AI Decision calls.
Use the same assessment policy from ingress and workflow decision nodes. Avoid running the same classification twice when the trusted, versioned admission decision is still applicable.
Apply guards throughout execution¶
Screen user input at admission, retrieved documents before model consumption, tool outputs before reuse, proposed tool actions before execution, and final output before release. Tool and retrieval content can introduce attacks after the initial request passes screening.
For business writes, enforce typed schemas, resource ownership, scoped credentials, value limits, approval requirements and idempotency outside model prompts. Keep read and write capabilities separate. Denied, uncertain, refused and timed-out assessments need explicit routes; none should silently select a privileged default.
Record policy/model version, decision, timing and reason codes with redacted evidence. Run routing and guardrails in shadow evaluation before making them blocking. Assess false routing, false rejection, attack misses, abstention, language slices and successful task completion together.
Complex teams and specialist workers¶
Keep TrueServe responsible for tenant identity, authoring, versioning, execution history, capabilities, evaluations and parent task lifecycle. Keep LangGraph as the default state machine and checkpoint owner. Add a framework adapter only where an evaluated task benefits from it.
| Runtime choice | Recommended use |
|---|---|
| Native LangGraph workflow/subworkflow | Default: branching, cycles, checkpoints, review, retries and explicit state |
| CrewAI worker | A bounded role-based research or synthesis task, returning a typed artifact |
| Microsoft Agent Framework worker | Candidate for Microsoft-oriented or conversational team patterns requiring another runtime |
| AutoGen worker | Compatibility for existing work; avoid making it the default new dependency |
LangGraph explicitly targets durable, stateful orchestration. CrewAI supports both collaborative Crews and event-driven Flows; describing it as sequential-only understates its scope. AutoGen's current maintainers place it in maintenance mode and direct new projects to Microsoft Agent Framework.
Do not let both the parent graph and an embedded framework independently own retries, approvals and authoritative task status. Define which failures the worker retries internally, which failures return to TrueServe, and how cancellation and budget exhaustion propagate.
Use a specialist task envelope containing:
- Parent execution and child task IDs; immutable workflow, worker and policy versions.
- A typed objective/input schema and explicit output schema.
- Necessary context and artifact references, rather than the whole conversation.
- Allowed tools, resource scope, credential grants and egress destinations.
- Deadline, token/cost/iteration budget and maximum parallel children.
- Idempotency key, retry class, cancellation token and trace context.
- A result status such as verified, rejected, needs-input or failed, with evidence and validation findings.
A specialist should often be a deterministic tool or subworkflow. An LLM is useful when interpretation is necessary. Narrow scope improves testability, but does not establish near-100% accuracy: validate SQL/query plans, reconcile writes through readback, test business invariants, and escalate uncertain results.
Example: a servicing request can route to a case-resolution workflow; parallel read-only workers gather account and policy evidence; a reasoning step proposes an action; deterministic policy and optional human review authorize it; a narrowly scoped ERP worker executes an idempotent write; a verifier reads back the result; the parent returns a structured outcome.
Strengthen durability before expanding autonomy¶
The current parallel node executes in-process branches over shallow state copies. Its all result can report overall ok while branch entries contain errors, and any/first waits for first completion, not necessarily first success. Define these semantics explicitly before relying on them for team success. Do not assume each branch is a durable independently scheduled agent.
Use child executions or the existing durable fanout machinery for work that must survive worker loss independently. Define state reducers or explicit result aggregation for concurrent writes; the graph's flat state is not a general shared-memory team coordination system.
The current execution lock has a 300-second TTL without renewal and releases through unconditional deletion. Long-running work needs ownership tokens, renewal, compare-and-delete and fencing or equivalent authoritative database claims. Expiry otherwise permits overlap, and a stale owner can delete a newer owner's lock.
Checkpoint recovery does not make external side effects exactly once. Use idempotent downstream operations, a side-effect journal and reconciliation or compensation where necessary. Fail required durable workflows closed when checkpoints are unavailable.
The existing code-executor sandbox is a useful foundation, but it does not make every native HTTP/SQL tool or third-party agent runtime sandboxed. Isolate untrusted code/framework workers with bounded compute, restricted egress, scoped short-lived credentials and artifact limits. Retain warm pools where startup latency matters.
One execution contract across external clients¶
Introduce a business workflow facade supporting submit, get status, cancel, provide input, approve an authorized action, and fetch artifacts. Include idempotency, immutable versions, caller identity and correlation IDs. Return a durable receipt promptly; allow a bounded synchronous result for short lookups.
Expose a small set of understandable outcome tools, such as resolve_service_case, instead of the internal worker catalog. Carry structured results and opaque artifact references. Recheck tenant/resource authorization on status, cancellation, input and artifact retrieval as well as submission.
The existing workflow MCP exposure discovers tenant workflows and builds input schemas. However, the normal catalog trigger creates run records and writes to Redis async-agent:jobs; the inspected worker uses RabbitMQ. No reader of that Redis stream was found in tracked application source. Only the evaluation branch here calls the canonical HTTP execute endpoint.
This is a static integration mismatch, not a verified deployed outage. Test and replace that dispatch path before expanding MCP adoption. The same path also defaults to waiting for results, with a 120-second limit and two-second status polling; long business tasks need a receipt-and-status contract.
Existing management MCP/A2A serves the authoring/control plane. Reuse its OAuth resource/audience, scope and tenant-isolation patterns, but provide business execution scopes and tools separately. Management A2A advertises 1.0 while the conversation A2A server advertises 0.3.0; negotiate and test each actual interface rather than assuming one shared task implementation.
| Client or protocol | Native integration approach | Remaining work |
|---|---|---|
| Claude Cowork | Plugin with skills/commands and MCP tool connections | Package business outcomes, OAuth onboarding, task-status and input/approval UX |
| ChatGPT | Plugin backed by MCP, optional UI; supported Work clients can subscribe to MCP Events | Execution tools, auth, artifacts and client-specific event/protocol support |
| Gemini CLI / ADK agents | MCP tools; A2A where the deployed agent runtime supports it | Choose the specific Gemini product and adapter; consumer Gemini support is not implied |
| Microsoft Teams | Extend the existing Bot gateway; optionally connect through Copilot Studio MCP | Receipt/progress messages, Adaptive Cards, durable input/approval and identity mapping |
| Slack | Events/Bolt-style app adapter invoking the same facade | Signature verification, quick acknowledgment, deduplication and thread updates |
| Other agent services | A2A workflow agent with task/context IDs and artifacts | Task lifecycle, cancellation, input-required and negotiated streaming/push support |
Anthropic's Cowork plugin examples combine skills, commands and MCP connections. OpenAI's current plugin documentation covers MCP-backed plugins and optional interfaces. Its MCP Events integration requires protocol 2026-07-28 and signed webhook subscriptions in supported ChatGPT Work clients; the repository's management MCP 2025-11-25 interface does not automatically implement that.
Gemini CLI and Google ADK document MCP integration. Copilot Studio documents MCP tools/resources. The current Teams gateway starts with personal chat and final replies. Slack requires event acknowledgment within three seconds; acknowledge after durable intake and deliver work asynchronously.
Treat MCP and A2A as complementary interfaces. MCP lets a host invoke business capabilities; A2A represents delegated agent work with task state and artifacts. The MCP 2025-11-25 task extension is experimental, so negotiate support and retain ordinary submit/status tools. A2A supplies task, context, artifact and lifecycle concepts. Neither protocol alone creates the required UI, channel identity mapping or authorization.
Client consent dialogs and plugin instructions help users understand actions; the server still enforces the actual capability grants. Bind external identities to tenant/user accounts, avoid shared unrestricted service credentials, and make approvals specific to the proposed action and current state.
Delivery and verification sequence¶
| Stage | Concrete deliverable | Exit evidence |
|---|---|---|
| 1. Baseline and convergence | Canonical dispatch, trace milestones, MCP integration coverage | HTTP and MCP reach the same durable execution with truthful states |
| 2. Startup and recovery | Warm interactive pool, reduced setup, version pinning, reliable dispatch and locks | Measured startup distribution; worker-crash and duplicate-delivery recovery |
| 3. Routing and safety | Shared assessor, local classifier experiment, Decisions fallback, policy hooks | Labeled routing/safety evaluation with latency and abstention results |
| 4. Specialist teams | Typed child contract, native durable team, one optional framework adapter | Better task outcomes than the native baseline within declared budgets |
| 5. Client packages | Cowork/ChatGPT MCP packages, workflow A2A, Teams and Slack adapters | End-to-end submit, progress, approval, cancellation and artifact access per client |
| 6. Gateway decision | Equivalent-policy comparison if ingress remains material | Demonstrated latency or operational benefit under representative load |
Benchmark warm and cold workers, fresh and reused connections, cache hits/expiry, one/fifty-node graphs, small/large tool catalogs, short/long inputs and several tenants. Mix long jobs with bursts of interactive work. Measure local classifier contention alongside worker and database saturation.
Use a constant-arrival-rate load generator so slow responses do not silently reduce offered load. Record request rate, concurrency, payload, environment, image/revision, resource allocation, dependency placement and error/rejection rates. Compare p50/p95/p99 and queue age; summing individual stage p95 values does not produce an end-to-end p95.
Inject publisher failure, worker loss, expired leases, checkpoint unavailability, duplicate submissions, ambiguous intent, guard timeouts and revoked credentials. Confirm retries do not duplicate side effects and that task/approval/artifact access remains tenant-scoped.
For official Dev or QA, follow the official environment debugging guide, use the configured access path and target Grafana/Loki, and correlate with the deployed revision. Local probe results and the city checkout cannot substitute for those measurements.