Procedure supervision + SLM extraction¶

Status: Built (PR1–PR5, 2026-09-03; deviations recorded under Phased delivery, live-SLM eval fixtures still pending) | Last reviewed: 2026-09-24

GPT Live's backend receives the rendered procedure; its voice layer receives the backend's answer. Current-step briefs are simulation diagnostics, not extra voice prompts. Substantive Live speech also triggers evidence checks while procedure work remains, including record claims without words such as “completed.” Corrections retain the corroboration, scope, expiration and per-step budget checks. A responding correction uses Live's instruction channel; ordinary brief updates are not sent to the voice. See GPT Live for transport and playback limits. Live also relays external instructions to the monitor after backend delivery. Its ledger retains up to twelve application messages separately from dialogue and tool facts, so a delivered supervisor question or decision survives a long caller exchange. A new message clears queued advice for reassessment. Receipt alone never marks a procedure step complete, and a requested action still needs its actual result. The capability-attestation flag is distinct from the advisor message channel; caller quotes never become application evidence. For Live, an active conditional procedure with remaining applicable steps takes priority over the ordinary flow when selecting guidance. For example, a pending copay review takes precedence over the normal payment question. Once that branch completes, the ordinary flow resumes; completion and applicability checks remain authoritative.

A tool completion can specify result_path, for example $.data.entry[0].resource.id, in addition to its success or result_statuses criterion. The selected value must exist and be nonempty. This prevents an HTTP-successful empty search from completing a patient lookup. Keep kind: "tool" so the original tool prerequisites remain enforced.

When a successful response lacks that required value, the monitor leaves the step incomplete and returns its authored recovery guidance. It does not advance, fabricate a record, or execute a retry. The guidance must specify the permitted retry limit and no-match outcome. The result includes its empty-result number for the current topic visit when a tool-call ID is available. A repeated callback keeps the same number; changing topics resets this observation history. Verified identity remains explicitly verified in that guidance. These observations help the model apply the authored retry policy; they do not themselves enforce a retry limit.

Gated tool results can also carry the next eligible step's authored guidance. The runtime projects the current result into a temporary checklist snapshot to choose that step; it does not mark completion or execute an action. The ordinary evidence relay still owns completion. Consent, waiting, applicability and prerequisite rules remain in force, and a disabled monitor or zero correction budget suppresses this guidance. Guidance is computed after the idempotency cache so a replay cannot restore an obsolete next step.

Keep completion criteria as complete as the authored instructions. A read-back check that only names the medication, reaction and onset cannot establish that the agent also collected required history and follow-up preferences. Model collection and confirmation as separate steps when both are prerequisites for a write; include every required category in the collection assessment, and preserve explicit unknowns rather than treating them as negative answers.

The design for the platform's next reliability layer: procedure supervision — an author-editable, AI-derivable checklist attached to an agentic subagent, enforced by deterministic tool gates on irreversible actions and a trailing SLM monitor that checks the live model's behaviour against the checklist every turn — plus a structured-output SLM entity-extraction layer that pulls typed variables out of every caller utterance. Both ride one new shared substrate (a logic_slm auxiliary-service slot + a shared structured-output client), both are fully configurable (models, prompts, endpoints — including self-hosted vLLM), and both are metered into the cost ledger and forecast.

It builds directly on Agent Logic (agent-logic.md), the guided scripted subagents (scripted-subagents.md), and the aux-service resolution ladder those introduced. Read those first; this document assumes their vocabulary.

Why this layer exists¶

Live testing keeps producing one failure class that none of the existing layers catches: an agentic subagent on a live voice call skips steps, acts before asking, and hallucinates progress in multi-step procedures. The 3032 demo-path sessions are the worked example — the agent rendered appointment cards before any lookup succeeded, promised lookups it never performed, and repeatedly had to be herded back onto the flow by prompt hardening that the next model update can silently undo.

The existing layers each miss it for a structural reason:

Rules (config.logic.rules) are stateless per-trigger over a flat variable blackboard. There is no way to express "step 3 must not happen before step 2", no counters, no per-rule memory. Expressing a five-step procedure means inventing tracking variables, trusting the model to set_variables them, and hand-writing gate expressions — which is exactly the discipline the model keeps breaking.
Response guards check individual utterances against patterns/expressions/judges, but carry no notion of where the conversation is in a procedure.
Tool gates (available_when) are the right enforcement primitive but must be authored per tool against variables the author has to maintain by hand.
Guided scripted subagents are the only real sequencer — and they trade away the LLM entirely (deep pause, step machine takes the floor). Perfect for rigid flows; wrong for procedures that need conversational flexibility within a required structure.

The gap is a sequence/progress primitive for a free-running LLM, giving the platform a three-tier reliability story:

Tier	Who has the floor	Structure
Prompt-only (today)	LLM	none — prose instructions
Monitored procedure (this design)	LLM	checklist watched + gated at runtime
Guided script (built)	step machine	fully deterministic

What the market does (survey, 2026-09)¶

Researched for this design; full latency/cost figures in Costing and latency.

Sierra is the only direct precedent for the supervision half: a published four-layer runtime — grounded content, natural-language rules, supervisor models auditing conversations in real time that can block, rewrite, or nudge responses, and deterministic guards that "can't be talked out of doing their job" — running in production voice (sierra.ai/blog/defense-in-depth-in-the-age-of-agents). This validates both halves of the dual-enforcement decision.
Salesforce AgentScript enforces structurally — before_reasoning/after_reasoning deterministic hooks, available when gating that removes an action from the model's view entirely, typed mutable/linked state — but has no runtime critic: nothing checks utterances against the script (github.com/salesforce/agentscript). Our Agent Logic already covers that structural tier; the critic is the open ground.
Parlant (open source) is the best per-turn mechanism reference: a guideline-matching engine evaluates each guideline's condition against conversation state before every response, loads only matching guidelines into context, and tracks action-applied state so a satisfied guideline never re-fires (parlant.io/docs). We adopt the action-applied idea as correction dedup.
NVIDIA NeMo Guardrails / IORails runs its 8B-model checker rails in parallel and speculatively — input rails execute concurrently with main-LLM generation, verdicts are cached — which is the latency playbook a voice platform needs (docs.nvidia.com/nemo/guardrails).
OpenAI Agents SDK guardrails run in parallel with the agent and abort on tripwire — explicitly "fast/cheap model screens for expensive model" (openai.github.io/openai-agents-python/guardrails).
Process-supervision literature ("Let's Verify Step by Step", arxiv.org/abs/2305.20050) supports the premise: verifying each step against a plan outperforms outcome-only checking. The monitor is the runtime analog of a process reward model.
Voice-infra competitors (Retell conversation flows, Vapi workflows, Bland pathways, ElevenLabs Agents, Dialogflow CX playbooks) all reduce to graph structure + variable extraction + offline evals. None ships a live critic. Two candid data points: Vapi documents that extraction is "best-effort, not deterministic" and that models struggle to hold node instructions plus transition conditions; Bland runs variable extraction before dialogue generation and admits the "slight latency" cost. Schema-constrained extraction on a fast SLM is precisely the fix for the failure mode they describe.
Intercom Fin Procedures / Decagon AOPs author procedures in natural language with hard gates only on actions, leaning on large-scale simulation testing rather than runtime checking — worth stealing later for regression testing, orthogonal to this design.

Two conclusions drive the architecture: (1) a live supervision layer is differentiator territory no voice vendor occupies, with Sierra proving it fits voice; (2) the latency constraint is TTFT, not decode or cost — so the design must be deterministic-first, speculative, and cache-friendly rather than "SLM on every hop".

Architecture¶

One new substrate, two features on top:

                        ┌── logic_slm aux slot (NEW)
                        │   ladder: stage config.aux overlay → stack manifest → tenant
                        │   aux_services → built-in table; self-hosted via base_url;
                        │   every call metered as its OWN provider/model ledger rows
                        ▼
        ┌── slm_client.py (NEW shared 4-provider structured-output client)
        │        ▲                     ▲                        ▲
 entity_extractor         procedure_monitor        unified evaluate_llm_judge
 (per caller turn,        (per agent turn,         (tool-gate + response-guard
  writes variables)        verdict vs checklist)    judges — today env-key-only)
        ▲                     ▲
        └──── ConversationAPIPublisher relays ────┘
              (existing chokepoint: final user transcripts, assistant turns,
               tool results — identical on cascaded and S2S/Nova pipelines)

Three principles, each earned from an incident class:

One blackboard. Procedure state is ordinary SessionVariableStore variables (proc_current_step, proc_<step_id>_done). Everything that already reads the store — #when sections, rule conditions, tool gates, the InstructionRefresher, guided fast-forward — sees procedure state with zero new plumbing, and the state survives topic transitions the same way every other variable does.
Deterministic first. Steps whose completion is a variable condition or a tool success complete with no SLM call; irreversible-action blocking compiles to ordinary tool gates evaluated in-process. The SLM is consulted only for explicitly fuzzy judgments — which is what keeps adherence_checks_per_turn well below 1 and the hot path at zero network calls.
Nothing new takes the floor. The monitor corrects through the already-hardened injection channels; taking over the conversation remains the guided script's job.

The procedure artifact¶

Lives at topics[].procedure on an llm-engine topic and identically at stage.config.procedure for the main prompt. Stage-scoped config is collab pass-through, so no shape-converter (MISC_ROOT_FIELDS) change is needed — the config.variables lesson does not repeat here.

"procedure": {
  "enabled": true,
  "steps": [
    { "id": "verify_identity",
      "title": "Verify the caller",
      "guidance": "Ask for name and date of birth, then call authenticate_customer.",
      "requires": [],
      "complete_when": { "kind": "variable", "expr": "customer_auth_status == \"verified\"" } },

    { "id": "confirm_amount",
      "requires": ["verify_identity"],
      "expects": ["refund_amount"],                 // feeds the extraction schema
      "complete_when": { "kind": "judge",
        "prompt": "The agent stated the refund amount aloud and the caller agreed to it." } },

    { "id": "take_payment",
      "requires": ["confirm_amount"],
      "complete_when": { "kind": "tool", "tool": "collect_payment" },
      "irreversible": true,
      "irreversible_fail_mode": "closed" }          // closed (default) | open
  ],
  "monitor": { "enabled": true, "max_corrections_per_step": 1, "severity_default": "nudge" }
}

complete_when.kind ∈ variable | tool | judge. variable reuses the shared trueserve_logic expression grammar (identical to available_when — one grammar, both mirrors); tool means the named registered tool returned success; judge is a plain-English criterion the SLM evaluates.
requires is a DAG, validated acyclic at publish — procedures are ordered where order matters and parallel where it doesn't. The monitor flags order violations; only irreversible steps block.
Arming and state. When the topic is entered (or at session start for a stage-level procedure), the compiler seeds proc_current_step and one proc_<id>_done: false per step through the store (setdefault — inbound CCaaS-seeded values keep winning, the existing rule). Every completion is a store write, which fires variable_changed rules, guided fast-forward, #when re-render, and gate visibility. __return__ or a transfer disarms the monitor and clears that topic's proc_* names.
Coexistence. A scripted (engine: "scripted") topic ignores procedure — the StepMachine is its procedure — and publish warns it is dead config.
AI derivation (authoring-time, never call-time). A new ai-manager route app/api/agents/[agentId]/derive-procedure/route.ts sends the topic prompt + declared variables + the stage's tool list to a reasoning model behind the existing ai_generation triple gate, returning procedure JSON validated by a zod schema (lib/procedure-schema.ts, mirrored into the generated config schema). Results land as editable step cards, never auto-published. The same capability is exposed as a generation-MCP tool so Generate-with-AI and the editor Assistant can author procedures. Call-time derivation was considered and rejected: it adds session-start latency and cost, and produces plan drift between calls of the same agent.

The runtime monitor¶

No new pipeline processor. The ConversationAPIPublisher is already the one chokepoint that sees final user transcripts, assistant text, bot start/stop, and every tool-call publish on both cascaded and S2S pipelines — and it already relays lifecycle to the idle handler, the rules processor, and the refresher. It gains one more relay, notify_procedure_monitor(event) (user-final / assistant-turn-complete / tool-result), feeding pure logic in a new lib/pipecat/logic/procedure_monitor.py. This inherits the rules processor's linear-pipeline-only constraint, with the same publish + preflight warning.

Deterministic-first loop:

On tool results and variable writes (hooks exist: register_tool_result_handler, store subscription), variable/tool steps are marked done and proc_current_step advances — zero cost, zero latency, no model involved.
On caller-final or assistant-turn-complete, the SLM runs only if the current step is judge-kind, or drift heuristics fire (the assistant used completion-claim language while a required step is open — the "hallucinated progress" signature).

Verdict schema (strict structured output — the agent-assist intersection rules: additionalProperties: false, every property required, absence expressed as null):

{ "reasoning": "brief rationale | null",
  "current_step": "step_id | null",
  "satisfied": ["step_id"],
  "assessments": [
    { "step": "step_id", "status": "satisfied | unsatisfied | unknown",
      "evidence_ids": ["e17"], "missing": "missing requirement | null" }
  ],
  "violation": { "kind": "none | skipped_step | hallucination | off_procedure | premature_action",
                 "step": "step_id | null", "detail": "…" },
  "not_applicable": ["step_id"],
  "suggested_correction": "string | null" }

A verdict naming an undeclared step id is discarded (the consult_agent_id lesson: models invent plausible ids). Every monitor fault fails open — supervision silence must never degrade a call the way the thing it supervises can.

The assessor receives at most three eligible judge requirements per call. Other steps remain visible as state, without inviting the model to complete deterministic steps or future requirements. A satisfied entry must also have a satisfied assessment citing actual evidence IDs from the request. Invented references, missing evidence and unknown cannot complete a step. Prerequisites and applicability are checked again before applying completion.

procedure_evidence.py retains 40 utterances and 32 compact current tool/variable observations independently of the ten-entry transient transcript window. Each entry has an evidence ID and a source label. Later reads of the same tool and transaction replace its earlier observation; different transaction IDs remain distinct. Tool evidence retains allowlisted outcome fields and requested business identifiers, excluding capture directives, credential fields and whole patient records. Variable observations are not represented as authenticated external outcomes. This is bounded session memory, not an unlimited history or a durable cross-session evidence store.

The structured request allows 512 output tokens. The prompt distinguishes accepted requests from completed business outcomes, scopes yes/no answers to their questions, treats contradictions as unknown and recommends at most one permitted action or wait. The extractor also distinguishes caller corrections from claims of verification, payment or supervisor approval.

Generic tool completion excludes authorization_required, requires_action, capture_started and setup_required, as well as pending/error statuses and an explicit success: false. For a tool step whose purpose is to start an operation, retain kind: "tool" and explicitly set result_statuses: ["pending", "capture_started", "succeeded"]. This retains its irreversible-action prerequisite gate while recognizing an accepted request. Error/refusal results still cannot complete it. Changing that step to kind: "variable" would remove the tool association used to compile its gate. Keep successful business completion separate. For example, ask_copay can complete after an accepted payment request, while record_payment still requires the trusted payment receipt contract. Use a component-qualified tool criterion for show_appointment, so another successful A2UI render does not satisfy the AppointmentSummary step.

Latency. The check is speculative: an S2S provider streams a full turn's text seconds before playout finishes (measured: 12+ s of Nova audio pushed in ~2 s), so the check runs during playout — the InstructionRefresher's speculative-pass precedent — and the verdict is ready by turn end. Verdicts are cached on (procedure_state_hash, transcript tail, evidence revision) (the NeMo IORails pattern), and the debounce/in-progress guard is copied from SuggestionLLMProcessor.

Correction ladder — severity chosen per verdict, capped by max_corrections_per_step, with Parlant-style correction-applied dedup (an identical correction is never re-issued):

Severity	Mechanism	Provider notes
note (default) or non-severe nudge	Queue a deduplicated correction after corroboration	Nova attaches it to the next tool result's `instructions`; unclaimed notes expire after 60 seconds. Other providers receive a `run_llm=False` note at the caller's next final transcript.
severe irreversible-action violation	Preempt speech and deliver an immediate corrective instruction through the rules processor	Best effort; a failed interruption does not swallow the correction.

max_corrections_per_step: 0 observes and records violations without delivering corrections. It does not disable judgments or deterministic gates. A missing limit defaults to one. This enables comparison of assessment alone against assessment with corrective feedback.

Findings are checked against live state before consuming a correction budget. Completed, inactive and inapplicable steps cannot produce a new correction. Queued notes retain their owning procedure and step; delivery checks state again, including inside a scheduled severe-correction task. Leaving a topic invalidates pending notes and assessments from that visit, even if the agent re-enters it before the assessment returns. Budgets and severity come from the step's owning procedure, not the first active checklist.

Before judging whether a procedure is active, run collect_procedures against the exact executed stage config. Inline {{#procedure name}} / {{#step id}} blocks define real procedures even when procedure.steps is empty or absent. Plain numbered instructions do not. Replacing blocks with prose during a prompt rewrite removes their runtime supervision; preserve and compare the parsed procedure names and step IDs before and after an edit. Check logic_slm_consumers, resolved auxiliary model and live checklist/verdict events as separate evidence that assessment runs.

Blocking gates — the deterministic hot path¶

lib/pipecat/logic/procedure_compiler.py, invoked from tool_wiring.py phase 1, compiles each irreversible tool step's prerequisites into the separate config.tool_logic[T].procedure_gate entry. It runs before the authored available_when gate and input healing; it does not overwrite the authored gate. The veto is the standard informative tool RESULT, and it enumerates the open prerequisite steps so the model's retry is informed rather than a guess.

Deterministic prerequisites make zero network calls. When only judge prerequisites remain, tool_gates.py calls monitor.settle_steps under an eight-second total bound. It reads completed state first and waits for an existing background assessment instead of treating a busy monitor as a negative verdict. If that assessment satisfies the prerequisites, no duplicate model request is needed. Otherwise the monitor checks current evidence within the remaining timeout. Cancelling the gate's waiter does not cancel the independent background assessment. Per-step irreversible_fail_mode decides the timeout behaviour: closed (the default for irreversible steps) vetoes; open allows and logs. Nothing in the gate may raise — the provider-callback invariant.

Nova recovery guidance: a procedure veto remembers the blocked tool and prerequisite metadata, without retaining arguments or replaying the call. Subsequent tool results can carry one compact instructions note naming the next ready prerequisite and its authored guidance. When prerequisites complete, the note tells the agent it can resume the blocked action subject to its instructions, consent and ordinary tool checks. Completed actions, unchanged recovery states and exited procedures produce no repeated note. Monitoring off or a zero correction budget suppresses this guidance. It requires no extra LLM call and creates no new conversational turn. This supplements the existing Nova result channel; it does not add a forced-response recovery path for other providers. A guidance failure never relaxes the veto.

The entity-extraction layer¶

A new lib/pipecat/logic/entity_extractor.py, fed each final caller transcript from the same publisher relay. It is trailing — off the voice path entirely.

Eligibility is declared: config.variables[].extraction: true, optionally scoped per topic (default: the current topic's step expects union of the agent-wide eligible set). Publish validation requires a description on every eligible variable — it becomes the schema property description, the same LLM-legibility discipline set_variables enforces.
One structured call per caller turn: a single object schema over the eligible variables, every property typed ["<type>", "null"], with "not stated → null" instructed in-prompt and the schema echoed in-prompt (small self-hosted models need prompt/schema alignment; constrained decoding guarantees shape, not correctness). The OpenAI-compatible path sends response_format: {type: "json_schema", strict: true} — vLLM's xgrammar guided decoding makes any 4–8B model emit guaranteed-valid JSON; the Bedrock path reuses the Converse call shape already proven in slm_matcher.py.
Short answers and corrections retain caller context: the latest utterance is paired with at most six previous caller utterances, capped at 500 characters each. This allows a reply such as a four-digit birth year to complete a previously stated month and day. Intermediate turns remain in that context when the worker coalesces pending extraction. Assistant statements and application messages are not fact sources. The prompt requires null for fields the latest answer does not add, complete or correct; extraction is still a model assessment, not deterministic verification.
Write-through: non-null values go through SessionVariableStore.set (linked variables refuse, type coercion applies, sensitive names are ineligible at validation). Null never clobbers an existing value. Because writes ride the store, they automatically fast-forward waiting guided steps and fire variable_changed rules — extraction-before-generation with no new plumbing.
set_variables coexists: extraction owns caller-stated facts (structured, per-turn); set_variables remains the model's channel for derived values. Its auto-injected guidance gains one sentence noting auto-extracted variables need not be set for caller-stated values. The guided single-slot matcher is unchanged; routing guided capture.kind: entity through the extractor is a follow-up.
Skips empty/non-lexical utterances, synthetic turns, and active secure capture (whose gates already starve transcripts). Fails open. Metered.

Configuration — the `logic_slm` slot¶

One new aux slot serves all three consumers (adherence checks, extraction, and the tool-gate/response-guard LLM judges): one ladder entry to operate and self-host, with per-call-site model overrides still possible through the sparse stage overlay (config.aux.logic_slm.config.model).

shared/service-definitions/aux-services.json gains the logic_slm table (defaults mirroring script_slm: Nova Micro / gpt-4o-mini / Gemini Flash-Lite keyed by owner provider); every stack manifest gains an aux.logic_slm entry — including stacks/self-hosted.json, where the vLLM/Ollama base_url override makes self-hosting a pure configuration choice, per the v1 decision (no first-party serving infrastructure).
aux_resolution.py: AUX_SLOTS += ("logic_slm",), stage id __logic_slm, resolved once at build into runtime_context["_logic_slm_stage"] when any consumer is enabled; drift rows in aux_reference_rows() + tests/test_aux_resolution.py.
Shared client — new lib/pipecat/logic/slm_client.py: the four-provider structured-output path (OpenAI-compatible with json_schema + api_base_url, Google, Vertex, Bedrock) extracted from suggestion_llm_processor.py and the ledger write from slm_matcher.py, as one structured_completion(stage, schema, system, user, *, timeout, scope). Both existing modules delegate to it (move-only discipline).
Judge unification. evaluate_llm_judge (libs/logic-expressions/trueserve_logic/llm_rule.py) is half-built today: OpenAI env-key-only, hardcoded default model, unmetered. Its call sites (tool-gate llm conditions, response-guard llm checks) move onto the client/ladder, gaining tenant configuration, base_url support, and cost rows at once; the env-key path survives only as a final fallback. This design must not inherit that seam.
Schema/UI: pipeline-types.ts types, generate-config-schemas.ts (logic_slm in $defs/AuxSlotOverride + the procedure branch) and regeneration; the stage inspector's Auxiliary services section and the tenant Settings → Aux Services page gain the slot row (the tenant aux_services plumbing is slot-agnostic).
Flags: NEXT_PUBLIC_FEATURE_PROCEDURES gates the authoring surface; runtime engages purely on config presence with a PROCEDURE_MONITOR_ENABLED env kill-switch; derivation rides the existing ai_generation triple gate.

Costing and latency¶

Observed side. Every SLM call goes through events_from_usage/persist_usage_events (libs/costing/usage.py) as the aux model's own provider/model rows — never folded into the primary provider's billing product — with stage_id="__logic_slm" and scope_key ∈ {adherence, extraction, llm_judge}. This is the __script_slm precedent, fire-and-forget off the latency path.

Forecast side (cost-estimation.md):

New assumptions in assumptions.ts: extraction_{input,output}_tokens_per_turn (≈350/40), adherence_checks_per_turn (low/mid/high ≈ 0.3/0.5/0.8 — deterministic-first keeps it below 1), adherence_{input,output}_tokens_per_check (≈600/60). This PR also closes a pre-existing gap: the guided SLM matcher has no forecast assumption today (observed-only) — guided_slm_* knobs land beside these.
A new auxSlmMeters() builder in forecast-agent.ts resolves the ladder TS-side (honoring the stage overlay and stack manifest) and emits input_text_token/output_text_token meters under the aux model's own billing selector. Coverage gaps surface as partial, never silently zero; a self-hosted endpoint maps to an explicit zero-price book entry with an assumption note, so "free because we host it" is a visible statement, not an omission.

The latency budget (survey figures, 2026-09): fast tiers run a 10–30-token structured verdict in ~100–300 ms of decode (Nova Micro 267 t/s at $0.04/$0.14 per 1M; Llama-3.1-8B on Groq 654 t/s; Gemini Flash-Lite $0.10/$0.40) — but hosted TTFT is 0.7–0.9 s, which is why the design is speculative + cached rather than inline: the only synchronous SLM wait anywhere is the procedure_judge tool gate, bounded at 8.0 s and skipped whenever the speculative verdict already answered. Per-call cost at these rates is fractions of a cent even checking every turn; cost is a configuration concern (visible in the forecast card), not a design constraint.

Authoring UX¶

The frank assessment behind this section: Agent Logic's power features leak expressions through every "no-code" seam (topic entry_when, $. capture paths, guard regexes), the fail-open/closed matrix is invisible, and procedure semantics don't exist as an authoring concept. This layer is the chance to fix that for the 80% case.

ProcedureEditor.tsx (components/service-config/sections/) — step cards in the GuidedStepsEditor visual language: title, guidance text, and completion as a three-way segmented picker: a variable is set (WhenConditionBuilder — flat AND/OR rows, no raw expressions), a tool succeeds (picker over buildAgentToolsList), AI judge (one plain-English sentence — the LogicRulesSection AI-judge pattern). The Irreversible toggle renders a lock chip that names which tools become blocked; drag-to-reorder maintains the requires chain with a per-step "any order" opt-out; fail mode is plain words ("If the checker can't answer in time: allow / block").
Generate from prompt — one button on the procedure panel calling the derive route; output lands as editable cards.
Preview live checklist — the monitor emits pipeline_event rows of kind procedure_status ({step_id, state, violation?}) through the publisher; PhoneCallTest renders a live ticking checklist beside the transcript, and corrections appear as event-log entries (the reasoning-channel discipline — never in a spoken bubble). Watching the checklist tick during a rehearsal call is the authoring feedback loop this whole layer has been missing.
Publish validation (agent-logic-validation.ts) — errors: requires cycle, unknown step id, an irreversible gate naming a tool not on the stage, a judge step with the monitor disabled, an extraction-eligible variable without a description, writes to linked variables. Warnings: a procedure on a scripted topic (dead config), no resolvable logic_slm while judge/extraction features are used.
Preflight — _apply_procedure_gate mirroring _apply_guided_gate: resolve the logic_slm lane without connecting; unresolvable lane + any closed-mode judge = error, otherwise warning. Preflight mirrors, never exceeds, runtime strictness.

Phased delivery¶

PR	Content	Independently ships as
PR1 — Substrate	`logic_slm` slot (JSON + stacks + resolver + drift tests), `slm_client.py` (move-only extraction, both donors delegate), judge unification + metering	Tenant-configurable, metered, self-hostable LLM judges for every existing gate/guard
PR2 — Extraction (dep PR1)	`entity_extractor.py`, publisher relay, variable eligibility config + schema envelope, forecast assumptions (incl. the guided-SLM gap), eval fixtures	Deterministic caller-fact capture on every turn
PR3 — Artifact + gates (no runtime SLM)	Procedure schema (TS + zod + envelope regen), `procedure_compiler.py` (state seeding + phase-1 gate merge), publish validation + preflight gate	Deterministic step-gated irreversible tools with author-visible state
PR4 — Monitor (dep PR1 + PR3)	`procedure_monitor.py`, speculative check + verdict cache, correction ladder + `procedure_judge` gate kind, checklist events, adherence assumptions	The supervision layer
PR5 — Authoring (dep PR3; checklist UI dep PR4)	ProcedureEditor, derive-from-prompt route + generation-MCP tool, Preview checklist, userguide + this doc updated to Built markers	The non-technical authoring surface

Built (2026-09-03) — deviations from the design above¶

All five PRs are implemented. Where the build deviated, the code is right and the sections above should be read with these corrections:

The compiled gate is a SEPARATE tool_logic key (procedure_gate), evaluated BEFORE the authored available_when, never AND-merged into it — an authored llm-kind gate has no expression to merge into, and a separate key means the authored gate survives verbatim (procedure_compiler.py PROCEDURE_GATE_KEY; step 0 in tool_gates.py's wrapper). Judge-kind prerequisites settle via monitor.settle_steps under an 8 s bound (_settle_judge_prerequisites) instead of a new procedure_judge condition kind.
The unified judge transports a strict JSON {"allow": bool} schema on the ladder path (llm_judge.evaluate_judge), falling back to the legacy env-key evaluate_llm_judge ALLOW/DENY word protocol only when no logic_slm stage resolved.
Checklist status rides the existing chat-event channel, not a new pipeline_event kind: _publish_chat_event(kind="procedure_status", data={checklist}) → persisted conversation_events rows + live SSE CHAT_EVENT — which also settles the open question below about post-call persistence (yes, for free). Preview renders it via ProcedureChecklistCard (mounted by PhoneCallTest only when configHasProcedures).
Derive-from-prompt is POST /api/procedures/derive (ManageAgents + the ai_generation triple gate), a direct reasoning-model call through the runner integration's auth context (lib/procedure-derive.ts, riding callAnthropicViaAuthContext), zod-validated into editable cards. No separate generation-MCP tool was added on purpose: the sandboxed CLI is a reasoning model and authors procedure blocks directly through the existing draft-save tools, with validateAgentConfig's procedure checks as the gate.
The live-SLM eval fixtures (extraction/procedure_eval_cases.json under SLM_EVAL=1) are the one Verification bullet not yet built.

Inline authoring rework (2026-09-03, second pass — user review)¶

The separate step-card section was replaced the same day by inline authoring in the prompt document ("more intuitive and centralized"), plus per-step IF/THEN gating:

Steps are {{#step id}} … {{/step}} blocks in the prompt (stage template.inline; a topic's instruction_addendum). The block BODY is the step's guidance — typed inline, and it now actually SHIPS to the model in place (the old config-only guidance never reached the prompt at all). Blocks are authoritative for step existence, ORDER (the default requires chain is document order), and guidance; config.procedure.steps[] becomes a sparse per-id SETTINGS SIDECAR (done-when, requires override — [] = any order, irreversible + fail mode, applies_when). A settings-less block defaults to a judge over its own body, so authors get a working monitored checklist by just wrapping their prose. The original config-only steps array still works unchanged when the prompt has no blocks.
One scanner, two languages: trueserve_logic.procedure_blocks + the TS mirror apps/ai-manager/lib/logic-expressions/procedure-blocks.ts, parity pinned by libs/logic-expressions/fixtures/procedure-block-cases.json (fixture-first rule). Markers strip at render inside BOTH shared entry points (render_when_blocks and execute_prompt_program — Chevron would read {{#step}} as a falsy section and DELETE the body); has_when_directives also gates on step markers so every render site flows.
IF/THEN (applies_when): {kind: 'code', expr} evaluates live against session variables (fail open — a broken condition never deletes a step); {kind: 'llm', prompt} is assessed by the monitor beside the adherence verdict (not_applicable lane, sanitized to steps that actually carry an llm condition) and sticks as proc_<id>_na. A non-applicable prerequisite never blocks an irreversible gate, a gate whose OWN step doesn't apply doesn't bind, and the checklist renders such steps as skipped/dimmed.
Editor: STEP/END STEP pills over the markers (when-blocks machinery; body editable between them), pill click → StepSettingsDialog (title/id, done-when incl. the "step text itself" default, in-order toggle, irreversible + fail mode, applies-when); the Procedure controls live in the prompt editor header row beside Copy-full-prompt (ProcedureHeaderStrip: step count, issue badge, ＋ Step insert, Generate-steps — which now INSERTS blocks instead of writing a config array).
Bug fixed in the same pass: enabling procedures no longer errors instantly with steps — Too small: expected array to have >=1 items — zod dropped min(1), and an enabled-but-empty procedure is a publish WARNING ("arms nothing"), never a blocker.
Generate-steps WEAVES, never appends (third pass, same day): the model returns the whole prompt with marker lines inserted around the existing prose, verified marker-only by strip-comparing both sides (buildWeaveResult — the marker-strip function doubles as the tamper check); pre-existing blocks are preserved, settings naming an existing id are discarded, prose drift salvages to append mode with a placement warning, and the client refuses the document replace if the author typed during generation. Verified live against 3032's tenant model (Opus, ~7 s, byte-identical prose, hardship step correctly emitted as applies_when: llm). A failed check is REFUSED (reason recorded, one retry with the reason fed back, then an error) — never salvage-appended, which duplicated prose; a renamed author group is reconciled back to the author's name. Text + settings sidecar land in one config write (a deferred sidecar write raced React's commit and lost the woven text).

Multiple procedures + references (fourth pass, same day — user decision: "Groups + references")¶

{{#procedure name}} … {{/procedure}} groups wrap step blocks anywhere in a prompt: each named group is its OWN checklist (own monitor budget, own procedure_status grouping). It arms where it is defined plus in every prompt that {{PROCEDURE:}}-references it — never everywhere (the first cut armed groups stage-globally, and the 3032 live test showed the appointments subagent's whole flow live on Main from the first second: the judge scored Main against a subagent's checklist, nudged, and the agent performed the flow from Main without ever routing — see the live-test findings below). Its PROGRESS still survives subagent hops (only a topic's own ungrouped checklist resets on exit); per-context gating belongs to the group's applies_when (CODE-kind only at group level — deterministic; AI-judged applicability stays on individual steps; the condition is stamped onto members as _group_applies_when and both step + group conditions must hold). Ungrouped steps keep today's semantics (the defining scope's default checklist; requires-chains stay within their bucket). Group settings live in config.procedure.groups[name] (sparse; edited by the PROCEDURE pill's ProcedureGroupDialog). Duplicate group names across scopes: first definition wins with a runtime warning; publish errors.
{{PROCEDURE:name}} references (dropdown: the Procedure category, or typed) are RENDER-ONLY: the resolver expands them into the named group's numbered step guidance — which is what makes a flow defined on Main visible inside a subagent's takeover prompt — and they never re-arm or duplicate state. Unknown names stay literal + publish-warn.
Grammar shipped fixture-first (procedure-block-cases.json, 22 cases, both scanners); the runtime side is _resolve_scope_procedures + collect_procedure_guidance + expand_procedure_references (resolver step 1.5 for the main template, and per-topic addendum before token resolution).

First live test (2026-09-03, agent 3032) — three symptoms, one root cause¶

The caller heard the agent interrupt itself, re-ask for name/DOB after verifying, and got an unstyled appointment card. The persisted events explained all three: named groups were stage-global, so the appointments subagent's flow_1 armed on Main; the monitor fired four nudge corrections in 3.5 minutes (off_procedure 25 s in, off_procedure on authenticate four seconds after the passkey verification the judge never saw, skipped_step on confirm_identity_details once the confirmation had scrolled out of the 10-turn window); every nudge is a respond=True append, which on Nova is an interactive user turn — a new spoken response cutting off the current one (both doubled transcript lines are nudges landing); and the corrections pulled the agent into doing the flow from Main, so it never called transition_topic and built the card from a2ui.dev standard primitives instead of the healthcare catalog's AppointmentSummary (the "missing CSS"). Fixes, all runtime:

Scoping: named groups arm where defined + referenced; progress persists across hops.
Delivery: corrections are QUEUED and delivered at the caller's next final transcript — the one moment a note informs the reply the model is composing anyway; default severity is note (on Nova every append is a turn regardless of run_llm, so the timing is the fix); only severe (hallucinated completion of an unfired irreversible step) delivers immediately.
Judge discipline: off_procedure is recorded in the status event but NEVER corrects; a violation needs the same (kind, step) on two consecutive checks; the prompt marks steps done / CURRENT / upcoming and forbids judging platform-verified steps; every session-variable change is echoed into the judge transcript as a system fact (customer_auth_status is now "verified"), so out-of-band ceremonies are visible.
Debuggability: procedure_status events carry last_violation and each row's active_in scopes; every verdict is logged at INFO with the model's reasoning.

Fifth live test (2026-09-04, full run to payment) — five symptoms, four fixes¶

The flow completed end to end (auth link → lookup → card → confirm → headshot → copay → payment → summary). What the log showed behind the caller's five complaints:

"Repeated 'I've received the photo' twice." The photo tap was injected into Nova TWICE, 2 ms apart: a UI action originates downstream (the publisher pushes it UPSTREAM), so it passed the Nova wrapper on the way up, the S2SUserAggregator re-emitted the same frame DOWNSTREAM, and it passed the wrapper again — two interactive turns, two attach_patient_headshot calls, two identical answers. The wrapper now delivers each append frame once (dedup by frame id).
The judge never ticked lookup_patient although find_appointment_patient had plainly returned, so every later verdict was scored against a stale current step (a wrong premature_action on ask_copay note even rode a tool result). A settings-less step whose body carries exactly ONE {{TOOL:x}} pill now completes deterministically when x SUCCEEDS; one {{COMPONENT:Name}} pill completes when the A2UI tool renders that component (infer_complete_when + the TS mirror inferCompleteWhen; authored settings always win; pending/consent_required/refusal results are not success). Tool outcomes also render as platform facts ("tool X succeeded") instead of dialogue lines.
"Says it'll send the camera widget but doesn't until I speak." The model said "I'll send a camera widget… please keep it handy", ended its turn, and sent nothing for 19 s until the caller spoke. The monitor's promise watchdog now arms on a promise phrase spoken while the current step is a tool step; if no tool result and no caller turn arrive within 4 s it delivers ONE spoken nudge for that step (on an idle line a short prompt beats the silence — and this is the "promised progress" class the layer exists to catch). It never arms on a question ("…is it okay to send that now?" is the model waiting for consent) and disarms the moment the caller starts speaking, not only on a final transcript.
"Card was missing buttons." The model omitted doneAction on AppointmentSummary, which is what renders the Done button. The registry guidance now marks it REQUIRED and says what happens without it.
"Didn't hear me say 'sure'." Two more unheard windows (29 s and 27 s). Inbound energy during both showed only short blips at amplitude ~430–670 — the level at which yesterday's "are you there" WAS heard — while the utterances that were heard peaked at 5,000–16,000. Every unheard utterance was one short word; every heard one was a phrase. Leading hypothesis: Krisp VIVA voice isolation (on by default, settings.turn_control.audio_enhancement.voice_isolation) attenuating quiet single words, and/or the browser AGC not ramping in time. Not a procedure-layer issue; test with voice isolation off.

Third live test (2026-09-04) — the repeats were not corrections¶

The caller heard the consent question twice and "I don't see any upcoming appointments" twice. The piped orchestrator log showed no [PROCEDURE CHECK] was ever injected (the only platform-notice turns were the greeting, the topic takeover, and the identity-verified notice). Both repeats were Nova answering a late tool result: the wrapper's deferred tool-result poll slept a geometric ladder (0.05 → 5 s), so a 3.9 s FHIR lookup was delivered 2.9 s after it finished — behind a later, faster call — and the model, handed a stale contradicting result mid-answer, re-planned, re-issued the lookup, and spoke the same sentence again when that second result landed after its turn. The poll now runs at 50 ms (250 ms past 2 s, 30 s ceiling) with the same two safety gates. Two procedure-layer findings from the same call:

Nova has no silent text channel. AWS documents cross-modal text input with interactive: true only; non-interactive text is pre-audio history. So a "note" injected at any moment — including the caller's turn boundary — is a spoken turn on Nova. Non-severe corrections on Nova now ride the NEXT tool result's instructions field (the same same-turn channel the topic takeover falls back to), attached by the wrapper's _send_tool_result and expiring after 60 s if no tool call comes; other providers keep the turn-boundary note.
The judge ignored an authoritative fact. With customer_auth_status is now "verified" interleaved in the dialogue, Nova Micro flagged authenticate as skipped three times. Facts now render in their own "Platform facts (authoritative)" section, the system prompt says a fact outranks speech, and the verdict schema puts reasoning FIRST (a small model that writes its rationale before the verdict is markedly more accurate). It did correctly catch confirm_identity_details being skipped — the model looked the patient up with a guessed date of birth before the caller answered.

Per-provider risks are concentrated in PR4 and named up front: on Nova, corrections travel as exactly one platform-notice user turn or via the tool-result instructions channel (never a mid-session SYSTEM, never two turns); every append must ride S2SUserAggregator or the user aggregator consumes it; OpenAI Realtime takes append + session.update; cascaded takes the plain corrective append.

Verification¶

Unit/drift: aux-slot drift rows; byte-identical delegation pins on the client extraction; compiler merge-not-assign + DAG-cycle tests; tool-gate additions; a latency budget test asserting zero network calls on the deterministic gate path; monitor fixture suite over skip/hallucinate/order/off-procedure transcripts plus correction-dedup, fail-open-on-outage, and one-correction-one-turn pins; extraction never-clobber-with-null, linked-refusal, secure-capture starvation; costing round-trips + a coverage-partial case.
Eval tier: tests/fixtures/{extraction,procedure}_eval_cases.json run live under SLM_EVAL=1 (the test_slm_eval.py harness pattern) against the configured logic_slm — including a self-hosted vLLM endpoint smoke.
E2E on vanguard: author a procedure on 3032's Appointments subagent (authenticate → confirm identity → lookup → confirm appointment → headshot → copay-ask → payment, with payment irreversible); Preview call watching the live checklist tick; deliberately provoke a skip ("just charge me now") and verify the deterministic veto plus spoken recovery; kill the SLM endpoint mid-call and verify fail-open; confirm cost_usage_events rows for __logic_slm and the forecast card showing the new meters.

Open questions (decide before PR4)¶

Should the monitor's per-turn verdict also be exposed to authors' own rules (a procedure_violation trigger kind) so tenants can script custom reactions? Cheap to add on the existing trigger vocabulary; deferred until the built-in ladder proves insufficient.
Whether the guided script's capture.kind: entity should route through the multi-slot extractor immediately (PR2) or stay single-slot until the eval harness compares accuracy.
Whether procedure_status events should persist for post-call review (Activity Log checklist replay) in PR4 or trail as a follow-up. Persisting is one more conversation_events kind; leaning yes.

Blocked consequential tools now return the next ready prerequisite’s authored guidance in their refusal result, including a ready ancestor when necessary. This is available to every provider and preserves the refusal; it never executes a retry or marks the prerequisite complete. Guidance respects the active procedure, monitoring setting and correction budget. Put the required component/tool contract in that step’s guidance so a model can repair the exact omission.

Next-step tool results and external-outcome continuations carry the first 900 characters of whitespace-normalized step guidance. Keep all branches needed for the immediate decision within that limit, including what to do when an answer has already been supplied. A longer full prompt does not restore a branch omitted from the immediate continuation. In the Capabilities demo, an overlong payment step clipped its already-authorized branch and repeated the permission question.

Terminal trusted payment notifications also carry the next ready authored step when delivered to the conversational model. The notification is evidence of the payment outcome, not of a later record write. This context uses actual procedure state and preserves consent/gates; pending notifications and live secure keypad handback keep their existing paths.

The builder also advertises existing prerequisite titles in tool descriptions before model tool selection, preserving the schema and handler. A blocked judge step may report missing facts from a background verdict only when that verdict matches current evidence/state; a later caller answer invalidates the hint. These descriptions and hints never replace the runtime gate or complete a step.

Caller-final utterances now start eligible background assessments as well as assistant-turn completion. An immediate judge-gated write can reuse that in-flight check instead of requiring another spoken confirmation. The write gate waits at most eight seconds; deterministic gates and ordinary speech do not wait on this check. A timeout remains closed by default, and a still-running assessment produces pending guidance that preserves the caller's answers rather than asking the same question again.

For GPT Live, a write whose remaining prerequisites are all judge-based first waits for native caller-caption delivery, including tools without extracted bindings. Otherwise a report-confirmation judge can assess the transcript before the caller's actual answer arrives. The existing eight-second assessment bound then applies. Pending caption delivery keeps the gate closed and schedules the bounded input-recovery continuation; it does not infer confirmation or repeat the write.

Checker instruction overrides¶

The stage's Properties → Auxiliary Services → Logic checker model exposes the regular model configuration form plus the default procedure-checker and entity-extraction instructions. Overrides are saved in config.aux.logic_slm.config.prompt_overrides.adherence and .extraction. The runtime chooses the override by completion scope, then appends the required JSON schema and supplies evidence separately. Per-condition AI judge prompts are unchanged. The prompt catalog in lib/pipecat/logic/aux_prompts.py is shared by runtime execution and the authoring preview; unset prompt keys continue inheriting tenant/stack defaults. See auxiliary resolution.