Runtime profiles¶

Status: Draft | Last reviewed: 2026-08-31

The design for multi-channel, multi-variant agents: one authored agent that serves both voice and chat, carries several pipeline variants (an S2S primary, a cascaded alternative, a backup on locally hosted or alternative models), and switches between them automatically — at session start and, where necessary, mid-call — based on channel, compliance policy, and provider health. It also defines stack-first authoring: pick a technology family (Amazon Nova, OpenAI, Gemini, self-hosted) and the platform derives the variants from curated best practice and what the tenant actually has, with every derived value overridable by advanced users. Read this before building any variant, failover, or stack-selection feature. This is a design doc: nothing below is built except where a section says otherwise.

Implementation status¶

Phase 1 is built (2026-08-30); phases 2–4 remain design:

Shared backoff policy — backoff.py.
OpenAI Realtime reconnect-on-drop — InstrumentedOpenAIRealtimeLLMService: the receive-loop exit now schedules reset_conversation() from a sibling task with backoff; exhaustion pushes a FatalErrorFrame so the session ends instead of hanging silent. The class forks the base receive loop as _tsf_receive_loop_impl — a method named _receive_task_handler added to it would shadow the reconnect wrapper and silently disable this fix.
Provider circuit breaker — circuit_breaker.py: keyed provider × service_type × region × credential_ref, in-process with best-effort Redis mirroring across replicas, half-open single-probe semantics, fed by the Realtime reconnect and the on-demand probe. Redis faults fail open.
Stack-first wizard path (default) — components/wizard/steps/StackStep.tsx + the create branch (lib/stack-initial-config.ts): the draft stores config.stack plus the family's primary pipeline; publish derives the rest. The step also picks integrations (2026-08-31): one Select per distinct provider the family's manifests use for the chosen channels (stackProvidersForChannels — derived from the manifests, never hardcoded), auto-selected when the tenant has exactly one active integration of that provider's source, with a visible non-blocking notice when there is none. The picks land in config.stack.integrations (provider slug → integration id) and both the seeding path and deriveProfiles stamp stage.integration_id from that map onto every stage whose provider has an entry — never over an explicit id already on the stage (authored profiles win whole and are never stamped).
ProfileSelector seam — profile_selector.py owns the resilience decision. health_check_strategy now defaults to circuit_breaker (O(1) read); on_demand keeps the legacy 2 s probe, whose findings feed the breaker. Two latent defects fixed on the way in: the old inline block only ever swapped the ServerHello copy (the session then rebuilt the primary, so the fallback pipeline never actually ran), and fallback_activated/original_agent_id were never written into runtime_context, so the resilience.fallback_activated trace attribute never fired. Selection now applies at the build-time fetch too and stamps the context.

Phase 4 is partially built (2026-08-30) — tier B in full, the MIGRATING transport contract in full, tier C bounded:

Tier B — in-place endpoint swap (built): config.resilience_endpoints: [{base_url, model?, api_key?, integration_id?}] on an OpenAI Realtime stage is resolved at BUILD time (resolve_resilience_endpoints — credentials through the same get_api_key machinery as the primary, never a DB read mid-call) and stashed on the service instance. When the tier-A reconnect budget exhausts, _tsf_swap_to_next_endpoint reassigns api_key/base_url/model in place and the backoff budget restarts against the new endpoint; each endpoint gets one full budget, each abandoned endpoint records a breaker failure keyed on its base_url as credential_ref (the primary keeps the pre-existing bare key), and a reconnect success records the active endpoint's breaker success. The list exhausting is exactly the pre-tier-B ending: FatalErrorFrame + breaker record. Absent config = byte-identical behavior. The cascaded OpenAI-compatible LLM is deliberately NOT wired — it is per-turn HTTP with no reconnect seam we own (see the module docstring in openai_services.py).
MIGRATING transport contract (built): Control.TYPE_SESSION_MIGRATING / TYPE_SESSION_ACTIVE in voice.agent.proto (stubs regenerated on both sides). media-proxy (internal/proxy/migration.go) arms a bounded suppression window (default 10 s, capped 30 s): an in-band TYPE_DISCONNECT inside the window is held and REPLAYED (params included) at window expiry so a failed migration resumes the exact normal teardown; ACTIVE disarms and discards it; the idle reaper is suppressed for the window via isIdle. ari-controller needed no change — it only tears a leg down on media-proxy's session-end callback, and its channel reconciler keys on Redis-session presence, which a migrating session keeps.
Tier C — cross-shape migration (bounded): the engine is midcall_migration.py, gated on settings.resilience.mid_call_migration: true (absent/false = the feature does not exist; the gate processor is not even inserted). Trigger, profile selection (voice-cascade → voice-backup via materialize_profile), the MIGRATING/ACTIVE signaling, the hold-line rule (only through an already-existing private handoff TTS), the context-carry-by-identity contract, and the tracer stamps are implemented and tested. The rebuild-behind-transport step itself is NOT wired in production: a faithful rebuild requires re-running most of GRPCProvider.initialize (conversation-row creation, recorder/publisher wiring, @transport.event_handler closures — several steps non-idempotent), which is beyond safe restructuring bounds for this phase. The engine returns a structured rebuild_unsupported outcome that disarms the window and falls through to exactly today's teardown — an honest bounded implementation instead of a rebuild path that could strand live calls. A future faithful rebuild plugs in as the engine's rebuild callable without changing its control flow.

Phases 2–3 are built (2026-08-31): the profiles storage model, publish-time derivation inside publishAgentVersion (profile-derivation.ts — authored profiles win whole, stack.author_overrides win field-by-field and survive re-derivation, denied providers drop with visible warnings, _profile_provenance stamped beside _capability_provenance), the stack-first wizard with integration selection (above), draft parity (the orchestrator expands drafts via POST /api/internal/capabilities/expand, so Preview's chat test on a stack draft selects the real chat profile), and the advanced-mode UI:

Variants panel — StackVariantsPanel.tsx, a collapsible card in Agent Settings' operations area (where the read-only Stack: badge used to be). The simple view previews the derivation client-side with the same deriveProfiles publish runs — possible because the compliance import was split: the pure deny-rule evaluation lives in browser-safe provider-compliance-rules.ts (re-exported by the fs-reading provider-compliance.ts, one implementation), and the tenant's effective deny rules come from GET /api/settings/data-protection/deny-rules (fail-open to [], matching publish's degradation). One row per variant: channel icon, pipeline summary, the integrations in use, a "customized" dot when overrides touch it; derivation warnings (including compliance-dropped profiles) render inline. Expanding a row exposes the high-value fields per stage (model + voice, options from the service definitions) editing stack.author_overrides['<profile>.<index>.<field>'] through the page-level config write, with per-field reset. Hand-authored profiles render read-only with an "authored" tag. Fold state is localStorage, never config.
Per-channel Preview — a voice agent with a chat profile (or stack.channels including chat) gets a Voice|Chat toggle in the Preview panel (TesterSidebarPanel.tsx chatProfileAvailable); the chat mode reuses the text-chat tester body verbatim, and the chat session machinery underneath is agent-type-agnostic.

materialize_profile and the profile ladder already exist in the selector.

Why¶

Today an agent is one agent_versions.config with one pipeline shape, so "voice and chat" or "S2S with a cascade fallback" means duplicate agents kept in sync by hand. Meanwhile the platform already contains four independent, ad-hoc selection mechanisms — each solving a slice of this problem without knowing the others exist:

Mechanism	What it selects	Where
Modality override	Text-only session forces `agent_type: "text_chat"` from the same agent row	grpc_provider.py#L212
Weighted A/B	Random published version by `traffic_weight`	db_pipeline_store.py#L36 → SQL `resolve_agent_version_weighted`
Resilience fallback	Whole different agent on a failed health probe	main.py#L859-L883 + health_checker.py
Version pin	`X-Pipeline-Version` / `:draft` / `:vN` suffix	main.py `_handle_stream`

Each has real limits. The modality override hands a chat session the voice config and hopes the text transport tolerates the STT/TTS stages. The resilience fallback points at a second agent id (a copy to keep in sync), probes only three providers with env-var keys, and costs up to 2 s of session-start latency per call. And the layered PII/PHI protection design created decisions with nowhere to live: its L0 compliance profile "forces cascade degradation" and its L4 matrix "denies a provider per profile" — both are variant selection acts.

Mid-call, the picture is worse and mostly undesigned:

AWS Nova Sonic self-heals well: SessionContinuationHelper pre-creates the next stream before AWS's ~8-minute limit, buffers audio through the handoff, and replays the persistent LLMContext.
OpenAI Realtime has no reconnect at all. A mid-call websocket drop ends the receive loop and the session hangs silent — the service's own reset_conversation() (disconnect, reconnect, replay context) exists and nothing calls it. This is a live reliability bug, and phase 1 of this design fixes it.
Moshi retries 3× with a 1 s delay; every other service dies with the socket. There is no shared retry/backoff utility and no runtime provider-health tracking anywhere — failed builds just log.

Design overview¶

Four components. Profiles are the storage substrate; the other three consume it.

Component	One-line role
Profiles	Named pipeline variants inside one agent version, sharing a single authored core
Stack-first derivation	Turns a family choice into concrete profiles at publish, best-practice tables × L4 matrix × tenant integrations
ProfileSelector	One session-start policy replacing the four ad-hoc mechanisms
Failover ladder	Provider self-heal → in-place endpoint swap → cross-shape migration, feeding a circuit breaker

flowchart LR
    AUTHOR["author picks a stack<br/>(family + channels + profile)"] --> DERIVE["publish-time derivation<br/>stack manifest × L4 matrix<br/>× tenant integrations"]
    DERIVE --> PROFILES["published profiles<br/>voice | voice-cascade<br/>voice-backup | chat<br/>(+ _profile_provenance)"]
    SESSION(("session start")) --> SEL["ProfileSelector<br/>pin → compliance → channel<br/>→ breaker → weights"]
    PROFILES --> SEL
    CB["circuit breaker<br/>(Redis, per provider ×<br/>service × region × cred)"] --> SEL
    SEL --> BUILD["universal builder<br/>(one profile = one config,<br/>unchanged)"]
    BUILD -->|"failures"| CB
    BUILD -->|"tier A/B/C"| LADDER["mid-call failover ladder"]
    LADDER -->|"exhaustion"| CB

The universal builder is deliberately untouched: a selected profile is a resolved config of exactly today's shape, so everything downstream — build phases, preflight internals, capability runtime — keeps working on one config at a time.

Data model: singular core, named profiles¶

AgentVersionConfig (pipeline-types.ts#L227) gains one optional key:

profiles?: Record<string, {
  pipeline: ServiceStage[];        // full pipeline, not a patch
  settings?: Partial<UniversalPipelineSettings>;  // sparse overlay on the core settings
  agent_type?: string;             // e.g. "text_chat" for the chat profile
}>

The authored core stays singular. Prompt, tools, variables, Agent Logic, capabilities, topics, copilot: authored once, shared by every profile. A variant is a different pipeline, never a different agent.
Full pipeline[] per profile, sparse settings overlay. Pipelines are ordered stage lists where S2S and cascade differ in shape; patching stage arrays is where overlay schemes die. Settings are flat-ish and overlay cleanly. The precedent is payment_handoff_tts_config, which derives a full ServiceStage, not a diff.
The bare config is the implicit default profile. Absent profiles means exactly today's behavior — zero migration, and every existing agent keeps working unchanged.
Standard derived profile names: voice (the stack's primary — S2S where the family has one), voice-cascade (STT→LLM→TTS on the same family), voice-backup (locally hosted / alternative models), chat.
Drafts store intent; publish materializes. The draft carries stack: {family, channels, author_overrides} — small and author-editable, so it must be added to MISC_ROOT_FIELDS in both Y.Doc shape converters (collab-gateway and the ai-manager copy) plus the round-trip test, or autosave silently drops it. The materialized profiles + _profile_provenance are written at publish, exactly like _capability_provenance — publish-only data never rides the collab doc, so no allow-list change is needed for it. Advanced-mode edits to profiles in the draft (see stack-first below) do require the profiles key in MISC_ROOT_FIELDS too.

Everything currently keyed on "one config per version" resolves as follows:

Coupling	Resolution
Publisher selection (`PUBLISHER_AGENT_TYPES`)	Reads the selected profile's `agent_type`; the chat profile carries `text_chat` explicitly
Preflight / editor test buttons	Runs per profile; the voice test button gates on the voice profiles, the chat button on `chat`
Cost estimation (config-hash pinning)	One hash per profile; see open questions for which profile prices the headline estimate
Capability expansion	Applies to the core once, then projects tools/prompt/gates into each profile's pipeline at the same publish step
`ccaas_pipeline_integrations`	Unaffected — keys on the base agent id

Stack-first authoring¶

The new-agent wizard leads with a stack picker: family + channels + (optionally) the tenant's compliance profile, replacing the blank-config default. generate / template / custom remain as paths in PathStep.tsx.

Stack manifests are data, one JSON per family under shared/service-definitions/stacks/ — completing the migration away from the deprecated provider-metadata.ts. A manifest names, per profile slot, the stage graph and (provider, service) choices with fallback ordering: the Nova stack derives Nova Sonic S2S for voice, Transcribe + Bedrock + Polly for voice-cascade, Bedrock for chat; the OpenAI stack derives Realtime / Whisper + GPT + TTS / GPT; self-hosted derives Moshi or an OpenAI-compatible endpoint (vLLM/Ollama — the base_url resolution chain "step config > integration config > provider default" already exists).
Derivation inputs: the stack manifest × the L4 provider matrix (a profile-denied provider is never derived) × the tenant's configured integrations (no credential, no stage — with a visible reason in the wizard, never silent omission). Stage defaults are baked from the service definitions via the same getConfigWithDefaults path Generate-with-AI uses (template-baseline.ts).
Pinned at publish with provenance. Derivation runs inside publishAgentVersion in the same slot as capability expansion (failure blocks publish); _profile_provenance records the stack manifest id + version, the derivation-input hash, and a per-field override map. An improved best-practice table changes nothing until the agent's next publish — no fleet-wide silent behavior change.
Advanced mode: every derived value is editable in the ordinary editor. Edits are stored as sparse overrides keyed by profile + stage path (the cost-estimation subject-profile semantics: store only differences so untouched defaults can evolve); re-derivation re-applies overrides on top and provenance marks each overridden field.

Note: this is the same lifecycle as capability packs — refs/intent in the draft, concrete expansion at publish, provenance on the published row — applied to pipelines instead of tools.

Selection policy¶

One ProfileSelector, evaluated at the single session chokepoint (_handle_stream, main.py), returning (agent_version, profile_name, reason). The reason rides to OTel, extending the existing resilience.fallback_activated attribute pattern. Precedence, strictest first:

Order	Input	Behavior
1	Explicit pin (`X-Pipeline-Version`, `:draft`/`:vN`, new profile pin for testing)	Honored, subject only to the compliance filter
2	Tenant compliance profile (data-protection L0/L4)	A filter, not a chooser: eliminates profiles whose providers the matrix denies; can force `voice-cascade` over `voice`
3	Channel / modalities	Text session ⇒ `chat`; voice ⇒ the voice ladder
4	Circuit-breaker state	Skip profiles with open breakers: `voice` → `voice-cascade` → `voice-backup`
5	Weights	A/B stays at the version level (existing SQL); the profile is selected within the chosen version

Failure stance: if the compliance filter eliminates every profile, refuse the session (fail closed — a policy violation must not degrade into a permitted-by-accident call). If only breakers eliminate everything, take the least-recently-tripped profile and flag the conversation (fail open — an outage heuristic must not take the platform down with it).

Circuit breaker¶

Orchestrator-owned (no new service), replacing the per-session 2 s on-demand probe:

Key: provider × service_type × region × credential_ref. Credential in the key so one tenant's revoked key does not trip the provider for everyone; a genuinely global provider outage trips every credential's breaker on its own evidence.
Feeds: pipeline build failures, service connect errors/timeouts, provider ErrorFrames from live sessions, and tier-A reconnect exhaustion (below).
State: Redis (the orchestrator runs multiple replicas; Redis is already a dependency) with a short-TTL in-process cache; standard closed → open (N failures per window) → half-open, where the half-open probe reuses an extended PROVIDER_CHECKS table — extended beyond today's three providers and taught to resolve tenant-integration credentials instead of env vars only.
Session start reads breaker state — an O(1) cached read — instead of probing. resilience.health_check_strategy gains "circuit_breaker" as the new default; fallback_pipeline_id (the different-agent fallback) survives as a deprecated escape hatch until profiles cover its uses.
Surfacing: the tool-health scheduler + status-column pattern (lib/tool-health/) applied to providers, feeding OperationsPanel.tsx.

Mid-call failover ladder¶

The expensive half, designed fully because "seamless" is the requirement. Three tiers, cheapest first; each exhaustion feeds the breaker and escalates.

Tier	Trigger	What changes	Caller-audible gap	Context guarantee
A: provider self-heal	Socket drop, provider error	Nothing — same service reconnects	≤1 s (Nova today)	Provider replays the persistent `LLMContext`
B: in-place endpoint swap	Tier A exhausted, same-shape alternative exists	One stage's endpoint/model via the existing `base_url` chain	≤1 turn	Chain untouched; context object untouched
C: cross-shape migration	Tier B impossible (S2S ⇒ cascade)	The whole pipeline, rebuilt behind the same transport	Target ≤2 s perceived, ≤4 s hard	Explicit carry, below

Tier A. Nova's SessionContinuationHelper is the reference implementation (pre-created next stream, audio buffered through the handoff, context replay). The design adds the missing OpenAI Realtime reconnect — backoff-wrapped calls to its existing reset_conversation() — and one shared retry/backoff utility (none exists; Nova, Moshi, and OpenAI each need it, and today each hand-rolls or lacks it). Exhaustion emits a ProviderFailedFrame upstream and increments the breaker.

Tier B. Where the failed stage has a same-shape alternative (cascade LLM → vLLM/Ollama backup, alternate region), swap the endpoint through the base_url/integration resolution the OpenAI wrappers already implement. The chain, the context, and the caller's audio path never change.

Tier C: rebuild behind the same transport. Chosen over "internal re-dial" deliberately: today media-proxy's grpcRxLoop treats gRPC stream end as session death (notifySessionEnd → ari-controller hangs the leg up), so a re-dial would be a distributed re-attachment problem across three services. Keeping the gRPC stream and transport objects alive keeps the whole migration local to the orchestrator:

The orchestrator signals MIGRATING on the session (a new signal in the media-proxy contract) so media-proxy and ari-controller suppress the hangup path for a bounded window; on timeout the normal teardown resumes — a failed migration must never strand a leg.
Hold/filler audio plays through the private-TTS mechanism the payment handoff already uses, in the agent's own voice.
The old pipecat chain is torn down (pipecat 1.4 has no dynamic pipeline mutation — rebuilding is the honest primitive) and the target profile's chain is assembled behind the same transport.
Context carries by identity: the persistent LLMContext (already proven to survive full provider reconnects), the SessionVariableStore-wrapped session_variables dict, consent and notice state under the HANDLED_NOTICE_PREFIX discipline (a migrated session must not re-announce or re-ask), and the deterministic conversation row id keeps persistence continuous. The InstructionRefresher's per-provider delivery matrix hands the system prompt to the new provider by its own mechanism.

Warning: voice continuity is a hard constraint, not polish. The payment-TTS engine lesson applies — a neural Polly Matthew is audibly a different person than the generative one Nova speaks with. A tier-C migration must select the cascade TTS voice for continuity with the S2S voice (the stack manifest pins matched voices per family), or the caller experiences the failover as being transferred to a stranger.

Multi-channel: chat as a real profile¶

The chat profile formalizes what the modality override fakes: no STT/TTS stages, agent_type: "text_chat" (so the publisher and text-turn machinery engage by declaration, not inference), text-appropriate settings. The grpc_provider.py#L212 override remains as the compatibility fallback for versions without profiles. One authored agent, both editor test buttons, both channels — no duplicate agent.

Configuration and gotchas for implementers¶

stack (and profiles, for advanced-mode draft editing) → MISC_ROOT_FIELDS in both shape-converter copies + round-trip test; _profile_provenance stays publish-only and out of collab.
New envelope keys go through generate-config-schemas.ts (re-run; every field described; drift CI).
Per-profile preflight is N calls to the existing preflight_agent — no new validation tier.
Stack manifests live beside the service definitions they reference and are validated against them in CI (a manifest naming an unregistered (provider, service) must fail the build, not the first caller — the seven-place registration lesson).

Failure policy¶

Component	Default	Rationale
Publish-time derivation	Fail closed	Same contract as capability expansion — a broken stack blocks publish
ProfileSelector: compliance filter empties the set	Fail closed (refuse session)	A policy violation must not degrade into a permitted call
ProfileSelector: breakers empty the set	Fail open + flag	An outage heuristic must not be its own outage
Circuit breaker internals (Redis unreachable)	Fail open (treat closed)	Selection degrades to today's behavior, never blocks sessions
Tier-C migration failure	Bounded, then normal teardown	`MIGRATING` window expiry resumes the existing hangup path — never a stranded leg

Phasing¶

Phase	Scope	Why this order
1	OpenAI Realtime reconnect fix + shared backoff utility + circuit breaker + ProfileSelector consolidating the four existing mechanisms	No schema change; fixes a live silent-hang bug; immediate reliability value on today's agents
2	`profiles` storage + `chat` profile + publish-time derivation + per-profile preflight	The substrate — multi-channel lands here
3	Stack-first wizard path (default) + advanced-mode sparse overrides + L4/tenant-aware derivation	The authoring face, on top of a substrate that can carry it
4	Tier B/C mid-call migration + the `MIGRATING` transport contract	The hardest runtime work, needing phases 1–2

Open questions¶

Voice continuity inventory — for each S2S voice per family, which cascade TTS voice matches? Needs listening tests, not just a mapping table.
Which profile prices the cost estimate shown to authors — the primary, the worst case, or a per-profile breakdown?
CCaaS allow-list vs profiles — a call admitted for an agent is admitted for all its profiles; is that acceptable when a fallback profile uses different providers?
Per-profile A/B weights — weights currently select versions; do we ever need weighted selection between profiles?
Breaker state in customer environments without a shared Redis across orchestrator replicas.
Does voice-backup require the tenant's self-hosted endpoint to exist at publish, or may it derive dormant and fail preflight only when selected?