Conversational authoring evaluation and delivery¶
Status: Draft | Last reviewed: 2026-09-10
Use this record to distinguish observed feasibility from proposed acceptance checks, then implement the smallest useful release. It includes offline observations from this review and a reproducible experiment plan; it does not contain a completed model comparison or a live catalog-to-agent trace.
Evidence collected¶
| Check | Observed result | What it establishes |
|---|---|---|
| Repository contract inspection | Completed | Current candidate reuse paths and gaps, linked in the capability matrix |
| Assistant permissions and management scopes | Selected tests passed | Advisory sessions reject draft writes; management tools require the relevant authority |
| Capability authoring/workspace tests | Selected tests passed | Prompt projection, source assembly/round-trip, capture preservation, path constraints, and no-overwrite behavior |
| Combined TypeScript test run | 18 tests passed, exit 0 | These local library contracts; no deployed tenant/runtime claim |
| Generation/assistant Python tests | 9 passed in an isolated dependency environment | Workspace seeding, command scope, and stream-event contracts |
| Compliance/governance/profile tests | 16 passed in the follow-up review | Policy signal extraction, artifact hashing, guardrail declaration, profile derivation and denied variants |
| Offline validator probes | Executed; observations linked below | Concrete generation-validator behavior on synthetic configurations |
| Documentation checks | Build passed; local links/JSON/style checked | Research pages render; repository-wide/source-link warnings remain |
| Model quality/retrieval A/B | Not run | Requires frozen fixtures, candidates, and measured model calls |
| Branded catalog-to-agent vertical slice | Not run | Requires isolated test tenant/workspace, authorized test connection, and runtime |
| Real phone fallback/action round-trip | Not run | Browser-only checks cannot establish telephony behavior |
| Cross-artifact crash/retry/manual-edit exercise | Not run | Requires implementation of the proposed coordinator/revision contract |
No live integration was invoked and no tenant catalog, agent, or workflow was published as part of these checks. This research changes documentation and research fixtures only.
Reproduce the offline checks¶
From the repository root:
cd apps/ai-manager
node --import tsx --test lib/assistant-mcp/tools.test.ts lib/management/tool-scopes.test.ts lib/capability-authoring-view.test.ts lib/capability-workspace.test.ts
node --import tsx --test lib/ai-governance.test.ts lib/compliance-policy.test.ts lib/profile-derivation.test.ts
node --import tsx ../../docs/_planning/conversational-authoring/validator-probe.cjs
The probe calls pure validation functions with synthetic inputs. It makes no tenant queries or model calls. The checked-in observations record the review run; rerun after validator changes instead of treating historical output as current behavior.
The runner venv initially lacked pytest, and global Python lacked pydantic. Run the selected tests in an isolated dependency environment from apps/workspace-agent-runner:
uv run --no-project --python .venv/bin/python --with 'pytest>=8,<9' --with 'pydantic>=2.8,<3' --with 'httpx>=0.27,<1' --with 'fastapi>=0.115,<1' python -m pytest tests/test_generation.py tests/test_assistant.py -q
This completed with 9 passed. It does not install packages into the project's venv or exercise a live model.
The existing documentation venv lacked the configured Material theme. From the repository root, the full handbook build completed using:
uv run --no-project --with mkdocs-material --with mkdocs-render-swagger-plugin --with mkdocs-include-markdown-plugin mkdocs build --config-file mkdocs.developer.yml --site-dir /tmp/trueserve-authoring-research-site
The build is not warning-free: existing handbook issues and relative links to repository source outside docs_dir produce warnings. All new local link targets were separately checked against the filesystem. No dependency manifest or product environment was changed.
Validator observations¶
| Synthetic case | Observed | Implication |
|---|---|---|
| Registered chat service with defaults | Valid, with prompt-style warnings | Establishes a usable positive control |
| Same stage with an invented saved-tool reference | Valid, with a reference warning | Generation validation alone does not establish tool existence/eligibility |
python_sandboxed using def handler |
Invalid | Confirms mismatch with current runner guidance |
| Unknown service in top-level chat pipeline | Invalid | Common validation does reject this unsupported combination |
| Unknown services inside current translation-flow fields | Valid, no warnings in this probe | Confirms a generation-validator coverage gap for this synthetic shape |
The translation result is not evidence that such a config runs; that is exactly why readiness must include runtime-aware validation. Add equivalent negative cases to the real validation suite when fixing traversal, rather than making a test assert that the gap should persist.
Prompt-style warnings on the positive control also warrant review: heuristic prompt lint should not become a substitute for testing the effective prompt.
Benchmark design¶
Use fixtures.json as scenario seeds. Its capability/integration situations are synthetic arrangements to instantiate in a test tenant, not actual tenant IDs.
These scenarios evaluate authoring behavior. Also generate and execute the distinct agent-specific suites tied to each brief, selected runtime, rules, procedures, subagents, and tools. Generated tests are artifacts; passed tests require execution evidence.
Compare:
- Baseline: current bundled guidance and generation toolset.
- Candidate A: one coordinator with the proposed protocol, versioned guidance, and shared discovery.
- Candidate B, optional: the same contracts and tools with a specialist worker for a task that A struggles with.
Hold model/version, runtime version, tool availability, budget, fixture data, and user-answer script constant within each comparison. Record exact differences in candidate prompt/tool versions. Do not give one candidate privileged retrieval results.
Start with at least three repetitions per scenario to expose variability; expand the sample where results are unstable. This is an initial experiment design, not a statistical-power claim. Keep a held-out set of paraphrases and later real authoring tasks separate from examples used to tune the prompts.
Evaluate model turns, compiled artifacts, and actual task results. Use deterministic checks for schema/references/access/idempotency, blinded human review for coaching/experience quality, and model-assisted grading only where calibrated against human judgments. This follows the task-specific approach in OpenAI's evaluation guidance.
Proposed release gates¶
Agree and calibrate these before the comparison. The hard invariants are design requirements; the quality/latency thresholds are starting proposals.
| Dimension | Proposed gate |
|---|---|
| Access and publication | Zero cross-tenant disclosure, unauthorized execution, or unintended publication |
| Grounding | Zero invented executable IDs and zero unsupported capability claims |
| Placeholder honesty | Every simulated result labeled; zero mock actions eligible for production execution |
| Config correctness | Every claimed test-ready artifact passes type-specific schema and reference validation |
| Business completion | Every required scenario proves its output/side effect, including error handling |
| Recovery | Repeated operation produces no duplicate artifact or business write |
| Collaboration | No lost manual edits; overlapping changes produce a specific conflict |
| Questions | No repeated answered question; required decisions captured before dependent actions |
| Experience quality | Human reviewers can describe outcome, exceptions, and remaining dependencies |
| Functional-user language | Business users understand questions, choices, and next steps without technical translation; no requirement to choose implementation mechanisms |
| Accessibility/branding | Required interaction tests pass against the chosen identity and fallback |
| Performance | Report p50/p95 authoring time, first useful draft time, model/tool calls, and cost per successful task |
Set an acceptable latency/cost budget from baseline measurements and product needs. Do not quote a universal budget or provider price from this research. Include failed attempts and retries in cost per successful task.
Branded vertical slice¶
Create a test support-intake fixture with a test support-system action and a tenant-specific brand. Label fixture-backed and real dependencies at each step.
| Step | Evidence to retain | Failure injection |
|---|---|---|
| Brief and discovery | Decisions, retrieved/rejected candidates, selected IDs | Misleading template, unavailable integration |
| Tool intent/binding | Input/output/error schema, binding, environment | Missing required field, wrong authority |
| Component source | Base/new source hash, brand revision, event contract | Missing source archive, unsupported component |
| Build/preview | Build result, manifest, responsive/error-state evidence | Build failure, worker disconnect |
| Publish test catalog | Actual version ID, source hash, operation ID | Lost response after successful publication |
| Bind draft | Expected/new revision, concrete catalog reference | Manual edit between read and write |
| Customer interaction | Render, captured fields, actual tool call/result | Invalid email, duplicate submit, timeout |
| Completion | Persisted support record or explicit simulated record | Accepted-but-not-completed backend response |
| Voice fallback | Spoken intake and result without visual dependency | User declines link; link expires/disconnects |
| Real phone test | Conversation/session routing and action return | UUID/session-ID distinction |
The existing A2UI process documents why browser and phone paths require separate checks. A successful component preview and a transport acknowledgement do not satisfy business completion.
For recovery, stop after publication but before binding. Resume twice with the same operation key. Require one catalog version, a correct draft reference, preserved manual edits, and an explicit runtime-test status. Do not delete the published artifact as an automatic compensation.
Agent-specific runtime tests¶
| Type | Required scenarios |
|---|---|
| Voice | Noise/unclear speech, interruption, entity correction, delayed/failed tool, transfer failure, optional surface declined |
| Chat | Grounded knowledge answer, no answer, attachment uncertainty, form correction, tool failure, result citation/reference |
| Translation | Both directions, names/numbers/negation, terminology, overlapping speakers, silence, disconnect, original-audio mixing |
| Background | Trigger payload, timezone/DST boundary, duplicate event, partial failure, safe retry, output schema, inactive draft |
Use bilingual human evaluation for translation fidelity. A target-language transcript or low latency alone does not establish meaning preservation.
Dependency-ordered backlog¶
Estimates are rough person-days for engineers familiar with this repository, including focused tests and docs. They come from the inspected seams, not completed feasibility experiments. Re-estimate after the first vertical slices. Rows can overlap across people; their sum is effort, not calendar time.
| Work item | Stream | Depends on | Range | Acceptance evidence |
|---|---|---|---|---|
| B0: Fix guidance drift; add validator negative cases and reference-readiness rules | Backend/content | None | 3–5 | Canonical Python example; translation and tool-ID cases |
| B1: Versioned protocol, four type guides, curated patterns and fixtures | Content/evaluation | B0 | 3–5 | Same briefs run against baseline/candidate; functional-user language reviewed |
| B2: Brief/revision persistence and operation journal | Backend | B0 | 5–8 | Restart/resume, cancellation, idempotent dispatch |
| B3: Unified discovery and eligibility descriptors | Backend | B0 | 4–7 | Relevant/misleading/unavailable/no-match fixtures |
| B11: Effective-policy discovery and eligible runtime/control selection | Backend/content | B0–B3 | 3–6 | Input/output boundaries distinguished; denied fallbacks rejected |
| B4: Tool intents, binding and non-executable placeholder UI | Backend/frontend | B2–B3 | 4–7 | Unknown systems preserved; no fake executable refs |
| B5: Conversation/editor integration and conflict-safe application | Frontend/backend | B2 | 5–9 | Manual-edit/rebase/overlap tests |
| B6: Multimodal evidence ingestion and source citations | Backend/frontend | B2 | 3–6 | Image/document observations with correction and access tests |
| B12: Requirement-linked agent test-suite generation and evidence adapter | Backend/evaluation | B1–B4, B11 | 4–7 | Test artifacts stay unrun until executed; hashes invalidate stale evidence |
| B7: Four-type draft release and evaluation gate | Evaluation/frontend | B1–B6, B11–B12 | 4–7 | Required scenarios pass; clear readiness |
| B8: Deterministic mock executor and publish/runtime exclusion | Backend/evaluation | B4 | 4–7 | Simulation honest across direct/inline/pack/workflow paths |
| B9: Scoped catalog worker and brand/source binding | Backend/frontend | B2–B3, B5 | 5–9 | Reusable source → build → preview with narrow scope |
| B10: Catalog publication/binding recovery and real multimodal tests | Backend/evaluation | B8–B9 | 5–9 | Full vertical slice, duplicate prevention, phone fallback |
B0–B7 plus B11–B12 define the first useful release; B7's acceptance gate also depends on B11–B12. B8–B10 add executable demonstrations and generated components. Use existing catalog selection before B9; do not label new-component generation available until its worker path exists.
Do not create a new orchestration service merely to start B2. Keep database state and adapters in the current ownership boundaries, then extract a service only if measured isolation/scaling requirements justify it.
Product decisions to settle during implementation¶
| Decision | Recommended default | Consequence if changed |
|---|---|---|
| Sketch with missing integrations | Allow explicit unbound intent | Blocking all drafts adds avoidable setup friction |
| Simulations | Test-only; visible execution provenance | Production mocks require a separate explicitly designed demo product |
| Catalog reuse | Pin versions; isolate adaptations | Floating shared references need pre-publication impact handling |
| Manual-edit ownership | Revision checks and narrow patches | Whole-config replacement risks lost work |
| Native voice authoring | Add after shared session/brief works | Earlier delivery expands transport and interruption scope |
| Retrieval infrastructure | Lexical/faceted baseline first | Embeddings add operations without proven value if failures are compatibility-related |
| Autonomous specialists | Add only for measured hard tasks | Default fan-out increases cost and conflict management |
Completion boundary¶
This review supplies the architecture, capability evidence, protocol, source research, contracts, seed fixtures, offline probes, and provisional backlog. The original spike's live experiment criterion remains open. Record real artifacts and observed failures in this page as those experiments run, then revise the architecture and estimates from the results.