Knowledge Agent delivery plan¶
Status: Draft | Last reviewed: 2026-10-09
This plan orders implementation into reviewable releases with observable acceptance criteria. Use it with the product proposal. It includes topic modeling, terminology alignment, connected policy evidence, and health from the first useful Vanguard release.
The 2026-10-09 qualification record distinguishes the implemented native slice and executed checks from the remaining acceptance criteria below. The public-corpus baseline and completed native graph have been tested; the reviewed graph generation is published. Wider domain and scale qualification remains open.
Delivery strategy¶
Build a narrow but complete workflow with the public policy corpus, then qualify the CDC/internal-policy scenario before expanding every source and destination. Start evaluation fixtures immediately. Ship the ability to understand why a result failed alongside the ability to construct a pipeline.
| Milestone | Exit result | Depends on |
|---|---|---|
| M0 | Verified storage/provider contracts and an evidence-task baseline | Representative source fixtures |
| M1 | Gated authoring, typed semantic/document graph, durable execution | M0 contract decisions |
| M2 | Usable native RAGFlow index/graph product with Test, Health, publication, refresh | M1 |
| M3 | More connectors and demonstrated large-volume behavior | M2 lifecycle contract |
| M4 | Qualified external destination adapters | M2 adapter contract |
| M5 | Full media workflows and advanced optimization | Stable evidence/lineage contracts |
The source, editor, runtime, and evaluation work can proceed in parallel once their contracts are agreed. Do not estimate calendar delivery from screen count: provider behavior, artifact consistency, and corpus processing dominate the uncertainty. Size the work after M0 using measured costs and team capacity.
M0 — prove the difficult boundaries¶
Representative fixture pack¶
Create the first versioned fixture collection from the 12-source public policy corpus and its selected extensions. The linked manifest records verified downloads and separates dated material and acquisition failures. Review the draft evidence requirements before using them as correctness labels.
Add selected public CDC pages and synthetic internal policies for domain qualification. Add authorized real internal documents when available; they are not a prerequisite for designing the contract.
Include HTML/Markdown, a long PDF, a scanned page, a cross-page table, and:
- Differently named concepts with a supported mapping.
- A deceptively similar term whose scope differs.
- A definition and an exception in separate documents.
- A condition requiring several pieces of evidence.
- A newer external source alongside an older internal citation.
- Missing or conflicting source support.
- The same content under different source permissions.
Create reviewed evidence tasks for lookup, crosswalk, policy comparison, scenario interpretation, change impact, and insufficient-evidence behavior. Include Q&A, Q-only, content-generated, and transcript-derived candidate workflows. Retain original source anchors rather than chunk IDs from one splitter.
Add synthetic transcript fixtures with multiple needs, paraphrases across calls, an implicit request, a context-dependent follow-up, and a question after turn 100. Include an incorrect observed agent answer, a misleading “resolved” label, an account-specific/action request, and differing source permissions.
Contract spikes¶
| Spike | Evidence required to finish |
|---|---|
| RAGFlow prepared-unit publishing | Exact boundaries read back; actual searchable fields and metadata types; model/dimension contract |
| RAGFlow update/retry/delete | No duplicate chunks after retry; confirmed removal; parse cannot overwrite pipeline-owned output |
| Native graph/compilation profile | Actual version, legacy GraphRAG versus current compilation, supported schema/job/inspection/query APIs |
| Native index/graph generation | A publication references matching verified artifacts; a failed graph build cannot activate |
| Metadata/access | Protected evidence cannot cross source/tenant boundaries during retrieval, expansion, or citation |
| Acquisition boundary | Selected public pages/linked files work through the actual crawler and enter the graph without independent RAGFlow ingestion |
| Parser comparison | Coverage and structure results on all fixture classes, including tail pages and tables |
| Terminology extraction | Candidate mapping evidence, hard negatives, review state, and reversible merge/split |
| Policy graph extraction | Conditions/negation/exceptions and source support survive native extraction, inspection, export, and retrieval |
| Retrieval baseline | Hybrid-only versus terminology expansion versus graph expansion, using the same evidence tasks |
| Transcript discovery | Complete-turn Analytics reader and direct upload path; extraction fidelity, source anchors, grouping, and reviewed support |
Record the installed/native-migration RAGFlow version, supported endpoints, and extension requirements. Current upstream documentation and existing Tooling clients may target different versions. Resolve this before building against an assumed API, including the GraphRAG/RAPTOR-to-Knowledge-Compilation transition.
Document which policy/terminology requirements native configuration satisfies and which need a review sidecar or supported extension. Introduce a separate graph database only for demonstrated requirements that those options cannot reasonably meet. Native graph support itself is already established in the source assessment.
Research access to a public page does not qualify the ingestion connector. Include real crawler access, raw HTML/locator retention, pagination/link scope, and access-error handling in each collection's acquisition proof.
If the RAGFlow API cannot preserve required units/fields, choose and test a specific adapter extension or synthetic-document packaging strategy. Do not quietly substitute another backend for the requested native RAGFlow path.
M1 — author and run a real pipeline¶
| Work package | Concrete output |
|---|---|
| Agent registration | knowledge_management_agents, labels, creation/filtering, routing, version/import handling |
| Availability | Server-only FEATURE_KNOWLEDGE_AGENTS throughout UI, APIs, tools, and workers |
| Configuration schema | Described versioned envelope and discriminated document/semantic node manifests |
| Collection/source model | Source identities, snapshot manifests, connection ownership, original artifact retention |
| Semantic model | Topic hierarchy, terminology/mapping records, assertion schema, review state |
| Canvas | Insertion, routing/merge, typed controls, undo/redo, keyboard/list alternative |
| Durable worker | Leases, dependencies, bounded batches, checkpoints, cancellation, idempotency, usage |
| Inspection | Original/parsed/chunk views, evidence links, mapping review, graph neighborhood |
| Discovery inputs | Ready Analytics release binding and shared text-transcript preparation; optional driver annotations |
Initial nodes cover structural parsing, selective OCR, deterministic routing, cleaning, prompt transformation, splitting, metadata, topics, term mappings, assertion/reference extraction, explicit merge, and validation.
Implement semantic nodes through qualified native RAGFlow stages where possible. Show combined native operations as managed stages and preserve their actual inspection/retry boundaries.
The first graph recipe uses a small policy ontology. Authors can configure its supported fields/prompts and inspect AI-populated values and source support. Schema controls follow the tested native capability profile. A complex ontology import system is not a prerequisite for the first release.
Acceptance: an author builds two document branches, merges them, inspects intermediate artifacts, changes one step, and sees only dependent work rerun. An unsupported input or branch failure is accounted for. A worker restart preserves completed work and cannot let an expired lease overwrite it.
M2 — first useful Vanguard release¶
Complete preparation and publication¶
Publish the prepared chunk bundle into a candidate RAGFlow dataset and build its native graph/compilation artifacts. Verify all required artifacts before creating the publication. Retain source, vocabulary, ontology, topic, model, prompt, and evaluator versions.
Expose a published Knowledge Agent tool through the normal authorized tool system. Consumers identify the agent/publication; the server resolves allowed sources, target IDs, and credentials. Preview, suites, and consuming agents use the same query executor.
A query profile can use hybrid retrieval alone or combine terminology resolution, reference/exception expansion, reranking, and source-grounded synthesis. Show which configuration ran; no hidden graph-to-vector fallback.
Test and Health¶
Implement manual/imported/generated cases, evidence requirements, case review, version comparisons, source/graph traces, and claim/evidence results. Support cases whose expected output is a comparison or structured analysis rather than a single answer string.
Health shows source/topic coverage, unmapped/ambiguous terminology, unsupported or unresolved graph relations, stale dependencies, missed evidence sets, qualifier retention, and index/graph consistency. Cases without verified labels are unmeasured or diagnostic, not part of a misleading correctness percentage.
Ship question discovery from text transcripts and prepared Analytics releases here. Review multiple extracted requests, group variants, retain driver/context evidence, and promote questions into requirements, verified cases, or content gaps. Show demand counts and sampling scope alongside knowledge coverage. Full audio/video processing is not a dependency.
Refresh and change impact¶
Support scheduled/manual website refresh and new upload snapshots. Preserve stable document identities, detect meaningful changes, reconcile confirmed deletions, rebuild affected artifacts, and identify potentially affected internal policy provisions/tasks.
Keep manual promotion as the default. A reviewable impact report is not an automatic edit to an organization's internal policy.
Release acceptance¶
The first release passes a recorded demonstration:
- Connect CDC material and a private synthetic internal repository.
- Inspect source authority, dates, structure, and citations.
- Review a terminology mapping and reject a misleading near-match.
- Inspect AI-populated graph fields and a cross-document exception path.
- Run a policy comparison/scenario task and inspect the complete evidence set.
- Remove required evidence and see an explicit unresolved/partial result.
- Change a source and view affected mappings, policies, and test cases.
- Compare candidates in Test/Health, publish one consistent generation, and return to the previous publication without restoring revoked access.
- Turn the feature flag off and verify that its routes, tools, and new work are unavailable in the same way as other disabled optional agent types.
- Select a transcript release, inspect a late-turn follow-up and grouped variants, then find policy support or create a gap. An incorrect spoken answer and an operational failure must not become a verified answer or an automatic content gap.
M3 — source breadth and scale¶
Add native S3/GCS acquisition with paginated enumeration, resumable fetching, provider version/change hints, complete inventory reconciliation, and source watermarks. Then add the chosen Salesforce source types through existing integration credentials and a declared content/visibility contract.
Do not interpret storage notifications as an exactly-once source of truth. Test lost/duplicate notifications, cursor expiry, renames, partial inventory, access revocation, and retry after an unknown write outcome.
Qualify large-volume behavior with a declared workload:
| Workload | What to establish |
|---|---|
| Small representative collection | Interactive sample latency and useful inspection |
| Many small files/chunks | Manifest/UI pagination, worker batching, database growth |
| Large individual document | Streaming, page/window coverage, peak memory/temp disk |
| Upload beyond existing Analytics limits | Resumption and verified finalization, not a buffered request |
| Incremental change in a large collection | Reuse rate, publication cost, graph dependency invalidation |
| Multiple tenants/builds | Fair scheduling, provider limits, budget enforcement |
| Large relation neighborhood | Traversal bounds, complete evidence tasks, controlled context size |
Record file count, source bytes, pages/media duration, chunk/assertion count, tenant concurrency, refresh rate, indexing delay, memory/disk, and cost. Publish supported limits based on these measurements. “Very large” is not an acceptance criterion until those dimensions are specified.
M4 — external destinations¶
Implement Bedrock first against one selected KB mode, then Google data stores and Data 360/Agentforce according to demand and access.
Each adapter must pass the same content/identity, metadata, filtering, update, deletion, async readiness, retry, and citation conformance suite. Add its exact limits and managed processing to the UI capability profile.
Qualify two consumption paths separately:
- TrueServe's query executor over the exported index, with native graph reuse where the adapter has qualified that combination.
- The provider's own retriever/answer consumer, with any required custom tool or graph integration explicitly installed and tested.
An external answer service does not inherit the TrueServe reasoning pipeline because its documents were exported successfully.
M5 — media and advanced optimization¶
Port Gallery patterns through the artifact contract: transcription with word times, exact-time scene extraction, duplicate-frame suppression, OCR/vision, aligned evidence segments, and chapter/topic hierarchy.
Require a long-video fixture with relevant evidence near the end, a visible-only fact, an audio-only fact, and rapid scene changes. Verify actual seek timestamps and complete coverage. Test missing visual/audio evidence rather than assuming a transcript captures everything.
Then evaluate native multimodal embeddings, richer table reasoning, topic community summaries, advanced semantic splitting, and bounded parameter search. Each remains a versioned experiment against held-out evidence tasks.
Implementation seams¶
| Area | Existing entry point or proposed boundary |
|---|---|
| Agent subtype | pipeline-types.ts, repository labels/edit routing |
| Runtime feature gate | feature-flags.ts, Studio feature provider and route guards |
| Editor | components/knowledge-management, sharing flow/value/Decision controls |
| Control plane | lib/knowledge-agents and /api/knowledge-agents/{agentId} routes |
| Processing | Dedicated worker role plus fixed document/media processors |
| Source service | Existing knowledge client, with explicit raw-acquisition contract |
| Model/usage helpers | Extract tested reusable Analytics/Framework interfaces |
| Graph | Native RAGFlow graph/compilation adapter, source evidence, review records, and verified publication manifest |
| Schema | Forward migrations and generated configuration schema |
| Evaluation | Shared Test presentation; knowledge-specific cases, executor, and evidence |
| Transcript discovery | Analytics normalization/annotation/evidence interfaces plus a new multi-request extractor and complete reader |
Use standard agent CRUD/versioning rather than a second identity API. Proposed knowledge routes manage collections/sources, model/mapping revisions, builds, paginated artifacts, graph inspection, evaluations, health, publications, and query sessions. All require the existing route-auth pattern and feature gate.
Required verification¶
| Boundary | Required cases |
|---|---|
| Graph compiler | Cycles, incompatible ports, missing outputs, route/fork/merge semantics |
| Document fidelity | Table headers, qualifiers, code/list boundaries, mixed OCR, tail coverage |
| Semantic extraction | Unsupported relation, wrong negation, ambiguity, scope/version mismatch |
| Worker lifecycle | Retry, lease expiry, cancellation race, duplicate enqueue, restart |
| Destination writes | Partial batch, timeout after acceptance, read-back mismatch, deletion |
| Publication | Failed required source/stage, incomplete native graph artifact, activation/rollback |
| Authorization | Tenant/source isolation, intermediate graph nodes, citation/history revocation |
| Evaluation | Held-out leakage, source-anchor stability, unknown references, judge disagreement |
| Discovery | Multiple/implicit requests, late turns, contextual follow-ups, false group merges, source revocation, duplicate/sample counts |
| UI | Visual configuration, preserved types, insertion/reflow, diff, inspect, live chat, health drill-down |
| Feature gating | Server/client consistency, imports/duplicates/direct URLs/tools/worker disabled |
Run appropriate unit, integration, browser, and schema-drift checks for each implementation slice. Keep expensive model/provider qualification separate from deterministic CI and retain its input/version/usage evidence.
Required regressions include a question with all relevant chunks retrieved but its exception dropped during context assembly, a false terminology merge that creates a plausible wrong graph path, and a source deletion during a pinned session. These exercise failures that an HTTP-success smoke test misses.
Decisions to close and documentation¶
Before assigning dates, confirm the native RAGFlow version/ownership boundary, graph/compilation controls, initial policy schema, source access rules, review ownership, corpus scale, retention, and target quality/freshness budgets. None requires waiting to build the M0 fixtures and conformance probes.
Every implementation slice updates its matching developer handbook and service README. UI slices also add/update the user guide and navigation. New flags, worker roles, schema, and APIs require reference/operations docs and the appropriate generated specs/schema artifacts in the same change.
Keep this proposal marked Draft until the design is reviewed. Do not present planned screens or API contracts as already available in current user guides.