Knowledge Agent experience and evaluation¶
Status: Draft | Last reviewed: 2026-10-09
This proposal describes how you build, inspect, and test a Knowledge Management Agent. Use it to design the editor and acceptance scenarios alongside the runtime contracts.
Workspace¶
Open the agent from the existing Agents repository. Reuse its identity, version, save, publish, and preview conventions. Give the editor five main views:
| View | Primary task |
|---|---|
| Prepare | Connect sources and edit the document-processing graph |
| Answer | Configure how questions become retrieved evidence and answers |
| Test | Maintain questions, inspect failures, and compare experiments |
| Health | Measure coverage, terminology, graph support, and retrieval/synthesis effectiveness |
| Activity | Inspect builds, refreshes, index generations, errors, and cost |
A Collection panel manages shared sources, documents, topics, terminology, and an evidence-linked graph explorer. A Knowledge model editor defines the ontology, extraction fields, relation types, and review policies. Show draft configuration, current candidate build, and published release distinctly. A completed preview is not automatically published.
Offer four starting choices: Content, Questions and answers, Questions only, and Transcripts. Each works before a full graph is configured. Transcripts can come from uploads or an authorized prepared Analytics release. Offer editable recipes for web articles, document libraries, and versioned product manuals; explain what each recipe assumes.
Preparation canvas¶
Use the Analytics visual vocabulary with a new editable graph. Lay out the main flow left to right. Branch lanes continue in the same direction and rejoin explicitly. Collapse a group of steps without hiding its input/output contract.
The + control on an edge offers only steps compatible with the upstream output. Inserting or moving a step previews which downstream fields or contracts change. Provide undo/redo, duplication, keyboard navigation, and an accessible list representation of the same graph.
Cards show their purpose and configuration summary. During a run, show actual counts, such as processed/total documents, output chunks, failures, and elapsed time. Distinguish waiting, running, cached, excluded, failed, and completed work.
Selecting a card opens an inspector with:
- Settings: ordinary fields, model/integration pickers, prompt editor, metadata rows, schema builder, and typed upstream-field selectors.
- Input/output: a representative document, structural view, chunk list, and before/after comparison.
- Quality: applicable checks and the questions affected by the step.
- Run details: the actual processor/model/prompt revision, usage, and errors.
Raw JSON and expressions are optional advanced views. They must preserve typed values when you switch back. Reuse the workflow value editor and Decision controls instead of introducing another configuration language.
Run sample to here executes the real processors against a frozen selection. Let you choose representative documents rather than silently taking the first few files. Run full build shows the source count, likely expansion, estimate basis, and budget. Closing the editor does not cancel durable work.
Step library¶
| Category | Operations | Initial scope |
|---|---|---|
| Acquire | Upload/folder, crawl, bucket/CRM source binding | Uploads and websites first |
| Inspect and route | MIME, language, structure, size, metadata, optional AI assessment | Deterministic routes first |
| Parse | HTML, Markdown, plain text, PDF layout, selective OCR | Core |
| Convert | Structured document to HTML/Markdown/text | Core with fidelity report |
| Clean | Boilerplate removal, whitespace, repeated headers, explicit exclusion | Core |
| Transform | Prompt reformatting, restructuring, supported structured output | Core |
| Split | Heading/element, paragraph/line, token ceiling, table row groups | Core |
| Enrich | Metadata, parent context, summaries, keywords, alternate questions | Core subset |
| Model topics | Assign taxonomy labels, discover clusters, review hierarchy | Core policy subset |
| Align terminology | Extract definitions, propose/approve scoped mappings | Core policy subset |
| Build graph | Extract assertions/fields, resolve references, validate/project | Core policy subset |
| Combine | Merge streams, deduplicate, join named representations | Explicit merge in core |
| Validate | Required metadata, source coverage, evidence links, schema/size checks | Core |
| Deliver | Destination capability check, index candidate, export bundle | RAGFlow first |
| Discover questions | Read transcripts, extract requests, group variants, review evidence needs | Text transcripts and prepared Analytics releases in core |
| Media | Transcribe, frame extraction, OCR/vision, temporal alignment | Later |
Add semantic splitting and advanced collection summaries after the basic contracts are measured. Parent/context expansion and explicit cross-reference retrieval belong in the policy baseline. Arbitrary user code and unbounded autonomous optimization are outside the first release.
For a prompt step, expose the input unit, selected fields, model, full prompt, output type, schema, and failure policy. Make “generate another representation” different from “replace this text.” The former preserves evidence by default. The latter requires a visible comparison and checks for lost or changed facts.
Document strategies¶
| Input | Suggested strategy | Preserve and test |
|---|---|---|
| HTML | DOM/heading hierarchy, boilerplate removal, element-aware split | Heading context, links, lists, tables, original URL/element |
| Markdown | Parse its syntax tree, split on structure, cap by tokens | Code fences, lists, links, frontmatter provenance |
| Digital PDF | Layout/text extraction, sections, table-aware split | Reading order, page locations, headers, notes and units |
| Scanned/mixed PDF | Detect pages needing OCR; merge into document order | Page confidence when available, original page image, omitted regions |
| Tables/CSV/XLSX | Structured rows/cells, repeated headers in row-group chunks | Row/column identity, units, merged cells, cross-page continuation |
| Images | OCR plus optional visual description | Original image/crop, spatial reference, distinction between text and interpretation |
| Audio | Timestamped transcription, speaker mapping, topic windows | Time ranges, uncertain words, attribution |
| Video | Transcript plus scene frames, OCR/descriptions, aligned segments | Actual frame timestamps, audio/visual evidence, seekable source |
File extension alone is insufficient. A PDF may contain both selectable text and scanned pages. Use inspection before expensive OCR and let authors override the chosen strategy for specific sources/documents.
Long documents need bounded windows and hierarchical context. Show coverage of all pages/sections, including the tail. A model's successful response to the first part of a document is not proof that the whole document was processed.
Large tables need multiple evidence units, not a single oversized embedding. Carry row-group headers and notes, preserve the full table for expansion, and test questions that require comparisons across groups.
For media, distinguish searching text derived from media from retrieving native multimodal embeddings. The first path can reuse a text index. The second requires compatible embedding/query models and destination capabilities and is a later adapter contract.
Metadata and chunk inspection¶
The metadata editor uses typed rows: field name, type, source or extraction instructions, allowed values, missing-value policy, and intended uses. Preview the extracted values across documents and show disagreements and null rates. Use the same registry to populate filter controls in Answer and Test.
The chunk inspector should show:
- Exact evidence text beside the text or fields used for retrieval.
- Parent section/document, adjacent chunks, token count, and structural boundaries.
- Source, authored, and generated metadata with their provenance.
- A table/page/HTML/media view highlighting the cited source locations.
- Generated summaries and alternate questions as labeled derived fields.
- Destination IDs and any export transformations or unsupported fields.
Do not imply that a generated summary or question became filterable merely because it appears in the inspector. The destination capability check and a retrieval probe establish what was actually indexed.
Question-driven authoring¶
Each evaluation case declares a task kind: Q&A, scenario interpretation, policy comparison, cross-source synthesis, terminology validation, or change impact. It can include a question/request, context, optional answer expectation, required claims and counter-evidence, source support, metadata/access scope, and tags. Offer CSV import, typed manual editing, source-grounded generation, and transcript-derived question discovery.
| Case state | Meaning |
|---|---|
| Requirement | A question the collection should support |
| Candidate | A generated or user-provided answer/evidence set awaiting review |
| Verified | Reviewed answer requirements and source support |
| Gap | Required support is missing or contradictory in the current snapshot |
| Held out | A case reserved for final comparison, excluded from optimization input |
Keep origin and review state separate: manually supplied does not necessarily mean verified; generated does not necessarily mean wrong.
Transcripts to knowledge requirements¶
Use Question discovery within Test to select transcripts or a Call Drivers release, inspect grouped questions, and follow each occurrence to its source turns. Extract multiple requests and contextual follow-ups per conversation. Driver, intent, resolution, and friction findings help grouping and priority.
The review shows original wording, normalized question, applicable context, representative variants, unique-conversation counts, and proposed knowledge support. An observed agent answer is separate from a verified expectation. Classify knowledge needs separately from account-specific or action requests.
Approve a requirement, find supporting sources, or create a content gap without inventing an answer. The discovery contract defines complete transcript coverage, Analytics reuse, review, sampling, and refresh. This recipe feeds the evidence suite without adding a reverse execution edge to the preparation canvas.
Content to questions¶
Generate candidate Q&A from parsed source evidence before enrichment. Require answer claims to cite source locations. Check that cited spans exist, and retain the generator model, prompt, source version, and selection method.
Sample across document classes, topics, languages, long-document sections, and tables. Include questions requiring several passages, questions about exceptions, and deliberately unanswerable cases when their expected scope can be reviewed. Do not generate the entire suite from summaries alone.
Synthetic cases cannot reveal facts that the parser omitted before question generation. Include original-source fixtures and independently authored policy tasks to measure that loss.
Authors review, edit, reject, or promote candidates. Generated alternate questions used to improve retrieval are a separate artifact from evaluation cases, even if both were derived from the same document.
Questions and reference answers to content requirements¶
Decompose a case into required facts and conditions. Locate support in the current raw/parsed sources and surface contradictions. A reference answer can describe desired behavior, but does not become source truth without support.
Create an editable Evidence requirement describing:
- Which claims must be supported together.
- Which source passages or structural units support them.
- Which qualifiers, headings, table headers, units, or exception notes must remain.
- Which product/version/language/audience fields must be present and filterable.
- Whether parent expansion or retrieval of several chunks is required.
From this, suggest a splitting/enrichment change or a content acquisition task. An “ideal chunk” is a reference evidence shape grounded in those sources; it is not fabricated replacement content created to match the supplied answer.
When evidence is absent, create a content gap with an owner and suggested source type. You may draft a proposed knowledge article, but it remains a separately reviewed source. It is not automatically added to the published collection.
Questions without reference answers¶
Run evidence discovery and propose an answer with citations for review. Until verified, show retrieval diagnostics and source coverage rather than an answer correctness percentage. “No support found in this snapshot” is not proof that no answer exists anywhere.
A question can explicitly expect abstention when its scope and absence of support have been reviewed. Keep such cases distinct from unanswered requirements.
Answer canvas¶
| Stage | Author controls |
|---|---|
| Question/context | Conversation use, language handling, bounded history |
| Task and terminology | Evidence plan, concept resolution, approved aliases, ambiguity handling |
| Rewrite | Optional model/prompt; preserve identifiers, negation, dates, and scope |
| Filters | Typed user/business filters intersected with trusted access restrictions |
| Retrieve | Published target, supported search mode, candidate count, thresholds |
| Graph expansion | Definitions, references, conditions/exceptions, allowed relations and traversal budgets |
| Rerank | Optional approved model, candidate limit, selected ranking fields |
| Assemble context | Deduplication, diversity, parent/neighbor expansion, token budget |
| Answer | Model/prompt, claim/evidence output, citation contract, unresolved scope/dependencies |
Allow a basic preset and advanced step editing over the same configuration. For initial release, the graph is bounded and acyclic. Multi-query expansion has explicit limits; it cannot become an uncontrolled agent search loop.
Changing a query model or prompt does not require document ingestion. Changing the embedding space, chunk schema, or filter indexing does require a compatible new index release. The UI explains which action is necessary.
The chat pane supports rich policy comparisons and scenario analyses as well as short answers. Its inspection panel shows original and rewritten requests, term mappings, filters, retrieved ranks, graph paths, missing dependencies, selected context, and actual model usage. Scores retain their provider-specific meaning; a retrieval score is not a probability of correctness.
Clicking a citation opens the original page/element/table or seeks the media. Display extracted evidence and derived commentary separately. A chat turn can be saved as a candidate case; the answer is not automatically a golden answer.
Evaluation layers¶
Use separate measures with visible denominators, failure counts, and unscored cases. A single “RAG score” cannot tell an author what to fix.
| Layer | Measures | Required evidence |
|---|---|---|
| Acquisition | Completeness, staleness, failed sources, confirmed removals | Committed manifest and source watermarks |
| Question discovery | Missed/invented needs, context retention, grouping and role attribution | Labeled complete transcripts, occurrence anchors, reviewed groups |
| Parsing | Missing pages/blocks, reading order, table/header fidelity, OCR error on labeled samples | Original and parsed fixtures |
| Transformation | Required-fact retention, changed numbers/units/qualifiers, schema failures | Before/after evidence anchors |
| Chunking | Required evidence co-coverage, oversized/orphan chunks, duplicate rate | Structural references and evidence requirements |
| Terminology/topics | False equivalences, unresolved terms, supported topic membership, task coverage | Definition evidence and reviewed mappings |
| Graph | Supported assertions, resolved dependencies, missed exceptions, sufficient evidence paths | Versioned assertions and source clauses |
| Retrieval | Evidence recall@k, relevance precision, MRR/nDCG where labels exist | Ranked retrieved units and graded relevance |
| Filtering/access | Correct scope, missing filter values, forbidden evidence | Trusted policy and scoped cases |
| Answering | Required-fact correctness, faithfulness, citation support/completeness, abstention | Answer, context, references, and rubric |
| Operations | Throughput, latency percentiles, retries, tokens, cost, refresh lag | Recorded work and provider usage |
Represent reference evidence as source spans or structural units, not only chunk IDs from a particular splitter. Support alternate valid evidence sets and multiple required passages within one set.
For a multi-passage case, report the fraction of required evidence units retrieved and whether at least one complete sufficient set was present. Also measure the context that actually reached the answer model: retrieving a passage and then dropping it during context assembly is still a failure.
Reference-free judges can assess support in retrieved context. They cannot establish that the collection is complete or that an unsupported reference answer is true. Score unknown-reference cases separately. Record judge versions, rubrics, repeated runs where needed, and human adjudication of disagreements.
Failure diagnosis¶
Provide a case detail view with source, parsed document, prepared chunks, retrieved context, and final answer side by side.
| Finding | Next investigation |
|---|---|
| No verified support in the source snapshot | Acquire/correct source content |
| Source support exists but is absent after parsing | Parser, OCR, reading order, truncation |
| Parsed support exists but not in usable chunks | Split boundaries, table context, transform loss |
| Prepared evidence exists but is filtered out | Typed metadata, scope mapping, access policy |
| Allowed evidence exists but is ranked too low | Rewrite, search mode, embeddings, reranker |
| Retrieved evidence is missing from final context | Context budget, deduplication, parent expansion |
| Sufficient context exists but answer is wrong | Answer prompt/model, citation or abstention behavior |
| Sources disagree | Source authority/version policy and content review |
| Terms were conflated or graph dependencies missed | Crosswalk scope, relation extraction, traversal and evidence coverage |
Offer an answer with reference evidence diagnostic: feed the same answer configuration the verified supporting passages. Compare that result with normal retrieval. This helps separate retrieval and generation failures; label the diagnosis as an inference, not a guaranteed root cause.
Experiment protocol¶
Pin source snapshot, graph, processors, target generation, query configuration, ontology/topic/terminology revisions, case revisions, access context, and evaluator policy for every experiment. Changing any of them creates a distinct run.
Compare preparation variants with the same query profile, then query variants with the same index. Larger factorial experiments need explicit limits and a cost budget. Show score deltas and case-level regressions; do not hide a severe failure behind an average improvement.
Keep development questions separate from held-out questions and group related paraphrases/source topics to prevent near-duplicate leakage. Held-out expected answers and questions must not enter enrichment prompts or optimization inputs. Source documents may remain in the index; the held-out test labels stay private. For transcript discovery, group splits by conversation and related questions, and exclude held-out occurrence transcripts from enrichment/optimization inputs. Show demand-weighted and distinct-question results with sampling and review coverage.
Report reviewed-reference and synthetic-diagnostic results separately. Include sample size and uncertainty for judge-based measures. Preserve failed provider attempts and exclude them from answer-quality denominators only with a visible reason; a failed execution never silently becomes a passing case.
Initial publication policy should require a complete manifest, valid schemas and citations, no forbidden evidence in scoped tests, and agreed quality and budget thresholds. Thresholds are collection-specific, versioned, and justified by a baseline. Automated tests support access control; they do not replace its runtime enforcement.