Knowledge Agent qualification — 2026-10-09

Status: Native public-corpus workflow qualified and published; broader release criteria remain open
Last reviewed: 2026-10-09

This records the implemented Vanguard slice. It does not close every exit criterion in the delivery plan, or qualify future cloud, Salesforce, external-destination or full-media adapters.

Persistent demonstration

The local Vanguard agent is Public Policy Knowledge Lab, knowledge-public-policy-lab, at /agents/knowledge-public-policy-lab/knowledge. It uses the 12 core sources in the public corpus. The baseline contains 12 acquired documents and 917 verified evidence units. The separate corpus inspection also parsed the four extension documents; those extensions are not part of the baseline's answer-accuracy denominator.

Artifact	Identity / result
Baseline build	7195b2de-7068-4940-9295-16dc6b938cc5
Baseline evaluation	fabb0281-e457-4a7a-8b86-b0dde0738220
Post-recovery baseline evaluation	6366b6b0-beab-4fba-9fb2-6ad8961b1723: 8/8, required-source recall 1.0
Baseline publication	Agent version 1, retained in publication history
Graph candidate	64841c12-c64c-4e66-bfc0-9b84b5503849
Final graph evaluation	00ac8c49-4e77-4388-be78-aebc5c23fc0a: 8/8, required-source recall 1.0
Current publication	Graph build, agent version 2
Complete graph	3,254 entities / 5,418 relations / all 917 native documents

The baseline uses native hybrid retrieval with vector weight 0.5, similarity threshold 0.05, 200 candidates and up to 12 initial evidence units. Rewriting and reranking are disabled. It uses text-embedding-3-small@OpenAI in RAGFlow and the tenant's gpt-4.1-mini integration for answers and the evaluation judge. Its frozen reviewed vocabulary is empty.

Reviewed corpus expectations

Eight source-backed engineering cases cover allowability versus allocation, sponsored hospitality, Fly America cost exceptions, the Department of Defense qualification, Boston seasonal rates, first-day meal arithmetic, the travel-expense appendix, and an unrelated employer whose private policy is absent.

The baseline judge reported 8/8 passing, with mean required-source recall 1.0 across the seven cases declaring required sources. The eighth case correctly remained unresolved. The expected claims and answers were checked against frozen acquired originals and source locations, and the resulting answers were reviewed as engineering fixtures. These are not institution- approved policy advice or a statistically representative corpus-wide score. The same model family generated and judged the answers; independent domain review remains necessary. The post-recovery run repeated that automated score. Manual reading found unnecessary missing-information entries and an overbroad parenthetical about the Open Skies restriction. These wording concerns remain separate from the judge's passing result; the score does not certify every generated sentence.

The completed graph was evaluated three times. The first run, 3080904a-6d77-4764-bae2-6a33edcc128d, passed automatically but added unnecessary detail and information gaps. The shared answer contract now limits gaps to facts needed for the question and asks for direct, scoped answers. The judge now receives numbered evidence and citations and checks added claims. A second run, 855471be-0d18-431c-be9e-34a47f8a1db3, removed the unnecessary gaps but overstated the sponsor-approval condition; it was not selected for publication.

The hospitality fixture was strengthened at revision 2 to reject a universal separate sponsor-approval requirement. The answer contract explicitly preserves modal and conditional language. The final eight-case run retained the conditional approval wording, the DoD restriction, seasonal rates, first-day arithmetic, the appendix's negative reimbursement value, and the unresolved employer-policy gap. Its source-linked answers were manually reviewed before publishing the demonstration. The first-day answer states the 75% rule without a departure-time restriction but does not repeat the source's “irrespective of time” wording. Some extra detail and inconsistent inline citation notation remain. These three runs are a development sequence, not independent held-out replications.

All 400 returned evidence entries across the final eight answers matched their frozen original text, retrieval text, native identities and source locators. Every citation number was within its returned evidence list. Required-source recall is source presence, not proof that each material claim was supported. Graph expansion produced 364–901 additional candidate units; a scoped native ranking pass selected 38 alongside the 12 initial hits. This broad graph needs further relevance and ontology-quality work; graph membership alone is not used as factual support.

The baseline and graph builds contain the same 917 source units, but their prepared text is not byte-identical. An offline comparison matched every unit by source, document and section/row: 648 differ only by HTML entity decoding (& becomes &), comprising 642 spreadsheet rows and six Word units. One Word unit's paragraph-end offsets also reflect the decoded text lengths. No other text differences were found. Both frozen builds remain unchanged. The paired case results therefore qualify the two saved configurations; they are not a controlled measurement of the graph's isolated effect.

Executed boundaries

Boundary	Result
Native prepared-unit contract	Exact content and typed string/number/boolean/array metadata read back; repeat writes kept identifiers; retrieval found the fixture; fixture deleted
Native graph contract fixture	Real graph task completed; complete-export marker returned; temporary dataset deleted
Repeatable native conformance test	Initial opt-in run passed; repeated runs exposed empty retrieval and metadata-write failures during an Infinity outage. Post-recovery runs passed in 63 and 33 seconds; the latter verified distinct initial/resumed graph tasks, complete receipts, unchanged evidence, and fixture cleanup
Connected graph answer fixture	Nine native entities/twelve relations across two original documents; one initial hybrid hit expanded to both policy and exception; excluding the exception produced an explicit information gap; source-support inspection passed in the browser; fixture deleted
Native compatibility regression	Duplicate-folder children preserved; complete 300-node export; cross-tenant denial
Baseline publication	Matching build and reviewed evaluation published through the real API
MCP authorization	agents:run token answered from the publication; draft preview and dataset override rejected; temporary token revoked
MCP SDK transport	Initialize, tool discovery and query passed; empty scope unresolved; untrusted Origin rejected; temporary token revoked
Browser authoring	Create agent/source, edit Split, reject stale save, edit reviewed claims, import quoted CSV, reject anonymous reads and oversized reservation
Real corpus browser inspection	Saved parsed operation and paginated original/prepared evidence opened; completed graph showed Boston entities with original spreadsheet rows and working source links
Corpus graph integrity	917 original full-field hashes unchanged; all 917 native documents covered; zero references outside that inventory; zero pending graph writes
Graph publication	Final reviewed evaluation and matching graph build published as agent version 2
Published graph MCP	Initialize/discovery/query passed; run-only draft preview and dataset override rejected; a metadata scope matching no units remained unresolved; temporary tokens revoked
Publication lifecycle fixture	Real uploaded policy indexed/evaluated; unchanged-config refresh published after draft deletion; prior publication restored; source disable blocked answers and restoration; fixture datasets and encrypted upload deleted
Scanned PDF fixture	Real vision extraction preserved both pages, the tail-page exception, written-approval qualifier, negation and footnote; missing OCR configuration rejected
Worker lease integration	Replaced/expired/cancelled leases could not commit; current lease could; other-tenant and disabled-source checks rejected
Worker concurrency	Two real discovery jobs held separate leases concurrently and both completed while the corpus graph continued; fixture deleted
Corpus graph observation retry	The one-hour observer timeout resumed through the real API in 2.1 seconds; all 917 native unit identifiers and 60 saved artifacts were unchanged, and the existing native graph task continued
Worker process recovery	After a forced exit, the replacement claimed the expired lease in 245 seconds; all 917 units and 60 artifacts were unchanged and the native merge task continued. Parser version 4 retained the frozen version 3 index
Change-impact integration	Changed content/units selected dependent cases and mappings; changed query configuration selected all cases; other-tenant and revoked snapshots rejected
Deterministic tests	DAG rules, document and unit routing, cache invalidation, parsing/locators, typed filters, graph references, incomplete graph receipts, imports and feature gates
Database	Migration 075 applied repeatedly; isolated fresh-boot schema drift check passed
Generated configuration	Descriptions complete; generated schemas match source
Global route audit	Knowledge routes accepted; six existing findings remain in Access Manager, Allianz review and Takeda embed routes

The lifecycle check exposed and corrected an API/worker preparation-version import problem and publication paths that attempted to update a deleted draft. The final lifecycle run used the real API after refreshing the local development server. All 750 pre-existing baseline schema constraints remain present.

Question discovery produced seven candidates from a declared two-unit sample after a retry resumed its completed window. A separate two-unit terminology sample produced twelve unreviewed candidates. Some candidates lose qualifiers or confuse a category with a synonym; they must not be activated automatically. The review UI retains their source quotations.

A synthetic 120-turn transcript retained needs after turn 100, including a context-dependent follow-up. The supplied incorrect spoken answer did not become a reference answer, and a synthetic email was redacted before job storage. The initial classifier produced reviewable account/action mistakes. This is a discovery diagnostic, not a claim of complete extraction or semantic grouping.

Native service change and observed limits

Tooling remains on RAGFlow 0.26.4, now with the tested Infinity 0.7.4 engine and matching client pair. A checked overlay fixes the duplicate-folder connection-close defect and adds an authorized full graph export. The matching Tooling chart templates and compatibility notes are present. The engine/client restoration completed with one ready replica for each service. Existing datasets were retained. The six-file graph-resume and resolution overlay passed its native contract fixture and completed the corpus with 917 successful documents and zero failures.

One graph-candidate indexing attempt was interrupted by a local database deadlock while the migration ran. Its completed units were retained and verified again on retry. The corpus graph uses native per-document extraction over the synthetic unit containers. Native progress 0.6 is a stage marker, not overall corpus completion; graph compilation is substantially slower than indexing. The worker additionally checks final per-document completion totals, because this RAGFlow version can mark a task complete after individual failures. The corpus task exceeded the worker's one-hour observation window. Retry resumed observation without acquiring, preparing or indexing the corpus again.

The full corpus finished per-unit extraction and reached global graph merging. A partial-graph diagnostic exposed 64–82 candidate units for several questions, exceeding the original answer window. Graph links now form a candidate pool; large pools receive scoped native ranking, with candidate/selected counts shown in the answer. All selected evidence remains intact during synthesis.

At 18:52 UTC, Tooling's Infinity container was OOM-killed at its 4 GiB limit. It had 45 recorded restarts and lost its ready service endpoint; the graph candidate correctly failed without publication. The host had about 15 GiB available memory. The image and existing PVC were preserved while the deployment limits changed to 8 GiB / 2 CPU, with 2 GiB / 500m requests. Helm rendering and server dry-run checks passed, and the operational rollback snapshot was saved privately on Tooling. Recovery initially scanned 250,457 WAL files, with advancing read counters. The Infinity chart now includes a startup probe allowing up to 60 minutes before normal readiness/liveness checks. The rendered chart and server dry run passed; host security scanning remains enabled.

The next startup failed with exit 139 during WAL replay, without an OOM event. Infinity was then quiesced and its active data archived with a recorded SHA-256 digest. The original PVC and an older quarantine directory were preserved. An isolated physical upgrade copy also failed on Infinity 0.7.4, ruling out an image-only upgrade of the old files. Binary WAL inspection found 207 transactions after the latest checkpoint: graph-candidate operations, temporary-fixture operations, and six metadata operations for the fixture's two documents. Recovery experiments run on copies; the original archive remains the rollback source. The published baseline has no transactions in this recovery window, but still requires readback and query verification after service restoration.

Recovery on the original engine succeeded after excluding the completed temporary fixture's 54 table operations and six metadata operations from an isolated replay copy. All 147 graph-candidate transactions were retained. The fixture was positively identified by its synthetic Northbank payload; its dataset and documents were already absent from RAGFlow. The original PVC and immutable archive were not edited during these trials.

Both 917-unit inventories matched their saved identifiers and exact content hashes. A subsequent logical migration to a fresh Infinity 0.7.4 store compared every field of all 6,645 live records across five tables, including both pre-existing datasets, embedding vectors and 1,870 metadata records. All 194 indexes were recreated. A synthetic JSON write was acknowledged at commit 510, beyond checkpoint 503, before an abrupt process kill. Its typed metadata survived replay, and all five table digests still matched afterward. The coordinated restoration preserved the existing PVC and service address. All five table digests and 194 indexes matched again in the live deployment. The first native API conformance run then passed exact evidence, typed metadata, scoped retrieval, full graph export, and temporary-dataset cleanup.

The matching client dependency profile in deploy/ragflow/knowledge-agent-compat/ uses Infinity SDK 0.7.4 and Thrift 0.24.0. An isolated staging check exposed both the original SDK's incompatible protocol version and its older Thrift runtime before changing the running RAGFlow application.

The recovered combined graph covered 110 of 917 native documents. Inspection confirmed that its writer deleted all subgraph rows while replacing that snapshot, including extraction checkpoints for documents not yet merged. Read-only inspection of the immutable transaction archive recovered 917 per-document graphs, with valid record checksums and exact source IDs. Those graphs were inserted as separate document_subgraph checkpoints; their contents and all 917 original evidence rows were verified afterward. The compatibility patch preserves these checkpoints across combined-graph writes, counts resumed chunks, recognizes already-merged documents, and clears checkpoints on an explicit graph wipe. Merges save groups of up to 32 documents, reducing repeated full-graph writes. A second scan explicitly selected the earliest transaction for each document. It found 31 later derived graphs in the first selection; all 31 belong to the 110 documents already present in the restored combined graph and were skipped during the resumed merge. The original archive was unchanged. Replacement of those cached records with the earliest extractions was verified across all 917 checkpoints after recovery. At that checkpoint, the corpus still required native completion, source-coverage verification and answer qualification. Those checks subsequently completed.

The initial graph profile was v4-batch32-sdk0.7.4. At 20:43 UTC, a native merge timed out after 180 seconds during record insertion. The task was cancelled without wiping evidence or checkpoints. The replacement v5-journal-batch32-sdk0.7.4 profile records a durable write journal, repairs unfinished writes before source-ID checks, replaces old entity/relation records, and saves the graph snapshot after derived records. Cancellation drains in-flight writes before releasing the dataset lock. The default batch timeout is 900 seconds. Seven tests against the pinned source pass, including interrupted insertion, lost completion acknowledgement, duplicate prevention and repeated cancellation. The deployed replacement passed the native fixture in 37 seconds, including distinct initial/resumed tasks and unchanged original evidence. The later corpus repair and answer qualification completed as recorded below. The chart's optional client override defaults off and is explicitly enabled for Tooling's 0.7.4 engine. The packaged RAGFlow chart retains its declared Bitnami MySQL 11.1.20 and MinIO 14.10.5 dependencies; both SDK-enabled/disabled renders and Helm lint passed.

During rollout, MySQL's one-second probes timed out under startup load and triggered a clean shutdown at 20:26 UTC. An all-database logical backup was saved privately before increasing the Tooling CPU allowance from 500m to one CPU and probe timeouts to five seconds. Liveness now requires six failures, and startup has a ten-minute allowance. Recreate deployment strategy prevents simultaneous writers to its single PVC. The MySQL 8.0.39 image, memory limit, and data volume were retained. MySQL, RAGFlow, and Infinity each reached one ready replica, and the expanded native conformance test passed afterward.

The corpus repair uses the earliest verified per-document extraction and the native graph_merge, PageRank and set_graph functions. It prepares one complete recovery write rather than replaying the failed sequence of partial writes. Preparation produced 3,675 nodes and 5,891 edges over all 917 documents. The candidate's 7,892 existing records were archived, copied outside the pod and hash-verified before mutation. This is a recovery procedure, not a measured fresh-corpus build benchmark. Post-write verification matched every entity and relationship attribute, all 917 extraction checkpoints, the planned graph hash, and full-field hashes of all 917 original evidence rows. No pending journal remained. The controller copied and hash-verified its archive outside the pod. Native entity resolution subsequently completed on this recovered graph. The native writer expands the 6,012,364-byte combined graph into 623,147,874 bytes of derived document-subgraph JSON before entity/relation records. Repeated relationship attributes account for this expansion. During insertion, Infinity used its two-CPU limit and continued disk writes while the client awaited index completion. This workload does not establish interactive graph build latency or a supported large-corpus limit. At 21:34 UTC, Kubernetes resized the existing Infinity pod's CPU limit from two to four for qualification. The pod UID, container ID and zero restart count were unchanged; the live cgroup showed the four-CPU allowance. The configured two-CPU limit was restored at 22:25 UTC after native qualification. The cgroup returned 200000 100000; the pod and container identities were unchanged, with zero restarts.

The first recovery write reached a native Thrift recursion limit in a 100-name entity deletion filter. Direct serializer checks confirmed that 32 names also fail while 16 serialize. A smaller-filter trial then exposed entity names being interpreted as full-text query operators. Both failed processes retained the write journal without an OOM event. The v7-journal-literal-cleanup-sdk0.7.4 profile instead scans stored identities, matches literal names, and deletes record IDs in groups of 128. Tests cover pagination, duplicate records, operator-like names, literal separators, and preservation of unchanged graph records, original units and extraction checkpoints. The deployed profile passed native conformance in 33 seconds. Native ID-filter serialization also passed at 128 and 256 records. Full-corpus journal replay completed with zero OOM events.

A source audit also found that native entity resolution could suppress a failed provider batch and still report its candidates as resolved. The prepared v8-complete-resolution-sdk0.7.4 profile adds a sixth overlay, rag/graphrag/entity_resolution.py: provider failures and missing results fail the phase, and every pair requires one explicit yes/no judgment. Partial or duplicate judgments are rejected. Outstanding batches are cancelled and drained before retry. Graph locks now use unique execution tokens and renew every 30 seconds through long writes and cancellation cleanup; their previous 20-minute expiry was shorter than this observed write. Failed renewal cancels the owner and fails the phase. Eleven source tests pass, including a failed batch alongside two in-flight batches, a valid no-merge result, renewal during cancellation, and lost lock ownership. Both SDK-enabled/disabled chart renders, Helm lint, source compilation and server dry runs passed. Deployment generation 33 reached one ready replica after the recovery archive was verified. Native API conformance passed in 34 seconds. A separate live Redis check held a lock past its original expiry, then confirmed that lost ownership failed the operation and preserved the replacement owner's lock. Its first run had a test-only bytes/string comparison corrected for the configured decoded Redis client.

Native task dcc15c68c42b11f19b1705702f2de37b started at 21:53 UTC. It reused the recovered graph and judged all 13,287 entity candidate pairs. It selected 511 pairs for merging, removed 421 duplicate nodes and 2,521 edges, and began writing the resolved graph at 22:09 UTC. The framework observer resumed through its normal Retry API. The graph committed at 22:17 UTC, and the native task reported ok=917 failed=0 total_docs=917 total_chunks=917. Final verification found 3,254 nodes, 5,418 edges, all 917 source documents, no references outside the original inventory, and no pending write journal. Every original evidence row's full-field hash remained unchanged.

The final integrity probe also compared both pre-existing datasets and the published baseline against the recovery receipts; all hashes matched. Its first metadata-table check found twelve added rows and no changed or missing pre-existing rows. Each addition matched the exact source identity, unit hash, typed fields and 42/64-credit payload of one of six completed synthetic native fixtures. Their datasets, documents and dataset tables were already absent. The twelve rows were archived outside the pod, hash-verified, and removed by exact document/dataset identity. All 1,870 pre-existing metadata rows then matched the original table hash. Native dataset deletion can leave this metadata residue; automatic retention cleanup remains unfinished.

The first release still needs wider CDC/internal-policy domain qualification, semantic demand grouping, richer ontology/qualifier quality measures, external source ACL propagation, large-volume memory/query pagination, and automatic native/original-asset retention cleanup. Native embedding/graph usage remains in RAGFlow's provider accounting; framework answer/preparation calls use the shared usage recorder. Do not equate the tested sample limits with a scale or cost guarantee.