Knowledge pipeline provider research¶

Status: Draft | Last reviewed: 2026-10-09

This research assesses destination and processing options for the Knowledge Management Agent. Use it to scope adapters and implementation spikes. Provider documentation establishes available mechanisms; it does not verify the capabilities, permissions, limits, or versions of a particular deployment.

Recommendation¶

Make RAGFlow the first destination for both retrieval indexes and native graph capabilities. Keep TrueServe's source-linked knowledge bundle as the portable boundary. Qualify RAGFlow before adding another graph store. Add Bedrock next; treat Google data stores and Salesforce Data 360 as separate integrations whose managed processing must be visible in the editor.

An export that uploads documents and lets a provider split them is useful, but it is different from publishing the exact chunks inspected in TrueServe. Represent that difference explicitly:

Mode	What the adapter promises
Exact prepared units	Verified exported units and metadata survive indexing as specified
Managed document ingestion	Provider parsing/chunking applies; inspect and evaluate its actual output
Retrieval only	Use an existing index without claiming ownership of its preparation

“Exact” refers to tested content boundaries and field mapping, not identical ranking across providers or ownership of their embedding implementations.

Destination assessment¶

Destination	Documented path	Initial position
RAGFlow	Chunk/document APIs, ingestion pipelines, native graph capabilities (R1, R2, R3)	First index and graph adapter; test installed-version behavior
Bedrock Knowledge Bases	Direct document ingestion, no-chunking, custom transformation (A1, A2, A3)	Next adapter, with an explicit KB mode
Google Vertex AI Search data stores	Managed parsing/chunking; bring-your-own chunks documented as preview/allowlist G1	Managed mode first unless preview access is confirmed
Salesforce Data 360 → Agentforce	Ingestion, search index, retriever, custom chunking (S1, S2, S3)	Feasible integration; needs an org proof

Native RAGFlow¶

The HTTP reference documents adding/updating chunks and retrieval. Added chunks can include content, important keywords, and questions used for embedding. Document metadata and chunk-related APIs need a version-specific mapping R1. RAGFlow also has its own Parser → Chunker → Transformer → Indexer ingestion pipeline and sample execution R2.

Native GraphRAG already includes AI graph extraction, entity deduplication, visualization, and retrieval R3. In v0.27, Knowledge Compilation replaced the legacy GraphRAG/RAPTOR UI with Graph/Tree; previously generated content remains searchable R4. Graph templates allow configurable entity types, relationship types, and extraction rules R5. Compilation supports document- and dataset-level artifacts, including topic/structure-oriented forms R6.

Existing Tooling exposes the older GraphRAG configuration and secondary-index jobs. Detect the actual native version and qualify its corresponding APIs; do not assume the new compilation controls are present on that deployment.

Proposed first adapter:

Resolve an explicitly authorized native RAGFlow connection.
Create a candidate dataset with a pinned embedding/index profile.
Create source-document containers and add prepared chunks through the supported API, without starting an independent parser/connector over those units.
Read back the complete chunk inventory and verify IDs, content, metadata, filter behavior, and retrieval.
Run the selected native graph/compilation stages over that candidate and inspect their source links, schema, readiness, and retrieval behavior.
Publish through the TrueServe publication binding when every required artifact is verified, and retain rollback mappings.

The API proof must determine whether arbitrary per-chunk metadata is both stored and filterable, whether supplied IDs are accepted, and how keywords/questions affect embedding versus returned evidence. Do not infer these capabilities from a successful HTTP response.

If required metadata is only supported per document, test one logical chunk per synthetic document with original-document lineage. Measure the resulting object count and ingestion overhead. If that is unsuitable, a small versioned native adapter extension is preferable to direct writes into RAGFlow's private tables.

Keep the canonical evidence text in TrueServe. Use supported retrieval fields for derived material; do not concatenate all generated text into the evidence and then present it as a source quotation.

Some compilation options change retrieval chunks; for example, Tree can replace retrievable chunks using leaf clusters R5. Disable such rewriting in an exact-unit profile, or model it as an explicit preparation step and evaluate the new outputs.

Amazon Bedrock Knowledge Bases¶

Direct ingestion accepts document content/metadata for supported S3 or custom data-source paths A1. “No chunking” treats each document as one chunk; AWS notes that automatic document-page-number citations/filtering are unavailable in that mode A2. A custom transformation Lambda can implement chunking or add chunk-level metadata using S3 intermediate files; its metadata overrides colliding file metadata A3.

Proposed exact-unit path: emit one prepared unit per document, include canonical source/locator metadata, and use a compatible no-chunking data source. Where preserving a larger original-document grouping matters, evaluate a thin custom transformation that delivers the prepared units from an immutable bundle. Keep expensive preparation in the knowledge worker.

Use Retrieve with the TrueServe query/answer executor to preserve model and prompt choice. Provider-managed answer generation can be a separate optional mode, evaluated as a distinct configuration.

Discover the KB type before selecting query settings. Customer-selected vector stores have provider-specific hybrid/filter constraints A4. Fully managed Knowledge Bases have different retrieval behavior and configuration A5. Do not apply one capability profile to both.

The spike must verify content/metadata limits, supported filter operators, embedding compatibility, document deletion, indexing visibility, and source citations in the chosen account/region. The first Bedrock implementation should target one explicitly selected KB mode; other modes remain separate capability profiles.

Google Vertex AI Search data stores¶

Google documents digital, OCR, and layout parsing; per-format parser settings; chunk retrieval; and inspection of processed output. The reviewed Vertex AI Search page marks both pre-parsed imports and bring-your-own chunks as Preview with allowlist. It also says chunking cannot be enabled/disabled after data-store creation, and parser changes do not reparse existing content G1.

Proposed first path without allowlist access: export prepared source documents and metadata, explicitly select managed ingestion, and evaluate the provider's actual returned chunks. This path does not promise the exact TrueServe chunk boundaries.

If bring-your-own-chunks access is available, prove its contract and switch the adapter to an exact-unit profile only after read-back, filter, deletion, and citation tests pass. Confirm behavior for arbitrary custom chunk metadata and original document identity rather than assuming document metadata has the same scope.

Keep the product binding specific to Discovery Engine/Vertex AI Search data stores. An alternative vector-store integration would be a separate adapter, not an interchangeable endpoint hidden under the same label.

Salesforce Data Cloud / Data 360 and Agentforce¶

Data 360's Ingestion API supports streaming and bulk record loading S1. Salesforce's Agentforce grounding guide connects ingested content to a search index and retriever S2. Its custom chunking extension accepts document elements and returns ordered text chunks with optional citation metadata S3.

The appropriate destination is therefore Data 360 plus a search index and retriever, with Agentforce as a consumer. Uploading rows alone does not finish the integration.

Test two packaging paths in an enabled org:

Prepared content records with source IDs, text, metadata, and a selected field for indexing. Determine whether the configured chunking preserves the desired units and whether source fields support the required filters.
A supported document source plus a custom chunking function that preserves the prepared units and citation mapping.

The custom function's citation dictionary is not evidence that arbitrary filterable chunk fields exist. Verify their schema and search mapping separately. Keep Salesforce provisioning, data spaces, permissions, ingestion completion, index readiness, retriever configuration, and Agentforce consumption in the acceptance flow.

Source acquisition from Salesforce and publication into Data 360 are independent connectors. A Salesforce Knowledge article used as input does not imply that the output must be a Salesforce index.

Adapter capability contract¶

Persist the discovered capability profile and adapter version with each run. Recheck it before writing or publishing to detect destination drift.

Capability	Why it affects authoring
Exact chunks versus managed parsing	Whether inspected boundaries survive
Document/chunk identity and idempotency	Retry and deletion correctness
Metadata types, scope, size, operators	Valid filter/schema controls
Pre-retrieval access restrictions	Whether protected sources are supported
Retrieval/evidence field separation	Safe use of summaries and alternate questions
Embedding ownership/model/dimensions	Whether supplied vectors or re-embedding are supported
Keyword, vector, hybrid, reranking	Valid query steps
Parent/neighbor/multivector support	Valid context expansion strategies
Native graph/compilation and evidence access	Supported semantic steps, review controls, and retrieval traces
Text/image/audio/video representation	Valid modality paths
Async indexing and consistency	Readiness and verification strategy
Isolated generations and activation	Publication and rollback contract
Quotas, batch limits, deletion semantics	Scheduling and lifecycle behavior

These are capability questions, not claims that every provider supports them. Unsupported requirements either block publication or select an explicitly different managed mode. An export warning alone is insufficient if the missing feature is required for access control or a release assertion.

Keep one embedding space per compatible target index. The canonical bundle may produce several destination representations, but vectors from different models or dimensions are not combined as if they shared a space.

For each adapter implement capabilities, validatePlan, prepareGeneration, upsertBatch, deleteBatch, getStatus, verify, retrieve, and activate/retire where supported. Define a normalized retrieval result carrying canonical IDs, evidence, applied filters, raw provider rank/score, and locators.

Native graph and semantic capabilities¶

Use RAGFlow's native engine to implement the logical connected policy model. The required proof covers:

Capability	Native behavior to verify
Schema and evidence	Custom relation types, qualifiers/conditions, stable IDs, exact source references, inspect/export APIs
Terminology	Cross-document resolution, scoped aliases, reviewable merges, rejection and reversal
Topics	Dataset-level organization, stable authored taxonomy mappings, source-linked summaries
Retrieval	Graph-derived chunks versus explicit paths; query controls, supported filters, authorized intermediate evidence
Lifecycle	Build readiness, candidate isolation, refresh/delete, changed relationships, consistent graph/index publication

These are conformance questions, not assumed omissions. Prefer native configuration, then a small supported extension or review/evidence sidecar for missing controls. TrueServe need not duplicate RAGFlow's extraction, graph storage, or retrieval.

A separate graph database such as Neo4j becomes an option only when measured requirements justify it, such as an independent graph query/export contract that the native engine cannot reasonably expose. Existing Neo4j infrastructure alone does not establish that need.

For later cloud vector destinations, qualify native graph reuse through TrueServe's query executor only if the graph APIs, canonical IDs, and access boundaries support the combination. A separate provider graph adapter is another future option.

Content export does not automatically give a provider's own answer service the TrueServe terminology resolver, graph traversal, or evidence-completeness checks. For Agentforce or another external consumer, either expose the published Knowledge Agent as an authorized tool or qualify a provider-native equivalent. Test semantic behavior separately from successful ingestion.

Keep legacy GraphRAG, current Knowledge Compilation, and any future external graph as explicit capability profiles. Their schemas, APIs, and lifecycle may differ.

Acquisition roadmap¶

Source	Reuse	New work needed
Website	Tooling crawl4ai client and scope controls	Preserve HTML/locators, manifest/checkpoint, confirmed deletion semantics
Files/folders	Framework storage/auth patterns and existing upload UX	Resumable direct upload, large manifests, checksums, path identity
S3	Integration selection and source configuration	Native raw enumeration/fetch, version/change handling, reconciliation
GCS	Integration selection and source configuration	Generation-aware enumeration/fetch and reconciliation
Salesforce	Tooling registry plus Framework integration patterns	Framework source type, selected-object/content contracts, versions/access/deletes
Azure/Confluence	Existing knowledge source configuration	Acquisition contract and access behavior when prioritized

Do not make RAGFlow the fetcher for every destination. That would force content through its ingestion before TrueServe could process it. Reuse source configuration and algorithms while making acquisition an independently testable boundary.

For Salesforce, define the intended input first: published knowledge articles, attached content, or selected business records. Include stable IDs, language, version/publication state, and visibility in that source contract. A snapshot index of CRM records has different freshness requirements from a manual library.

Parsing and media choices¶

Use the fixture results to choose a default parser, with per-document overrides. The initial candidates are existing RAGFlow parsing in managed mode and an independent processor such as Docling for pipeline-owned structured output. Docling documents hierarchy, table/picture objects, provenance, OCR, and Markdown/HTML/JSON export (P1, P2).

Docling is a starting candidate, not a claim of superior accuracy on TrueServe documents. Compare table headers/merged cells, reading order, scanned pages, language coverage, memory, throughput, and source-location fidelity.

Retain managed OCR/parser adapters for difficult inputs. Existing workflow OCR/provider integration patterns are useful, but knowledge jobs also need asynchronous large-document handling and complete page coverage. Standardize their result representation rather than requiring the same parsing engine for every input type.

The Gallery assessment identifies reusable transcription, frame OCR/description, and chapter patterns. For the first media milestone, produce text evidence linked to real timestamps. Native multimodal retrieval should follow a separate measured experiment.

Evaluation research¶

RAGChecker separates retrieval and generation diagnosis at a finer level than a single end-to-end score E1. Ragas documents reference-based context recall, including reference-context and ID-based variants E2.

Use these as metric/evaluator adapters, not as the product's database model. TrueServe still owns frozen cases, source anchors, permissions, run evidence, judge provenance, and release policy.

The proposed diagnostic flow is a design inference from those approaches: compare source support, parsed support, chunk coverage, retrieved context, and answer claims. Measure table/locator fidelity and source freshness separately; an answer judge alone cannot validate them.

Primary sources¶

Reviewed on 2026-10-09. Pin library/API versions in implementation and repeat contract probes against the selected deployment.