Work · 03Knowledge
Seven retrieval corpora, served into the assistant the client already used.
A field-services company's staff already used an AI assistant daily and ran their operation on seven pieces of software whose vendor documentation they could not search well — one of them with no usable search at all. The question was whether the assistant could answer "how do I do this in that tool" from the vendor's own documentation, with a citation, inside the assistant already open.
- Client
- A field-services company. Not named.
- Layer
- 3 of 4. Knowledge.
- Sources
- Seven vendor documentation sets.
- Measured
- 2026-09-19, counted from the corpus files.
Retrieval is worth building only where the cheaper substitutes fail, so those come first. A corpus under roughly 200,000 tokens — about 500 pages — belongs in the prompt with caching and needs no retrieval system. A public FAQ bot is a hosted chat widget at a monthly fee. An organisation on Microsoft 365 holding Copilot licences should test Copilot Studio grounding against its own store before paying for anything. This build exists for the remainder: several sources with separate provenance, content behind a login or with no export, citations required, a self-hosted index, and the client's existing assistant as the interface rather than a vendor's widget.
01The constraint
What was actually hard.
Not the first corpus; the seventh. Every vendor's documentation site is structured differently, extraction bugs surface weeks later as wrong answers, and the commonest retrieval failure in operating content is a chunk boundary that splits a numbered procedure. Making seven sources behave identically — so the eighth is added, not engineered — is the work.
02The mechanism
How it works.
- One pipeline shape per source, identical across all seven: ingest → extract → model → index → retrieve → serve. A new source is a copy of the folder with its own ingest and extract; nothing downstream changes.
- Plain files the client owns. Each corpus is an articles file plus a chunks file plus a manifest and a vector store — readable, diffable, re-indexable, with no vendor lock. The index is rebuildable from those files alone, on the client's own machine, which is the honest answer to "what happens when I stop paying you".
- Heading-aware chunking at 400–800 tokens, tables and numbered steps kept whole.
- A generated context prefix per chunk, 50–100 tokens naming the document and section, prepended before both the embedding and the keyword index.
- Hybrid retrieval: keyword and dense search run in parallel, fused by reciprocal rank, then reranked; the top twenty sections are returned, not a summary, so citations stay honest.
- Local embeddings — a 384-dimension open model — so the index self-hosts and a model change is a rebuild, not a project.
- Served as one search tool per corpus over the same protocol as 04, into the assistant the client already used. Search, not list-and-read.
- Acceptance is a golden-question set written by the client's own expert — the correct section named per question — re-run after every index change.
03The figures
What it measured.
Each figure carries the date it was taken. A number without a date is not on this page.
| Figure | Value | Measured |
|---|---|---|
| Corpora built and indexed, one pipeline shape | 7 | 2026-09-19 |
| Source articles across the seven | 5,899 | 2026-09-19 |
| Chunks across the seven | 15,788 | 2026-09-19 |
| Embedding | local open model · 384 dimensions | 2026-09-19 |
The published basis for the design: the vendor's benchmark, not this build's results
| Published finding | Value | Read |
|---|---|---|
| Reduction in top-20 retrieval failures, contextual embeddings | 35% | 2026-09-19 |
| Adding a contextual keyword index | 49% | 2026-09-19 |
| Adding a reranker | 67% | 2026-09-19 |
| Cost to generate the context prefixes, with prompt caching | about $1.02 per million document tokens | 2026-09-19 |
| Threshold below which the prompt replaces retrieval entirely | under ~200,000 tokens | 2026-09-19 |
First: below roughly 200,000 tokens, the whole corpus goes in the prompt with caching and there is nothing to build. That test is run per source and per client, in tokens, before anything is proposed; what justified a retrieval system here was seven sources with separate provenance and citation, at a combined size no prompt holds. Second: an organisation on Microsoft 365 with Copilot licences should test Copilot Studio grounding first, and if grounding answers the golden questions, that is the recommendation. One more fact, because buyers ask: no primary price benchmark exists for custom retrieval builds — every circulating figure is a vendor blog — so this line is priced against the outcome and the substitutes, never a "market rate".
04Doing it again
What it would take to do again.
The pipeline shape, the chunking, the context-prefix prompt, the retriever and the search tool are unchanged from client to client; the ingest and extract steps are rewritten per source, and the golden questions are always the client's. Any organisation already running one of the seven documented products can be served a corpus on day one, because the corpus files exist.
A fit
Several sources with separate provenance, citations required, and an assistant already in daily use.
Not a fit
One corpus under ~200k tokens, or a tenant where Copilot grounding answers the golden questions. This is an attach line, sold inside a pipeline or agent-integration engagement, not on its own.
Up against a constraint like this one?
Thirty minutes, no prep, no pitch. Bring the thing that is slow; if it is not a fit, I say so and point you somewhere useful.