Executive summary
- Client: an enterprise software company with a large suite of business applications and tens of thousands of pages of product documentation.
- Problem: the in-product help assistant took tens of seconds to answer and gave a usable answer only about half the time.
- What we built: a full rebuild of the retrieval and answer pipeline on the same documentation, in three weeks. It includes a fine-tuned embedding model, a custom reranker, intent-aware retrieval, and a two-tier answer design that serves human-approved answers first.
- Result: answers in 3 to 5 seconds in the working prototype, with better retrieval quality and answer accuracy on the benchmark set. Some domains still needed more work, which set the priorities for the optimization program that followed.
- Status: delivered as a proof in March 2026. A seven-stage optimization program for end users followed.
The problem
The company's applications are dense. Each product line has its own screens, fields, codes, and workflows, and the documentation reflects that: tens of thousands of pages across user guides, field definitions, training video scripts, and troubleshooting notes.
Two groups ask the help assistant questions, and they need different things:
- Technical support staff want depth, full citations, and diagnostic detail.
- Customers' own employees are often on a call or standing at a counter. They type the way they talk, using shorthand and screen numbers, and need the answer in three sentences. A wrong screen number sends them to the wrong place and can lead to changes on the wrong record.
The existing pipeline served both groups the same way, slowly. It had been tuned in place for months without a clear path to reliability.
Why we rebuilt instead of tuning
When a retrieval system has been tuned for months and is still slow and unreliable, the problem is usually structural: how the documents were split, which embedding model is used, and how the system decides how much context to retrieve. Adjusting prompts on top of that has limited effect.
ALLTIPLY proposed a fixed-length rebuild on the same corpus, measured against the existing system. The decision framework behind that choice is in Rebuild or tune?
What we built
Corpus and indexing
- Re-ingested and re-chunked the documentation with a strategy tuned to the product's screen and code conventions. Training video scripts were split at step boundaries so a "how do I" question retrieves one clean procedure.
- Hybrid search: keyword search with headings weighted above body text, plus vector search, merged with rank fusion.
- A fine-tuned embedding model trained on the company's own vocabulary. Off-the-shelf embeddings treated product codes and screen names as noise. The tuned model did not.
- A custom reranker on top of retrieval to put the most precise passages first.
Query understanding
- An intent classifier sorts each question into types such as screen lookup, field definition, how-to, diagnostic flow, and error resolution.
- Shorthand and abbreviations are expanded before search, and screen numbers are mapped to the application they belong to, so results can be filtered to the right product.
Intent-aware retrieval and generation
- Each intent gets its own retrieval depth and context budget. A screen lookup can be answered with about 1,000 tokens of context. A diagnostic flow needs 5,000 or more. Smaller budgets for simple questions mean faster answers.
- The reranker is skipped for single-passage intents, saving 200 to 400 milliseconds where it adds nothing.
- Procedural sources are boosted for how-to questions.
- End-user mode keeps answers short, leads with the action, and simplifies citations. Support-staff mode keeps full citations.
- A semantic cache returns instant answers to repeated questions, since the same questions come up across many customers.
Human-approved answers first
The system checks a library of human-approved answers before generating anything. If a question matches an approved answer closely enough, that answer is served as written. Only when nothing matches does the retrieval and generation pipeline run. An admin dashboard compares approved answers against generated ones, so the product team can see where to add answers next. The pattern is written up in Premium and fallback answers.
Results
- Latency: 3 to 5 seconds per answer in the prototype, compared with tens of seconds before.
- Quality: measurable improvements in retrieval quality and answer accuracy on the benchmark questions.
- Honest gaps: reviewers found answer quality still uneven in some product areas, where domain details were not yet captured. That feedback shaped the next phase instead of being hidden.
What came next
The proof led to a seven-stage optimization program focused on end users: corpus and chunking, query understanding, search and retrieval, context assembly, generation, caching, and evaluation. Corpus, retrieval, and context assembly work shipped first. The evaluation stage adds retrieval-only metrics (recall and mean reciprocal rank at k), more test questions for under-covered applications, a dedicated test set of real end-user questions, and an evaluation gate that blocks a release if accuracy drops. The technical detail is in Intent-aware retrieval.
What was hard, and what we would do differently
- Lock the benchmark before you deliver the proof. Agree on the method, question set, and date in writing up front. Once one side looks likely to win, benchmarks tend to slip.
- Build the proof with the internal team. A rebuild that outperforms an internal effort creates political cost for the people who own that effort. Building it together works better for everyone.
- Stream carefully. A wrong screen number that flashes on screen mid-answer and is corrected at the end still misleads the user. Validate critical tokens before showing them.
Related
Stuck on a retrieval system that will not get reliable? Talk to us.
Related services: Enterprise Knowledge Assistants and AI Assistants.
