Enterprise help assistant rebuild: data and lessons

Executive summary

Client: an enterprise client exploring a help assistant built on its product documentation.
Goal: see how a new retrieval and answer pipeline would perform on that documentation, judged on speed, retrieval quality, and answer accuracy.
What the company could see and test: a working assistant on its documentation and an evidence trail on every answer. ALLTIPLY also offered a test endpoint for a domain-tuned model the company could own.
What the system recorded on our side: an intent label on every question, and a product area when the question names a mapped product term.
Main lesson: most of the work is not the model. It is matching how people ask to how the documentation is written, and finding the quiet defaults that degrade retrieval.

The problem

Help assistants built on product documentation face a common mismatch. The documentation uses formal product terms. Users type abbreviations and half-sentences.

ALLTIPLY built a new pipeline on the company's documentation so its team could test it with real questions. Closing the gap between how users ask and how the documentation reads turned out to be most of the work.

What the company could see and test

The rebuild put three things in front of the company's team: answers drawn from its own documentation, the evidence behind each answer, and the offer of a domain-tuned model it could test itself.

A working assistant on the company's own documentation

For each question the team put to the assistant, it saw the answer alongside the sources cited for it, drawn from documentation the team already knows well. A wrong answer or a weak source is visible to the people best placed to spot it.

Built for: the company's team, as evaluators. Supports: judging the approach on real questions before a larger build.

An evidence trail on every answer

In support-staff mode, each citation shows the source type, the source title, and the product terms it relates to. In end-user mode, citations reduce to simple numbered references. A reviewer can check an answer against its source, and see what kind of source backed it, instead of taking it on trust.

Designed for: support staff (full trail) and end users (simplified). Supports: whether to trust an answer, and where in the documentation it came from.

A domain-tuned model the company could own

ALLTIPLY fine-tuned a single model that handles both embedding and generation, following published research on training one model for both tasks. It was trained on a large set of question-answer pairs generated from a seed set of verified questions. ALLTIPLY offered the company's engineers a test endpoint for running their own questions or batch evaluations. An owned model can be cloud-hosted at first and self-hosted later.

Offered to: the company's engineering team. Supports: whether a domain-tuned model it could own fits its needs.

What we learned

Each lesson is stated as general practice for teams building help assistants on product documentation.

Match how people ask to how the documentation reads

ALLTIPLY made repeated review passes over the documentation, matching everyday phrasing to its terms. Shorthand lists work best when they are drawn from how users actually type, so build them from real query logs, not from the manual.

Classify the question before you retrieve

One configuration for every question serves neither simple nor complex questions well. ALLTIPLY made retrieval depth and the context budget depend on the question type. In the rebuild, the budget was set to about 1,000 tokens for a simple lookup and 5,000 or more for a multi-step troubleshooting question, in place of a flat 6,000 for every question. For single-passage questions in end-user mode, the pipeline skipped the reranker, a step estimated to add 200 to 400 ms.

Audit your defaults, not only your code

While tuning the pipeline, ALLTIPLY found that search-widening rules, a fallback that counts words instead of tokens, and incomplete mapping tables can each look harmless and still degrade results without any visible error. Check mapping tables for completeness, not only correctness. Put a measurement behind every threshold, because an untuned semantic cache, one that reuses answers to similar questions, is a correctness risk as well as a speed feature. Make re-ingestion part of the release: a chunking change does nothing until the documentation is re-indexed.

Seed with verified questions, and invest in embeddings first

Our view is that even a small set of expert-verified entries can seed a strong generated training set. On this documentation, we observed that fine-tuned embeddings handled the product's own terms better than off-the-shelf ones, with the reranker adding precision on top. Invest in embedding quality first.

Measure retrieval on its own, per area, with real phrasing

An end-to-end score can hide a retrieval decline while the model compensates. Track retrieval separately, give every product area enough questions to catch a regression, and test with real questions that are abbreviated and missing context. Define the scorecard before tuning starts, so every change can be compared against the same baseline.

Shape answers for the reader, and check before you stream

Support staff want full citations. End users want a short answer that leads with the action. A wrong detail shown mid-stream and corrected at the end still misleads, so validate critical details before display.

What the system recorded on our side

Two further signals, question intent and product area, lived inside the system for the ALLTIPLY engineers tuning it.

An intent label on every question, and a product area for mapped terms

Each question is classified by intent, such as quick lookup, definition, how-to, or troubleshooting. Questions that name a known product term are also routed to the matching part of the documentation, so search stays focused.

Together these create a per-question signal of what kind of help users ask for and which part of the product they ask about. Aggregated, that signal could show which question types and product areas drive the most demand. Some product terms had no mapping yet, so those questions fell back to broader search. Finding unmapped terms early is a useful by-product of area resolution: it shows where a mapping table needs extending.

Used by: ALLTIPLY engineers in the system. Supports: context needs per question type, per-intent evaluation, and mapping-table extensions. The unmapped terms were filed as an engineering issue for follow-up.

The measurement layer we designed

The system had an end-to-end evaluation set and baseline evaluation tooling. ALLTIPLY also designed the layer that turns these signals into release decisions:

Retrieval-only metrics (recall and mean reciprocal rank at k): did retrieval itself improve after a change.
Per-area coverage with real end-user questions: do changes help actual users, area by area.
A release gate: ship or block a release when accuracy drops.

A measurement layer like this would typically serve engineering and release owners.

How it works

Corpus: documentation re-ingested and re-chunked around the product's own terms. Two further ingestion changes, heading weighting in full-text search and step-boundary sub-chunking of procedures, are complete in the ingestion code and take effect in the index at the next re-ingestion.
Hybrid search: Postgres full-text search and an HNSW vector index, merged with reciprocal rank fusion.
Ranking: a fine-tuned embedding model and a cross-encoder reranker.
Intent routing: retrieval depth and context budget set per question type, with procedural sources favored for how-to questions.
Two answer modes: support-staff mode with full citations, and end-user mode with simplified numbered references. An answer-length limit is set in configuration.

Want to make your help assistant's answers easier to trust? Talk to us.

What an enterprise help assistant rebuild taught us