Finding: in enterprise retrieval, most speed and accuracy gains came from treating different kinds of questions differently. When each question type gets its own retrieval depth, context budget, reranking rule, answer length, and cache key, the system beats one that runs every question through the same pipeline, even on the same model.
Summary
- Many RAG pipelines apply one configuration to every query. That makes simple lookups slow and complex diagnostic questions short of context.
- An intent classifier at the front of the pipeline lets every later stage adapt: how much to retrieve, whether to rerank, how much context to include, how long the answer should be, and how to cache it.
- This paper walks through a seven-stage optimization program for end users of an enterprise product help assistant, with the specific settings we changed and why.
- The evaluation stage matters most and is the most often skipped. Measure retrieval separately from answer quality, or retrieval problems stay hidden until the model can no longer compensate for them.
Where this comes from
After rebuilding a product help assistant for an enterprise software company, ALLTIPLY ran an optimization program aimed at a specific user: a customer's employee on a call or at a counter who needs the right answer in a few sentences. The underlying architecture is hybrid keyword and vector search with rank fusion, a fine-tuned embedding model, a cross-encoder reranker, and approved answers served ahead of generated ones (see Premium and fallback answers).
The intent classes
Questions are sorted into a small set of intents, each needing different handling:
- Screen lookup and field definition: one or two precise passages. Speed matters most.
- How-to: a single step-by-step procedure.
- Diagnostic flow: several related passages and a lot of context.
- Error resolution: wording must be exact, because a slightly different phrasing can suggest the wrong fix.
Stage by stage
1. Corpus and chunking
- Training video scripts were split at step boundaries, so a how-to question retrieves one clean procedure instead of half of two.
- Full-text search weights headings above body text.
- Token counts were backfilled for every chunk, and inverse document frequency statistics are computed during ingestion for every source.
2. Query understanding
- Shorthand expansion. Users type the way they talk. A 15-entry abbreviation dictionary missed common shorthand, so keyword search missed too. The fix is to mine production query logs for the shorthand people actually use.
- Screen-to-product mapping. One range of screen numbers had no mapping to a product. Questions referencing those screens skipped product filtering and got noisy results. Check mapping tables for completeness, not only for correctness.
- Per-intent broadening thresholds. A global rule said to widen the search whenever fewer than three results came back. For a screen lookup, one exact match is a perfect result, yet the rule still triggered a broader, slower, noisier search. Thresholds should depend on intent.
3. Search and retrieval
- Vector index search depth was tuned for recall.
- Single-passage intents retrieve two candidates instead of three, so there is less to rerank and a smaller prompt.
- The cross-encoder reranker is skipped for single-passage intents. It added 200 to 400 milliseconds where it could not change the result.
- Procedural sources get a boost for how-to questions.
4. Context assembly
- Every intent used to get the same 6,000-token context budget. A screen lookup needs about 1,000. A diagnostic flow needs 5,000 or more. Smaller budgets for simple questions reduce time to the first token.
- A fallback path counted words instead of tokens, which left the context budget 10 to 20 percent under-filled on technical text. It now uses the real tokenizer.
- End-user mode drops breadcrumb-style citations and keeps simple numbered references. Support staff still see full citations.
5. Generation
- An answer-length limit that existed only as a configuration value is now an explicit instruction in the prompt: stay short and lead with the action.
- When the user is already on a specific screen, the prompt says so, and the answer skips navigation steps.
- Error-resolution answers use a lower sampling temperature than other intents.
- Screen numbers are checked before they are shown on screen. Without this, a wrong screen number could appear mid-answer and only be corrected at the end, which still misleads a user who is watching the answer appear.
6. Caching
- The semantic cache matched at 0.92 cosine similarity, but nobody knew whether that threshold was right. Too strict, and paraphrases miss. Too loose, and users get wrong cached answers. Track the hit rate and review a sample of hits.
- Add intent, product, and screen as a secondary cache key, so two phrasings of the same question match with confidence, and a similar-looking question in a different context does not.
7. Evaluation
- Measure retrieval on its own. An end-to-end evaluation of 100 questions can look stable while retrieval gets worse, because the model compensates, until it cannot. Add recall at k and mean reciprocal rank at k.
- Cover every product. Some products had only two or three test questions. A regression there would never show. Target 10 to 15 questions per product.
- Test with real users' questions. Build a set of about 50 real end-user questions: abbreviated, action-oriented, often missing context.
- Gate releases. Run the evaluation in continuous integration and block a deploy if accuracy drops below the threshold.
What to take from this
- Classify intent first. Every later stage improves once it knows what kind of question it is handling.
- Defaults set for the average question hurt both simple and complex questions.
- Check what the user sees, not only the final text. Streaming exposes intermediate mistakes.
- If you only measure end-to-end answer quality, retrieval problems stay hidden.
Method and limitations
These findings come from one enterprise help assistant in 2026. Corpus, retrieval, and context-assembly changes shipped first. Generation, caching, and evaluation changes are specified and being worked through. Specific values, such as token budgets and similarity thresholds, depend on the corpus and should be tuned on your own evaluation set. For the broader design space, see 12 enterprise RAG architectures and Enterprise context engineering.
About ALLTIPLY Labs
ALLTIPLY Labs publishes what we learn building AI systems that people rely on at work. If your retrieval system is accurate in testing but slow or unreliable for real users, talk to us.
