Finding: an enterprise help assistant should not generate every answer. It should check a library of human-approved answers first and run retrieval and generation only when nothing in that library matches.
Summary
- Most AI help assistants send every question through the same retrieval-augmented generation (RAG) pipeline, so a common, well-understood question carries the same hallucination risk as an unusual one.
- A two-tier design separates them. The premium tier is a curated library of human-approved answers that are served word for word. The fallback tier is the RAG pipeline, used only when the premium tier has no match.
- The most important design decision is the match threshold between the two tiers. Set it too loose and approved answers get served for questions they do not answer. Set it too strict and the system behaves like plain RAG.
- The premium tier improves only through a deliberate promotion process, and it works best when product and support experts own it, not engineers.
Where this comes from
This architecture came out of ALLTIPLY's rebuild of a customer-facing help assistant for an enterprise software company in early 2026. The documentation ran to tens of thousands of pages, and a wrong answer could send a user to the wrong screen and lead them to change the wrong record. The full project is described in the product help assistant case study. This paper describes the pattern in general terms so it can be applied elsewhere.
Why one pipeline for every question is the wrong default
Question traffic to a help assistant is uneven. A small set of questions ("how do I void a transaction," "what does this field mean") makes up much of the volume. They are asked in many slightly different ways, and their correct answers rarely change.
Sending those questions through RAG means that every time, the system retrieves passages, assembles context, and generates new text. Each step can go wrong. A fluent answer that is slightly off on a high-volume question does more damage than an error on a rare one, because many more people see it.
These are exactly the questions where a human can write the answer once, check it, and serve it indefinitely.
The architecture
Premium tier: approved answers
- Entries are question and answer pairs, with source documents and metadata, written or approved by subject matter experts.
- Matching is strict: exact match, semantic similarity above a high threshold, or both.
- Every entry records who approved it, when, and on what basis. Entries can be retired.
- Answers are served exactly as approved. Nothing is generated.
Fallback tier: retrieval and generation
- Runs only when no premium entry clears the threshold.
- Returns an answer with sources and a confidence signal.
- Where the product allows, fallback answers are labeled in the interface as not yet reviewed.
- Every fallback answer is logged for the promotion process.
The promotion loop
This is what makes the system improve over time. Fallback answers that are asked often and consistently rated good are queued for expert review. Once approved, they move to the premium tier. In the engagement this pattern came from, the approved library was planned to grow from roughly 100 entries toward more than 10,000. An admin dashboard compared approved answers with generated ones, so the product team could see which questions to review next.
Setting the threshold
The match threshold between the tiers has more effect than any model choice.
- Too loose: an approved answer about voiding a transaction fires for a question about reversing one. The answer reads as authoritative, so the error is hard for the user to spot.
- Too strict: the premium tier almost never matches, and the system works like plain RAG with extra overhead.
Set the threshold per domain and per corpus, and plan to keep adjusting it. A practical starting point is to hold out paraphrased versions of approved questions, plus near-miss questions that should not match, and tune until the paraphrases match and the near-misses do not.
Trade-offs to plan for
- Cold start. Until the premium tier has meaningful coverage, the system performs like plain RAG. Plan for several months before it makes a visible difference.
- Review costs people time. Every entry needs expert approval. If the review queue stalls, the system stops improving.
- Ownership. Engineering teams that own the premium tier tend to underinvest in it. Give it to the product, support, or subject matter experts who deal with answer quality every day.
- Approved answers go stale. An answer approved six months ago may be wrong after a product release. Give entries an expiry date and tie reviews to release notes.
When to use it
- Customer-facing, regulated, or paid help experiences where wrong answers are costly.
- Domains with a stable set of frequently asked questions.
- Organizations that want non-engineers to control answer quality without touching the retrieval pipeline.
It adds little where nearly every question is new, or where content changes faster than anyone can review it.
Method and limitations
This is a practitioner write-up, not a controlled study. It is based on one enterprise implementation plus ALLTIPLY's broader retrieval work. Threshold values and ramp times vary by corpus and should be measured on your own data. For the retrieval side of the system, see Intent-aware retrieval and 12 enterprise RAG architectures.
About ALLTIPLY Labs
ALLTIPLY Labs publishes what we learn from building and running AI systems in production. If you want this pattern applied to your help or support experience, talk to us.
