Approved and generated answers: the data a two-tier help assistant can give you

Reviewed and updated October 8, 2026.

Finding: A help assistant that serves approved answers first and generates only as a fallback is also a measurement system: each answer's source label, broken out by product area, can show owners where help content still rests on generated text.

Summary

• A two-tier help assistant checks a library of human-approved answers first. Retrieval and generation run only as a fallback.
• The split can support source control and review. Our design judgment is that its owner-facing data may also be useful: a source label on each answer, a review queue for experts and a log of near-misses at the match threshold.
• We recommend starting with answer-source share by product area to identify where expert review may be needed. This is an author recommendation, not a tested ranking of reports.
• What our demonstration proof gave the client: an assistant on its own documentation, a per-answer evidence trail in two modes, and a domain model trained on that documentation, with a test endpoint offered.
• Main lessons: match approved answers to real user phrasing, give every threshold its own evidence, and treat verified answers as seed data for training.

Where this comes from

This paper draws on a short demonstration proof ALLTIPLY ran for an enterprise client: a retrieval and answer pipeline on the company's documentation. That pipeline could serve as the generated fallback in a two-tier design. The approved-answer tier and promotion loop described below are recommendations, not features delivered in this proof.

What the proof gave the client

The proof gave the company three things to use in a demonstration environment, plus a domain model with a test endpoint offered.

• A working assistant on its own documentation. The company's evaluators could put their own questions to it and judge speed, retrieval quality, and answer accuracy on material they know well. Decision it supports: whether answer quality on real questions meets the bar for the company's users.
• A full evidence trail for support staff. In support-staff mode, each citation showed the source type, the source title, and the parts of the product it relates to. Decision it supports: whether staff can trust an answer before passing it on, and where in the documentation it came from.
• Simple citations for end users. In end-user mode, citations collapsed to numbered references, so answers stay uncluttered and can still be traced to a source. Decision it supports: whether a short answer is backed by the documentation.
• A domain model, with a test endpoint offered. A single model, trained on the company's documentation, handles both embedding and generation. ALLTIPLY offered a test endpoint so the company's team could run its own questions or batch evaluations against it. Decision it would support: whether a domain-tuned model earns its training effort compared with a general-purpose one.

The pattern in brief

• Approved tier. Question and answer pairs approved by subject matter experts. We recommend storing each with its sources and an approval record (who, when, and why), matching strictly, and serving answers exactly as approved.
• Generated fallback. Retrieval-augmented generation (RAG), used only when no approved answer clears the match threshold.
• Promotion loop. Generated answers that are asked often and rated well go to expert review. Approved ones join the approved tier, and experts retire entries that no longer hold.

In our proof, the retrieval and answer pipeline used hybrid search. Keyword search (Postgres full-text) and vector search (an HNSW index) were merged by reciprocal rank fusion. Heading weighting for keyword search is applied at ingestion. Retrieval used a fine-tuned domain embedding model. A cross-encoder reranker ordered the results, except for end-user questions that a single passage answers, where skipping it avoids an estimated 200 to 400 ms. Tuned embeddings retrieved better than off-the-shelf ones on the product's own terminology. This was a qualitative observation, not a measured result.

Each query also carried a question-type label and, where it used a mapped product-specific term, a product area. ALLTIPLY engineers used the question type to set retrieval depth, reranking, and context size, and the product area to filter retrieval to the relevant part of the documentation. These labels are also the raw material for the owner reports below.

Research reference and implementation record

The unified embedding and generation approach is described in Generative Representational Instruction Tuning. The research authors publish GritLM code and a GritLM-7B reference checkpoint. These links identify the research approach, not the exact model build used in this ALLTIPLY proof. Its checkpoint, training-data revision and code commit are not identified in the published record.

The proof supplies no controlled model comparison. A synthetic training set and a shared embedding/generation architecture do not establish that the tuned model improves either task.

The data a two-tier design can produce

These views are general practice for two-tier assistants. ALLTIPLY did not build them in this project.

Answer-source share by product area

Label every answer approved or generated. Store the question type and product area with the label, and roll the labels up by area. The result is the answer-source share: one number per area, the portion of its questions answered from reviewed content. Each area's owner can see how much of that area still rests on generated text. Joined with the question type, the same data shows which kinds of questions the library covers. An area might have its simple how-to questions covered while its troubleshooting questions still depend on generation. Decision it supports: which area gets expert review time next, and for which question types.

Read the share with care. A low share in a young area can mean the library is new, not that answers are poor. A high share means little if the approved entries are out of date.

A review queue from generated answers

Log each generated answer with the question, the sources it cited, its question type and area, how often similar questions arrive, and how users rate it. Sorted by volume, the generated tier becomes a work list for experts. Frequent answers that rate well are candidates to promote. Frequent answers that rate poorly often point to a documentation gap, which a new approved answer would not fix at the source. Decision it supports: what experts review this week, and whether the fix is a new approved answer or better documentation.

A near-miss log at the match threshold

Log the questions that score just above and just below the match threshold, each with the approved entry it nearly matched. Have experts label a sample: should this question have received that answer? Paraphrases that fell below the line mean the threshold is too strict. Different questions that cleared it mean it is too loose. Keep the log by question type, since a simple how-to question and a troubleshooting question may need different thresholds. A semantic cache that reuses answers to similar past questions needs the same evidence: its hit rate and an expert-reviewed sample of its matches. Decision it supports: where to set each threshold, based on labeled examples rather than a default.

Evaluation that separates retrieval from answers

End-to-end evaluation alone can miss regressions in individual product areas, and a strong generator can hide a decline in retrieval. For any two-tier assistant, we recommend four practices.

• Retrieval-only metrics (Recall@K and MRR@K, which show whether the right source was found and how high it ranked), tracked apart from answer quality.
• 10 to 15 test questions per product area.
• About 50 real end-user questions, written the way users type them.
• A release check (CI gate) that blocks a release when accuracy drops below a threshold.

Decision it supports: whether a change helped retrieval, answers, or neither, area by area, and whether a release ships.

The suggested 10 to 15 questions per area and about 50 real user questions are planning starting points, not validated sample requirements. Add coverage for difficult cases and consequential failures, and retain a held-out acceptance set. No release threshold has been validated by this report.

What we learned

1. Approved answers must match how users actually ask

Users do not phrase questions the way documentation reads. Much of the work in our proof went into aligning everyday phrasing with the documentation. Keyword search misses whenever users type wording the index does not cover. In a two-tier design, the same gap hits the match step: approved answers written in documentation language will miss users' own wording. The fix is to mine production query logs for the wording users type, and to feed the same list into approved-answer matching.

2. Every threshold needs its own evidence

The match threshold between tiers needs its own evaluation. If it is too loose, an approved answer about resetting another user's password could be served for a question about changing your own. The approved label may create false confidence. If it is too strict, useful approved entries are missed. A fallback that widens retrieval whenever too few results return can also replace a strong single match with a noisier result set. Tune thresholds against real paraphrases and near-misses, separating question types where needed.

3. Verified answers are training data, not only a serving layer

The expert-verified pairs in a proposed approved tier can also seed training data. In our proof, a verified seed was used to generate a larger synthetic question-answer set, then to fine-tune one model for embedding and generation following GritLM. This records the training approach, not a measured improvement. Serving an approved answer directly and testing whether it helps training are separate decisions. See RAG vs fine-tuning vs long context and custom AI development.

4. Correct is not the same as useful

In this proof, a configured answer-length limit was not reflected in the prompt. Add a concise-answer instruction and test whether outputs follow it; a prompt instruction does not guarantee a strict limit. Where a strict limit is required, validate the completed output in application code and handle noncompliance explicitly, for example through a bounded retry or a fallback response. Check that any shortening retains necessary evidence and meaning. Have domain experts write or check test questions so they reflect tasks users perform.

5. Per-area reporting needs a complete area map

Any per-area view depends on mapping each question to an area. Mappings from product terms to product areas can be incomplete in ways that are hard to spot, and a single missing entry can affect many questions. A question that uses an unmapped term loses area filtering and falls back to broader, unfiltered search. In an answer-source report it would also have no area to count against, so the gap would stay invisible. Mapping tables need a completeness check, not only a correctness check.

Trade-offs and fit

• Cold start. Until the approved tier covers real traffic, the system performs like plain RAG and the answer-source share looks low.
• Review costs expert time. If the queue stalls, the system stops improving.
• Ownership. The approved tier works best when the experts who know each product area own it.
• Staleness. An approved answer can go wrong after a product release. Give entries a review date and tie reviews to release notes. A software change does not update answers until the corpus is re-ingested, so make re-ingestion a standard step in every release.
• Fit. Best where wrong answers are costly and questions recur, not where most questions are new.

Method and limitations

This is a practitioner write-up from one engagement, not a controlled study. The work was a short proof in a demonstration environment, so we report no production outcomes, controlled comparisons, or measured model results. Latency figures are engineering estimates, not benchmarks. ALLTIPLY's part was the retrieval and generation pipeline, including intent-aware retrieval and the two response modes, and a domain model offered for testing through an endpoint. The approved tier, the owner views, and items marked recommended describe the general pattern. Related reading: RAG architecture selection and enterprise context engineering beyond basic RAG.

Related reading

For the question-routing method and proof limitations, see intent-aware retrieval for enterprise RAG. For implementation scope, see enterprise knowledge assistants and custom AI development.

About ALLTIPLY Labs

ALLTIPLY Labs publishes what we learn building AI systems. To scope an approved-answer and fallback evaluation, request a project review with recurring questions, review owners and the current documentation.