Intent-aware retrieval for enterprise RAG

Reviewed and updated October 8, 2026.

Finding: Labeling each question's intent before retrieval let an enterprise help assistant fit retrieval and context to the question, and the same pipeline produced per-question data on question type, product area and answer evidence.

Summary

In the proof system, every question yields three signals: its intent, its product area when it names a product-specific identifier, and the evidence behind its answer. Aggregated, they could show what kind of help people need, where the product-area mapping needs extending and what each answer rests on.
Each question gets one of five intent labels: reference lookup, definition, how-to, troubleshooting or error resolution. The label sets the context budget, and in end-user mode it also sets retrieval depth, reranking and source weighting.
The most useful findings were defects in the proof build, many of them silent. This paper lists seven, each with its status.
ALLTIPLY built a retrieval and answer pipeline for an enterprise client's product help assistant, as a working proof on the company's own documentation. It was designed to serve support staff and end users from one index.
The measurement layer that would prove wins and catch regressions was not part of the proof build. This paper sets it out in full as recommended next steps.

What the company could examine

The proof ran as a demonstration environment. Its technical and business owners could put their own questions to it and judge speed, retrieval quality and answer accuracy.

Answers traced to their own documentation. Every answer cites the passages it drew on, so owners could check retrieval against content they know well.
A signal produced on every question. Intent, product area and cited passages sat inside the proof system, not in a report. They are the raw material for the views proposed below.
A domain-tuned model to test. A test endpoint for the combined embedding-and-generation model described in lesson 4 was part of the proof, for running questions or batch evaluations against it.

What each question yields

Once the pipeline labels a question, the label is data. Each signal below is produced per question in the proof system, where it steers retrieval or shapes the answer. The proposed views are ours and were not built.

Intent

Reference lookup: one precise passage about a specific item.
Definition: what a specific term means, also from one passage.
How-to: a step-by-step procedure, favoring procedural sources.
Troubleshooting: several related passages and a lot of context.
Error resolution: precise wording, since a near miss can point to the wrong fix.

Proposed view: the intent mix across all questions, so documentation and support owners can see what kind of help people ask for and decide where to write or update content first. The label is also a natural key for evaluating results by question type, and it is planned as part of the cache key.

Product area

Questions that name a product-specific identifier are mapped to the product area it belongs to, and retrieval is filtered to that area. When an identifier has no area in the mapping, the question falls back to unfiltered, noisier search.

Proposed view: questions that named an unmapped identifier, so the team tuning the assistant can see where to extend the mapping before relying on filtered retrieval there.

Evidence

In support-staff mode, each citation shows the source type, source title and related product identifiers, so staff can judge whether to trust an answer and trace where it came from. In end-user mode, citations collapse to simple numbered references.

Proposed view: which documents carry the load and which are never cited, so documentation owners can see what the assistant relies on. It could feed the corpus audit in the measurement plan below.

What we learned

1. Users do not phrase questions the way the documentation reads

Most of the work was aligning everyday phrasing with specialized technical documentation, which took many rounds of iteration. Abbreviation dictionaries rarely cover all the shorthand users type, and keyword search misses whatever the dictionary misses. Planned next: mine query logs for the shorthand people actually type.

2. Classify intent first, then size retrieval and context to it

A shared configuration can fit different question types poorly. In this proof, the initial single 6,000-token budget gave simple reference lookups context they did not need. The team expected that extra context to add answer time, but did not record a measured timing difference. Built in the proof system:

Context budget now varies by intent. The design target was about 1,000 tokens for reference lookups and 5,000 or more for troubleshooting.
In end-user mode, reference lookups and definitions (the single-passage intents) retrieve 2 candidates instead of 3 and skip the cross-encoder reranker, which added an estimated 200 to 400 ms.
In end-user mode, procedural sources get a 1.2 boost for how-to questions.

3. One index, two answer shapes

Support staff want full citations. End users want a short answer that leads with the action. The end-user mode, which replaces full source citations with numbered references, is built in the proof system. Planned: prompt instructions to lead with the action and to skip navigation steps the user does not need, plus a lower temperature (0.05 instead of 0.1) for error resolution. Our recommendation: design for how people grade usefulness, not only correctness. Give the direct answer first, then the reason.

4. Invest in domain embeddings first

Fine-tuned domain embeddings plus the cross-encoder reranker gave substantially better retrieval on specialized vocabulary than off-the-shelf embeddings. We recorded no figures, so this is a qualitative result.

As part of the proof, ALLTIPLY also fine-tuned a single 7B model for both embedding and generation, following published research on unified embedding and generation. A seed of human-verified question-answer pairs was expanded into a much larger synthetic training set. No measured results from this model are recorded. It was described only qualitatively, so we report no results.

Working hypothesis, not a measured result: expert-verified entries can be a starting point for domain training, but the resulting model needs its own evaluation.

Research reference and implementation record

The unified embedding and generation approach is described in Generative Representational Instruction Tuning. The research authors publish GritLM code and a GritLM-7B reference checkpoint. These links identify the research approach, not the exact model build used in this ALLTIPLY proof. Its checkpoint, training-data revision and code commit are not identified in the published record.

Configuration and change record

Reported configuration or change	Status in this proof	Evidence still needed
Intent routing, candidate counts, context budgets and source weighting	Implemented in the demonstration proof as described above	Versioned configuration export and matching evaluation run
Shared tokenizer on the fallback path	Recorded as fixed in the engineering issue log	Code revision and regression examples
Procedural chunking at step boundaries	Code complete; index rebuild and cache invalidation open	Ingestion revision, index version and post-change evaluation
Domain embedding and generation model	Fine-tuned model with a test endpoint in the proof; no measured results reported	Checkpoint, training-data revision and comparison conditions

This is a summary of the published engineering record, not a supplied release manifest. Configuration values and timing estimates should not be read as recommended defaults or production benchmarks.

The defects we found

Several of the defects below caused no visible failure, so they surface only when someone looks for them. Each entry gives the defect, the general lesson and its status in the proof system.

A fallback rule that fired on perfect matches. A global rule broadened the search unless results met a 3-result minimum and a rank-fusion score threshold of 0.008. A reference lookup with one perfect match misses the minimum, so it triggers a slower, noisier search. Lesson: global fallback rules can quietly degrade good results. Status: per-intent thresholds planned.
Words counted as tokens. A fallback path counted words instead of tokens. We estimated this left the context budget 10 to 20 percent under-filled on technical text whenever it triggered. Lesson: token accounting errors waste context without any visible sign. Status: fixed with one consistent tokenizer.
Gaps in the identifier-to-area mapping. Some identifiers had no product area, so questions naming them lost area filtering and got noisier results. Lesson: check mapping tables for completeness, not only correctness. Status: gap identified, fix planned.
An unvalidated cache threshold. The semantic cache matched at 0.92 cosine similarity, a threshold not yet validated. Too strict misses paraphrases. Too loose serves wrong cached answers. Lesson: an untuned cache is a correctness risk, not only a speed feature. Status: hit-rate tracking, a review of matches and an intent-aware cache key planned.
A limit set but not enforced. The end-user answer limit of 200 words existed in configuration but was not enforced in the prompt. Lesson: confirm that every configured limit is enforced where the answer is produced. Status: a prompt instruction to stay under 150 words planned.
Wrong values visible mid-stream. Wrong product identifiers could stream to the user and be replaced only in the final event, so wrong values could flash by. Lesson: streaming shows users your intermediate mistakes. Status: buffering and validating critical tokens such as identifiers before display planned.
A chunking change with open follow-on steps. Splitting long procedural content at step boundaries was complete in code. Lesson: a chunking change is not finished until the index is rebuilt and the cache is invalidated, so track those steps with the change. Status: code complete, index rebuild and cache invalidation open, so this paper reports no retrieval effect from the change.

What to measure next

The proof system had an end-to-end evaluation of about 100 questions and baseline evaluation tooling. An end-to-end score can hide a retrieval decline while the model compensates, and some product areas had only a handful of questions, so a regression there would go unseen. Everything below was specified, not built. Each item answers one question:

Retrieval-only metrics (Recall@K and MRR@K), kept separate from answer quality so the generator cannot hide retrieval decline. Did retrieval itself get better or worse?
Coverage by product area, 10 to 15 questions each. Did a change help or hurt this area?
About 50 real end-user questions: abbreviated, action-oriented and often missing context. Does it work for the people who use it?
A release gate in CI that blocks a deploy when accuracy drops below a threshold. Ship or block?
Cache hit rate, a review of matches and a secondary cache key built from intent, product area and identifier. Where should the threshold sit?
A challenge matrix with regression reporting, plus reranker and candidate-set diagnostics. Which change broke which kind of question, and at which stage?
A view for inspecting citations and assembled context. What did this answer actually draw on?
A corpus audit that flags duplicate, stale or out-of-scope content in the live index. What should be cleaned up first?
Escalation for weak-context answers and an operator release checklist that lists the evidence required before launch. Is it ready to launch?

As general practice, have domain experts write or review evaluation questions, so the evaluation measures the system rather than the wording of the question.

The proposed 10 to 15 questions per area and about 50 end-user questions are starting points for planning coverage, not validated sample requirements. Expand the set for the range of intents, difficult cases and consequences of errors; keep acceptance examples separate from tuning data.

The system in brief

Questions often use product-specific codes and names, so documents were chunked to fit the documentation's naming conventions. Search is hybrid. Postgres full-text search weights headings above body text, an HNSW vector index runs at ef_search=100, and reciprocal rank fusion merges the two result lists. The vector index uses fine-tuned domain embeddings, and a cross-encoder reranker reorders the top candidates. The heading weighting is a corpus change, so it carries the same re-indexing caveat as the last defect above.

Method and limitations

This paper draws on one engagement: a scoped proof on one help assistant and one documentation corpus, plus ALLTIPLY's engineering issue log from tuning the proof system. The system was built and demonstrated in a proof environment, and all results come from it. This paper reports no production data and no end-to-end accuracy or answer-time figures. Items marked planned were specified, not built.

Budgets, candidate counts and thresholds depend on the corpus, so tune them on your own evaluation set. Treat this as a single-case report.

A review protocol for the help-assistant proof

Applied evaluation protocol · Version 1.0 · October 10, 2026. This is a proposed procedure derived from the published proof and its defect record. It is not a completed experiment or a record of passed release checks.

The proof implemented intent routing and citations. Its published end-to-end evaluation covered about 100 questions; that count does not establish accuracy. The record lacks the versioned configuration and matched results needed to reproduce a performance claim.

Make a comparison reproducible

Freeze the test package. Record the question-set revision, authorized corpus snapshot, document dates, ingestion and index versions, model checkpoint, prompts, tokenizer, reranker, routing settings, cache state and code revision. Record environment, run time, concurrency, timeout and retry settings. Keep acceptance questions separate from tuning examples.
Agree the questions and reference evidence. Include reference lookup, definition, how-to, troubleshooting and error resolution across the product areas in scope. A domain reviewer identifies acceptable supporting passages, expected answer content and cases that should stop or escalate. Record exclusions and missing coverage.
Run the baseline and candidate on the same inputs. Hold the corpus and environment fixed; document every intended difference. Run cache-cold and cache-warm comparisons separately. Record repeated runs where generation varies and retain failures, timeouts and incomplete responses in the denominator.
Inspect retrieval before judging the answer. Report whether known relevant passages were retrieved and their ranks. Review citation support and usable answer correctness separately. Measure first visible response and complete usable answer time separately. Keep results grouped by intent and product area so aggregate results cannot hide a weak slice.
Record the decision. Before the run, agree failure severity, acceptance criteria, who reviews disputed answers and who can authorize release. Afterward, record numerator, denominator, run conditions, unresolved defects and the decision to continue, narrow the scope or stop. A changed model, index or cache requires an explicitly scoped regression review.

Challenge the failures already found

Tests motivated by the published issue record
Case to include	What to inspect	Published status
A reference question with one exact supporting passage	Whether fallback broadens a good result and adds irrelevant context.	Per-intent fallback rules planned.
A product identifier missing from the area map	Whether unfiltered retrieval supplies the wrong product's evidence.	Mapping gap identified; fix planned.
Similar wording with different intent or identifiers	Whether the cache returns an answer for the wrong task.	Cache validation and intent-aware keys planned.
An answer containing a critical product identifier	Every visible streamed event, as well as the final answer.	Buffering and validation planned.
A procedure affected by new chunk boundaries	Matching ingestion revision, rebuilt index and cleared cache before measuring retrieval.	Chunking code complete; index and cache steps open.
Technical text on the fallback path	Consistent token accounting and a regression example.	Shared tokenizer recorded as fixed; code revision not supplied.

Tradeoff: less context and fewer retrieval steps may reduce work, but can remove evidence needed for a correct answer. A cache can save generation work while returning a stale or mismatched answer. Test quality and latency together; no measured improvement is established by this protocol.

A compact result record

For each question retain: question ID; intent; product area; reference passage IDs; baseline and candidate versions; retrieved passage IDs and ranks; citations; reviewer decision and reason; first-response and complete-answer timings; cache state; errors; and run ID. Keep protected source content in the agreed review environment. Public summaries should contain only authorized examples and aggregate results.

Evidence limit: the original question set, code/configuration revisions and run outputs were not supplied for this editorial reconstruction. The procedure can guide a new evaluation; it cannot reproduce an undisclosed historical experiment.

Source: Intent-aware retrieval: implementation, defects and proposed measurement. For the engagement boundary, read the rebuild field report.

About ALLTIPLY Labs

ALLTIPLY Labs publishes what we learn building AI systems for use at work. If you need to evaluate a help assistant, request a project review with its corpus, failure examples and current evaluation record.

Intent-aware retrieval for enterprise RAG: lessons and the data it produces