RAG, SFT, Distillation and RLVR: Choose the Change You Need

Reviewed and updated October 8, 2026.

Choose a model adaptation method by the behavior you need to change and the evidence available to train and evaluate it. Do not treat retrieval, supervised fine-tuning, distillation and reinforcement learning as interchangeable steps in a required sequence. They solve different problems and can be combined.

Separate runtime evidence from learned behavior

• Retrieval: supply selected source material at request time. The original RAG paper connects a generator to a retrieved document index. Retrieval does not itself change model weights.
• Supervised fine-tuning: train on examples of desired outputs or language-model sequences. TRL's SFT documentation describes supported dataset forms and training objectives. The quality and suitability of examples matter.
• Distillation: train a student using a teacher's behavior or outputs. Hinton, Vinyals and Dean's distillation paper describes transferring knowledge into a deployable model; its experiments do not establish your student's task performance.
• Reinforcement learning with verifiable rewards: optimize against outcomes that a reliable checker can verify. TRL's GRPO documentation provides custom reward functions, including correctness checks. A reward function's existence does not make it a good verifier.

An illustrative document-extraction decision

Suppose a team extracts a fixed set of fields from supplier forms. This is a proposed comparison, not a training result. Start with a base model, instructions and a deterministic output validator. Keep representative forms and expert labels in a holdout set.

If the model lacks a changing supplier policy, retrieve the current policy. If it consistently formats an otherwise correct extraction poorly, compare better instructions and constrained validation with SFT on reviewed examples. If a teacher produces useful labeled extractions and a smaller serving artifact is desirable, evaluate distillation after checking model and output-use rights.

Use verifiable-reward training only if the checker can establish the required outcome. A valid JSON object is not proof that the extracted amount is correct. Exact field agreement may be checkable on labeled forms; ambiguous handwriting or a business interpretation may require expert judgment. An exploitable reward can improve the training score while worsening the actual task.

Account for the work beyond inference

List label creation, data cleaning, train/holdout separation, compute, artifact management and future retraining. Review permissions to use source data, teacher outputs and model weights. Check whether the resulting artifact can be distributed or deployed in the intended environment.

Evaluate uncommon forms, missing fields, changed policies and refusals. Test capabilities outside the fine-tuning examples where the product still relies on them. Training improvements must be compared with a simpler baseline under the same task rubric.

Prefer a bounded combination

A tuned model can use retrieval for current facts and application tools for exact records. A distilled model still needs access controls and task evaluation. Choose the smallest combination that addresses a demonstrated failure, and keep the release reversible.

For runtime knowledge choices, see retrieval, long context and tools. For adaptation work, see AI development. Request a project review with the behavior to change, available labels and the deployment rights to confirm.