Preference Tuning: RLHF and DPO

Updated Nov 16, 2026 · 5 min read
On this page

    After supervised fine-tuning, a model follows instructions, but its answers are uneven: some are clear, some ramble, some are confidently wrong, and a few are unsafe. Writing a perfect answer for every possible prompt is impossible. Saying which of two answers is better is easy and fast for people. Preference tuning turns those comparisons into training signal. The two names you need for interviews are RLHF (the original, reward-model-plus-reinforcement-learning approach) and DPO (a simpler loss that skips the reward model).

    How it works

    The data: preference pairs

    For a prompt, sample two (or more) answers from the current model, and have a rater (a person, or a model guided by written rules) pick the better one. Each record is (prompt, chosen answer, rejected answer). Tens of thousands of pairs is a common scale for a product fine-tune; frontier labs use far more.

    RLHF learns a reward model and optimises against it with reinforcement learning; DPO uses the same pairs directly in one loss.
    RLHF learns a reward model and optimises against it with reinforcement learning; DPO uses the same pairs directly in one loss.

    RLHF

    1. Reward model: a copy of the model with a scalar output head, trained so that score(chosen) > score(rejected).
    2. Reinforcement learning (usually PPO): the policy model generates answers to prompts, the reward model scores them, and the policy is updated to get higher scores.
    3. KL penalty: the reward is reduced the further the policy's token probabilities drift from the frozen SFT model, so it keeps writing fluent text instead of whatever games the reward.

    It is powerful, because the model learns from its own fresh samples and can find better answers than those in the data. It is also heavy: the policy, a reference copy, the reward model and a value model are all in memory, and training is sensitive to hyperparameters.

    DPO (Direct Preference Optimisation)

    DPO shows that, for the same objective, you can skip the reward model and the RL loop. It trains the policy directly on the pairs with a classification-style loss, keeping a frozen reference model (the SFT model) for comparison.

    The loss pushes the chosen answer up and the rejected answer down, measured relative to the reference model.
    The loss pushes the chosen answer up and the rejected answer down, measured relative to the reference model.
    • β sets how far the model may move from the reference: small β allows larger changes.
    • It is an ordinary supervised training loop over a fixed data set: two models in memory, easy to run with LoRA.
    • Variants change the details (no reference model, single good/bad labels instead of pairs, iterated "online" rounds with fresh samples).

    Related: RL with checkable rewards

    For maths and code, the reward can come from a checker (the answer matches, the tests pass) instead of a learned reward model. Algorithms such as GRPO compare a group of sampled answers to each other instead of training a value model. This is how many reasoning models are trained.

    Trade-offs and costs

    • RLHF: best ceiling, explores beyond the data; complex, expensive, and prone to reward hacking (longer answers, flattery, confident tone that the reward model happens to like).
    • DPO: simple, stable, cheap; limited to the answers in the data set (offline), can over-optimise and make answers longer or drift if β is too small or the data is narrow.
    • Both depend on the quality and consistency of the preference labels; disagreeing raters give a noisy signal.

    In production

    • Most teams fine-tuning open models use SFT then DPO (or a variant), often with LoRA.
    • Collect pairs from your own model on your own prompts; preferences over another model's outputs teach less.
    • Write a rating guide (what makes an answer better) and measure agreement between raters.
    • Watch length: compare answer length before and after; a judge or rater bias toward longer answers is the most common failure.
    • Evaluate on held-out preference pairs (win rate against the SFT model) plus task and safety evals.

    When to reach for it

    • SFT gives correct behaviour, but quality varies and you can tell good from bad more easily than you can write the ideal answer.
    • You need to reduce specific bad behaviours: verbosity, unhelpful refusals, unsafe content, a tone users dislike.
    • RLHF or RL with checkable rewards when you have the infrastructure and an objective you can score reliably; DPO otherwise.

    Example

    A coding assistant's SFT model gives correct answers, but they are long and often re-explain the question. The team samples two answers for 15,000 real prompts and asks engineers to pick the better one using a short rating guide ("correct first, then concise, then explains only what's needed"). They train DPO with LoRA on top of the SFT model, β = 0.1. On held-out prompts, engineers prefer the new model about two times out of three, average answer length drops by a third, and the coding eval score holds. (Numbers illustrative.)

    Common mistakes

    • Calling SFT "RLHF". Different data and different training.
    • No reference model or KL limit, so the model drifts into odd text that wins on the reward.
    • Trusting the reward model as the eval. A model optimised against it will score well on it; evaluate with fresh human or judge comparisons.
    • Ignoring length bias in raters and judges.
    • Preference data with low rater agreement, which teaches noise.

    Say this in the interview

    "Preference tuning learns from pairs where one answer was preferred. RLHF trains a reward model on those pairs and then uses PPO to maximise reward with a KL penalty to the SFT model; it's powerful but heavy and prone to reward hacking. DPO gets the same kind of objective with a single loss directly on the pairs, against a frozen reference, so it's a normal training loop. For most product teams I'd do SFT then DPO with LoRA, and watch answer length and win rate on held-out prompts."

    Interview questions on this

    1. Why compare answers instead of writing ideal ones? Comparisons are faster and more consistent for raters, and capture quality differences that are hard to write down.
    2. What does the KL penalty prevent? The policy drifting into degenerate text that exploits the reward model.
    3. What is reward hacking? Give an example. Optimising quirks of the reward, like length or flattery, instead of real quality.
    4. DPO vs RLHF in one sentence each? DPO: one supervised loss on pairs against a reference. RLHF: learn a reward, then do reinforcement learning against it.
    5. What does β do in DPO? Controls how far the trained model can move from the reference.
    6. How would you train a model on maths without human raters? Use rewards from checking the final answer or running tests, with an RL method such as GRPO.

    Used in

    • AI Engineering: How a Chat Model Is Made; Supervised Fine-tuning; Labeling and Preference Data; Evals and LLM-as-Judge; Evaluating a Fine-tuned Model and Catastrophic Forgetting.
    • AI System Design: Design a Fine-tuning Platform.