Labeling and Preference Data

Updated Nov 16, 2026 · 5 min read
On this page

    A team pays a vendor to label 10,000 chatbot answers as "helpful" or "not helpful" and trains a reward model on them. The model barely beats chance. An audit finds three raters who answered in under two seconds per item, guidelines that never said what to do with answers that were polite but wrong, and responses always shown in the same order, so "pick the left one" was an easy habit. Human labels and preference data are the input to fine-tuning, reward models and evaluation, and their quality comes from the labeling program around them: guidelines, calibration, agreement checks and aggregation.

    How it works

    Guidelines, a calibration pilot, production with gold items, adjudication and versioning, with low agreement fed back into the guidelines.
    Guidelines, a calibration pilot, production with gold items, adjudication and versioning, with low agreement fed back into the guidelines.

    Kinds of labels

    • Classification and extraction: intent, topic, safety category, entities. Ground truth for classifiers and evals.
    • Demonstrations: humans write the ideal response; used for supervised fine-tuning (see Supervised Fine-tuning).
    • Ratings: a score per response on a rubric (correctness, helpfulness, tone). Useful but noisy: raters use scales differently.
    • Pairwise preferences: which of two responses is better. Easier and more consistent for humans than absolute scores; the standard input for RLHF reward models and DPO (see Preference Tuning: RLHF and DPO).

    The labeling program

    1. Guidelines with definitions, worked examples, edge cases and an explicit "unsure / can't judge" option. Most disagreement comes from unclear guidelines.
    2. Pilot and calibration: every rater labels the same 50–100 items; compare, discuss disagreements, and fix the guidelines.
    3. Production with quality controls: hidden gold items with known answers, a fraction of items labeled by 2–3 raters to measure agreement, and time-per-item checks.
    4. Adjudication: domain experts resolve disagreements on the overlap set and on flagged items.
    5. Versioning: each label carries rater ID, guideline version and timestamp, so you can drop a bad rater's work or re-label when guidelines change.

    Preference data

    A preference record: prompt, two responses, the choice, its strength and a reason; Bradley-Terry turns many pairs into scores.
    A preference record: prompt, two responses, the choice, its strength and a reason; Bradley-Terry turns many pairs into scores.

    A record holds the prompt, two responses (from different models, sampling settings or prompt versions), the choice, optionally its strength and a short reason, plus "tie" and "both bad" options. Show responses in random left/right order. Many comparisons are combined with the Bradley-Terry model, P(A ≻ B) = σ(score A − score B): a reward model learns such scores from pairs for RLHF, while DPO trains the policy model directly on the pairs.

    Measuring label quality

    Raw agreement and Cohen's kappa for a careful pair of raters and for a rater answering at random; scores fitted from noisy pairs still recover the true order.
    Raw agreement and Cohen's kappa for a careful pair of raters and for a rater answering at random; scores fitted from noisy pairs still recover the true order.
    • Inter-annotator agreement: raw agreement is misleading because random answers already agree 50% of the time on two options. Cohen's kappa (two raters) or Krippendorff's alpha (many) correct for chance. In the included preference_demo.py, two careful raters reach 65% agreement and kappa 0.30 on close pairs; a coin-flipping rater reaches 50% and kappa 0.00. Preference tasks have genuinely hard pairs, so moderate agreement is normal; near-zero means a broken rater or guideline.
    • Gold accuracy per rater, and drift over time.
    • Aggregation: majority vote or rater-weighted models for categorical labels; Bradley-Terry for pairs. In the demo, 400 noisy comparisons still recover the true order of five responses.

    LLMs in the loop

    LLM judges can pre-label, flag likely errors, or produce AI-feedback preferences at scale (RLAIF). Calibrate them against human labels on a sample, watch for their known biases (position, length, self-preference) and keep humans on high-stakes and disputed items.

    Trade-offs and costs

    • Experts vs crowd: experts are slower and costly but needed for medicine, law, code correctness; trained general raters work for tone and basic helpfulness.
    • Pairwise vs ratings: pairwise is more reliable per judgement but needs more comparisons to rank many items.
    • Overlap rate: labeling items 2–3 times costs more but is the only way to measure quality.
    • Speed incentives: paying per item rewards speed; mix in gold items and time checks.
    • Privacy: production conversations need consent, redaction and access controls before raters see them.

    In production

    • A labeling tool (open-source or vendor) with gold items, overlap, rater analytics and versioned exports.
    • Dashboards: agreement per task, gold accuracy per rater, time per item, label distribution over time.
    • Re-calibrate when guidelines or products change; keep old labels tagged with their guideline version.
    • Sample production failures (thumbs-down, escalations) into the labeling queue, so labels follow what users actually hit.

    When to reach for it

    Building eval sets, fine-tuning datasets, reward models or classifiers; auditing an existing dataset; and "how would you collect data to improve the model?" interview questions.

    Example

    The team from the opening restarts: written guidelines with 30 worked examples and a "polite but wrong = not helpful" rule; a 100-item calibration round (kappa rises from 0.18 to 0.52 after two guideline revisions); 10% of items triple-labeled; 5% gold items; raters below 80% gold accuracy retrained or removed; pairwise comparisons with randomised order instead of binary ratings. The new reward model agrees with expert judgements 74% of the time on a held-out set, up from 55%. (Example numbers.)

    Common mistakes

    • No guidelines or examples for edge cases.
    • Measuring raw agreement instead of chance-corrected agreement.
    • Fixed left/right order in pairwise tasks.
    • No gold items, so careless raters go unnoticed.
    • Mixing labels from different guideline versions without tracking them.
    • Trusting an LLM judge without calibrating it against humans.

    Say this in the interview

    "Label quality comes from the program: clear guidelines with edge cases, a calibration round, gold items and overlap in production, adjudication by experts, and versioned labels with rater IDs. For preference data I'd collect pairwise comparisons with randomised order, ties and reasons, measure agreement with kappa rather than raw agreement, and aggregate with Bradley-Terry; those pairs train a reward model for RLHF or go straight into DPO. LLM judges can pre-label at scale, but only after calibration against human labels."

    Interview questions on this

    1. Why use pairwise preferences instead of 1–10 ratings? People compare more consistently than they score; scales vary per rater.
    2. What does Cohen's kappa correct for? Agreement expected by chance given each rater's label rates.
    3. How do you catch low-quality raters? Gold items, agreement with peers, time per item.
    4. How do preference pairs become a training signal? A Bradley-Terry reward model for RLHF, or directly in DPO.
    5. What biases do LLM judges have? Position, verbosity, self-preference; mitigate by swapping order and calibrating.
    6. How do you choose which items to label? Production failures, uncertain model predictions, and coverage of important slices.

    Used in

    • AI Engineering: Synthetic Data for LLMs; Preference Tuning: RLHF and DPO; How a Chat Model Is Made; Evals and LLM-as-Judge; Online Evaluation and A/B Tests for LLM Features.
    • AI System Design: Design an Eval Platform; Design a Fine-tuning Platform.