Synthetic Data for LLMs

Updated Nov 16, 2026 · 5 min read
On this page

    A team needs 20,000 examples to fine-tune a support model on their product's tone and tools, and they have 300 real conversations. They ask a strong model to write the rest. The fine-tuned model sounds great in demos, but in production it confidently quotes refund rules that don't exist (the generator made them up), handles only the five scenarios the prompt mentioned, and phrases everything the same way. Synthetic data works, and most modern training pipelines use it, but only as a pipeline with seeds, verification, deduplication and real-data anchors, not as a single prompt.

    How it works

    Seeds go into generation; a verifier and diversity checks decide what survives; eval failures feed back as new seeds.
    Seeds go into generation; a verifier and diversity checks decide what survives; eval failures feed back as new seeds.

    What it's used for

    • Instruction and chat data: prompts and responses in the style and domain you need.
    • Distillation: a large "teacher" model's answers (sometimes with reasoning) train a smaller "student" (see Distillation: Making a Small Model Act Big).
    • Hard and rare cases: edge cases, adversarial inputs, rare languages, safety refusals, which real logs rarely contain.
    • RAG and search: question–answer pairs generated from your own documents, to tune retrievers or build eval sets.
    • Code and math with checkable answers: problems whose solutions can be executed or verified, the best case for synthetic data.
    • Preference data: pairs of responses ranked by a judge model (AI feedback) to complement human labels (see Labeling and Preference Data).

    Generation

    1. Seeds carry the diversity: real examples, a taxonomy of topics and skills, user personas, difficulty levels, and your documents. Without varied seeds, outputs collapse to a few patterns.
    2. Prompts ask for both the input and the output, with constraints (format, length, difficulty). Methods like self-instruct (the model proposes new tasks from a few seeds) and evolving instructions (rewrite tasks to be harder or more specific) expand coverage.
    3. Grounding: for domain facts, generate from your documents, not from the model's memory, or the data will contain invented policies.

    Verification and filtering

    In the included demo, a verifier removes every wrong answer, but the remaining data still comes from only three templates.
    In the included demo, a verifier removes every wrong answer, but the remaining data still comes from only three templates.
    • Programmatic verifiers wherever possible: unit tests for code, exact answers for math, schema validation for JSON, tool calls replayed against a sandbox.
    • Rejection sampling: generate several candidates per prompt and keep the ones that pass the check (or score best).
    • LLM judges for open-ended text, calibrated against human ratings on a sample (see Evals and LLM-as-Judge).
    • Deduplication and diversity: near-duplicate removal (see Building a Training Data Pipeline) and coverage checks over topics, lengths and difficulty.

    The included synth_filter.py generates 1,000 word problems with a generator that's wrong 20% of the time: the verifier removes all 213 wrong items, dedup leaves 772, and yet every item comes from 3 templates. Correct is not the same as diverse.

    Model collapse

    Repeatedly training on model-generated data narrows the distribution; rare cases vanish first.
    Repeatedly training on model-generated data narrows the distribution; rare cases vanish first.

    Training models on the outputs of earlier models, generation after generation, loses the tails of the distribution: rare words, unusual cases, minority styles. Research on recursive training has shown this degradation. Practical defences: keep real data in every mix, filter for quality and diversity rather than generating blindly, and never let synthetic data replace your real evaluation sets.

    Trade-offs and costs

    • Cheap volume vs hidden errors: generation is fast; verification is where the effort goes, and it is task-specific.
    • Teacher bias: the student inherits the teacher's mistakes, style and refusals.
    • Licence terms: some model providers restrict using outputs to train competing models; check the terms of the model you generate with.
    • Contamination: synthetic data generated by a model that saw your benchmark can leak it; decontaminate as with real data.
    • Overfitting to the generator's style: blend with real examples and evaluate on real traffic.

    In production

    • Treat it like any dataset: version the seeds, prompts, generator model, filters and outputs together.
    • Measure pass rates per filter and per topic; low pass rates point at a bad prompt or a hard area.
    • Human-review a random sample of every batch.
    • Evaluate on real, held-out data, never on synthetic data from the same generator.
    • Close the loop: failures found in evals and production become new seeds.

    When to reach for it

    When real data is scarce, expensive, private or missing the cases you care about, and especially when outputs can be verified automatically (code, math, structured extraction, tool use). Be most careful with open-ended factual content.

    Example

    The support team from the opening rebuilds its dataset: seeds are the 300 real conversations, a list of 40 intents, 6 customer personas and the actual policy documents. Generation produces 30,000 candidate dialogues grounded in the policy text; a checker verifies that every quoted rule exists in the documents and every tool call matches the tool schema; an LLM judge scores tone; near-duplicates are removed. 14,000 survive. The final mix is 70% synthetic and 30% real (upsampled). On 500 held-out real conversations, policy hallucinations drop from 9% to 1.5%. (Example numbers.)

    Common mistakes

    • Generating from the model's memory for domain facts instead of from your documents.
    • No verifier, so errors are trained in.
    • Low diversity: one prompt template, one persona.
    • Evaluating on synthetic data from the same pipeline.
    • Ignoring the generator's licence terms.
    • Replacing real data entirely, inviting collapse and style drift.

    Say this in the interview

    "I treat synthetic data as a pipeline: diverse seeds (real examples, a topic taxonomy, personas, our documents), generation grounded in those sources, then verification (unit tests, exact answers, schema checks, or a calibrated LLM judge), rejection sampling, dedup and diversity checks. I blend it with real data, keep evals real and decontaminated, version everything, and feed eval failures back as seeds. The risks are invented facts, low diversity, teacher bias, licence terms and collapse when models train on their own outputs."

    Interview questions on this

    1. When is synthetic data most reliable? When outputs can be verified automatically, such as code with tests or math with answers.
    2. How do you keep synthetic data diverse? Varied seeds, personas, difficulty, formats; dedup and coverage metrics.
    3. What is model collapse? Loss of rare patterns when models train repeatedly on model outputs.
    4. How do you avoid invented facts in synthetic domain data? Generate from source documents and verify claims against them.
    5. Why not evaluate on synthetic data? It shares the generator's blind spots and inflates scores.
    6. What is rejection sampling here? Generate several candidates and keep only those that pass the verifier.

    Used in

    • AI Engineering: Labeling and Preference Data; Building a Training Data Pipeline; Supervised Fine-tuning: Data and Recipe; Distillation: Making a Small Model Act Big; Evals and LLM-as-Judge; Evaluating a RAG System.
    • AI System Design: Design a Fine-tuning Platform; Design an Eval Platform.