Building a Training Data Pipeline: Dedup, Filtering, Mixing

Updated Nov 16, 2026 · 5 min read
On this page

    A team pretrains a small model on 2 TB of crawled text and gets disappointing results. Inspecting the data shows why: one in five documents is a near-copy of another (cookie banners, mirrored sites, templated pages), large parts are menus and boilerplate, and the benchmark questions they report on appear verbatim in the training set, so even the good-looking scores are inflated. Model quality is decided by the data pipeline as much as by the architecture: extraction, filtering, deduplication, decontamination and mixing, run at a scale of billions of documents and fast enough to keep thousands of GPUs fed.

    How it works

    Typical stages from raw crawl to training shards; most raw documents never reach the model (illustrative shares).
    Typical stages from raw crawl to training shards; most raw documents never reach the model (illustrative shares).

    1. Extract

    Turn HTML, PDFs and code repositories into clean text: strip navigation, ads and boilerplate, keep headings, lists, tables and code blocks. Extraction quality alone changes downstream model quality noticeably, so it's worth evaluating, not just running.

    2. Filter

    • Language identification (a fast classifier) to keep the languages you want, in the proportions you want.
    • Heuristic quality filters: document length, share of symbols or non-alphabetic characters, repeated lines, ratio of stop words, "lorem ipsum" and other junk markers.
    • Model-based quality filters: a small classifier trained to tell high-quality text (reference-like, educational) from low-quality text, or perplexity under a small reference model. Used by many recent open datasets.
    • Safety: remove toxic or illegal content, and PII (emails, phone numbers, IDs) with detectors and redaction.
    • Licensing and opt-outs: respect robots.txt and licence rules for each source.

    3. Deduplicate

    Duplicates waste compute, make the model memorise and repeat text, and skew the mix toward whatever was copied most.

    • Exact dedup: hash each document (or each line or paragraph) and drop repeats.
    • Near-dup dedup with MinHash + LSH:
    Shingles become a MinHash signature whose matching rate estimates Jaccard similarity; LSH bands put likely duplicates in the same bucket.
    Shingles become a MinHash signature whose matching rate estimates Jaccard similarity; LSH bands put likely duplicates in the same bucket.
    1. Turn each document into a set of shingles (overlapping character or word n-grams).
    2. Compute a MinHash signature: for each of, say, 128 hash functions, the minimum hash over the shingles. The fraction of matching positions between two signatures estimates their Jaccard similarity.
    3. LSH banding: split the signature into bands (16 bands × 8 rows); documents sharing any whole band land in the same bucket. A pair with similarity s becomes a candidate with probability 1 − (1 − s⁸)¹⁶: 6% at 0.5, 95% at 0.8. Only documents in the same bucket are compared, so the work grows roughly linearly with the number of documents instead of quadratically.
    4. Verify candidates and keep one document per cluster.

    The included minhash_dedup.py flags an exact copy, a punctuation change and a light rewrite (Jaccard 0.73) as duplicates, and keeps a reworded sentence with the same meaning (Jaccard 0.53). Semantic dedup with embeddings is an extra step some pipelines add for paraphrases.

    4. Decontaminate

    Remove training documents that overlap with evaluation sets (n-gram overlap, typically 8–13-gram matches), or results on those benchmarks measure memory, not ability. Keep the list of removed items so you can report it.

    5. Mix, tokenize and shard

    Example mixture weights by source, and how mixes are chosen and delivered to the GPUs.
    Example mixture weights by source, and how mixes are chosen and delivered to the GPUs.
    • Mixture weights decide how much of each source the model sees. They differ from raw sizes: high-quality sources (code, books, curated reference text, math) are upsampled and seen a few times; web text is downsampled. Weights are tuned by training small proxy models on candidate mixes and comparing evals.
    • Annealing: many recipes shift toward the highest-quality data near the end of training.
    • Tokenize once and write shuffled, fixed-size shards (with an index) to object storage. Training streams shards to every GPU with a deterministic, resumable order (see Checkpoints and Fault Tolerance).

    Trade-offs and costs

    • Aggressive filtering vs diversity: strict quality classifiers can remove dialects, niche topics and non-English text; check what you lose, by language and domain.
    • Dedup granularity: document-level dedup misses repeated paragraphs; line-level dedup can break documents.
    • Compute: MinHash over billions of documents needs a distributed job (Spark, Ray, Dask, or dedicated tools); still far cheaper than training on the duplicates.
    • Reproducibility: every filter threshold changes the dataset; version code, thresholds and outputs together.

    In production

    • Run as batch jobs over object storage with a workflow orchestrator; each stage writes versioned outputs and statistics (documents in and out, per reason, per source and language).
    • Sample and read data at every stage. Many problems are obvious to a human in five minutes.
    • Feed the GPUs: data loading must deliver GPUs × tokens per GPU per second. Pre-tokenized shards, enough CPU workers per GPU and local caching avoid a starved training run.
    • Keep lineage: which dataset version and mixture trained which checkpoint.

    When to reach for it

    Pretraining and continued pretraining, large-scale fine-tuning datasets, and "design a data pipeline for LLM training" interviews. The same dedup and decontamination steps apply to fine-tuning and evaluation data.

    Example

    A team preparing 500B tokens for continued pretraining of a code model: extract from repositories, drop generated and minified files, keep permissively licensed code, exact dedup by file hash (removes 30% of files), near-dedup with MinHash at a 0.8 threshold (removes 12% more), scrub secrets and emails, decontaminate against the coding benchmarks they report, then mix 70% code, 20% natural language about code, 10% math. A 1B proxy model trained on the deduplicated data matches the undeduplicated baseline's eval scores with 35% fewer tokens. (Example numbers.)

    Common mistakes

    • Skipping near-dup removal and training on thousands of copies of templates.
    • Not decontaminating, then reporting inflated benchmark results.
    • Mixing by raw source size instead of by quality and goal.
    • No per-stage statistics, so nobody can explain why the dataset changed.
    • Data loaders slower than the GPUs, wasting expensive compute.

    Say this in the interview

    "I'd build it as versioned batch stages: extract clean text, filter by language and quality with heuristics plus a small quality classifier, scrub toxicity and PII, remove exact duplicates by hash and near-duplicates with MinHash and LSH, decontaminate against eval sets, then mix sources by weights tuned on small proxy models, upsampling high-quality data, and write pre-tokenized shuffled shards streamed to the GPUs with a resumable order. Every stage logs counts by reason, and I'd read samples at each step."

    Interview questions on this

    1. How does MinHash estimate Jaccard similarity? The probability that two sets share the same minimum hash equals their Jaccard similarity.
    2. Why use LSH bands instead of comparing all pairs? All-pairs is quadratic; bands bucket likely matches so only a few comparisons are needed.
    3. What is benchmark contamination and how do you prevent it? Eval items in training data; n-gram overlap filtering and reporting.
    4. How do you choose mixture weights? Proxy-model experiments on candidate mixes, judged on target evals.
    5. How would you detect that training is data-loader bound? GPUs idle between steps, low SM activity, data-wait time in the profiler.
    6. What can go wrong with a quality classifier filter? It encodes a narrow idea of quality and removes dialects or domains.

    Used in

    • AI Engineering: Synthetic Data for LLMs; Labeling and Preference Data; How a Chat Model Is Made; Tokens and Tokenization; Checkpoints and Fault Tolerance in Long Training Runs; Document Ingestion for RAG.
    • AI System Design: Design a Fine-tuning Platform; Design a Web Crawler.