Evaluating a Fine-tuned Model and Catastrophic Forgetting
On this page
A team fine-tunes a model to write SQL for their warehouse. On their SQL test set it jumps from 61% to 88% correct, and they ship it. A week later other teams who share the model complain: it answers ordinary questions in SQL, its summaries got worse, and it now helps with requests the base model correctly refused. The fine-tune did exactly what it was trained to do, and it forgot some of what it knew before. Evaluating a fine-tuned model means checking both what improved and what got worse.
How it works
Four things to measure

- The target task, on a held-out set that never touched training: accuracy, exact match, pass rate of generated code, or rubric scores from people or an LLM judge.
- Output and production behaviour: valid JSON or schema rate, length, stop behaviour, latency, tool-call format.
- General ability: a small regression suite of things the model must still do (general Q&A, summarising, following instructions, other languages your users use), plus a few standard public benchmarks if they are relevant.
- Safety: refusal of harmful requests, absence of over-refusal on harmless ones, leaking of training data or personal information.
Always compare against two baselines: the base model with your best prompt (did fine-tuning beat prompting?) and the previous production model (is this a regression?).
Catastrophic forgetting
When a model is trained on a narrow data set, the weight changes that help the new task can damage abilities it had before. The effect is stronger with more epochs, higher learning rates, narrower data, and full fine-tuning. Safety behaviour is particularly fragile: even a small, harmless-looking fine-tune can weaken refusals, because the training data contains no refusals.

Trade-offs and costs
- Specialisation vs generality: some loss of general ability is acceptable for a model that only ever does one task behind a fixed interface; it is not acceptable for a shared assistant.
- Mixing in general data reduces forgetting but dilutes the task signal; tune the ratio on the eval sets.
- LoRA forgets less than full fine-tuning in most reports, partly because it changes less, and the adapter can be switched off.
- Evaluation costs time and money (judge calls, human review), so automate the bulk and sample for human checks.
In production
- Gate releases on the scorecard: the target metric must improve, and regression and safety metrics must stay within a set tolerance of the baseline.
- Shadow or A/B test before full rollout; offline evals miss real traffic.
- Keep the eval sets versioned and growing: every production failure becomes a test case.
- Watch the training curve: stop at the checkpoint with the best held-out score, not the last one.
- Route by task if one model can't do both well: the fine-tuned model for its task, the base model for everything else.
When to reach for it
Every fine-tune, before it ships. A fine-tune without a regression and safety check is an untested release.
Example
The SQL team adds a scorecard. Target: SQL correctness on 300 held-out questions. Format: query parses and runs. General: 200 mixed tasks scored by an LLM judge against the base model's answers. Safety: 150 harmful and 150 harmless-but-sensitive prompts. The first model fails the general and safety gates. The second run uses LoRA instead of full fine-tuning, 1 epoch instead of 3, and mixes 10% general instruction data and 300 refusal examples into training. Its SQL score is 86% instead of 88%, general ability is within 2 points of the base, and safety matches the base. They ship it, and route non-SQL requests to the base model anyway.
Common mistakes
- Only measuring the target task.
- Test examples that leaked into training (near-duplicates count too), giving inflated scores.
- No prompt baseline, so nobody knows fine-tuning was needed.
- Trusting training loss instead of held-out evals.
- Forgetting safety: assuming a harmless data set can't weaken refusals.
Say this in the interview
"I evaluate a fine-tune on four things: the target task on a held-out set, output format and production behaviour, general ability through a regression suite, and safety. I compare against the prompted base model and the current production model. Narrow fine-tuning causes catastrophic forgetting, so I limit it with LoRA, fewer epochs, a lower learning rate, and mixing general and refusal data into training, then gate the release on all four scores and A/B test before full rollout."
Interview questions on this
- What is catastrophic forgetting? Loss of earlier abilities when a model is trained on new, narrow data.
- How do you detect it? A regression suite of general tasks and safety prompts, compared with the base model.
- How do you reduce it? Replay or general data in the mix, LoRA, fewer epochs, lower learning rate, early stopping, weight averaging.
- Why compare against the prompted base model? To prove fine-tuning beat the cheaper option.
- How do you check for test-set leakage? Deduplicate test against training with exact and near-duplicate matching before training.
- Offline evals look good. What else before full rollout? Shadow traffic or an A/B test with production metrics.
Used in
- AI Engineering: Evals and LLM-as-Judge; Supervised Fine-tuning; LoRA and QLoRA; Online Evaluation and A/B Tests for LLM Features; Versioning Prompts and Models, and Rolling Them Out.
- AI System Design: Design an LLM Evaluation Platform; Design a Fine-tuning Platform.