Fine-tuning

  1. 1 Fine-tune, RAG or Prompt? Choosing the Right Tool "I separate knowledge from behaviour. Knowledge that is private or changes goes in through RAG, so it's fresh and citable. Behaviour, meaning format, tone or a narrow skill, is what fine-tuning is good at, and it can also let a smaller model do the job cheaply. I always start with a strong prompt... 4 min read
  2. 2 LoRA and QLoRA "LoRA freezes the pretrained weights and learns each layer's update as a low-rank product A·B, with B initialised to zero so training starts from the base model. Only a fraction of a percent of parameters are trained, so gradients and optimizer state nearly disappear, and adapters are small enoug... 5 min read
  3. 3 Supervised Fine-tuning: Data and Recipe "SFT trains the model with next-token loss on input–output examples so it imitates the outputs. The hard part is the data: a few thousand clean, consistent, representative examples beat a large noisy dump. I'd use the base model's exact chat template, mask the loss to the assistant tokens, train ... 4 min read
  4. 4 Preference Tuning: RLHF and DPO "Preference tuning learns from pairs where one answer was preferred. RLHF trains a reward model on those pairs and then uses PPO to maximise reward with a KL penalty to the SFT model; it's powerful but heavy and prone to reward hacking. DPO gets the same kind of objective with a single loss direc... 5 min read
  5. 5 Distillation: Making a Small Model Act Big "Distillation trains a small student model to reproduce a large teacher on our task. The usual version is sequence-level: run the teacher over real prompts, filter its answers, and fine-tune the student on them. If we control both models, logit-level distillation matches the teacher's full distri... 4 min read
  6. 6 Evaluating a Fine-tuned Model and Catastrophic Forgetting "I evaluate a fine-tune on four things: the target task on a held-out set, output format and production behaviour, general ability through a regression suite, and safety. I compare against the prompted base model and the current production model. Narrow fine-tuning causes catastrophic forgetting,... 4 min read