How a Chat Model Is Made: Pretraining, SFT and RLHF
On this page
Download an open "base" model and ask it "What is the capital of France?". Instead of answering, it may write "What is the capital of Germany? What is the capital of Spain?", because on the internet a line like yours is often part of a quiz list. The same architecture, after more training, answers "Paris" politely. Knowing what each training stage adds explains a lot of production behaviour: why models follow formats, why they refuse some requests, why they sound confident when wrong, and when fine-tuning is worth it.
How it works


Stage 1: pretraining
The model learns one task: predict the next token on a huge amount of text (web pages, books, code, papers), often trillions of tokens. To get good at it, it must pick up grammar, facts, reasoning patterns and coding conventions, because all of them help guess the next token.
- Takes most of the compute, often thousands of GPUs for weeks or months.
- Result: a base model that knows a lot and continues any text plausibly, but has no notion of being an assistant.
- Its knowledge stops at the training cutoff; anything later is unknown unless provided in the prompt.
Stage 2: supervised fine-tuning (SFT)
Next, the model is trained on curated instruction → response examples written or checked by people (and increasingly generated by other models and filtered): answer a question, follow a format, write code, refuse an unsafe request. The training objective is still next-token prediction, but only on good responses.
- Far less data (tens of thousands to millions of examples) and compute.
- Result: the model follows instructions, uses a chat format with roles (system, user, assistant), and produces the expected structure.
Stage 3: preference tuning (RLHF, DPO and friends)
SFT teaches by imitation, but writing perfect examples is hard. It is much easier to compare two answers and say which is better. Preference tuning uses those comparisons.

- RLHF: collect pairs ranked by raters, train a reward model to predict which answer people prefer, then use reinforcement learning (for example PPO) to make the chat model produce higher-reward answers, with a penalty for drifting too far from the SFT model.
- DPO (Direct Preference Optimisation): skip the reward model and RL loop; a simple loss raises the probability of the preferred answer relative to the rejected one. Easier and more stable to run, now very common.
- AI feedback: some pipelines use another model, guided by written principles, to produce the preferences.
- Reasoning-focused training: newer models also get reinforcement learning on tasks with checkable answers (maths, code with tests), which rewards correct step-by-step reasoning.
Result: answers that are more helpful, better calibrated in tone, and safer.
Trade-offs and costs
- Knowledge comes from pretraining. Later stages mostly shape how the model uses what it knows; they add little new knowledge. This is why fine-tuning is a poor way to teach facts (use RAG) but a good way to teach format, style and narrow skills.
- Alignment can over-shoot. Preference tuning can make models verbose, over-cautious, or eager to please (agreeing with the user even when the user is wrong).
- Reward hacking. Optimising a learned reward can find answers that score well but aren't actually better, such as long confident answers that raters liked.
- Cost. Pretraining is the big bill; SFT and preference tuning are affordable for many companies with open models and parameter-efficient methods like LoRA.
In production
- Pick the right checkpoint. Use instruct/chat models for applications; base models for further pretraining or special completion tasks.
- Use the chat template. Instruct models expect the exact role format they were trained with; a wrong template quietly lowers quality.
- Knowledge cutoff. For anything recent, supply facts in the prompt (RAG, tools).
- Failure modes: confident wrong answers (pretraining rewards plausible text, not truth); refusals of harmless requests; sycophancy; behaviour changes when the provider updates the model under the same name, so pin versions and keep evals.
When to reach for it
This knowledge decides real choices: prompt vs RAG vs fine-tune; whether to start from a base or instruct model; how to read a model card; and how to explain odd behaviour (refusals, flattery, format drift) in an incident review or interview.
Example
A legal-tech team wants answers in a strict clause-by-clause format and in the firm's tone, using the firm's current documents. Splitting the problem by stage: the documents are knowledge that changes, so they go in through RAG; the format and tone are behaviour, so they fine-tune an open instruct model with a few thousand example answers (SFT with LoRA); and they use DPO on pairs where lawyers marked the better of two drafts to cut verbosity. Pretraining is left alone; it would cost orders of magnitude more and isn't needed.
Common mistakes
- Fine-tuning to add facts that change, instead of retrieving them.
- Calling every post-training step "RLHF". SFT, RLHF and DPO are different steps with different data.
- Prompting a base model like a chat model and concluding the model is bad.
- Ignoring the chat template when self-hosting an instruct model.
- Assuming alignment means truthfulness. Preference tuning improves helpfulness and tone; it doesn't guarantee facts.
Say this in the interview
"Pretraining on trillions of tokens with next-token prediction gives a base model its knowledge. Supervised fine-tuning on instruction-response pairs teaches it to follow instructions and formats. Preference tuning, with RLHF via a reward model or DPO directly on preferred-vs-rejected pairs, shapes helpfulness and safety. Knowledge mostly comes from pretraining, so I'd use RAG for facts and fine-tuning for behaviour and format."
Interview questions on this
- Why doesn't a base model answer questions? It was trained only to continue text, so it imitates whatever document a prompt resembles.
- RLHF vs DPO? RLHF trains a reward model and optimises with RL; DPO optimises directly on preference pairs with a simple loss, cheaper and more stable.
- Why is fine-tuning a bad way to teach facts? It adds little reliable knowledge and the facts go stale; retrieval is cheaper and updatable.
- What is reward hacking? The model exploits weaknesses of the learned reward, like length or flattery, instead of truly improving.
- What does the KL penalty in RLHF do? Keeps the tuned model close to the SFT model so it doesn't drift into degenerate text that games the reward.
Used in
- AI Engineering: Fine-tune, RAG or Prompt?; Supervised Fine-tuning; Preference Tuning: RLHF and DPO; Labeling and Preference Data; Why LLMs Hallucinate.
- AI System Design: Design a Fine-tuning Platform.