Fine-tune, RAG or Prompt? Choosing the Right Tool

Updated Nov 16, 2026 · 4 min read
On this page

    A support team's chatbot gives answers that are outdated, too long, and in the wrong tone. Someone proposes fine-tuning "on all our docs". Two months later the model writes in the right tone, still quotes last year's refund policy, and has to be retrained every time a document changes. The team picked the right tool for the tone and the wrong one for the facts. Prompting, retrieval (RAG) and fine-tuning change different things, and choosing between them is one of the most common AI engineering interview questions.

    How it works

    Prompting changes the instructions, RAG changes what the model can see, fine-tuning changes the model's weights.
    Prompting changes the instructions, RAG changes what the model can see, fine-tuning changes the model's weights.

    Prompting

    You change the instructions and examples sent with every request: the role, the rules, the output format, a few worked examples. Nothing is trained. Changes take minutes and are easy to roll back, so it is always the first thing to try. Its limits: the model can only use what it already knows plus what fits in the prompt, and long prompts cost tokens and latency on every call.

    Retrieval-augmented generation (RAG)

    At request time you search your own data (documents, tickets, product catalogue) and put the most relevant pieces into the prompt. The model answers from that context. This is how you give a model knowledge it doesn't have: private, recent, or per-user. Updating knowledge means updating the index, not the model, and answers can cite their sources. Its limits: if retrieval misses the right passage, the answer is wrong, and it doesn't teach the model a new style or skill.

    Fine-tuning

    You continue training the model on examples of the input and the output you want, so the change lives in the weights. This is how you change behaviour: a consistent format, a house tone, a narrow skill like classifying tickets into your 40 categories, or tool-calling in your exact schema. It also lets a small model do a job that otherwise needed a big one, which cuts cost and latency. Its limits: you need hundreds to thousands of good examples, an eval set, training runs and a deployment pipeline; and it is a poor way to add facts, because facts learned this way are unreliable and go stale.

    Trade-offs and costs

    PromptRAGFine-tune
    Changesinstructionscontextweights
    Time to first resultminutesdaysdays to weeks
    Needsgood instructionsindexed data, retrieval tuninglabelled examples, GPUs, evals
    Fresh knowledgeonly what you paste inyesno (retrain)
    Per-request costgrows with prompt lengthretrieval + longer promptcan be lower (shorter prompt, smaller model)
    Riskprompt brittlenessretrieval missesoverfitting, forgetting, maintenance

    In production

    • Start with an eval set. 50–200 real questions with expected answers or grading rules. Without it you can't tell whether a change helped.
    • They combine. A common end state is a fine-tuned small model that follows the format perfectly and still uses RAG for facts.
    • Fine-tuning creates a model to maintain: retrain when the base model is upgraded, version it, evaluate it like any release.
    • Long context is not free RAG. Pasting everything into a 200k-token prompt is slow and costly, and models use the middle of very long prompts less reliably.

    When to reach for it

    • Prompt: always first; enough when the model has the knowledge and just needs direction.
    • RAG: the answer depends on information the model doesn't have or that changes; you need citations or per-user access control.
    • Fine-tune: prompting can't get the format, tone or skill consistent; prompts have grown huge with examples; or you want a smaller, cheaper model for a high-volume narrow task.

    Example

    The support bot again, split by problem. Outdated answers: the refund policy changes quarterly, so it goes into a RAG index that is rebuilt when documents change, and answers cite the policy page. Too long and wrong tone: a tighter system prompt with two examples fixes most of it. The remaining inconsistency, on 300,000 conversations a month, justifies fine-tuning a small open model with LoRA on 3,000 approved replies; it matches the tone and costs a fraction of the large model per request. The eval set of 150 real tickets is run after every change.

    Move to the next step only when the eval set shows the previous one is not enough.
    Move to the next step only when the eval set shows the previous one is not enough.

    Common mistakes

    • Fine-tuning to teach facts that change; use retrieval.
    • Skipping the prompt baseline, so nobody knows whether fine-tuning helped.
    • No eval set, so every choice is a guess.
    • Treating RAG and fine-tuning as rivals. They solve different problems and often work together.
    • Forgetting maintenance: a fine-tuned model must be retrained when the base model changes.

    Say this in the interview

    "I separate knowledge from behaviour. Knowledge that is private or changes goes in through RAG, so it's fresh and citable. Behaviour, meaning format, tone or a narrow skill, is what fine-tuning is good at, and it can also let a smaller model do the job cheaply. I always start with a strong prompt and an eval set, and only move to the next tool when the eval says the current one isn't enough. Often the answer is both: a fine-tuned model that still retrieves."

    Interview questions on this

    1. Our model doesn't know our product. Fine-tune it? Usually no: use RAG, because product facts change and need sources. Fine-tune only if the style or format is also wrong.
    2. When is fine-tuning cheaper than prompting? At high volume, when a small fine-tuned model replaces a large model with a long few-shot prompt.
    3. Why not just use a long context window? Cost and latency on every request, and weaker use of information buried in the middle.
    4. How much data do you need to fine-tune? Often a few hundred to a few thousand high-quality examples for a format or narrow skill; quality matters more than quantity.
    5. How would you show fine-tuning helped? Compare against the prompt and RAG baseline on the same held-out eval set, and check general ability didn't drop.

    Used in

    • AI Engineering: LoRA and QLoRA; Supervised Fine-tuning; Distillation; RAG End to End; Prompt Engineering for Production; Evals and LLM-as-Judge.
    • AI System Design: Design an Enterprise RAG System; Design a Fine-tuning Platform.