How Would You Reduce Hallucinations?

Free walkthrough Updated Nov 16, 2026 · 3 min read
On this page

    Problem statement

    "Our support assistant sometimes makes things up: refund timelines, plan features, even links. Users act on these answers. How would you reduce hallucinations?"

    This is one of the most common practical questions for AI application and agent roles. The interviewer wants a structured answer: causes first, then layered fixes, then how you'd measure progress.

    Why it happens

    The model fills missing or ignored context with plausible text, and nothing checks the claim before it reaches the user.
    The model fills missing or ignored context with plausible text, and nothing checks the claim before it reaches the user.
    • The model fills gaps. It is trained to produce likely text; when the facts aren't in its context, it writes what such an answer usually looks like (see Why LLMs Hallucinate).
    • Context problems: the right document wasn't retrieved, a stale one was, or the right passage was buried in a long prompt and ignored.
    • Pressure to answer: prompts that demand a complete answer and give no way to abstain.
    • Sampling: high temperature makes unlikely (often wrong) tokens more likely, especially numbers and names.
    • No checks: the answer goes straight to the user.

    Options

    Weaktell the model not to hallucinate

    Add "Do not make things up" to the prompt and set temperature to 0. It helps a little with sampling noise but does nothing about missing knowledge or bad retrieval; a deterministic model can be deterministically wrong. There is also no measurement, so nobody knows whether it worked.

    Goodground, allow abstaining, cite

    Retrieve relevant sources (RAG) and instruct the model to answer only from them, with clear delimiters. Define an explicit "I don't have that information; here's how to reach billing" answer and reward it in evals. Require a citation for each factual claim, and show citations in the UI so users can check. Use low temperature and structured output for facts. This removes most hallucinations caused by missing knowledge.

    Stronglayered defences with measurement

    Everything in Good, plus:

    Each layer removes a share of the problem; measurement tells you when to stop.
    Each layer removes a share of the problem; measurement tells you when to stop.
    • Fix retrieval first: measure retrieval recall; most "hallucinations" in RAG systems start with the right passage missing (see Your RAG Answers Are Wrong).
    • Verify before showing: split the answer into claims and check each against the retrieved passages with a judge or NLI model; drop, rewrite or flag unsupported claims.
    • Constrain risky outputs: links only from an allowlist or from retrieved documents; prices, dates and policy terms filled from structured data, not generated.
    • Risk-based routing: for refunds, legal or medical topics, require higher confidence or human review.
    • Close the loop: user reports and sampled live traffic feed new test cases.

    How to measure it

    • Faithfulness: share of claims supported by the provided sources (judge scored, calibrated against human labels).
    • Correctness: against reference answers on a labelled test set.
    • Abstention quality: on questions the sources can't answer, how often the system says so instead of guessing; and on answerable ones, how often it wrongly refuses.
    • Citation accuracy: cited passages actually support the claim.
    • Online: user "this is wrong" reports, escalations, and judge-scored samples of live traffic.

    Report them together: pushing abstention too far hides hallucinations by refusing to answer useful questions.

    Example answer flow

    "I'd start with 200 real support questions plus 30 that our docs don't cover, and measure faithfulness and abstention. Suppose 12% of answers contain an unsupported claim. I'd check retrieval recall first, since a missing passage is the most common cause, then add answer-only-from-sources instructions, an abstain path, citations, and a claim-check step for refund and pricing answers. I'd re-run the test set after each change and watch that correct answers don't drop."

    Follow-up questions

    1. How would you build the claim-verification step, and what does it cost in latency?
    2. Users complain the bot now says "I don't know" too often. What do you do?
    3. How do you stop the model from inventing URLs?
    4. Would fine-tuning reduce hallucinations here? When would it help?
    5. How do you calibrate an LLM judge for faithfulness?
    6. What changes for an agent that takes actions based on its beliefs?

    Related

    • Concepts: Why LLMs Hallucinate; RAG End to End; Evaluating a RAG System; Evals and LLM-as-Judge; Structured Output.
    • Practice: Your RAG Answers Are Wrong: How Do You Debug It?
    • AI System Design: Design a Customer Support Agent; Design Perplexity.