RAG End to End
On this page
A new employee asks the company assistant how many vacation days they get. The model has never seen the HR handbook, so it answers with a typical number from the internet, confidently and wrongly. Fine-tuning on the handbook would be slow, expensive and out of date the day the policy changes. Retrieval-augmented generation (RAG) takes the other route: find the relevant passages at question time and give them to the model to answer from.
How it works

Two halves: indexing and answering

Offline: build the index
- Ingest documents from their sources (wikis, PDFs, tickets, databases), keeping metadata: title, URL, owner, permissions, last updated.
- Clean and chunk: remove boilerplate, split into passages of a few hundred tokens that each make sense alone (see Chunking Strategies).
- Embed each chunk and store it in a vector index, usually alongside a keyword index.
- Keep it fresh: re-index changed documents and delete removed ones.
Online: answer a question
- Rewrite the query if needed: resolve "it" and "that" from the chat history, expand abbreviations, or split a compound question.
- Retrieve candidates, often 20–50, with vector and keyword search together, filtered by what this user is allowed to see.
- Rerank the candidates with a stronger model (a cross-encoder) and keep the best few, often 3–8.
- Generate: put the passages in the prompt with clear source labels and instructions to answer only from them, cite sources, and say so when the answer isn't there.
- Return the answer with citations the user can click.
Why RAG instead of fine-tuning
- Fresh: update the index, not the model.
- Traceable: every claim can point at a source.
- Permission-aware: retrieval filters by the user's access, so people only get answers from documents they may read.
- Cheaper to build and change. Fine-tuning remains useful for style and format, and the two combine well (see Fine-tune, RAG or Prompt?).
Trade-offs and costs
- More context isn't always better. Each extra chunk adds tokens, latency and distraction; models answer better from 5 sharp passages than 30 vague ones.
- Every stage can fail independently, and the final answer hides which one did.

- Latency: retrieval and reranking add tens to hundreds of milliseconds before the model starts; prompts with retrieved text raise TTFT.
- Cost: embedding the corpus, running the indexes, reranking, and longer prompts on every request.
In production
- Evaluate each stage separately: retrieval recall (is the right passage in the top k?), reranking precision, and answer faithfulness and correctness (see Evaluating a RAG System).
- Permissions in the index, enforced as filters at query time, never just in the prompt.
- Freshness SLA: how soon a changed document is searchable; monitor ingestion lag.
- Observability: log the query, retrieved chunk IDs and scores, and the final prompt for every answer, so bad answers can be traced.
- Failure modes: stale or missing documents; answers stitched from two contradicting versions of a policy; prompt injection hidden inside a retrieved document; the model answering from memory when retrieval found nothing.
When to reach for it
Whenever answers must come from a specific, changing body of knowledge: internal docs, product manuals, support articles, legal and policy texts, codebases, or the web (search-backed assistants). For general knowledge or pure style, prompting or fine-tuning may be enough.
Example
An HR assistant indexes 3,000 policy pages into about 40,000 chunks with region and employee-type metadata. A question from an employee in Germany is rewritten using the chat context, retrieves 30 candidates by hybrid search filtered to the German policies the employee can access, reranks to 5, and asks the model to answer only from them with citations, replying "not covered in the policies I can see" otherwise. When answers are wrong, the trace shows that most cases were a missing or outdated page, fixed in ingestion, not in the prompt.
Common mistakes
- Blaming the model first when retrieval never returned the right passage.
- Stuffing too many chunks into the prompt.
- No permission filtering, leaking documents through answers.
- No "not found" path, so the model fills gaps from memory.
- Never re-indexing, so answers cite last year's policy.
Say this in the interview
"RAG has an offline part: ingest, clean, chunk, embed and index with metadata and permissions; and an online part: rewrite the query, retrieve candidates with hybrid search filtered by access, rerank to a few passages, and generate an answer grounded in them with citations and a 'not found' option. I'd evaluate retrieval recall and answer faithfulness separately, and log every stage so wrong answers can be traced."
Interview questions on this
- Why rerank after retrieval? First-stage retrieval is fast but rough; a cross-encoder reads query and passage together and orders the short list much better.
- How do you enforce permissions? Store access metadata with each chunk and filter at retrieval time by the user's identity.
- The right document exists but answers are wrong. Where do you look? Retrieval recall, chunk boundaries, ranking, then the prompt and model, in that order.
- How many chunks should you pass to the model? As few as give high recall, often 3–8; measure, since more adds cost and distraction.
- How do you keep the index fresh? Event-driven or scheduled re-ingestion of changed documents, deletes for removed ones, and monitoring of ingestion lag.
Used in
- AI Engineering: Embeddings; Vector Search; Chunking Strategies; Hybrid Search and Reranking; Evaluating a RAG System; Document Ingestion for RAG; Your RAG Answers Are Wrong (practical question).
- AI System Design: RAG and grounding; Design Perplexity; Design Enterprise RAG; Design Cursor.