RAG interview questions and answers

What are the two pipelines in a RAG system?

The indexing pipeline runs ahead of time: load documents, parse them, split them into chunks, embed the chunks and store them with metadata. The query pipeline runs per question: process the query, retrieve candidates, rerank them, build a prompt with the best chunks and generate a grounded answer.

Does RAG stop hallucinations completely?

No. It reduces them by giving the model relevant facts, but the model can still ignore the context, mix sources or answer from memory when retrieval misses. You also need a prompt that tells it to answer only from the context, citations, an "I do not know" path and faithfulness checks.

What is the difference between naive RAG and advanced RAG?

Naive RAG embeds the raw question, takes the top few chunks and sends them to the model. Advanced RAG adds steps before and after retrieval: query rewriting, hybrid search, metadata filters and reranking before, and context compression, citations and answer checks after.

Why not just put all the documents into a long-context model?

Long context costs more per call, adds latency, and models use facts in very long prompts less reliably. RAG sends only the relevant parts, scales to far more data than any context window, can respect per-user permissions and gives traceable sources. Long context helps when the document set is small and changes rarely.

What is the "lost in the middle" problem?

Research has shown that models use information at the start and end of a long prompt better than information in the middle. In RAG this means stuffing many chunks can hide the important one. Send fewer, better chunks and put the most relevant ones first or last.

How do you decide how many chunks to retrieve?

Retrieve a wider set, such as 20 to 50 candidates, for recall, then rerank and keep only the top few, often 3 to 8, for the prompt. Tune the numbers on an evaluation set by watching recall at k and answer quality together, within your token and latency budget.

Why is document parsing often the weakest part of a RAG system?

PDFs, slides and scanned files lose structure when turned into plain text: tables flatten, columns interleave, headers repeat on every page and scanned pages need OCR. Bad text in means bad chunks and bad answers, so check parsed output by eye and use layout-aware parsers for complex files.

Compare fixed-size, recursive and semantic chunking.

Fixed-size chunking cuts every N tokens: simple, but it splits sentences. Recursive chunking tries separators in order, such as headings, paragraphs, then sentences, so chunks follow structure. Semantic chunking splits where embedding similarity between neighbouring sentences drops. Recursive splitting on document structure is the usual strong default.

Why add overlap between chunks?

An answer can straddle a chunk boundary. Repeating a small part of the previous chunk, often 10 to 20 percent, keeps that context in at least one chunk. Too much overlap wastes storage and returns near-duplicate chunks.

def chunk_words(text, size=300, overlap=50):
    words = text.split()
    step = size - overlap
    return [" ".join(words[i:i + size]) for i in range(0, len(words), step)]

What metadata would you store with each chunk?

Source document ID, title, section heading, page or URL, created and updated dates, language, document type and access-control fields such as tenant or team. Metadata powers filters, citations, freshness rules and permission checks, and lets you delete or re-index one document cleanly.

What is parent-child or small-to-big retrieval?

You embed small chunks, such as sentences or short passages, because they match questions precisely, but each one points to a larger parent section. At query time you search the small chunks and send their parents to the model, so it gets enough surrounding context.

What is contextual chunk enrichment?

A chunk like "revenue grew 3 percent" means little alone. Before embedding, you prepend context such as the document title, section path or a one-line summary of where the chunk sits. Embeddings and keyword search then match questions that name the company or period, which improves retrieval noticeably.

How do you handle tables and images in RAG?

Keep each table as a unit, convert it to Markdown or HTML with its caption and headers, and add a short text summary for search. For charts and images, store a generated description or use a multimodal embedding model, and keep a link to the original so it can be shown with the answer.

How do you choose an embedding model for RAG?

Check public retrieval benchmarks such as MTEB for a shortlist, then test on your own questions, because domain matters. Also weigh language coverage (Hindi or mixed-language queries), maximum input length, vector size and its storage cost, latency, price and whether you can self-host.

What is the difference between a bi-encoder and a cross-encoder?

A bi-encoder embeds the query and each document separately, so document vectors are computed once and search is fast. A cross-encoder reads the query and one document together and outputs a relevance score: more accurate, but too slow for the whole corpus. Use bi-encoders to retrieve and cross-encoders to rerank.

What happens when you change the embedding model?

Vectors from different models live in different spaces and cannot be compared, so the whole corpus must be re-embedded and the index rebuilt. Store the model name and version with every vector, build the new index in parallel, compare on your evaluation set, then switch traffic.

Why do some embedding models use different prefixes for queries and documents?

Questions are short and documents are long, so some models are trained asymmetrically and expect markers such as "query:" and "passage:". Leaving them out quietly lowers retrieval quality. Always follow the model card for how to format queries and documents.

How do you reduce the storage and memory cost of vectors?

Use fewer dimensions if the model supports truncation (Matryoshka-style embeddings), store vectors as half precision or int8 with scalar quantization, or use binary quantization for a fast first pass followed by rescoring with full vectors. Measure recall before and after each change.

How do you tune an approximate vector index?

Each index trades recall for speed. For HNSW, a larger graph degree and build effort improve recall at the cost of memory and build time, and a larger search effort at query time improves recall but adds latency. Compare against exact search on a sample to measure recall at your chosen settings.

Why does keyword search (BM25) still matter in RAG?

Embeddings capture meaning but can miss exact tokens: product codes, error messages, names, acronyms and rare terms. BM25 matches those exactly. Combining both, hybrid search, usually beats either alone, especially for technical and enterprise documents.

How does reciprocal rank fusion combine result lists?

Each document gets 1 / (k + rank) from every list it appears in, and the scores are added; k is often 60. It uses only ranks, so it works even though BM25 scores and cosine similarities are on different scales, and documents that rank well in both lists rise to the top.

def rrf(result_lists, k=60):
    scores = {}
    for results in result_lists:
        for rank, doc_id in enumerate(results, start=1):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

What is query rewriting and when does it help?

An LLM turns the user's question into a better search query: it fixes spelling, expands abbreviations, removes chit-chat and, in a conversation, replaces words like "it" with what they refer to. It helps most with short, vague or follow-up questions.

What is multi-query retrieval?

The LLM writes several differently worded versions of the question, each is searched, and the results are merged, for example with reciprocal rank fusion. Different wordings reach different relevant chunks, which raises recall at the cost of extra calls.

What is HyDE?

Hypothetical Document Embeddings: the LLM first writes a short hypothetical answer to the question, and you embed that answer instead of the question. An answer looks more like the documents than a question does, so it often retrieves better. The hypothetical text is only used for search, never shown to the user.

How do you handle questions that need several pieces of information?

Use query decomposition: the LLM splits a question such as "compare the leave policy in 2025 and 2026" into sub-questions, each is retrieved separately, and the answers are combined. For multi-hop questions, the answer to one step can shape the next query.

How do metadata filters improve retrieval?

Filters narrow the search to the right subset, such as one product, one year or one tenant, before or during the vector search, so similar but wrong documents never compete. An LLM can extract filters from the question, for example a year or a document type, and pass them as structured fields.

What is maximal marginal relevance (MMR)?

MMR picks results one at a time, balancing relevance to the query against similarity to results already picked. It stops the top results from being five near-copies of the same passage, so the prompt covers more distinct facts. A lambda setting controls the balance.

How many candidates should you rerank, and what does it cost?

A cross-encoder reranker scores each query-document pair, so cost and latency grow with the candidate count. Reranking 20 to 100 candidates is common. Measure how much recall and answer quality improve as you raise the number, and stop where the gain no longer justifies the extra latency.

What is GraphRAG?

GraphRAG uses an LLM to extract entities and relationships from documents into a knowledge graph, often with summaries of clusters of related entities. It helps with questions that connect many documents or ask for themes across a whole corpus, which plain chunk retrieval answers poorly. Building and updating the graph is costly.

What is agentic RAG?

Instead of one fixed retrieve-then-answer step, an agent decides what to do: which source or tool to search, whether the results are good enough, whether to rewrite the query and search again, and when to answer. It handles harder questions but is slower, costlier and harder to test.

What are corrective and self-reflective RAG?

They add checks to the loop. Corrective RAG grades the retrieved documents and, if they look irrelevant, rewrites the query or falls back to another source such as web search. Self-reflective approaches have the model judge whether retrieval is needed and whether its draft answer is supported by the context.

How do you build RAG for a multi-turn chat?

Rewrite each follow-up into a standalone question using the chat history, so "what about its price?" becomes "what is the price of the X200 plan?", then retrieve with that. Keep recent turns in the prompt but retrieve fresh context for every question.

Should you use RAG or text-to-SQL for structured data?

For questions about numbers in tables, such as totals, counts or trends, generating a SQL query and running it is more accurate than retrieving text chunks. RAG suits unstructured text. Many assistants route between the two, and text-to-SQL needs read-only access, schema descriptions and query validation.

What is semantic caching in a RAG system?

You store past questions with their answers and, when a new question's embedding is very close to a cached one, return the cached answer without retrieval or generation. It cuts cost and latency for repeated questions, but the threshold must be strict and cache entries must expire when documents change.

How do you write the generation prompt for RAG?

Put the instructions first: answer only from the provided context, cite the source of each claim, and say clearly when the context does not contain the answer. Then add the chunks, each with an ID and source, and finally the question. Ask for a short answer format your UI can show.

How do you add citations to RAG answers?

Label every chunk with an ID, ask the model to put the ID after each claim, then map IDs back to titles, pages or URLs in your code. Validate that cited IDs were actually in the context, and drop or flag sentences without support.

context = "\n\n".join(f"[{i}] ({c.source}) {c.text}" for i, c in enumerate(chunks, start=1))
prompt = f"Answer using only the sources below. Cite like [1].\n\n{context}\n\nQuestion: {question}"

How do you make a RAG system say "I don't know" when it should?

Tell the model to answer only from the context and give it an explicit fallback sentence. Also check retrieval scores: if the best reranker score is below a threshold, skip generation and return the fallback, or ask a clarifying question. Test with questions your documents cannot answer.

What do you do when retrieved documents contradict each other?

Prefer the newest or most authoritative source using metadata such as dates and document type, tell the model to mention the conflict instead of picking silently, and fix the root cause by removing outdated documents from the index.

How can retrieved content be used for prompt injection?

A document or web page can contain text such as "ignore your instructions" that the model may follow. Treat retrieved text as data: mark it clearly in the prompt, never let it trigger tools or actions without checks, limit what the model can do, and filter suspicious content during indexing.

Which metrics measure retrieval quality?

Recall at k asks whether a relevant chunk appears in the top k. Precision at k asks how many of the top k are relevant. Mean reciprocal rank rewards putting the first relevant chunk high, and nDCG also weighs graded relevance. Measure retrieval separately from answers so you know which stage fails.

def recall_at_k(retrieved_ids, relevant_ids, k=5):
    hits = set(retrieved_ids[:k]) & set(relevant_ids)
    return len(hits) / len(relevant_ids)

Which metrics measure answer quality in RAG?

Faithfulness checks that every claim is supported by the retrieved context. Answer relevance checks that the answer addresses the question. Correctness compares with a reference answer when you have one. Context precision and context recall judge whether the right chunks were retrieved. Frameworks such as Ragas compute these with an LLM judge.

How do you build an evaluation set for a RAG system?

Collect real user questions where possible, add questions written by domain experts, and generate more with an LLM from your documents. For each, record the expected answer and the source chunks. Include unanswerable and tricky questions, and keep the set versioned so changes can be compared.

What are the risks of using an LLM as a judge?

Judges can prefer longer answers, answers in their own style, or the first option shown, and they may miss domain errors. Use clear rubrics, ask for a short reason before the score, check a sample against human ratings, and keep the judge model fixed when you compare versions.

How do you enforce document permissions in RAG?

Store access fields such as tenant, team or user groups with every chunk and apply them as filters inside the retrieval query, before results reach the model. Filtering after generation is too late. For strict isolation, use a separate index or namespace per tenant.

How do you keep a RAG index fresh?

Re-index incrementally: detect new, changed and deleted documents with timestamps or content hashes, re-chunk and re-embed only those, and delete the old chunks by document ID. Show the document date in citations so users can judge freshness.

A RAG system gives a wrong answer. How do you debug it?

Trace the request stage by stage. Was the right document parsed correctly? Was it split so the fact sits in one chunk? Did the query, or its rewrite, retrieve that chunk, and did the reranker keep it? Was it in the prompt? Only if all of that is right is it a generation problem.

What does the latency budget of a RAG request look like?

Typical stages are query rewriting (one LLM call), embedding the query, vector and keyword search, reranking, and generation, which usually dominates. Cut latency by skipping rewriting for clear questions, running searches in parallel, limiting rerank candidates, caching, and streaming the answer.