Home · About · Services · Industries · Projects · Products · Process · Blog · FAQ · Contact
How to evaluate a RAG assistant before launch How to evaluate a RAG assistant before launch: build a real question set, score retrieval and answers separately, test refusals and permissions, set thresholds.
A RAG assistant usually looks finished a week before it is ready. The demo questions work and the citations appear. Then real users ask about an edge case, a superseded policy or something the documents never covered, and the assistant answers confidently anyway.
To evaluate a RAG assistant before launch, build a test set of 100 to 200 real questions with known source passages, then score retrieval (is the right passage in the top results?) separately from the answer (is it correct, grounded in that passage and properly cited?). Add questions it should refuse, set pass thresholds, and rerun the set on every change.
This is the test plan we use before putting a knowledge assistant in front of staff or customers. It assumes you already have a working pipeline; if you are still deciding how to split documents, read our post on RAG chunking strategies first.
Build the test set from real questions The test set is the most valuable thing you will produce, and the step most teams rush. Questions you write after reading the documents are too easy, because they reuse the documents' own wording. Real users ask in their own words, with typos, abbreviations and missing context.
Gather questions from support tickets, email, chat logs and short sessions with future users. For each, a subject expert records the expected answer in a sentence or two and the document and section that support it. Aim for a spread like this:
Common questions, the ones asked every week, roughly half the set. Questions whose answer is split across two documents or two sections. Questions where an old and a new version of a document disagree. Questions involving numbers, dates, tables or eligibility conditions. Questions the documents do not answer at all, about one in ten. Questions a given user is not allowed to see the answer to, if you have access groups. LLM-generated questions are useful for coverage on large corpora, where experts cannot write hundreds of questions. Treat them as a supplement and review a sample by hand: they tend to paraphrase the chunk they came from, which makes retrieval look better than it is.
Store the set as JSON Lines in version control next to the code, so changes to it are reviewed like any other change. The figures below are placeholders; use what your own policy says, checked by its owner.
{"id": "leave-014", "question": "can i carry forward unused EL to next year?", "expected": "Yes, up to 30 days of earned leave carry forward; the rest lapses on 31 March.", "sources": ["hr-leave-policy-2026#carry-forward"], "groups": ["staff"], "type": "common"}
{"id": "oos-003", "question": "what is the wifi password in block C?", "expected": null, "sources": [], "groups": ["staff"], "type": "unanswerable"}Score retrieval on its own first Most bad answers start with bad retrieval. If the right passage never reaches the model, no prompt will save the answer. Two numbers are enough to start: recall at k (did any expected source appear in the top k results?) and mean reciprocal rank (how high did the first correct one appear?).
import json
def evaluate_retrieval(cases, retrieve, k=5):
hits, rr = 0, 0.0
misses = []
for case in cases:
if not case["sources"]:
continue # unanswerable questions are scored elsewhere
results = retrieve(case["question"], groups=case["groups"], top=k)
ids = [r["source_id"] for r in results]
ranks = [i + 1 for i, s in enumerate(ids) if s in case["sources"]]
if ranks:
hits += 1
rr += 1 / ranks[0]
else:
misses.append(case["id"])
n = sum(1 for c in cases if c["sources"]) or 1
return {"recall_at_k": hits / n, "mrr": rr / n, "misses": misses}
with open("eval/questions.jsonl", encoding="utf-8") as fh:
cases = [json.loads(line) for line in fh if line.strip()]The list of misses matters more than the averages. Read each one and sort it into a cause: the chunk was split badly, the question uses a term the document does not, a metadata filter excluded the right document, or the expected source in the test set is itself wrong. Each cause has a different fix, and on a first run a fair share of misses are errors in the test set itself.
Run this with the same k the application uses, and with hybrid search switched on if production uses it. Measuring pure vector search and then shipping hybrid with a reranker tells you very little.
Score the answers: correct, grounded, cited Once retrieval is reasonable, judge the generated answers. We score each answer on four separate criteria rather than one overall grade, which would hide what failed.
Correctness: does it agree with the expected answer on the facts that matter? Groundedness: is every claim supported by the retrieved passages, rather than the model's general knowledge? Citation accuracy: do the cited sources actually contain what the answer attributes to them? Completeness: does it include the conditions and exceptions, not just the headline? Expert review is the reference standard, but it does not scale to every change. The usual approach is an LLM judge: a separate model call that receives the question, expected answer, retrieved passages and generated answer, and returns a verdict with a short reason. Ragas, DeepEval, the evaluators in Azure AI Foundry and promptfoo all package this pattern. A plain prompt you control also works:
You are grading an answer from a document assistant.
Question: {question}
Reference answer: {expected}
Retrieved passages:
{passages}
Assistant answer: {answer}
Return JSON with these fields:
- "correct": true if the answer agrees with the reference on every key fact
- "grounded": true if every claim is supported by the retrieved passages
- "citations_ok": true if each cited passage supports the claim it is attached to
- "reason": one sentence explaining any false valueDo not trust the judge until you have checked it. Have an expert grade around 50 answers by hand, run the judge on the same answers and study the disagreements. If it is lenient on partly correct answers, tighten the prompt or split the criterion. Use a strong model, ask for binary verdicts rather than scores out of ten, and keep the judge prompt fixed so runs stay comparable.
Test refusals, permissions and awkward inputs A launch-ready assistant knows when to say it does not know. For unanswerable questions, the correct behaviour is a clear statement that the documents do not cover it, ideally with a pointer to who does. Count any confident answer as a failure, even if it happens to be true, because it came from the model rather than your documents.
Then test what only shows up once real people use it:
Permission leaks: run questions as a user without access and confirm restricted passages never appear in retrieval results, not just in the answer. Stale content: ask about a policy that changed and check the current version wins. Prompt injection: include documents or questions with instructions such as "ignore your rules" and confirm the assistant keeps to its system prompt. Off-topic and abusive inputs: check the tone and that it steers back to its purpose. Follow-up questions: test short conversations, where "what about contract staff?" depends on the previous turn. Language mix: if users write in Hindi, Hinglish or another language, include those questions. Permission checks belong in the retrieval layer, enforced by filters on access metadata. The evaluation is there to prove the filters work, not to replace them.
Set launch thresholds and measure the running costs Agree the pass marks with the business owner before the first full run, so the numbers are not negotiated afterwards. Thresholds depend on the stakes: an internal IT helpdesk can launch with rougher edges than an assistant answering patients or students about fees and deadlines. As a starting point, we ask for:
High retrieval recall on common questions, with every miss explained. No ungrounded answers on the questions tagged as high-risk, such as money, health, eligibility and deadlines. Correct refusals on nearly all unanswerable questions. Zero permission leaks. This one is not a percentage. Latency measured at the 95th percentile, not the average, and cost per question within budget. Record tokens and time per question during the run, then multiply by expected daily volume to get a monthly running cost before launch, not on the first invoice. If it is too high, test a smaller model or fewer retrieved passages against the same set and see what quality you give up.
Keep evaluating after launch The test set is not a one-off gate. Run the retrieval checks in CI on every change to chunking, prompts, models or search settings, and the full answer evaluation before each release. When a provider retires a model version, as Azure OpenAI does on a published schedule, the same set tells you whether the replacement behaves.
In production, log every question, the retrieved sources and the answer, with a thumbs up or down for users. Review the negative feedback weekly, and add the real failures to the test set. Over a few months the set becomes a record of how your users actually ask.
Pre-launch checklist 100 to 200 real questions with expert answers and source passages, in version control. Retrieval recall and MRR measured with production settings, and every miss reviewed. Answers scored separately for correctness, groundedness, citations and completeness. LLM judge checked against human grades before you rely on it. Unanswerable, permission, stale-content and injection cases included and passing. Thresholds agreed with the business owner in writing before the run. Cost and 95th-percentile latency per question recorded. Evaluation wired into CI, with production feedback flowing back into the set. If you can tick all of these, you launch knowing where the assistant is weak.
Infigini IT Services · 11 Kanwali Road, Dehradun 248001, India · info@infigini.com · +91 95684 87789