RAG Evaluation: Metrics, Test Sets, and Frameworks

RAG evaluation checks two things separately: whether your application retrieves the evidence a question needs, and whether its answer uses that evidence correctly. Build a representative test set, run a fixed version of the application, and inspect both stage-level scores and individual failures before accepting a change.

RAG, retrieval-augmented generation, supplies outside information to an AI before it answers. A fluent response can still rely on the wrong passage or omit an important exception. Tools like fenbs keep the team’s evaluation decisions, follow-up tasks and lessons available to connected assistants; your evaluation runner performs the measurements.

What is the evaluation process?

1. Define the task and failure conditions. Say what a useful answer must contain, when it should decline to answer and which failures block release.
2. Create questions with expected evidence and reviewed answer requirements. Include ordinary questions, difficult cases and questions the sources cannot answer.
3. Run one recorded application version against the set. Save retrieved evidence, generated answers, timing and the configuration needed to repeat the run.
4. Score retrieval and answers separately. Review automated judgments against examples that a person has already checked.
5. Compare changes on the same cases, investigate regressions and create specific fixes before approving the next version.

If the architecture is unfamiliar, read what RAG does first. The evaluation should describe success for your application, rather than borrowing a benchmark’s definition of a good answer without checking whether it fits.

What belongs in a representative test set?

Write questions in the language your users actually use. For a hypothetical product-support assistant, include a straightforward setup question, an exact error message, a question using an older product name and a procedure with an exception. Keep expected source passages with each case.

Add cases with missing evidence, conflicting revisions and restricted documents. An unauthorized user should not receive private information, even when the answer would otherwise be accurate. A question with no supported answer should reward an appropriate explanation of the gap.

A useful case contains the question, user access context, required evidence IDs, answer requirements and failure category. Record source versions so a later document edit does not silently change what counts as correct. Describe required facts rather than demanding one exact sentence.

Keep some cases separate from daily tuning. Otherwise, you can improve the examples you see while missing the same failure in unfamiliar wording. Save the person’s approved test scope and release conditions in fenbs Decisions and rules so another assistant does not quietly redefine success.

Which metrics explain retrieval quality?

Define what counts as relevant before calculating a score. If two passages express the same rule, either may be acceptable; if a question requires a rule and its exception, retrieving only the rule is incomplete. Document the unit you score: whole documents, passages or factual requirements.

Ragas documents context recall↗ (opens in a new tab) as a measure of whether required information was retrieved. Its model-based version can use claims from a reference answer as the comparison. Read the metric’s inputs carefully; that differs from counting matched document IDs.

How do you judge the final answer?

Score correctness, support and usefulness as different properties. Correctness asks whether the answer meets the reviewed requirements. Support asks whether its claims follow from the evidence the model received. Usefulness asks whether it addresses the user’s question clearly enough to act on.

Ragas faithfulness↗ (opens in a new tab) checks claims against retrieved context. That is valuable, but it cannot make an outdated source current. An answer may faithfully repeat an old procedure and still be wrong for today’s task.

Inspect citations independently. Open the cited passage, check that it supports the claim and confirm that the answer includes necessary conditions. Also test refusal behavior: declining every difficult question may avoid unsupported claims while making the application unhelpful.

For example, a support answer might name the correct setting but omit that it requires a restart. Mark the missing condition explicitly rather than assigning an unexplained overall score. That gives the developer a fix to investigate and the next reviewer a clear retest.

Which evaluation framework should you use?

Choose after you know the evidence and scores you need. Ragas can supply individual retrieval and answer metrics, as its metric documentation shows. LangSmith’s RAG evaluation tutorial↗ (opens in a new tab) demonstrates datasets, experiment runs and separate evaluators for answer correctness, relevance, grounding and retrieval relevance.

A small script may be enough for an early application if it saves inputs, evidence, outputs and scores consistently. A framework becomes useful when you need repeatable experiments, multiple evaluators and a manageable review workflow. Keep your test cases exportable so changing tools does not erase the meaning of earlier results.

Model judges, AI systems that score another answer, need their own checks. Give them a specific rubric and compare their decisions with human-reviewed examples. Review disagreements, record the judge model and prompt, and keep those fixed while comparing application changes.

How do you decide whether a change is better?

Compare the same questions, source snapshot and access context across versions. Change one major component at a time when possible. Look at which questions improved or regressed, rather than only the average score across the set.

Break results into meaningful groups: exact lookups, questions needing several passages, missing-evidence cases and access tests. Review response time alongside quality. A better average may conceal a serious failure in the group your users rely on most.

Keep the lesson from each accepted fix in fenbs: what failed, what you learned and what to check differently. That gives the next assistant more direction than an unexplained score change.

Broader action-taking systems need additional checks covered in AI agent evaluation. If stored work state affects the answer, distinguish it from source retrieval using agent memory versus RAG. These are separate causes of failure, even when they appear in the same application.

How to do this with fenbs?

1. Create a free account, then add a project for the application and tasks for the next evaluation run.
2. Record the person’s chosen release criteria in Decisions and rules. Keep pending changes to those criteria open for a person to decide.
3. Create tasks for specific failures, with the failed case reference, suspected stage and planned retest. Keep sensitive source content in its approved storage.
4. Save Lessons learned after reviewing failures, and save Where we left off with the next experiment. Ask your connected assistant to read that context before tuning again.

fenbs does not calculate RAG metrics or run experiments. Use your evaluation framework for scores and detailed traces; use fenbs when decisions and follow-up work must survive changes of assistant. It is free for one person, with no credit card.

The context engineering hub connects evaluation with the wider question of what evidence an assistant receives.