RAG evaluation: test retrieval, answers and access

Three layers of evidence

A RAG system retrieves context and uses it to generate an answer. Evaluation should distinguish whether the right evidence was found, whether the answer is supported by that evidence and whether the caller was allowed to receive it. Frameworks may name their metrics differently; preserve the definition as well as the label.

LangChain’s RAG evaluation tutorial separates retrieval relevance, groundedness, answer relevance and correctness. Those distinctions are useful starting points, but a product still needs a task-specific rubric and an access-control test suite.[1]

Build cases with answer requirements and source identity

For each case, record the question, caller context, corpus version, relevant source IDs, expected claims and acceptable no-answer behavior. Include questions with several required documents, conflicting versions, ambiguous wording and no accessible answer. A list of questions without an evidence reference is hard to diagnose when a score changes.

Define relevance at the unit the retrieval system returns: document, section or chunk. If several chunks from one document all count as separate relevant items, the denominator can reward duplication. Keep a mapping back to the authoritative source and decide how duplicates affect the metric.

• Ordinary answerable cases with current evidence.
• Multi-source cases where one document alone is insufficient.
• No-answer cases, including an answer that exists only in inaccessible content.
• Freshness cases with superseded or deleted documents.
• Adversarial content and cross-tenant requests.

Worked example: two required documents

Consider a synthetic five-document corpus. A question requires documents A and B. The top three retrieved documents are A, C and D. For this example, relevance is binary at the document level: only A and B are relevant. Precision@3 is 1/3, or 33.3%. Recall@3 is 1/2, or 50%.

The second retrieval result provides all required documents, but it does not prove that the answer uses them correctly. Evaluate whether each required claim is supported, whether the answer handles disagreement and whether citations resolve to the correct versions. A generator can still fail with perfect recall on this small case.

Grade the answer at the level of its claims

Separate required information from optional helpful detail. For a policy question, the rubric might require the current limit, its effective date and the relevant exception. It should reject an answer that states the wrong limit even if the tone and formatting are excellent.

Automated judges can help scale some judgments, but their agreement with a human rubric needs calibration. Keep critical policy checks deterministic where possible and inspect disagreements. A score generated by another model is evidence to validate, not ground truth by default.

Test access as a separate release condition

A retrieval benchmark should include at least two tenants, several roles and a permission change after indexing or caching. Verify that denied source content is not passed to the generator, leaked through a snippet or returned through a citation. An answer can be factually correct and still be unacceptable because the caller should not have seen it.

Do not average a cross-tenant disclosure into ordinary relevance scores. Make it a named failure that blocks the affected release. Track attempted but blocked access separately so the team can distinguish a working boundary from a committed disclosure.

Change one layer and preserve the comparison

When retrieval recall is poor, inspect the corpus and search path before changing the generator. When evidence is present but support is poor, inspect context assembly and generation. Keep the question set, corpus snapshot and grader version fixed for a controlled comparison; document any intentional changes.

The final report should include per-case failures, not only averages. A CTO should be able to see which tasks improved, which regressed and which boundary remains unresolved. That is the evidence needed to decide whether to ship, investigate or narrow the feature.