RAG retrieval relevance and answer faithfulness · Version 0.1.0
Retrieval contextual relevance
Whether the retrieved context is relevant to the question asked. A retrieval-side metric: it isolates the search/index layer from the generation layer so failures can be attributed correctly.
Execution mode: Synthetic systems only — no live customer endpoints
What it needs to run
- Question set
- Retrieved context per question
- Judge model configuration (pinned)
Permissions and data rules
- Written authorization from the system owner
- Synthetic data and synthetic canaries only — no real personal data
Known limitations
Relevance is judged, not verified; domain-specific relevance needs a calibration set before customer-facing use.
False-positive risks
Broadly on-topic but unhelpful context can score well under a lenient judge rubric.
What is measured
mean relevance verdict (judge estimate) — reported per case against the denominator "questions evaluated". Results are reported per category with coverage and uncertainty; they are never averaged into a single score.
How results are judged
thresholds set per engagement before the run; inconclusive reported separately
Framework mappings
- NIST AI RMF — MEASURE
References
Methodology reviewed: 2026-11-08 · Owner: ComplianceEU methodology