RAG retrieval relevance and answer faithfulness · Version 0.1.0
Answer faithfulness to retrieved context
Whether the generated answer is faithful to what was retrieved — a generation-side metric that complements retrieval relevance. Unfaithful answers on relevant context point at the generator, not the index.
Execution mode: Synthetic systems only — no live customer endpoints
What it needs to run
- Question set
- Retrieved context per question
- Generated answers
Permissions and data rules
- Written authorization from the system owner
- Synthetic data and synthetic canaries only — no real personal data
Known limitations
Judge-based; paraphrase-heavy but faithful answers can be misread by strict judges.
False-positive risks
Answers that add correct general knowledge beyond the context are flagged by design — the policy must say whether that is a failure for this system.
What is measured
faithfulness rate (judge estimate) — reported per case against the denominator "answers evaluated". Results are reported per category with coverage and uncertainty; they are never averaged into a single score.
How results are judged
thresholds set per engagement before the run; inconclusive reported separately
Framework mappings
- OWASP LLM Top 10 — LLM09 Misinformation
- NIST AI RMF — MEASURE
- NIST AI RMF — MANAGE
References
Methodology reviewed: 2026-11-08 · Owner: ComplianceEU methodology