Faithfulness (also called groundedness) is the most critical generation-side metric in RAG evaluation. It measures whether the claims in a generated answer can be entirely inferred from the retrieved context. In other words: did the LLM stick to the facts it was given, or did it make things up?
This is the metric that directly measures hallucination. When an LLM generates a response that sounds authoritative but contains claims not supported by the provided context, the faithfulness score drops. Unlike retrieval metrics (Context Precision and Recall) that evaluate the retriever, faithfulness evaluates the generator — it tells you whether the LLM is using the retrieved information correctly rather than drawing on its parametric knowledge or fabricating facts.
In production RAG systems, faithfulness failures are the most dangerous kind of error because they’re the hardest for end users to detect. A confident, well-written response that contains one unsupported claim can erode user trust in the entire system. This is why faithfulness monitoring is considered table stakes for any production RAG deployment.
Faithfulness is computed as the ratio of supported claims to total claims in the generated response:
Faithfulness = |Claims in Response Supported by Context| / |Total Claims in Response|
Values range from 0 (no claims are grounded) to 1 (every claim is supported by retrieved context). A score of 1.0 means zero hallucination relative to the provided context.
RAGAS computes faithfulness through two sequential LLM calls:
Step 1 — Claim Extraction: The generated answer is decomposed into atomic claims. Each claim is a single, independently verifiable factual statement. The LLM is prompted with instructions like: “Given the following response, break it down into individual factual claims. Each claim should be a single sentence that can be verified independently.”
Response: "Python was created by Guido van Rossum in 1991.
It uses indentation for code blocks and supports multiple
programming paradigms including OOP and functional programming."
Extracted Claims:
1. Python was created by Guido van Rossum
2. Python was created in 1991
3. Python uses indentation for code blocks
4. Python supports object-oriented programming
5. Python supports functional programmingStep 2 — Natural Language Inference (NLI) Verification: Each extracted claim is checked against the full retrieved context. The LLM acts as an NLI judge, classifying each claim as:
Question: "When was Python released and who created it?"
Retrieved Context:
"Python is a high-level programming language created by
Guido van Rossum. The first version, Python 0.9.0, was
released in February 1991."
Generated Response:
"Python was created by Guido van Rossum and first released
in 1991. It quickly became one of the most popular
programming languages in the world."
Claim Extraction:
1. Python was created by Guido van Rossum → SUPPORTED ✓
2. Python was first released in 1991 → SUPPORTED ✓
3. Python quickly became one of the most
popular languages in the world → NOT SUPPORTED ✗
(context says nothing about popularity)
Faithfulness = 2/3 = 0.667The third claim about popularity sounds reasonable and is probably true, but it cannot be inferred from the retrieved context. This is exactly the kind of “plausible hallucination” that faithfulness is designed to detect — the LLM drew on its parametric knowledge instead of limiting itself to the provided context.
There are two main approaches to the NLI verification step:
LLM-as-a-Judge (RAGAS default): Uses a general-purpose LLM (like GPT-4 or Claude) to perform the entailment check. Advantages: high accuracy on nuanced claims, handles paraphrasing well, no model training needed. Disadvantages: expensive (requires LLM calls per claim), non-deterministic, susceptible to judge bias.
Classifier Models (e.g., HHEM, DeBERTa-based NLI): Uses a fine-tuned classification model that takes (context, claim) pairs and outputs entailment probability. Vectara’s HHEM-2.1-Open is a popular choice. Advantages: fast, deterministic, cheap (runs locally). Disadvantages: less accurate on complex or domain-specific claims, requires a model to be hosted.
Use LLM-as-a-judge for high-stakes evaluations (pre-deployment audits, compliance reviews) where accuracy matters most. Use classifier models for continuous monitoring in production where you need low-latency, low-cost evaluation on every response.
from ragas import evaluate
from ragas.metrics import faithfulness
from datasets import Dataset
eval_data = Dataset.from_dict({
"question": [
"What are the side effects of aspirin?"
],
"contexts": [
[
"Common side effects of aspirin include stomach upset, "
"heartburn, and nausea. Serious side effects include "
"gastrointestinal bleeding and allergic reactions."
]
],
"answer": [
"Aspirin can cause stomach upset, heartburn, nausea, "
"and gastrointestinal bleeding. It should not be given "
"to children under 16 due to the risk of Reye's syndrome."
]
})
result = evaluate(eval_data, metrics=[faithfulness])
print(result)
# {'faithfulness': 0.8}
# The Reye's syndrome claim is NOT in the context — hallucinationA crucial distinction: faithfulness measures whether the answer is grounded in the retrieved context, not whether the answer is factually correct. An answer can be:
This is why faithfulness is paired with retrieval metrics: the retriever’s job is to find correct information; the generator’s job is to stay faithful to whatever was retrieved.
The primary application of faithfulness scoring is detecting hallucinations before they reach users. In production systems, faithfulness is computed on every response (or a statistical sample), and responses below a threshold trigger fallback behavior:
When evaluating whether to switch LLM providers or model versions, faithfulness provides a direct comparison of generation quality. Some models are inherently more “obedient” to context — smaller models tend to hallucinate more because they have less capacity to distinguish between their parametric knowledge and the provided context. Testing on a standardized evaluation set reveals these differences quantitatively.
Faithfulness scores provide objective feedback when iterating on system prompts. Adding instructions like “Only answer based on the provided context. If the information isn’t in the context, say you don’t know” can significantly improve faithfulness. You can A/B test prompt variants and measure the impact:
# Common prompt patterns that improve faithfulness:
# 1. Explicit grounding instruction
"Answer ONLY based on the following context. Do not use
any external knowledge."
# 2. Citation requirement
"For each claim in your answer, cite the specific passage
from the context that supports it."
# 3. Uncertainty expression
"If the context does not contain enough information to
fully answer the question, explicitly state what is missing
rather than guessing."Faithfulness thresholds vary by domain:
When faithfulness is low, the claim-level decomposition helps pinpoint why. Common patterns include: