RAGAS (Retrieval Augmented Generation Assessment) is the industry-standard open-source framework for evaluating RAG pipelines. Its defining advantage is reference-free evaluation: you can measure retrieval quality, generation faithfulness, and answer relevance without requiring human-annotated ground-truth answers for every test case. This makes it practical to evaluate RAG systems at scale, where manual annotation would be prohibitively expensive.
RAGAS provides a suite of complementary metrics that decompose RAG performance into its constituent parts. Rather than asking “is this answer good?” as a single question, RAGAS separately measures whether the retriever found the right documents (context precision and recall), whether the generator stayed faithful to those documents (faithfulness), and whether the answer actually addresses the user’s question (answer relevancy). This decomposition is critical because RAG failures are notoriously hard to diagnose—when an answer is wrong, you need to know whether the retriever or the generator is at fault.
Originally introduced in a 2023 arXiv paper by the Exploding Gradients team, RAGAS has become the default evaluation framework used alongside tools like LangSmith, Langfuse, and DeepEval. It supports both LLM-based metrics (which use an evaluator LLM to score outputs) and non-LLM-based metrics (which use embeddings or string matching).
RAGAS evaluates RAG pipelines along four primary dimensions. Together, these form the composite “RAGAS score” that gives an overall quality assessment:
Faithfulness measures the factual consistency of the generated answer against the retrieved context. It answers the question: “Can every claim in the response be traced back to the provided context?” This is the primary hallucination detection metric.
Faithfulness = (Number of claims in the response supported by retrieved context) / (Total number of claims in the response). Score ranges from 0 to 1, where 1.0 means every statement in the answer is grounded in the context.
The calculation works in two steps. First, an LLM decomposes the generated answer into individual atomic claims (statements of fact). Then, for each claim, a second LLM call checks whether it can be inferred from the retrieved context chunks. Claims that introduce information not present in the context are marked as unfaithful.
# Faithfulness evaluation example
from ragas.metrics import faithfulness
from ragas import evaluate
from datasets import Dataset
# Prepare evaluation data
eval_data = Dataset.from_dict({
"question": ["What is the capital of France?"],
"answer": ["The capital of France is Paris, which has a population of 2.1 million."],
"contexts": [["France is a country in Western Europe. Its capital city is Paris."]]
})
# Run evaluation
result = evaluate(eval_data, metrics=[faithfulness])
print(result)
# The claim "capital is Paris" is supported (faithful)
# The claim "population of 2.1 million" is NOT in context (unfaithful)
# Faithfulness score: 0.5 (1 out of 2 claims supported)Answer relevancy measures how well the generated answer addresses the original question. A high score means the answer is focused and pertinent; a low score means the answer wanders off-topic or includes irrelevant information, even if that information is factually correct.
RAGAS reverse-engineers the process: it generates N variant questions from the answer using an LLM, then computes the mean cosine similarity between these generated questions and the original question. If the answer truly addresses the question, you should be able to reconstruct the original question from it.
This is a clever approach because it avoids needing a reference answer. Instead, it tests whether the answer contains sufficient information to reconstruct the question’s intent. An answer that includes lots of irrelevant content will generate questions that diverge from the original, resulting in lower cosine similarity scores.
Context precision evaluates the signal-to-noise ratio of the retrieved context. It asks: “Are the retrieved chunks that were actually relevant to the answer ranked at the top of the results?” High precision means your retriever is not wasting context window space with irrelevant documents.
The metric is computed as the mean precision@k across all retrieved chunks. For each chunk, RAGAS checks whether it was useful in answering the question (by comparing against a reference answer or using LLM judgment). Chunks that contribute to the answer should ideally appear at ranks 1, 2, 3—not buried at positions 8, 9, 10.
# Evaluating all four core metrics
from ragas.metrics import (
faithfulness,
answer_relevancy,
context_precision,
context_recall
)
from ragas import evaluate
eval_data = Dataset.from_dict({
"question": [
"How does hybrid search combine semantic and keyword search?"
],
"answer": [
"Hybrid search combines dense vector embeddings for semantic matching "
"with sparse BM25 scoring for keyword precision. Results from both "
"systems are merged using fusion algorithms like RRF or RSF."
],
"contexts": [[
"Hybrid search merges semantic dense embeddings with keyword-based "
"sparse BM25 search to leverage strengths of both approaches.",
"Reciprocal Rank Fusion (RRF) ranks documents by ordinal position "
"from multiple search systems.",
"Vector databases store high-dimensional embeddings for similarity search."
]],
"ground_truth": [
"Hybrid search combines semantic search using dense vector embeddings "
"with keyword search using BM25/sparse vectors, merging results via "
"fusion algorithms such as RRF or RSF."
]
})
result = evaluate(
eval_data,
metrics=[faithfulness, answer_relevancy, context_precision, context_recall]
)
print(result)
# Returns individual scores for each metric plus a composite RAGAS scoreContext recall measures the completeness of retrieval—did the retriever fetch all the information necessary to answer the question? Unlike precision (which measures noise), recall measures coverage. A retriever with high precision but low recall returns few results that are all relevant but misses critical information.
Context recall requires a ground-truth reference answer. RAGAS decomposes the reference answer into individual claims, then checks how many of those claims can be found in the retrieved context. The score is the ratio of claims covered by the context to the total claims in the reference.
Context precision and recall capture the classic information retrieval trade-off. Increasing the number of retrieved chunks (e.g., top-20 instead of top-5) typically improves recall but hurts precision. RAGAS measures both so you can find the optimal balance for your use case.
RAGAS metrics fall into two categories:
The composite RAGAS score is the harmonic mean of the four core metrics. This penalizes pipelines that score well on some dimensions but poorly on others—a pipeline with 0.95 faithfulness but 0.3 context recall will score lower than one with 0.8 across all four metrics. The harmonic mean ensures balanced performance across all aspects of the RAG pipeline.
In production, RAGAS evaluations are typically integrated into CI/CD pipelines. After any change to the chunking strategy, embedding model, retriever configuration, or prompt template, the full evaluation suite runs against a synthetic test dataset (generated by RAGAS’s own testset generator). Results are tracked over time in observability platforms like Langfuse or LangSmith, enabling trend analysis and regression detection.
When users report bad answers, RAGAS metrics pinpoint the root cause. Low context recall with high faithfulness means the retriever missed relevant documents but the generator was honest about what it had. Low faithfulness with high context precision means the retriever did its job but the generator hallucinated beyond the provided context. This diagnostic capability is RAGAS’s most valuable feature in production.
When evaluating whether to switch from pure semantic search to hybrid search, or whether to add a reranker, RAGAS provides apples-to-apples comparison. Run both pipeline configurations against the same test dataset and compare context precision and recall scores. Teams typically see 10–25% improvements in context precision when adding a cross-encoder reranker, which RAGAS quantifies precisely.
RAGAS metrics reveal the impact of chunking decisions. Smaller chunks (200 tokens) typically show higher context precision (less noise per chunk) but lower context recall (relevant information split across chunks). Larger chunks (1000 tokens) show the inverse pattern. RAGAS enables data-driven chunk size optimization rather than guesswork.
Production RAG systems evaluate a random sample of queries daily using RAGAS metrics. Faithfulness scores trending downward may indicate model drift or corpus staleness. Context recall drops after a corpus update may indicate broken chunking for new document formats. These metrics serve as the “vital signs” of a RAG system in production.