Citation and Coverage Validators are regex-driven evaluators that verify whether an LLM’s response correctly cites sources from its retrieved context. In retrieval-augmented generation (RAG) systems, the model is expected to ground its answers in specific documents by referencing them with citation markers like [resource_id] or [doc_3]. These validators parse those citation spans, compare them against the set of actually-retrieved documents, and score the response for citation accuracy and coverage.
The problem these validators solve is trust verification at scale. When an RAG system serves thousands of queries per day, manual spot-checking of citations is impractical. Automated validators run as part of the evaluation pipeline, flagging responses that hallucinate citations (referencing documents not in the context), miss key sources (failing to cite highly relevant documents), or cite out-of-range identifiers that don’t correspond to any document.
In a Go-based RAG evaluation pipeline, these validators plug into the embeddable evaluation runner as Evaluator implementations. They receive the generated response along with metadata about which documents were retrieved, extract all citation markers via regex, and produce a structured result containing precision (what fraction of citations are valid), recall (what fraction of relevant documents are cited), and a list of specific violations.
The validator scans the LLM output using a configurable regular expression pattern (typically \[([a-zA-Z0-9_]+)\]) to extract all citation markers. Each match is parsed into a resource identifier that is then validated against the set of documents provided in the retrieval context. The regex pattern is configurable because different systems use different citation formats: bracket notation, superscript numbers, or inline footnotes.
Citation precision measures the fraction of extracted citations that reference valid, actually-retrieved documents. A precision of 1.0 means every citation in the response points to a real document from the context. Low precision indicates hallucinated citations—the model is referencing documents that were never retrieved, which undermines user trust and can propagate misinformation.
Coverage measures what fraction of the relevant retrieved documents are actually cited in the response. A coverage score of 1.0 means the model referenced every relevant document. Low coverage suggests the model is ignoring parts of its context, potentially missing important information. The “relevant” set can be all retrieved documents or a subset marked as ground-truth relevant by human annotators.
Beyond precision and recall, the validator detects out-of-range citations: identifiers that syntactically match the citation pattern but fall outside the valid ID space. For example, if the context contains documents doc_1 through doc_5, a citation of doc_12 is out-of-range. These are a specific subclass of hallucinated citations that often indicate the model is confabulating document IDs rather than copying them from the context.
The validator operates in three phases: extraction, validation, and scoring. Each phase produces intermediate data that feeds into the final evaluation result.
package eval
import (
"regexp"
"strings"
)
type CitationExtractor struct {
pattern *regexp.Regexp
}
func NewCitationExtractor(pattern string) *CitationExtractor {
if pattern == "" {
pattern = `\[([a-zA-Z0-9_]+)\]`
}
return &CitationExtractor{
pattern: regexp.MustCompile(pattern),
}
}
type Citation struct {
ResourceID string
StartPos int
EndPos int
RawMatch string
}
func (ce *CitationExtractor) Extract(text string) []Citation {
matches := ce.pattern.FindAllStringSubmatchIndex(text, -1)
citations := make([]Citation, 0, len(matches))
for _, match := range matches {
fullStart, fullEnd := match[0], match[1]
groupStart, groupEnd := match[2], match[3]
citations = append(citations, Citation{
ResourceID: text[groupStart:groupEnd],
StartPos: fullStart,
EndPos: fullEnd,
RawMatch: text[fullStart:fullEnd],
})
}
return citations
}
func (ce *CitationExtractor) UniqueIDs(text string) []string {
citations := ce.Extract(text)
seen := make(map[string]bool, len(citations))
unique := make([]string, 0, len(citations))
for _, c := range citations {
id := strings.ToLower(c.ResourceID)
if !seen[id] {
seen[id] = true
unique = append(unique, c.ResourceID)
}
}
return unique
}package eval
import (
"context"
"fmt"
"strings"
)
type CitationValidator struct {
extractor *CitationExtractor
}
func NewCitationValidator(citationPattern string) *CitationValidator {
return &CitationValidator{
extractor: NewCitationExtractor(citationPattern),
}
}
func (cv *CitationValidator) Name() string {
return "citation_coverage"
}
func (cv *CitationValidator) Evaluate(
ctx context.Context, ex Example, out Output,
) (EvalResult, error) {
retrievedIDs, ok := ex.Input["retrieved_ids"].([]interface{})
if !ok {
return EvalResult{}, fmt.Errorf("missing retrieved_ids in example input")
}
retrievedSet := make(map[string]bool, len(retrievedIDs))
for _, id := range retrievedIDs {
retrievedSet[strings.ToLower(fmt.Sprint(id))] = true
}
citedIDs := cv.extractor.UniqueIDs(out.Response)
validCitations := 0
invalidCitations := 0
var violations []string
for _, id := range citedIDs {
if retrievedSet[strings.ToLower(id)] {
validCitations++
} else {
invalidCitations++
violations = append(violations,
fmt.Sprintf("hallucinated citation: [%s]", id))
}
}
coveredCount := 0
for _, id := range retrievedIDs {
idStr := strings.ToLower(fmt.Sprint(id))
for _, cited := range citedIDs {
if strings.ToLower(cited) == idStr {
coveredCount++
break
}
}
}
totalCited := validCitations + invalidCitations
precision := 1.0
if totalCited > 0 {
precision = float64(validCitations) / float64(totalCited)
}
recall := 0.0
if len(retrievedIDs) > 0 {
recall = float64(coveredCount) / float64(len(retrievedIDs))
}
f1 := 0.0
if precision+recall > 0 {
f1 = 2 * precision * recall / (precision + recall)
}
pass := precision >= 0.9 && recall >= 0.5
reason := ""
if !pass {
reason = fmt.Sprintf(
"precision=%.2f recall=%.2f; violations: %s",
precision, recall, strings.Join(violations, "; "))
}
return EvalResult{
Pass: pass,
Score: f1,
Reason: reason,
Keywords: violations,
}, nil
}package eval
import (
"context"
"fmt"
"strconv"
"strings"
)
type OutOfRangeDetector struct {
extractor *CitationExtractor
prefix string
}
func NewOutOfRangeDetector(prefix string) *OutOfRangeDetector {
return &OutOfRangeDetector{
extractor: NewCitationExtractor(""),
prefix: prefix,
}
}
func (d *OutOfRangeDetector) Name() string {
return "citation_out_of_range"
}
func (d *OutOfRangeDetector) Evaluate(
ctx context.Context, ex Example, out Output,
) (EvalResult, error) {
maxID, ok := ex.Input["max_document_id"].(float64)
if !ok {
return EvalResult{}, fmt.Errorf("missing max_document_id")
}
citations := d.extractor.Extract(out.Response)
outOfRange := 0
var violations []string
for _, c := range citations {
if !strings.HasPrefix(c.ResourceID, d.prefix) {
continue
}
numStr := strings.TrimPrefix(c.ResourceID, d.prefix)
num, err := strconv.Atoi(numStr)
if err != nil {
continue
}
if num < 1 || num > int(maxID) {
outOfRange++
violations = append(violations,
fmt.Sprintf("[%s] out of range [1,%d]",
c.ResourceID, int(maxID)))
}
}
pass := outOfRange == 0
score := 1.0
if len(citations) > 0 {
score = 1.0 - float64(outOfRange)/float64(len(citations))
}
return EvalResult{
Pass: pass,
Score: score,
Reason: strings.Join(violations, "; "),
Keywords: violations,
}, nil
}In a production RAG pipeline, citation validators run as part of every evaluation suite. When a new retrieval model or prompt template is deployed, the validators measure whether the model’s citation behavior has regressed. A drop in citation precision below 0.9 blocks the deployment, since hallucinated citations directly erode user trust.
The coverage score feeds into a retrieval quality dashboard. If coverage drops, it often indicates that the retrieval stage is returning too many documents, overwhelming the model’s context window and causing it to ignore some sources. This signal helps the team tune the top-k parameter for retrieval.
Out-of-range detection catches a specific failure mode where the model generates plausible-looking but fabricated document IDs. This is particularly common when the model has been fine-tuned on examples with sequential IDs—it learns the pattern and extrapolates beyond the actual document set.
Different systems use different citation conventions. Academic-style systems might use [1], [2], while document-grounded systems use [doc_id]. The extractor’s regex pattern is injected at construction time, making the same validator reusable across projects. For multi-format systems, chain multiple extractors and merge their results.
Not all retrieved documents are equally important. A weighted variant assigns relevance scores to each document (from the retrieval stage’s ranking) and computes coverage as the sum of weights of cited documents divided by the total weight. This penalizes missing the top-ranked document more than missing a low-ranked one.
The pass/fail thresholds (precision ≥ 0.9, recall ≥ 0.5) are tuned using replay mode from the evaluation runner. Teams replay historical outputs with different threshold values and examine the resulting pass rates. The goal is to set thresholds that flag genuine regressions without creating excessive false positives that erode trust in the CI signal.
\[\w+\] matches non-citation brackets such as [NOTE] or [TODO], inflating the invalid citation count. Use a pattern that matches the specific ID format used by the system, and test it against a corpus of known outputs before deploying.Doc_3 vs doc_3) but the validator does case-sensitive comparison, valid citations are incorrectly flagged as hallucinated. Normalize both sides to lowercase before comparison.[doc_1] might accidentally match against doc_10, doc_11, etc. if using substring matching instead of exact matching. Always compare full, normalized identifiers.