Most model benchmarks reward attempts. A model that guesses on every question can score well if enough guesses are correct, even when its wrong answers would be unacceptable in production. For factual assistants, legal workflows, health triage, financial analysis, and software support, the important question is not only “does the model know the answer?” It is also “does the model know when it does not know?”
Abstention-aware knowledge reliability benchmarks measure factual recall and calibrated uncertainty together. They give positive credit for correct answers, negative credit for confident wrong answers, and neutral or less-negative treatment for explicit abstention. This turns refusal from a nuisance into a first-class evaluation outcome: a model that cannot answer should say so instead of manufacturing a plausible detail.
AA-Omniscience is a concrete example of this pattern. It evaluates models on thousands of short-answer questions across economically relevant domains, instructs models to abstain when uncertain, grades responses into correct, incorrect, partial, or not attempted, and reports an index that penalizes hallucinated guesses. The transferable lesson is the benchmark design, not the leaderboard: production teams can adapt the same scoring structure for their own domain-specific knowledge evaluations.
Knowledge reliability combines factual coverage with restraint. A model with high coverage answers many questions correctly. A reliable model also avoids answering when its internal knowledge is insufficient. For high-stakes domains, the second property can be as valuable as the first because a wrong answer can trigger bad downstream decisions.
An abstention-aware benchmark treats “I do not know” as a legitimate output class. It is not equivalent to a correct answer, but it is better than an unsupported claim. This changes model-selection pressure: the benchmark no longer rewards blind attempt rate, and it exposes the tradeoff between raw accuracy and hallucination rate.
The cleanest knowledge-reliability questions have compact targets: dates, names, versions, numeric values, locations, identifiers, or short phrases. Short targets simplify grading, reduce ambiguity, and make it easier to distinguish a correct answer from a partial answer or a contradiction.
Do not evaluate factual systems as if every prompt must be answered. Evaluate them under the policy you want in production: answer when sufficiently confident, abstain when unsupported, and never guess just to maximize attempt rate.
The benchmark starts with a curated set of questions that are difficult enough to avoid saturation, precise enough to grade, and distributed across the domains that matter to the application. AA-Omniscience uses a large full set and a representative public subset; an internal team might instead maintain private evaluation shards for finance, healthcare policy, product support, and software engineering.
type KnowledgeQuestion = {
id: string;
domain: string;
topic: string;
question: string;
target: string;
source_refs: string[];
answer_format: "date" | "name" | "number" | "identifier" | "short_phrase";
};
const q: KnowledgeQuestion = {
id: "python-stdlib-042",
domain: "Software Engineering",
topic: "Python standard library",
question: "In CPython 3.8 or later, which -X option value is equivalent to setting PYTHONPYCACHEPREFIX?",
target: "pycache_prefix=PATH",
source_refs: ["python-docs"],
answer_format: "identifier"
};The model should receive a clear instruction that abstention is allowed and preferred over guessing. This is important because many instruction-tuned models have learned to be helpful by always producing an answer. If the evaluation does not explicitly allow non-answers, it measures a mixture of knowledge and social compliance rather than calibrated factuality.
const answerPrompt = `
You are answering questions about ${domain}, especially ${topic}.
Answer with just the answer, no explanation.
If you do not know the answer, or you would need more context or tools,
say that you do not know rather than guessing.
Question: ${question}
`;Responses should be classified into at least four states: correct, incorrect, partial, and not attempted. The partial bucket matters because many answers are directionally right but at the wrong level of precision. For example, returning a major version when the target requires a full semantic version should not receive full credit, but it is meaningfully different from contradicting the target.
type Grade = "CORRECT" | "INCORRECT" | "PARTIAL" | "NOT_ATTEMPTED";
function scoreGrade(g: Grade) {
switch (g) {
case "CORRECT": return 1;
case "INCORRECT": return -1;
case "PARTIAL": return 0;
case "NOT_ATTEMPTED": return 0;
}
}A single headline score is useful, but it should not hide behavior. Report accuracy, attempt rate, not-attempted rate, partial rate, hallucination rate among non-correct outcomes, and a reliability index that subtracts incorrect answers from correct answers. This makes it visible whether a model wins by knowing more, refusing more, or guessing more.
When selecting a model for analysis support, compliance analysis, support automation, or technical documentation, raw benchmark averages can be misleading. An abstention-aware evaluation reveals whether the model is safe to use without retrieval, whether it needs tool access for most questions, or whether it tends to fabricate details when its memory is weak.
Teams can adapt the benchmark to internal domains by writing short-answer questions from authoritative sources: API references, policy manuals, internal product docs, customer contracts, runbooks, and database catalogs. The goal is not to memorize trivia. The goal is to test whether the assistant can distinguish stable known facts from questions that require retrieval or escalation.
Changing a system prompt, refusal instruction, reasoning mode, or decoding configuration can alter the balance between helpfulness and caution. Abstention-aware tests catch regressions where a model becomes more willing to answer but also more willing to hallucinate. This is especially useful when product teams tune prompts to sound more confident.
A useful reliability index can be implemented as correct minus incorrect, normalized by the number of questions. This bounded score is easy to explain: positive values mean the model produces more correct answers than wrong ones; negative values mean incorrect guesses dominate. Partial answers and abstentions are neutral by default, though a team can choose to penalize excessive abstention separately.
def reliability_metrics(grades):
n = len(grades)
correct = sum(g == "CORRECT" for g in grades)
incorrect = sum(g == "INCORRECT" for g in grades)
partial = sum(g == "PARTIAL" for g in grades)
abstained = sum(g == "NOT_ATTEMPTED" for g in grades)
index = 100.0 * (correct - incorrect) / n
accuracy = correct / n
attempt_rate = (n - abstained) / n
hallucination_rate = incorrect / max(1, incorrect + partial + abstained)
return {
"reliability_index": index,
"accuracy": accuracy,
"attempt_rate": attempt_rate,
"hallucination_rate": hallucination_rate,
"partial_rate": partial / n,
"abstention_rate": abstained / n,
}Store the raw model response before grading. Then run a deterministic parser or a pinned grading model with examples and strict output labels. Aggregation should consume only the normalized grade records. This separation lets the team re-grade a fixed response set when the grader improves, audit disagreements, and compare model behavior independent of grading changes.
EvaluationRun
question_id
model_id
prompt_version
raw_response
grade
grader_version
latency_ms
output_tokens
created_atPublic questions are useful for reproducibility, but public-only benchmarks can be overfit. Maintain private shards for release gating and rotate in fresh questions as authoritative sources change. When using a public subset, treat it as a smoke test rather than the sole evidence for production readiness.
Overall scores can hide deployment risk. A model may be reliable on software questions and weak on finance, or strong on names and weak on exact numeric targets. Report slices by domain, topic, answer format, source recency, and difficulty. Model choice should follow the use case, not the global leaderboard.