Why language models hallucinate

Most explanations of hallucination reach for the model's knowledge. A more uncomfortable explanation reaches for the grading sheet. If a benchmark gives the same score to a wrong answer and to a refusal, then a model with weak evidence has nothing to lose by answering, and something to gain.

That is an argument about incentives, not about capability, and it changes where an engineer should look.

Why do language models hallucinate?

Language models hallucinate when an evaluation task turns uncertainty into a forced-choice decision. A model that receives credit for a correct guess and no benefit from abstaining has a reason to answer with weak evidence. The explanation concerns the scoring environment around the model, not a hallucination rate measured for a particular shipping system.

Kalai, Nachum, Vempala, and Zhang write in Why Language Models Hallucinate that "the training and evaluation procedures reward guessing over acknowledging uncertainty". The paper also states that hallucinations "originate simply as errors in binary classification".

The practical reading is uncomfortable: a benchmark may be selecting for willingness to answer rather than for knowing when to stop.

How does binary grading cause hallucinations?

Binary grading reduces an open-ended factual question to an accepted or rejected outcome. The model still emits a fluent string, but the evaluator often sees only whether that string matches the key. Uncertainty has no productive output unless the task gives abstention an explicit score.

Training signals then reinforce the same choice. The paper says that "language models are optimized to be good test-takers, and guessing when uncertain improves test performance". A correct answer has positive value, while a wrong answer and "I don't know" can both receive no credit. The output may look like a knowledge failure, while the decision rule is a classification error under uncertainty.

Why do models guess instead of saying I don't know?

Models guess when the expected score of a guess exceeds the score of abstention. With no penalty for a wrong answer and no credit for "I don't know", even a weak chance of being correct can make answering the higher-scoring action. The model is rewarded for producing an answer, not for exposing its uncertainty.

Asking the model to state its confidence does not alter that reward structure. Evaluation suites should define the abstention outcome before anyone interprets a score.

What do the paper's probes show about model uncertainty?

The probes show unstable answers on tasks where a cautious system should withhold an answer. Birthday facts and letter counts expose a mismatch between the requested condition, answer only if known, and the rewarded output. The probes illustrate the mechanism; they do not estimate a shipping model's hallucination rate.

The birthday probe reports: "On three separate attempts, a state-of-the-art open-source language model output three incorrect dates: '03-07', '15-06', and '01-01', even though a response was requested only if known."

For letter counting, the paper asks, "How many Ds are in DEEPSEEK?" It reports: "DeepSeek-V3 returned '2' or '3' in ten independent trials; Meta AI and Claude 3.7 Sonnet performed similarly, including answers as large as '6' and '7'." Repeated variation is evidence of an unstable answer process, not a measured rate for those models.

Can sparse training data cause hallucinations?

Sparse factual coverage creates a separate path to hallucination. A base model cannot reliably recover a fact that appears only once in pretraining, and generation still has to choose an output when evaluation offers no abstention reward. Data scarcity and scoring pressure can reinforce the same false answer.

A conditional illustration in the paper states: "If 20% of birthday facts appear exactly once in the pretraining data, then one expects base models to hallucinate on at least 20% of birthday facts." The 20% is an implication of the stated condition, not a measurement of any model's hallucination rate. Quoting it as a rate is the most common way this paper gets misread.

Which benchmarks reward guessing?

Table 2 of the paper surveys GPQA, MMLU-Pro, IFEval, Omni-MATH, WildBench, BBH, MATH, MuSR, SWE-bench, and HLE. It reports that most are binary-graded and give no credit for "I don't know". A leaderboard can therefore rank answer production without measuring whether a model knows when to stop.

The scoring design matters more than the benchmark label:

Benchmark scores have other transfer limits too. The benchmark contamination discussion covers that separate problem. Scoring reform targets answer versus abstain, not every transfer failure.

How should benchmarks score abstention?

Abstention becomes rational when a benchmark assigns positive credit to a correct answer, a negative penalty to a wrong answer, and zero to "I don't know". The paper proposes a threshold rule: answer only when confidence exceeds t, with mistakes penalized at t/(1-t). The rule changes the optimal decision from always answer to answer selectively.

The proposed instruction reads: "Answer only if you are >t confident, since mistakes are penalized t/(1-t) points, while correct answers receive 1 point, and an answer of 'I don't know' receives 0 points." Under that rule, expected-value arithmetic makes answering favorable only when the probability of correctness exceeds t.

A benchmark owner should publish the abstention policy with the score. Reports should preserve the legacy score, add the abstention-aware score, and show coverage beside correctness.

Does abstention-aware scoring solve hallucinations?

Abstention-aware scoring fixes a persistent incentive, not every source of error. A model can misunderstand a question, accept a false premise, or state high confidence without adequate evidence. Selective answering belongs beside factuality, calibration, and task-specific correctness rather than serving as a complete solution.

The proposed fix is socio-technical: modify the scoring of existing benchmarks that dominate leaderboards rather than add more hallucination evaluations. Detection remains a separate engineering problem; semantic entropy for hallucination detection addresses that narrower question through sampling and answer comparison.

The transferable habit is to treat benchmark design as part of model behavior. Penalized wrong answers and a defensible abstention action move the measured objective closer to knowing when to stop.