Your LLM Judge Is Grading on Length

Grading model output with another model scales in a way human review never will, which is why most teams end up doing it. It also quietly introduces a second model whose preferences become part of your measurement error.

The important thing is that this is a separate failure from the ones you may already be guarding against. A benchmark can be well designed and its runs repeated properly while the grader itself is skewed. Run-to-run variance and gameable benchmarks are a different problem. This post is about the judge.

What is LLM-as-a-judge evaluation?

It uses a language model to assign a score, pick the better answer, or decide whether an output satisfies a rubric. The result reflects the rubric, the prompt, the context, and the judge's own behaviour, so a single aggregate score hides which criterion failed and how stable the decision was.

The strongest use case is a bounded comparison against a clear criterion. A judge can spot a direct factual error, check that an answer follows an explicit format, or compare two responses against the same rubric. Each criterion should have an observable pass condition, because "sounds excellent" cannot be evaluated consistently and "contains the required fields" can.

Which biases affect an LLM judge?

Three are well documented: position, verbosity, and self-preference. These patterns do not prove every judge fails on every task. They establish that evaluation design has to test for these errors rather than assume neutrality.

Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge evaluates a broader set of these effects. The reported results hold under that paper's tested conditions, which is not the same as invalidating every automated score. The assistant-side sibling of these effects, answers that bend toward whoever is asking, is measured separately and worth keeping distinct.

Does an LLM judge prefer the first answer?

It can, and position bias is the easiest of the three to test for. Evaluate the same pair twice with the positions swapped, then check whether the judge agrees with itself.

If a judge picks answer A when A is shown first and picks B when B is shown first, that pair has an order disagreement. The decision was driven by layout rather than content, and a production evaluator should record it and route it to a tie rule or human review.

Randomising order across a batch reduces systematic exposure to one position, but it hides which individual decisions flipped. The two-order test gives you that visibility, which is why it is worth the doubled cost on any comparison that matters.

Does an LLM judge reward longer answers?

Often, yes, and this is the bias most likely to be silently shaping your numbers. A judge can rate a longer answer higher even when the extra text adds nothing.

Be careful with the language here. A correlation between response length and judge score is evidence of an association, not proof that length caused the rating. Longer answers are sometimes genuinely better, so the association has to be tested against answer quality rather than assumed to be bias.

The diagnostic is straightforward. Build answers of comparable quality at different lengths, include concise answers that fully satisfy the task and long answers padded with repetition, then ask the judge to score relevance, correctness, and completeness separately. If scores rise mainly with token count while rubric performance stays flat, your evaluator is measuring verbosity.

Does an LLM judge favour its own model family?

That is the effect Self-Preference Bias in LLM-as-a-Judge studies directly. A score advantage for the judge's own family is not proof the favoured output is better, and it matters most in exactly the situation where teams reach for a judge: deciding whether model A beats model B.

The cleanest test labels candidate outputs by their generating model, hides those labels from the judge, and compares results across judge families while holding the candidate text and rubric constant. If one judge family consistently favours its own generator, that belongs in the evaluation report rather than in a footnote.

Teams cannot always use a separate family. When you cannot, treat self-preference as a measured risk and raise human review for close comparisons, disputed outputs, and any model-versus-model claim you plan to publish.

How reliable is a judge on ordinary tasks?

Reliable enough to be useful, and that deserves saying plainly. Studies report LLM judges agreeing with human experts on many evaluation tasks, with failures concentrating on edge cases, long-tail inputs, and adversarial examples. The evidence supports selective use rather than either unconditional trust or a blanket ban.

Human agreement does not erase bias, though, and the two get conflated. A judge can agree with experts on routine examples and still prefer one answer position or one model family. Report task agreement and bias diagnostics separately, because they measure different properties.

What protocol makes an LLM judge usable?

Separate routine scoring from diagnostic checks and human escalation, and preserve enough per-evaluation detail to explain a result that changes.

1. Define observable criteria. State what counts as correct, complete, and relevant. Give the judge a reference answer or verification source when the task needs factual judgment.
2. Prefer pairwise decisions for comparative questions. Ask which answer better satisfies the rubric and require criterion-level reasons. Keep absolute scores only when the scale has a defined interpretation.
3. Run both orderings. Present A then B, then B then A. Store both decisions and treat a flip as an order disagreement.
4. Test length sensitivity. Include quality-matched answers of different lengths and concise answers that meet the rubric. Track score against token count as a diagnostic.
5. Test model-family sensitivity. Compare decisions across generating families, and use a separate judge family whenever the decision concerns model superiority.
6. Set an escalation rule. Send order disagreements, close scores, novel failure modes, and adversarial cases to humans. Sample routine agreements as a quality check.
7. Report the limits. Publish task scope, judge identity, rubric version, sample construction, disagreement rate, and human-review policy beside the aggregate.

Record the raw output, prompt, rubric version, judge version, ordering, and decision for every evaluation. Aggregates without those fields cannot explain why a number moved, and the number will move.

This protocol does not make the judge unbiased. It makes the failure modes visible and limits the claims the resulting score can support, which is the honest goal for any measurement instrument you did not build yourself.