LLM as judge
A language model scores another system's outputs against a rubric. The judge must be calibrated against human reviewers before it is trusted.
LLM as judge is the practice of using a language model to score the outputs of another model or system, usually against a written rubric. It is useful when the outputs are too many to read by hand, or when the criteria are about tone, completeness or reasoning rather than exact matches.
The judge is itself a model, so it has errors of its own. A judge is trusted only after it has been checked against human reviewers on the same cases.
In practice
I start with a rubric that a human can apply: a short list of criteria, each with a scale and examples of each level. I ask two people to score a sample of outputs and measure how often they agree. Then I run the judge on the same sample. If the judge agrees with the humans at a level the owner accepts, the judge can score the larger set. Agreement is re-checked when the judge or the system changes.
The judge is never the only evidence for a high-stakes decision. Exact checks, such as a value that must match a record, stay as code.
Example
[Composite] A summary judge was scoring most outputs as acceptable. When two reviewers checked a sample, they accepted far fewer. The judge had been rewarding length. The rubric was rewritten to score omissions and unsupported claims separately, and the judge was re-calibrated before its scores were used again.
Common mistakes
- Trusting a judge because its scores look sensible.
- Using the same model family as judge and system without checking for shared blind spots.
- Writing a rubric that reviewers cannot apply consistently.
Related terms
Cite this definition
LLM as judge. Mayur Patil, mayurpatil.ai, updated 2026-11-01. https://mayurpatil.ai/glossary/llm-as-judge/