Rubric

A rubric is the written scoring guide for an output. It names the criteria that matter, such as accuracy, completeness and the use of the right source. For each criterion it gives levels, from poor to good, and examples of what each level looks like. A rubric makes a judgement repeatable: two people applying it to the same output should reach the same score most of the time.

Without a rubric, a score reflects the reviewer's mood. With one, the score can be compared across reviewers, across time, and across models.

In practice

I write the rubric with the business owner, using real outputs as the examples. I keep the number of criteria small, usually three to five, so that reviewers can apply them. I then test agreement: two reviewers score the same sample, and the disagreements are used to tighten the wording. A rubric that two reviewers cannot apply consistently is not ready.

The same rubric is used by any LLM as judge that scores outputs, and the judge is checked against the human scores.

Example

[Composite] A reviewer team scored summaries of supplier contracts on a single scale, and the scores varied widely. The rubric was split into three criteria, each with examples. Agreement between two reviewers rose, and the disagreements that remained showed where the wording still needed work.

Common mistakes

• Writing one overall criterion, which makes the score hard to act on.
• Giving levels with no examples, so that reviewers interpret them differently.
• Changing the rubric without rescoring earlier outputs, which breaks comparison over time.