Eval set

An eval set is a collection of cases that the system must handle, each with an expected outcome agreed in advance. The cases are drawn from real inputs wherever possible, including the awkward ones. The set is the yardstick: it tells the team whether a change made the system better, worse or no different.

An eval set is different from a test set written by the builders. The owner should agree the expected outcomes, because the owner is the person who will judge the system after handover.

In practice

I build the first version from real records in the readiness review, with the owner marking the correct outcome for each. The set includes easy cases, hard cases and cases that should be refused. Each case has a short note on why it matters. The set grows during the slice and the build, whenever a failure reveals a case that was missing.

The set is run through the evaluation harness on every change, and its results are stored with the change, so the history of the system is visible.

Example

[Composite] An assistant for claims handlers started with forty cases. In the second month a new claim type appeared in production that the set did not cover. The failure was added as five cases, and the next release was checked against them. The owner now adds cases each month from the escalations log.

Common mistakes

• Building the set from the cases the builders find easy.
• Letting the expected outcomes be decided by the builders alone.
• Freezing the set after launch. A set that never changes stops reflecting production.