Regression eval

A regression eval is the run of an evaluation set after a change. The change may be a new model, a changed prompt, an updated tool, or a new data source. The purpose is to find anything that got worse, including cases that were correct before the change and are now wrong.

An agent is a chain of parts, and a change in one part can alter behaviour in another. A regression eval is the only reliable way to see that, because a single new case shows the new behaviour but not the old behaviour that was lost.

In practice

Every change goes through the same run. The run produces a score for each case, and the scores are compared with the last accepted run. A drop in any case that matters to the business owner blocks the change until it is explained. The results are kept with the version of the system, so the history is visible.

The set must include the cases that have failed before. Each production failure that is fixed becomes a permanent case, so the same failure cannot return unnoticed.

Example

[Composite] A team updated a tool that looked up customer accounts. The new scores were higher overall, but one case for accounts with two currencies had started to fail. The regression run caught it before release, and the tool was fixed before the change went live.

Common mistakes

• Running the evaluation set only for new features, not for every change.
• Comparing only the overall average, which hides a drop in one important case.
• Removing a case because it is hard to pass, rather than fixing the system.