Model routing

Model routing is the practice of sending each step of a workflow to the model that suits it. A simple classification or extraction step may run on a small, cheap model. A step that needs careful reasoning may run on a larger one. The workflow decides which step goes where, and the decision is recorded.

Routing is an economic decision and a quality decision at once. A cheaper route that causes rework raises the cost per completed task, so routing is only worth doing when the evaluation set shows that the cheaper model meets the bar for that step.

In practice

I start with one model for the whole workflow, so that the baseline is clear. Then I measure each step separately: its quality on the evaluation set, its cost and its latency. Steps that pass on a smaller model are routed there. Steps that do not stay on the larger model. Each route is a rule with a test, and the test is rerun when either model changes.

Example

[Composite] A document workflow used one large model for every step. Classifying the document type passed the evaluation on a smaller model, while drafting the summary did not. The team routed only the classification step to the smaller model, and the summaries kept their quality.

Common mistakes

• Routing before measuring the baseline, so no one knows what was lost.
• Routing on price alone, without the evaluation set.
• Forgetting to retest the routes when a model is updated.