Most teams come into the build with a model already chosen. Usually the frontier model of whichever vendor sold them their last engagement. The eval does not care which vendor was in the room — only which model passes.
The matrix we run, not the matrix we read about
For every production path, we score the eval against a matrix of candidates. The matrix tends to include three frontier-class models from different vendors, two or three mid-tier models, one small fast model, and at least one open-weights option deployed on our own infrastructure. The score is not a single number. It is four columns: quality against the eval, p95 latency, dollars per million tokens, and behavior under malformed or adversarial input.
A model does not "win" by scoring highest on quality. A model wins by being the cheapest one that passes the thresholds the business wrote down.
What tends to actually win
In the matrices we run, the frontier model wins on quality more often than not. It wins on the full scorecard less often than teams expect.
The pattern that tends to hold: a mid-tier model with a tighter prompt and one structured-output retry passes the same threshold as the frontier model, runs in a third of the latency, and lands the production path under $0.50 per million tokens. The frontier model lands the same path at roughly four to five times that cost. Impressive-in-isolation does not enter the scorecard; the cheaper model that clears the thresholds wins, because that is the rule.
Why this matters beyond the cost line
The model bill is the visible win. It is not the most important one.
The most important win is portability. When a new model lands — and one lands every six to ten weeks now — re-running the eval matrix is a Wednesday afternoon, not a migration project. The team owns the harness. The harness picks the model. There is no vendor lock-in to negotiate out of, because the eval is the thing the system was built against, not the model.
The second-most-important win is honesty under stress. Production load is not the load the demo ran under. Smaller models tend to degrade more predictably when load goes up, retries fire, or the input gets weird. The matrix exposes that in week three instead of in the first incident.
What this is not
This is not an argument against frontier models. Frontier when we must — and when a frontier model is the right call, you can often compress it rather than replace it. The agents that need long-horizon reasoning, the workflows where a wrong answer is catastrophic, the tasks where the cheaper models genuinely fail the eval — those run on the frontier model. The decision is made by the scorecard, not by the salesperson who landed the room first.
The rule is straightforward: the scorecard makes the call, and the cheapest model that clears it wins. Which vendor sold you the idea gets no vote.