When the inference bill arrives, the standard advice is "use a smaller model." But a smaller model is a different model — different training, different failure modes, a fresh eval run to find out whether it still does the job at all. There's another option that gets skipped: keep the model that already passes, and make it physically smaller.
Substitution and compression both lower the bill. Only one of them starts from the artifact you already trust.
Compression versus substitution
Substitution swaps your model for a smaller sibling and hopes the capability transfers. Compression takes the exact model that clears your eval and removes the capacity it isn't using. SVD-based compression — SVD-LLM — factors each weight matrix and drops the least-important singular values, the directions that carry the least signal. Same model, fewer parameters, behaviour you can measure against the version you already validated.
The naive version of this degrades quality fast, and that's where most people stop. The part that makes it usable is making the truncation loss-aware: SVD-LLM maps each cut directly to its effect on the output, and SVD-LLM V2 assigns a different compression ratio to each layer instead of one blunt setting across the whole network. The cut stops being a guess and becomes a measured trade.
The eval is what turns a cut into a decision
Here's the part that matters for anyone shipping. Every compression method has a knob: how far to cut. Turn it too far and the model breaks; too little and you left money on the table. Without an eval, "we compressed it 30%" is a number with no meaning — 30% of what, costing you what?
With an eval, the knob becomes the same engineering decision as model selection: find the most-compressed version that still clears the bar on your task, not the vendor's benchmark. And this is exactly where the money is — inference cost is the line that compounds across every call, and a smaller artifact that still passes is the durable cost win, not a one-time discount.
What we won't do
We won't compress a model and ship it on the vendor's perplexity numbers. The only compression ratio that means anything is the largest one that still passes your eval, on your workload.
We won't treat "smaller and cheaper" as free. Compression has a cost in quality; the eval is the instrument that prices it, so you cut with a number in front of you instead of a hope.
"Use a smaller model" is advice. "Compress the model that passes until the eval says stop" is engineering. The cheapest thing that does the job is rarely a different model — it's the right one with the fat trimmed off, and a number proving it still works.