Model Leaderboards Need a Cost Denominator

The short version: a model leaderboard without a cost denominator is buying theater. The useful ranking is not smartest model overall. It is cheapest reliable model for the workflow you actually run.

Benchmarks are still helpful. They tell you which systems deserve a test. The problem starts when teams treat benchmark rank as the buying decision itself. A model can win an abstract leaderboard and still lose the actual business case if it is materially more expensive, slower, or harder to control inside the workflow that matters. That is why I now treat leaderboard position as a starting signal, not a conclusion.

The denominator I want is cost per successful task. That number forces quality and economics into the same sentence. If one model scores three percent higher but costs two or three times more for the same workflow, the burden of proof shifts. Does it reduce human edit time? Does it lower escalation rate? Does it help close a task the cheaper model cannot do safely? If not, the prettier benchmark is mostly status signaling.

This connects directly to AI Spend Should Be Reviewed Like Pipeline and Every AI Workflow Needs an Eval Before a Seat. Spend review tells you whether the budget is justified. The eval tells you whether the output is good enough. The Measurement Layer is what ties those two together. Without that link, model selection becomes a prestige purchase disguised as technical rigor.

My practical loop is simple. Pick one workflow. Run the same sample across two or three models. Score task success, factual error rate, edit time, and cost. Then write down the cheapest model that clears the bar and the expensive model that only wins when stakes are unusually high. That gives the team a routing rule, not just an opinion. Once you have a routing rule, the tier list becomes useful in the right way: as an input to operational choice rather than a spectator sport.