Your fastest model is probably not your fastest model

What shows up on LLM leaderboards and benchmark pages doesn't always match what you experience in practice.

For instance, the third-fastest model we tested actually finished tenth.

Unfortunately, the LLM leaderboards we all eagerly check as all the new models drop don't measure task completion time, and the gap between what they show and what your task actually takes is bigger than most teams expect.

Two numbers dominate model selection pages: tokens per second and dollars per million tokens. Both are denominated in tokens (roughly a word or word fragment), yet tokens aren't a unit of work.

We asked fourteen models to do the same job and they used anywhere from 1,800 to 6,400 tokens to finish it, which obviously makes a big difference in both cost and time to complete the task.

The truth is: a model that emits fast but thinks in paragraphs can take longer, and cost more, than one that emits slowly and gets to the point.

That's why we made our own benchmark to test all these new models like Luna, Grok 4.5, DeepSeek V4, MiniMax M3, Kimi K3, and Fable.

Our test structure:

We measured this the way a customer would experience it: One fixed task (review a Python function, write an improved version with tests) sent to fourteen current models through OpenRouter.

Specifically:

• For each model, we gave them three runs capped short to probe speed, and three runs uncapped, so each model finished naturally.
• Speed and latency come from OpenRouter's own per-generation statistics, so they reflect the provider the router actually selected.
• Token counts are native counts including reasoning tokens.
• Prices are based on OpenRouter's list price.

Best of all, this is the kind of test anyone can afford to run, even for your side project: The whole experiment cost under three dollars.

Here's what we learned...

Dividing by tokens per task reorders the leaderboard

Surprisingly, GPT-5.6 Luna won both columns: fastest emitter (131 tps) and fastest to finish (15s).

Below it the two rankings disagree, in places by seven positions.

KAT-Coder-Pro V2.5, the third-fastest emitter at 113 tps, finished tenth: it generated 5,536 tokens where GPT-5.5 needed only 1,777. That verbosity (using over 3X the tokens) costs more time than a fast emission rate saves. Meanwhile, despite a lot of hype, Kimi K3 was both the slowest emitter and the most verbose model in the set, finished last at 11.5x Luna's time.

The harsh reality: Speed rankings on model pages don't predict the finish order on real tasks. Verbosity determines how long a model actually takes.

The reordering happens inside a single product line, too.

GPT-5.6 ships in three tiers, and on this task the cheapest finished first:

• Luna took 15s and $0.012
• Terra 28s and $0.042
• Sol, the $30/Mtok flagship tier, 59s and $0.079

Paying 5x more per token bought 3.8x more waiting.

Slower and more expensive.

This is why you have to take a more nuanced approach to choosing the right model for the right task.

Cheap tokens are not cheap tasks

The same split we just saw also applies to price.

Kimi K2.7 Code's tokens cost more than KAT's ($3.49 vs $2.96 per Mtok), but Kimi finished the task for 37% less, because it used half the tokens.

GLM-5.2's tokens cost 2.2x MiniMax M3's, yet per task the two are within 4% of each other, because of the difference in verbosity.

You see this same effect across other model comparisons:

• Claude Sonnet 4.6's tokens cost 12.5x M3's, but per finished task the gap is 6.4x.
• Kimi K3's tokens cost 3.3x less than Claude Fable 5's ($15 vs $50 per Mtok), but because K3 used the most tokens in the set, the two are within 30% of each other.

As you can see, a cheap token rate doesn't mean a cheap task.

In our next analysis, you can see that neither ranking survives: the cheapest-per-token model isn't the cheapest per task, and the fastest emitter isn't the fastest to finish.

DeepSeek V4 Pro is both cheap per token and terse, so it finished the task for $0.0017, roughly 20x less than Claude Sonnet 4.6 and 73x less than Claude Fable 5, the most expensive finish in the set.

Where public data allows a clean comparison, it agrees with our probe.

Artificial Analysis publishes output tokens per Intelligence Index task: MiniMax M3 at 24.0k, Kimi K2.7 Code at 17.7k, GPT-5.5 (high) at 10.1k, the same verbosity ordering we measured.

For several of the newer closed models, no public per-task figure exists, or the published data uses reasoning-effort settings that don't match what a router actually serves. That's part of why we ran this ourselves.

For Ito, the stakes are high. Tokens per task isn't an academic metric for us; it's how we decide which models stay in production. Our QA agents run the same categories of tasks on every pull request, and a model that uses 3x the tokens to complete the same step costs 3x more and takes 3x longer per review cycle.

The per-token price on the spec sheet tells you the cost of one token, not one task. You need to run your actual workload to find out the real difference.

The router adds a second distribution on top

The speed column above is a median hiding a wide spread. Across six runs, Kimi K2.7 Code measured 223 tps when OpenRouter routed it to Together and 28 tps on DeepInfra. MiniMax M3 ranged from 25 tps (Parasail) to 130 tps (MiniMax's own API). GLM-5.2 hit six different providers in six runs and ranged 20 to 85 tps.

First-party serving narrows the spread but doesn't close it: Sonnet 4.6 and KAT-Coder held within about 15%, while GPT-5.6 Sol varied 2.4x between OpenAI and Azure and GPT-5.6 Luna varied 1.7x across runs on OpenAI's own endpoint.

A leaderboard tps entry is a claim about one provider on one day. If your traffic goes through a router, your realized speed is a draw from a distribution the leaderboard doesn't show. For open-weight models the spread of that distribution was larger than most gaps between models.

If you're routing through OpenRouter or a similar service, the speed you see in production isn't the speed on the leaderboard. It's whatever provider got selected for that call.

What you should measure instead

Once both axes are per-task, your picture clears up. Of the ten models (fourteen minus the four without a published Intelligence Index), six are on the frontier: no single model is cheaper, smarter, and faster than DeepSeek V4 Pro, MiniMax M3, GLM-5.2, GPT-5.6 Luna, Grok 4.5, or Claude Fable 5.

For the other four, there's always at least one model in the set that beats them on all three axes. Grok 4.5 alone beats both GPT-5.5 and GPT-5.6 Sol.

Tokens per task is as easy to publish as tokens per second, and it's the denominator that turns both speed and price into decision-grade numbers: seconds to finish and dollars to finish.

Until a more realistic standard emerges, the probe above is a cheap alternative that's easy to reproduce: choose a batch of the latest models, assign them all one task you care about, give them six runs each, then analyze your results. And you can do all of it for less than $3 of token spend.

To truly know which models are best for your company, measure it yourself.

The leaderboard won't tell you what you need to know, but a simple test you run yourself will.

It costs just a few dollars to find out.