The AI Model Plateau: Why Your Infrastructure Matters More Than the Next Release

The days of 1:1 model comparisons are over.

At one point, it felt like a game of leapfrog, with each release from OpenAI beating the last release from Anthropic, until Anthropic jumped ahead again. Some people stuck to their preference, knowing the latest release would temporarily put them back on top, while others switched back and forth.

Now, with open-source options like KimiK3 competing on intelligence, and OpenAI and Anthropic bragging about pricing optimization as often as they do model quality, the field is murkier.

Sheer model quality is no longer the differentiator it once was, which means your choice of model isn’t either. The value chain is shifting, and pipelines are increasingly becoming the primary constraint.

How do you route work? What do you cache? How do you evaluate the results? How quickly can you adapt your harness when something new or better comes along?

The companies that can answer these questions will have better, cheaper results than the companies that can’t, even if they work with the same models and in the same problem space.

The best returns will come from investing in your infrastructure and in the continuous optimization necessary to remain loosely coupled as the field keeps changing. In short: Keep your options open, and don’t tie your harness too tightly to one specific model. That’s the TL;DR, but we’ve come to this opinion only after a lot of experimentation.

Why we’re writing this: Ito is the only AI code review that actually runs your code (where we use roughly 100 billion tokens a week). The numbers here come from our own experience experimenting with many different models and pipeline configurations.

Listen to the labs: Progress is plateauing

The vendors describe their own releases in increasingly modest terms over time, and the features they lead with tend to focus on workflow and efficiency rather than on output quality.

To be clear: a plateau is not a collapse. Progress is still happening (as our experience switching to Luna demonstrates), but it is becoming more incremental and less paradigm-shifting.

Take, for example, Anthropic's Opus 4.8 announcement (published in May of 2026), which says, "Users will find Opus 4.8 to be a modest but tangible improvement on its predecessor."

Simon Willison, co-creator of Django, pored through the spec sheet and agreed, writing, "Not much has changed since 4.7. [...] Both the reliable knowledge cutoff and the training data cutoff are January 2026, the same as for 4.7. The context window is still 1,000,000 tokens, and the max output is 128,000 tokens."

The big feature accompanying Opus 4.8 was dynamic workflows. Dynamic workflows let Claude plan a job, dispatch hundreds of parallel subagents, and verify the results before reporting back. Cool? Of course. But if the model itself were dramatically better, they wouldn't need workflow tooling to paper over its limitations.

The case is similar with OpenAI. In its GPT-5.6 announcement (published in July of 2026), they focus on efficiency rather than capability, writing, "By making every layer more efficient, OpenAI is delivering stronger performance per dollar."

Their companion engineering post walks through where the gains came from:

• Better load balancing across geographies and accelerator types.
• Kernel rewrites that cut end-to-end serving cost by 20%.
• Speculative decoding improvements worth more than 15% on token generation.

Price cuts are exciting, and they can make a big difference in what users can afford to actually do. But a price cut is not a model improvement, and it results from fixing routing and caching. It’s incremental and iterative.

The plateau even shows up within product lines. On Artificial Analysis, the intelligence gap between Luna and Terra is only a few points. Opus to Fable looks similar.

The tiers are converging on quality and separating on cost and speed. Your choice of one model over another is no longer about intelligence gains.

Their benchmarks are not your benchmarks

Each model provider compares their models against benchmarks to disprove this, to signal that model choice remains the most important decision. But their benchmarks don’t necessarily correlate with what you need the models to do.

That’s why we ran our own research: the benchmark results didn't predict what we saw in production. We tested 14 models on a single fixed coding task, with 6 runs each.

KAT-Coder-Pro V2.5 was the third-fastest emitter at 113 tokens per second and finished tenth because it burned 5,536 tokens. GPT-5.5 needed over 3 times fewer tokens at only 1,777. Meanwhile, the once-hyped Kimi K3 came in last at 11.5x Luna's wall clock.

Within a single product line, Luna took 15 seconds and $0.012, Terra took 28 seconds and $0.042, and Sol took 59 seconds and $0.079. Paying 5x more per token bought 3.8x more waiting time.

Price per token also failed to predict price per task.

Kimi K2.7 Code has more expensive tokens than KAT-Coder and finished the job at 37% less cost. GLM-5.2's tokens cost 2.2x MiniMax M3's, yet per task the two landed at almost identical costs. And on the extremes, DeepSeek V4 Pro finished at a cost of only $0.0017, roughly 73 times cheaper than Claude Fable 5, despite the task being identical.

Our data isn’t unique in this regard. Artificial Analysis compares output tokens per Intelligence Index task, and the verbosity ordering matches what we measured: MiniMax M3 at 24.0k, Kimi K2.7 Code at 17.7k, GPT-5.5 high at 10.1k.

DoorDash came to the same conclusion running their code review agent in production. In the engineering blog post on the topic, the authors write, “The cheapest model is not always the cheapest review. Several stages produce structured JSON. For simple schemas, a faster model works well. For more complex schemas, weaker models sometimes produced invalid output and retried multiple times, while a stronger model produced valid output on the first attempt.”

They hit on a good summation, which applies well beyond building code review agents: “The unit that matters is not token price. It is cost per successful review.”

That’s why you need to try the models against your system and use cases, not just look at benchmarks and leaderboards. The more you do so, the more you’ll find that your pipeline influences your results as much (or more) than the model leaderboards and benchmarks.

The constraint moved from the model to your pipeline

Many companies (including ours) just had an especially vivid moment comparing models to see how the results varied.

It feels like everyone spent the last month or two furiously trying new open source models, then trying the newest cool models again, only to end up back with OpenAI. OpenAI won this round because their speed and cost are (so far) impossible for even open source to beat.

Our collective surprise points to an underlying point: Model improvements, until you measure them in the real world, are hard to predict.

That’s why your pipelines are increasingly the primary constraint.

If you can’t easily test new models, swap them in and out for different tasks, and optimize the pipeline itself, you’ll be left behind; if your competitors can easily swap models and you can’t, they will have have major gains in speed, cost, and quality that you’ll struggle to keep up with.

Now, if you want to better keep up, we’ve found three questions to ask that can guide your efforts.

1. Which tasks go to which model?

Different models handle the same tasks differently. Given the same task, some will perform better or worse (or fail altogether). Some will also cost more or fewer tokens. Further complicating things, often the quality doesn’t correlate well with the cost.

DoorDash's DashBench numbers, for example, show how much your results can vary depending on the model mix inside your architecture:

For context, weighted recall is the share of known issues the system catches, with higher-severity issues counting more. Weighted F1 balances precision and recall, giving more weight to higher-severity issues.

Of these configurations, the Sonnet 5 pairing costs the most but has the weakest coverage. The Composer pairing is most precise, but catches far fewer of the known issues. No configuration dominates, so the right choice depends on what you are optimizing for.

As the DoorDash engineers write, 'Best' is meaningless until you say best at what, in which cases, at what cost.

2. Which probabilistic steps can you make deterministic?

Not every step needs to be probabilistic. Thoughtworks calls this pattern “feedback sensors for coding agents”: compilers, linters, type checkers, and test suites wired into agent workflows so failures trigger auto-correction before humans need to see anything.

DoorDash applies similar logic to what they put in front of the model. When they designed their AI code reviewer, they ensured that every potential review rule survived a filter before it made it into a profile. They write: “If CI would already catch it, drop it. If the LLM knows it from general training, drop it. If we can't point to a concrete file-and-line in the codebase as evidence, drop it.”

The stuff that remains is review knowledge that's actually specific to their codebase. Only after that do they start routing, as they write, "The routing is a big part of why acceptance rates hold up across 56 very different repos. The agent isn't applying a single universal standard. It's applying the standard that matters for that specific change."

3. Can you optimize the provider layer?

The provider layer has more optimization space than you might think. Researchers from the University of Pennsylvania, Columbia University, and OpenMesh AI built a continuous benchmark that measures inference at endpoint granularity across five axes:

• Output speed
• Time to first token
• Workload-blended price
• Effective context
• Quality on the live endpoint

They found, “The same model name on different providers is not the same product.”

In their research, open weights such as Llama 3.3 70B and gpt-oss-120B were often served by 20 or more providers. As a result, they write, “Quantization and serving choices change the model’s behavior in measurable, sometimes undisclosed, ways.” Despite that, leaderboards tend to hide this variation by aggregating at the model or provider level.

That means there’s optimization potential, and often quite a lot of it, at the provider layer.

Our own experience demonstrated a similar pattern. We tested GLM-5.2 and MiniMax M3 on Vercel AI Gateway, OpenRouter, and Together AI.

Vercel had by far the worst cache hit rate for both models, and scored the worst on our internal evals against the same underlying providers reached through OpenRouter. Same model, very different performance.

Cache behavior turned out to be the biggest point of divergence. About 97% of our input tokens are cache reads, so a provider that degraded caching upgraded the bill.

If you like a model, but find yourself experiencing performance issues, you may want to try another provider before you swap models.

Build infrastructure that enables hot swapping

Vendor (and model) lock-in is one of the biggest risks AI-enabled companies face. Building tightly coupled infrastructure that doesn’t support hot swapping different models means you can’t be on the bleeding edge today and will be behind the times tomorrow.

To keep your infrastructure flexible, ensure you can evaluate candidate models effectively, write good tests, and encourage experimentation so you’re not wedded to any component of your stack.

Evaluate candidate models by specific agent job, and test the results and quality by task

“Best model” is not a useful category, nor an effective way to experiment and build today. The better question is: “What is the best model for this step, and at this quality floor?”

Fortunately, this change in framework doesn’t require a big expenditure.

The cost of tokens is dropping, and that benefits both what you do in production and your experiments.

Remember: our experiment where we ran the same test across thirteen different models? It cost us three dollars. Total.

And the learnings from our small experiment proved out when we started using the best-scoring models in production.

DoorDash gets at the same idea from the measurement side, writing, “The useful question stops being 'which model is best?’ It becomes: for a given architecture, context policy, tool policy, runtime budget, and model mix, what tradeoff between coverage, precision, cost, and latency do we actually get, and where does it fail?”

The key phrase here is “actually get.” Experiments like ours close the gap between what’s hyped on X or wowing on benchmarks, and what actually will work best for your use case. You need to experiment to find out.

Write tests that mimic tasks you need AI to do in your system

Start simple. Pick an agent step, and build a test around what that step really does in production. For us, that meant a real test batch running against a real browser in a devcontainer. That was our largest LLM cost.

The tests need to fail for reasons you can act on. Capture the input, the tokens it burned, the wall clock, and the output, and then score the output against something you wrote in advance. Include cases where the right behavior is doing nothing.

Then, keep the set alive. Retire cases that stop discriminating between models, and add production misses as new cases. That corpus turns into a durable asset for you.

Models will keep getting better, and your eval set is how you find out whether the improvement means something for your workload. We only learned that leaderboard rankings didn’t survive contact against our own workload when we tested the models ourselves.

Encourage a culture of experimentation

Don't shoot the messenger because one of your AI-pilled staff is excited about a new model. Ask how you can let them experiment safely.

That might mean offering them a smaller product surface. That might mean working with a subset of customers who opt into the latest and greatest, knowing there will be some issues. Or you can hold semi-regular hackathons where it’s safe to try new tooling, setups, and models.

The goal is to make bad results cheap, then reward the behavior of experimenting in your actions and in words.

If a developer experiments and the results show their idea isn’t worth pursuing, the results are still useful. Now, the next three developers who might have tried the same idea know not to repeat it. Empower experimentation with real budget, real access, and a slot in the roadmap.

After all, breakthroughs can only happen when you put in the time. It’s similar to increasing your luck surface area. If you don’t experiment, breakthroughs will never happen. If you do experiment, breakthroughs become at least possible and, over a long enough period of time, inevitable (albeit unpredictable).

That was our experience. Ito's average runtime dropped by 50% over 4 weeks due to model switches, while costs fell by over 10x. We learned a lesson we intend to repeat, and we hope you do, too.

Avoid tightly coupled systems (again)

Have you heard this one before? Avoid tight coupling. Again.

Tight coupling feels efficient right up until the thing you coupled to changes underneath you.

This is the same lesson the industry has learned repeatedly with databases, cloud providers, and orchestrators. The only new dynamic here is the half-life. Model generations turn over in weeks now, not years.

Remember that this is a good thing. You win either way. If the plateauing trend holds, you can route queries and swap model versions as necessary, optimizing costs and improving results while everyone else waits for the next big leap. And if a genuinely significant shift does happen, your pipeline is ready to integrate it and take advantage as soon as it’s released.

There’s a balance to strike. Some of this is outside of your control. After all, when OpenAI dropped Luna, our average runtime fell by 50%, and costs fell by more than 10x. We could have never planned for that.

What you can control is your pipeline, and that’s why it will be a key differentiator going forward. Theoretical improvements are one thing. Putting them into practice is another at a rate that keeps up with the pace of AI is another. Is your team ready?