I think it's pretty evident that Artificial Analysis loves to change the benchmark to place US models on the top, it's no the first time either.
The current placement of many models simply doesn't makes sense, GLM 5.3 is so much better than Astra at backend tasks, Astra does the most weird nonsensical bugs.
This is what happens when the benchmark team is funded by VC capital.
I find it fascinating that some of the main areas people are interested in are using benchmarks with quite few questions: ~200 for coding and ~150 for science.
As long as that's the case, it's no surprise that there is often a perceived difference of quality between benchmarks and real-world use.
I know that running the full composite benchmark set is probably already quite costly, and also that the benchmarks have to be rotated regularly, but it feels like a significant untackled problem.
chinese labs are probably benchmaxxing but i'd like to see when the index changes relative to the position of the best chinese models (last suspiciously-timed update was when glm-5.3-flash was looking really strong)
It is hard to tell what people are trying to say when they use undefined words with no particular meaning. This doesn't seem to change the relative position of the Chinese models much. The Chinese are a month or few behind and the state of intelligence is so volatile that the top 10 will be completely different in 6 months. I'd expect it to still be a mix of US and Chinese models with the US models leading.
Neither is the primary cause, I'd argue. If you look at the coding index specifically, you're getting overall more plausible results (in particular, Terra no longer outperforming Fable). [1, 2]
I don't find it surprising that especially smaller Flash models that have been optimized primarily for coding fall behind on non-coding tasks and that their overall score suffers in comparison.
It does give rise to the question of how useful a single aggregate numeric index of very heterogeneous capabilities is, though.
Look also at the cyber index; while somewhat difficult to extrapolate because of refusals, even if for Opus 5.5 and Fable 5.1 you optimistically count all refusals as successes, the results are ... interesting.
Tinfoil hat time: but the way they waited a couple weeks to evaluate Mimo 2.6 Flash, allowing GPT-6 Luna to sit on the Pareto frontier in the meantime, really did make me think they might have some bias going on... (Luna was displaced when they finally evaluated Mimo 2.6 Flash, which had been released the day before Luna)
edit to add: there totally is benchmaxxing going on, but I'd be really surprised if it wasn't across the board. After all, how can any lab tell if their model has improved over a previous iteration, if not by comparing benchmark performance?
Or, are we witnessing reverse-benchmaxxing? Meaning that first some entities let players optimise for a given benchmark, then they update the measurement methodology so as to retain some players on the top ranks, while dropping others. As former Italian prime minister, and highly controversial public figure, Giulio Andreotti once remarked "Thinking the worst is a sin, but you’re often right."
I keep seeing that MiMo is in the frontier in terms of performance and cost but I see very few talking about it and the only ones that do talk about it are doing it in context of AA
This seems to be a hit and a miss, there's no visual indication that opus 5.5 is better even though in 9 tests out of 10 it will outperform astra especially when it comes to niche domains and will absolutely destroy them in anything requiring raw intelligence rather than a naive search, naive search is what makes openai models extremely capable in scientific fields though.
The score hit on smaller models is a welcome change especially when compared to models that nearly 10 times larger.
Grok being punished quite a lot is also a very welcome change as the model is frankly not that good.
I would love to see few more points knocked off haiku by introducing steerability/creativity benchmarks.
(EDIT: A child comment has refuted the AA score comments, AA runs using their own harness.)
One of the new benchmarks is FrontierCode 2.0 by Cognition. AA seems to have accepted Cognition's benchmark scores without questioning, but they should have looked more closely.
On this benchmark, Anthropic's models are run using Claude Code; OpenAI's using Codex, and Gemini is run using... Cognition's own Devin (not Antigravity). Thus its score is much lower than it should be, knocking ~3 points off its AA score. All open-source models are also inexplicably run using Devin, damaging their scores as well.
This may even be intentional, if Cognition has beef with Google, since Antigravity was built using Windsurf's technology. But aside from that, it is to their advantage to make Anthropic/OAI look like leaders in the AI market. They are a company known to lie and exaggerate again and again, so I have trouble giving them benefit of the doubt.
It's interesting that the Stirrup scores are somewhat close to the specialized harness scores. Either Stirrup is quite excellent or is it that harness doesn't matter so much after a point?
Yes, that is true, you might be right. But I think such, because the moat in their wrapper layer feels very thin and does not make sense in a world full of 20+ wrappers around similar models. It feels better to "only" have to deal with Ant/OAI (and other pure wrapper cos). Competing vs. Google, Meta, and every Chinese lab on earth, every one using their own bespoke wrappers, is much scarier.
Additionally: competitive frontier = more commodified = high valuations under more scrutiny.
I think it's pretty evident that Artificial Analysis loves to change the benchmark to place US models on the top, it's no the first time either.
The current placement of many models simply doesn't makes sense, GLM 5.3 is so much better than Astra at backend tasks, Astra does the most weird nonsensical bugs.
This is what happens when the benchmark team is funded by VC capital.
I find it fascinating that some of the main areas people are interested in are using benchmarks with quite few questions: ~200 for coding and ~150 for science.
As long as that's the case, it's no surprise that there is often a perceived difference of quality between benchmarks and real-world use.
I know that running the full composite benchmark set is probably already quite costly, and also that the benchmarks have to be rotated regularly, but it feels like a significant untackled problem.
Interesting that changes in the index always happen when „non us big tech“ models seem to be a little to good for comfort.
It makes sense for Muse Wonder though as that model does not trade blows with the best frontier models by a longshot.
chinese labs are probably benchmaxxing but i'd like to see when the index changes relative to the position of the best chinese models (last suspiciously-timed update was when glm-5.3-flash was looking really strong)
Well either AA is biased bribed riged whatever or chinese models are benchmaxxed or maybe its both
> ...or chinese models are benchmaxxed...
It is hard to tell what people are trying to say when they use undefined words with no particular meaning. This doesn't seem to change the relative position of the Chinese models much. The Chinese are a month or few behind and the state of intelligence is so volatile that the top 10 will be completely different in 6 months. I'd expect it to still be a mix of US and Chinese models with the US models leading.
Neither is the primary cause, I'd argue. If you look at the coding index specifically, you're getting overall more plausible results (in particular, Terra no longer outperforming Fable). [1, 2]
I don't find it surprising that especially smaller Flash models that have been optimized primarily for coding fall behind on non-coding tasks and that their overall score suffers in comparison.
It does give rise to the question of how useful a single aggregate numeric index of very heterogeneous capabilities is, though.
Look also at the cyber index; while somewhat difficult to extrapolate because of refusals, even if for Opus 5.5 and Fable 5.1 you optimistically count all refusals as successes, the results are ... interesting.
[1] https://artificialanalysis.ai/#coding-agents-tabs
[2] To what extent Terminal-Bench 4.0 can be trusted, is a separate question, but unlikely to be biased towards one particular group of models: https://preferencemodel.com/blog/terminal-bench-4-is-saturat...
Tinfoil hat time: but the way they waited a couple weeks to evaluate Mimo 2.6 Flash, allowing GPT-6 Luna to sit on the Pareto frontier in the meantime, really did make me think they might have some bias going on... (Luna was displaced when they finally evaluated Mimo 2.6 Flash, which had been released the day before Luna)
edit to add: there totally is benchmaxxing going on, but I'd be really surprised if it wasn't across the board. After all, how can any lab tell if their model has improved over a previous iteration, if not by comparing benchmark performance?
The drop for K3.1 Pro & MiMo V2.6 Pro is quite surprising. Are they benchmaxxed?
Or, are we witnessing reverse-benchmaxxing? Meaning that first some entities let players optimise for a given benchmark, then they update the measurement methodology so as to retain some players on the top ranks, while dropping others. As former Italian prime minister, and highly controversial public figure, Giulio Andreotti once remarked "Thinking the worst is a sin, but you’re often right."
Transparently so. MiMo's training dashboard showed the DeepSWE hillclimb in real time.
I keep seeing that MiMo is in the frontier in terms of performance and cost but I see very few talking about it and the only ones that do talk about it are doing it in context of AA
This seems to be a hit and a miss, there's no visual indication that opus 5.5 is better even though in 9 tests out of 10 it will outperform astra especially when it comes to niche domains and will absolutely destroy them in anything requiring raw intelligence rather than a naive search, naive search is what makes openai models extremely capable in scientific fields though.
The score hit on smaller models is a welcome change especially when compared to models that nearly 10 times larger.
Grok being punished quite a lot is also a very welcome change as the model is frankly not that good.
I would love to see few more points knocked off haiku by introducing steerability/creativity benchmarks.
It's wild seeing opus 5.5 up there so high, while i'm still using Fable 5.1 because I couldn't stand how opus writes.
Opus 5 writes like its off it's meds, opus 5.5 is a lot less weird that way.
Where is GLM 5.3 Flash?
Interesting to see the open weight models drop so drastically from the changes here.
They perform pretty badly on the new benchmarks introduced.
[dead]
(EDIT: A child comment has refuted the AA score comments, AA runs using their own harness.)
One of the new benchmarks is FrontierCode 2.0 by Cognition. AA seems to have accepted Cognition's benchmark scores without questioning, but they should have looked more closely.
On this benchmark, Anthropic's models are run using Claude Code; OpenAI's using Codex, and Gemini is run using... Cognition's own Devin (not Antigravity). Thus its score is much lower than it should be, knocking ~3 points off its AA score. All open-source models are also inexplicably run using Devin, damaging their scores as well.
This may even be intentional, if Cognition has beef with Google, since Antigravity was built using Windsurf's technology. But aside from that, it is to their advantage to make Anthropic/OAI look like leaders in the AI market. They are a company known to lie and exaggerate again and again, so I have trouble giving them benefit of the doubt.
I don't think this is relevant on AA's FrontierCode score. AA runs FrontierCode 2.0 benchmark using their own Stirrup agent harness [1].
[1] https://artificialanalysis.ai/evaluations/frontiercode-2-0
My mistake, thank you for catching.
It's interesting that the Stirrup scores are somewhat close to the specialized harness scores. Either Stirrup is quite excellent or is it that harness doesn't matter so much after a point?
What's the advantage for cognition?
I would think it's the opposite as a more competitive frontier for llms makes the labs shed some margins and enable a broader ecosystem.
Yes, that is true, you might be right. But I think such, because the moat in their wrapper layer feels very thin and does not make sense in a world full of 20+ wrappers around similar models. It feels better to "only" have to deal with Ant/OAI (and other pure wrapper cos). Competing vs. Google, Meta, and every Chinese lab on earth, every one using their own bespoke wrappers, is much scarier.
Additionally: competitive frontier = more commodified = high valuations under more scrutiny.
[flagged]