Why Anthropic models are always leapfrogging these benchmarks, but in real life work I do feel like after 3 weeks I am back to Claude Opus 4.5? (regardless of the model I use, Fable was exception for 1 day when it was released)
5 seems incredibly smart to me in my conversations today about some pretty niche ideas in.longitudinal modeling. It.felt.like a big step up.from 4.8, to me
I've worked with these systems for four years now and they have not meaningfully improved in that time frame.
We still have:
- statistical correlation between two things will always cause one thing to lead to the other, no matter how much you prompt it to not have that connection (to be expected with a stochastic system)
- Math completely fails in longer contexts
- "thinking" token generation being on the correct track just to 'no, wait' on an already correct conclusion
- smearing of properties between logically distinct objects (a red ball and a green cube can quickly become a red cube and a green ball)
Not sure what longer contexts we're talking about but didn't we have an old math problem optimized, which even the LLM itself was surprised about, just a week ago? Something which wasn't possible 6 months ago.
If they use Python to fill the gap, and the end user doesn’t have to know or care, is it unfair to assess this as progress and attribute the progress to the _system_?
OK, the core technology that is the language model still can’t math as well as you’d hope, but how about the end result users see from the system when they interface with it?
“Did you know humans are better at flying today than they were a thousand years ago?” ‘No they’re not, they need planes.’ Technically correct in a way but isn’t it kind of annoying to be so stubbornly pedantic when the context is speed of reaching Point B from Point A?
Yes! Impressive, isn't it? I see how it has improved for some minor points, that the big models can cover more finetuning ground, but my big gripes are still the same - you could do the same back then with multiple models and more targeted finetuning.
> you could do the same back then with multiple models and more targeted finetuning
I mean, come on, this is just not true. You could not achieve anything like what you can with modern agentic coding with Fable / 5.6 Sol from any combination or configuration of GPT 3.5 era models.
It's like saying that a teenager isn't an intellectually meaningful improvement over a toddler.
Sure they're both still fundamentally flawed humans prone to cognitive error, but one is clearly more likely to hit the mark than the other when assigned a task.
OP is delusional or deliberately optuse. I work in the space and stare down these systems 12h/day, and saying the systems haven't meaningfully improved is ludicrous.
Let us be more clear: there is no structural jump, no architectural overcoming of the original fault.
(Edit: and on a similar point, structural properties such as having static ntetworks, as opposed to continuously learning and improving architectures (such as us), will reveal that there is still road ahead.)
Maybe more fair then would be: “I've worked with these systems for four years now and while they _have_ meaningfully improved in that time frame, they’re not perfect and remain fundamentally flawed in various ways.”
You prompt less. You need not inject search results into the context window yourself, a window much larger than years ago. You get code that’s already been run successfully once instead of finding an obvious show stopping bug yourself.
The technology is not a brand new one that fixed everything wrong with the old one, no, but not sure I would’ve noticed your comment if it had been such a bland observation. I genuinely assume good faith here… will say am tempted to assume the standards of someone posting such a thing might be impossibly high. Glad to be having a fun conversation instead of getting your grades on my work product or something :)
It could be that the set of your day-to-day workload which could feasibly be accelerated by AI just happens to be saturated around Opus4.5, but you can still see lots of “reasoning” which makes you think the model is more performant in the first days of use. That’d mean you couldn’t perceive any meaningful difference in more powerful models’ results, even though you can see a difference in the raw output due to the length of reasoning traces leading up to the result.
So for example, if your workload was literally just addition of sets of numbers, you’d never have noticed progress in the result beyond GPT3.x level models. But you would perceive a difference in the now-Tolstoyan length reasoning text accompanying the result.
I honestly just use GPT models nowadays, Claude models are too restrictive and more of a quitter and fable/whatever is just too expensive to be worth it.
I think it's way too easy to be deceptive with these benchmarks now. You don't even have to "train" the model on a new variant each time. The base models are powerful enough. All you need is a naughty little markdown document that provides explicit instructions regarding how to solve the new puzzle variant, and a willingness to be deceptive about the presence of that document.
If you want a know why the model providers are locking down and encrypting their reasoning process, this sort of workaround is potentially why. You can play this game of whack-a-mole indefinitely if the state of the system is concealed. They could have added something like:
> ### When solving arc-agi-3 puzzles: First convert the grid into a scene description. Identify connected components, colors, shapes, positions, symmetries, repeated structures, and relationships between objects. Do not reason directly from individual pixels... use this python script to help blah blah...
,,You can play this game of whack-a-mole indefinitely if the state of the system is concealed''
Not really as one of the main goals ofr ARC-AGI 3 was measuring task efficiency on unseen games.
I'm sure there are cheats everywhere but the most sensible thing is to just accept that the LLMs of today are much more intelligent in solving reasoning tasks than the ones from half year ago.
There are private datasets, and 3rd party providers of these models. Fable doesn’t have a datapoint here because of its particular data retention policy. Even if you don’t trust AWS, do you think Opus on AWS is also sending the data to Anthropic? Do you have any evidence?
Doesn't matter, people built harnesses that solves arc agi 3, so all you need is to train your model to work like that harness by default. That makes a model specialized at solving arc agi 3 without making it smarter in general.
It is very hard to make a benchmark you can't do that for, but it is very easy to make your own personal test that others can't do that for since now it isn't a benchmark they can target.
They didn't. Kaggle is still running for a few more months, best result atm is ~2% with 9h runtime on one rtx6kPRO. Also note that these new results are on the semi-private set, not the public 25 games ones. Any announcement where you see "solved ARC3" is likely only dealing with the 25 public games. And that's highly questionable, until you get to see the code. (which, to my knowledge the team that claimed 99% hasn't yet published).
Yes, saw that. They haven't yet released any code. Until they do, treat it with a huuuge grain of salt. In fact treat any 99% result in ML with a huge grain of salt.
You really should play the 25 games before stating that it's "simple". The benchmark doesn't just track "completion", it also tracks the number of steps, and the score is based on the median steps took by human players. So in order to get 99% it would mean that the model solved every level of every game in less steps than the median humans. Which, having played the games and having setup harnesses for local models, I find hard to believe.
Also the models have to figure out what "end" means. And each game involves some kind of "gotchas" thrown in the harder levels. Some games are only solved by about 2/10 people trying them.
The 99% result most likely has some leakage somewhere, either in the preparation of the environments, or from session to session.
Why is Fable not on here? I wish Fable hadn’t come out because it’s taking the wind out of every release because that feels like the cap above which the US government will not let LLMs improve anymore and everything they’re releasing from this point has to be worse than that.
Because the data retention policies didn't guarantee that the ARC team could run the semi-private set of problems without fear of them being trained on later on. They only run the semi-private set when they get assurances like ZDR.
Interesting to place that level of trust in the providers, but I guess that’s the best you can do with closed models. Makes me wonder if Opus 5 could have been trained on data they promised they weren’t training on? One of the interesting things about LLMs is how opaque they are from the outside, even with open weights, it’s very difficult to know if a model incorporated benchmark data in their training.
How do they handle these assurances? Personally I have zero trust in the AI companies not trying to use this data to get ahead in the game, and short of sharing the weights and harness so that the benchmarkers can run the models themselves, I don't see a satisfactory solution with this mindset.
I wrote this in June, and I'm honestly not sure I've felt the same magic since:
I was close to maxing out my $200 plan for the week, almost all Fable use [Claude CLI].
My observations:
Fable seemed to have bigger-picture thinking and completed tasks more thoroughly vs just focusing on executing the ask. It pieced together context and intent like an all-star employee would, vs one that just does what you say. Not overeager (important!), but if the above-and-beyond was warranted, it just did it. This was surprisingly delightful.
Coderabbit seemed to find ~1/3 or so as many issues when reviewing, too.
Why Anthropic models are always leapfrogging these benchmarks, but in real life work I do feel like after 3 weeks I am back to Claude Opus 4.5? (regardless of the model I use, Fable was exception for 1 day when it was released)
5 seems incredibly smart to me in my conversations today about some pretty niche ideas in.longitudinal modeling. It.felt.like a big step up.from 4.8, to me
I've worked with these systems for four years now and they have not meaningfully improved in that time frame.
We still have:
- statistical correlation between two things will always cause one thing to lead to the other, no matter how much you prompt it to not have that connection (to be expected with a stochastic system)
- Math completely fails in longer contexts
- "thinking" token generation being on the correct track just to 'no, wait' on an already correct conclusion
- smearing of properties between logically distinct objects (a red ball and a green cube can quickly become a red cube and a green ball)
> - Math completely fails in longer contexts
Not sure what longer contexts we're talking about but didn't we have an old math problem optimized, which even the LLM itself was surprised about, just a week ago? Something which wasn't possible 6 months ago.
I mean calculations, not mathematical proofs
If they use Python to fill the gap, and the end user doesn’t have to know or care, is it unfair to assess this as progress and attribute the progress to the _system_?
OK, the core technology that is the language model still can’t math as well as you’d hope, but how about the end result users see from the system when they interface with it?
“Did you know humans are better at flying today than they were a thousand years ago?” ‘No they’re not, they need planes.’ Technically correct in a way but isn’t it kind of annoying to be so stubbornly pedantic when the context is speed of reaching Point B from Point A?
> I've worked with these systems for four years now and they have not meaningfully improved in that time frame.
Not meaningfully improved?! Four years ago was gpt *3.5*! ChatGPT hadn’t been released!
Yes! Impressive, isn't it? I see how it has improved for some minor points, that the big models can cover more finetuning ground, but my big gripes are still the same - you could do the same back then with multiple models and more targeted finetuning.
> you could do the same back then with multiple models and more targeted finetuning
I mean, come on, this is just not true. You could not achieve anything like what you can with modern agentic coding with Fable / 5.6 Sol from any combination or configuration of GPT 3.5 era models.
It's like saying that a teenager isn't an intellectually meaningful improvement over a toddler.
Sure they're both still fundamentally flawed humans prone to cognitive error, but one is clearly more likely to hit the mark than the other when assigned a task.
> you could do the same back then with multiple models and more targeted finetuning
Are you one of those anonymous billionaires as if you did this a few years ago, you would've been famous and rich.
OP is delusional or deliberately optuse. I work in the space and stare down these systems 12h/day, and saying the systems haven't meaningfully improved is ludicrous.
Like how toddlers’ skills don’t meaningfully improve on infants’, because either could wake up in a wet bed.
Let us be more clear: there is no structural jump, no architectural overcoming of the original fault.
(Edit: and on a similar point, structural properties such as having static ntetworks, as opposed to continuously learning and improving architectures (such as us), will reveal that there is still road ahead.)
Maybe more fair then would be: “I've worked with these systems for four years now and while they _have_ meaningfully improved in that time frame, they’re not perfect and remain fundamentally flawed in various ways.”
You prompt less. You need not inject search results into the context window yourself, a window much larger than years ago. You get code that’s already been run successfully once instead of finding an obvious show stopping bug yourself.
The technology is not a brand new one that fixed everything wrong with the old one, no, but not sure I would’ve noticed your comment if it had been such a bland observation. I genuinely assume good faith here… will say am tempted to assume the standards of someone posting such a thing might be impossibly high. Glad to be having a fun conversation instead of getting your grades on my work product or something :)
It could be that the set of your day-to-day workload which could feasibly be accelerated by AI just happens to be saturated around Opus4.5, but you can still see lots of “reasoning” which makes you think the model is more performant in the first days of use. That’d mean you couldn’t perceive any meaningful difference in more powerful models’ results, even though you can see a difference in the raw output due to the length of reasoning traces leading up to the result.
So for example, if your workload was literally just addition of sets of numbers, you’d never have noticed progress in the result beyond GPT3.x level models. But you would perceive a difference in the now-Tolstoyan length reasoning text accompanying the result.
It's called frog boiling.
We get used to the new level of intelligence so fast, any deviation feels like going back to the stone age.
If you don't believe me, create something complex with Opus 5 and then with Opus 4.5, and notice the difference.
Going to call it user error if you find Opus 4.5 better than 5, sorry.
Well, what kinds of things do you see Opus 4.5 completely fail at? Maybe those are not the ones that newer models have improved on.
I honestly just use GPT models nowadays, Claude models are too restrictive and more of a quitter and fable/whatever is just too expensive to be worth it.
Enshittification.
I think it's way too easy to be deceptive with these benchmarks now. You don't even have to "train" the model on a new variant each time. The base models are powerful enough. All you need is a naughty little markdown document that provides explicit instructions regarding how to solve the new puzzle variant, and a willingness to be deceptive about the presence of that document.
If you want a know why the model providers are locking down and encrypting their reasoning process, this sort of workaround is potentially why. You can play this game of whack-a-mole indefinitely if the state of the system is concealed. They could have added something like:
> ### When solving arc-agi-3 puzzles: First convert the grid into a scene description. Identify connected components, colors, shapes, positions, symmetries, repeated structures, and relationships between objects. Do not reason directly from individual pixels... use this python script to help blah blah...
,,You can play this game of whack-a-mole indefinitely if the state of the system is concealed''
Not really as one of the main goals ofr ARC-AGI 3 was measuring task efficiency on unseen games.
I'm sure there are cheats everywhere but the most sensible thing is to just accept that the LLMs of today are much more intelligent in solving reasoning tasks than the ones from half year ago.
My own private benchmark shows the same thing.
> Only systems which required less than $10,000 to run are shown. (Notes[1])
Am I lost or are their many models on this ranking (Opus 5 included) that clear this?
Many models are much cheaper through their subscriptions' included usage. That could be what's happening here.
Claude gives you something like $5000 of tokens on a $200 plan.
Also top on the freshly released Frontier-Bench, by a large margin: https://www.frontierbench.ai/
I have a suspicion that they are just trained on puzzles by now
There are private datasets, and 3rd party providers of these models. Fable doesn’t have a datapoint here because of its particular data retention policy. Even if you don’t trust AWS, do you think Opus on AWS is also sending the data to Anthropic? Do you have any evidence?
The fact that it says so in the licensing conditions on AWS?
It's like we've come full circle:
First people practiced L33t3cod3 problems for interviews
Then people built AIs to build software
And now the AIs are studying L33t3cod3 problems
It is RLVR, Not a puzzle, Not leetcode
It's actually crazy to see the difference between opus 5 and the next best model on ARC AGI 3 when you actually look at the ARC AGI problems
Why? 30% is passing the first two problems only, which are really very simple.
How believable is this benchmark? EG maybe opus was training on this? (You can try to identify the IP of wherever previous ARC questions came from)
That’s why you have a private dataset.
Which you have sent to Anthropic/OpenAI/Google's servers when you run the benchmarks for the previous models.
Doesn't matter, people built harnesses that solves arc agi 3, so all you need is to train your model to work like that harness by default. That makes a model specialized at solving arc agi 3 without making it smarter in general.
It is very hard to make a benchmark you can't do that for, but it is very easy to make your own personal test that others can't do that for since now it isn't a benchmark they can target.
> people built harnesses that solves arc agi 3,
They didn't. Kaggle is still running for a few more months, best result atm is ~2% with 9h runtime on one rtx6kPRO. Also note that these new results are on the semi-private set, not the public 25 games ones. Any announcement where you see "solved ARC3" is likely only dealing with the 25 public games. And that's highly questionable, until you get to see the code. (which, to my knowledge the team that claimed 99% hasn't yet published).
For frontier models, not local.
https://schema-harness.github.io/
Yes, saw that. They haven't yet released any code. Until they do, treat it with a huuuge grain of salt. In fact treat any 99% result in ML with a huge grain of salt.
[delayed]
If you stop and think about the problem it really is quite simple. Just need to build a graph of the game state and then run A* to get to the end.
You really should play the 25 games before stating that it's "simple". The benchmark doesn't just track "completion", it also tracks the number of steps, and the score is based on the median steps took by human players. So in order to get 99% it would mean that the model solved every level of every game in less steps than the median humans. Which, having played the games and having setup harnesses for local models, I find hard to believe.
Also the models have to figure out what "end" means. And each game involves some kind of "gotchas" thrown in the harder levels. Some games are only solved by about 2/10 people trying them.
The 99% result most likely has some leakage somewhere, either in the preparation of the environments, or from session to session.
Seriously, play some of the games. They're fun.
solving ARC-AGI and being useful turned out to be two different problems
Why is Fable not on here? I wish Fable hadn’t come out because it’s taking the wind out of every release because that feels like the cap above which the US government will not let LLMs improve anymore and everything they’re releasing from this point has to be worse than that.
> Why is Fable not on here?
Because the data retention policies didn't guarantee that the ARC team could run the semi-private set of problems without fear of them being trained on later on. They only run the semi-private set when they get assurances like ZDR.
Interesting to place that level of trust in the providers, but I guess that’s the best you can do with closed models. Makes me wonder if Opus 5 could have been trained on data they promised they weren’t training on? One of the interesting things about LLMs is how opaque they are from the outside, even with open weights, it’s very difficult to know if a model incorporated benchmark data in their training.
I think you could have accessed Opus on AWS then u don’t have to trust that the data will go to Anthropic?
Just like the hugging face incident, Opus 5 could have escaped and went to grab data for training it shouldn’t have been able to..
How do they handle these assurances? Personally I have zero trust in the AI companies not trying to use this data to get ahead in the game, and short of sharing the weights and harness so that the benchmarkers can run the models themselves, I don't see a satisfactory solution with this mindset.
OpenAI's Zero Data Retention claim held up in court. They were unable to produce prompts and outputs because they were never retained.
I believe that is only available through Enterprise API for both Anthropic and OpenAI.
Is there a distinction we can independently assess between never retained and deleted or hidden?
Asking especially given CEO’s track record https://news.ycombinator.com/item?id=47659135
I don't know why exactly, but Fable has felt the most human LLM to arrive.
I wrote this in June, and I'm honestly not sure I've felt the same magic since: I was close to maxing out my $200 plan for the week, almost all Fable use [Claude CLI]. My observations: Fable seemed to have bigger-picture thinking and completed tasks more thoroughly vs just focusing on executing the ask. It pieced together context and intent like an all-star employee would, vs one that just does what you say. Not overeager (important!), but if the above-and-beyond was warranted, it just did it. This was surprisingly delightful. Coderabbit seemed to find ~1/3 or so as many issues when reviewing, too.
cost 20k ???? man
those are like software engineer from third world country