Performance is insane for the cost if those benchmarks hold up. Looks like they did time this for right before Anthropic's IPO. Curious what Anthropic releases in response, and how these do on real world tasks.
Feels like we're way past the pelican benchmark being useful at this point. Anyone have a better benchmark for a quick vibe check on how this compares to Fable 5.2 and Opus 5.5?
I didn't find Sol 6.1 nearly as impressive in real SWE use compared to what the benchmarks implied. It wouldn't surprise me if the models have internalised some kind of 'benchmarking mode' which only kicks in when some attributes that are peculiar to how benchmark problems are presented.
We've recently moved away from Anthropic models at $PAID_JOB and over towards GPT models. I'm trying to be open-minded but the GPT models we have (5.6 Terra and Sol) are just not that great. They're fine, but even simple things such as "check my work on this small Bash function" get it to point out stuff that's already implemented in my work. I don't know if it's me/how I use it/whatever, but just some N=1 evidence, I guess.
I've been having fun with having it build interactive Three.js demos of devices, with setup screen, a few different operational modes, mechanical movements etc
Office work has never been OpenAIs strength, no shame in that by itself, but these Slideshow examples look particularly awful, far worse than what I myself got out of GPT-5.6 Sol about two weeks ago in a single prompt.
Same for Workbooks and memos, that is objectively inferior to some pre Opus 4.5 files I have lying around. It's ok for a model not to be great at these, but these are some very poor examples. I am really confused what is going on at OpenAI in this regard.
I remember the same with their frontend blog post a few months ago that they proudly shared while filled with visual glitches, bugs and issues. Also, the vibe charts from a bit before that. Please, get someone with fresh eyes to look at your communications before you publish, these leave a very bad first impression.
Regardless of office work, I can't wait to find out whether these models finally adhere to a prompt or are still willing to carry out explicitly disallowed, irrecoverably destructive tasks, just because those might (but don't even always) get the model to a result more quickly along with some of the other regressions that have been present since the release of GPT-5.5.
Happy for the new Sol. It's my workhorse on company account.
But more than that I'm waiting for new Luna. To me it was the game changer that ended "Tokenpocalypse", and is still almost unbeataben in "infinitely using a decent model" use case.
I find that Luna tends to misinterpret instructions in READMEs and Agents.mds. Maybe it is something specific about my setup, but I found that Terra is the lowest I can comfortably use on its own (Luna can be driven by larger models in the form of a subagent and it works great for that)
Seeing them keep prices low is awesome. Was worried for a while that it would be like the GPT5.x series where progressively the models just got more and more expensive.
Cool, looks like 6.2 Sol is cheaper than Sol 6.1. I've stopped using Astra because it also goes and does things that I didn't want or didn't ask for, and burns a bunch of credits while doing so.
Am I reading these charts wrong? It seems like in most cases 6.2-sol : 6.1-sol Isn't that big of a change? It is a little smarter but costs a little more.
I find that 6 Astra already had very good taste in frontend design, but relatively poor taste in "nice and readable code". I really hope this release cracks that. Excited to try it out!
Its been stated often, but I appreciate that OAI is now bringing the topic of conversation to "task completion cost" rather than brute forcing solutions
I have yet to find an application for Astra vs Sol where the former is noticeable better. Genuinely asking, does anyone feel a big difference in their work between the two? If so, what are you working on?
Subagent coordination, Astra also performs better on Opus/Fable thinking/modeling shape problems imo, also computer use/thinking about shapes. Astra is able to hear waveform images which is pretty nuts.
For just doing things or coding things up, Sol all the way.
Wow, that's the opposite of what I see. I always simply use Astra (high), despite the cost and latency overhead -- it's just a lot more reliable. I find that on average it uses fewer tokens for a task (and I lose less hair per task), simply because I don't have to steer it as precisely .
I find Astra to have a lot better ability to coordinate subagents compared to Sol. I haven’t found a distinction in the complexity of work they can handle.
Benchmarks look good, it most often is the case with Astra and Sol.
And then i start doing some actual work and everything breaks down, and i have to go back to Claude to fix the mess.
I'll probably give it a try but i guess i should be prepared for some serious over engineering once again.
You ask for a 'hello world' and you get load bearing sealed artifacts with hashes and p99 performance testing for 3 days without having the actual 'hello world' in the end
Performance is insane for the cost if those benchmarks hold up. Looks like they did time this for right before Anthropic's IPO. Curious what Anthropic releases in response, and how these do on real world tasks.
Feels like we're way past the pelican benchmark being useful at this point. Anyone have a better benchmark for a quick vibe check on how this compares to Fable 5.2 and Opus 5.5?
I didn't find Sol 6.1 nearly as impressive in real SWE use compared to what the benchmarks implied. It wouldn't surprise me if the models have internalised some kind of 'benchmarking mode' which only kicks in when some attributes that are peculiar to how benchmark problems are presented.
We've recently moved away from Anthropic models at $PAID_JOB and over towards GPT models. I'm trying to be open-minded but the GPT models we have (5.6 Terra and Sol) are just not that great. They're fine, but even simple things such as "check my work on this small Bash function" get it to point out stuff that's already implemented in my work. I don't know if it's me/how I use it/whatever, but just some N=1 evidence, I guess.
You make it sound as though the models are like not working to their full potential instead of being a little bit overfit on the benchmarks.
I've been having fun with having it build interactive Three.js demos of devices, with setup screen, a few different operational modes, mechanical movements etc
Office work has never been OpenAIs strength, no shame in that by itself, but these Slideshow examples look particularly awful, far worse than what I myself got out of GPT-5.6 Sol about two weeks ago in a single prompt.
Same for Workbooks and memos, that is objectively inferior to some pre Opus 4.5 files I have lying around. It's ok for a model not to be great at these, but these are some very poor examples. I am really confused what is going on at OpenAI in this regard.
I remember the same with their frontend blog post a few months ago that they proudly shared while filled with visual glitches, bugs and issues. Also, the vibe charts from a bit before that. Please, get someone with fresh eyes to look at your communications before you publish, these leave a very bad first impression.
Regardless of office work, I can't wait to find out whether these models finally adhere to a prompt or are still willing to carry out explicitly disallowed, irrecoverably destructive tasks, just because those might (but don't even always) get the model to a result more quickly along with some of the other regressions that have been present since the release of GPT-5.5.
Happy for the new Sol. It's my workhorse on company account.
But more than that I'm waiting for new Luna. To me it was the game changer that ended "Tokenpocalypse", and is still almost unbeataben in "infinitely using a decent model" use case.
I find that Luna tends to misinterpret instructions in READMEs and Agents.mds. Maybe it is something specific about my setup, but I found that Terra is the lowest I can comfortably use on its own (Luna can be driven by larger models in the form of a subagent and it works great for that)
I used Luna Max a lot, until Deepseek 1017. Then never again.
I wonder why they didn't show Gemini 4 Carbon's scores on the science benchmarks, especially since Google seems to be pushing it for that -
https://x.com/pushmeet/status/2120814064863461440?s=20
(FD: Google employee, I don't work on Gemini)
Yeah it feels shady how Carbon was selectively plotted (or not), unlike Opus/Fable
Two posibilities. 1. They couldn't figure out how to actually use those Google models.
2. They launched the benchmark days ago and Gemini is still rereading the file with tasks.
The video is pretty funny. It's like a callback to 90s/00s office life?
I'm immediately reminded of the famous Excel elevator commercial. [1] Perhaps that was intended?
[1] https://www.youtube.com/watch?v=HegoJaXOIqQ
Seeing them keep prices low is awesome. Was worried for a while that it would be like the GPT5.x series where progressively the models just got more and more expensive.
I feel like the fact that they're not public lets them play the subsidization game though. And the same playbook that Anthropic started at that.
Would be awesome if it stays this way though
we've seen anthropic numbers, inference is not being subsidized by these companies
Tweeted 11 minutes ago https://x.com/OpenAI/status/2122750124543926515, posted to HN 18 minutes ago
The promo video seems to be a pretty strong rip on the 90's Microsoft Excel elevator ad. [1]
Maybe it found its way into the training data?
[1]: https://www.youtube.com/watch?v=HegoJaXOIqQ
Caught this on my lunch break, their linked system card (https://deploymentsafety.openai.com/gpt-6-2-astra-sol) isn't up yet.
Cool, looks like 6.2 Sol is cheaper than Sol 6.1. I've stopped using Astra because it also goes and does things that I didn't want or didn't ask for, and burns a bunch of credits while doing so.
Am I reading these charts wrong? It seems like in most cases 6.2-sol : 6.1-sol Isn't that big of a change? It is a little smarter but costs a little more.
Looks like 6.1-sol high+ is equivalent to 6.2-sol medium at the same price. So a bit smarter at a cheaper cost.
I'm still waiting for the shoe to drop. It's literally impossible for the prices to remain this low for much longer.
I find that 6 Astra already had very good taste in frontend design, but relatively poor taste in "nice and readable code". I really hope this release cracks that. Excited to try it out!
Very much on time (scroll down to "Release-Prognose"): https://www.heise.de/en/news/Alibaba-launches-Qwen4-Max-more...
Its been stated often, but I appreciate that OAI is now bringing the topic of conversation to "task completion cost" rather than brute forcing solutions
No reset?
Will need to play around with it to get a feel but those benchmarks with those costs look incredible!
It seems like they were saving these for just before ANT's IPO
I care less about the models being good and more on if I can fallback to them if Opus 5.5 gets a massive price hike after the IPO
I have yet to find an application for Astra vs Sol where the former is noticeable better. Genuinely asking, does anyone feel a big difference in their work between the two? If so, what are you working on?
Subagent coordination, Astra also performs better on Opus/Fable thinking/modeling shape problems imo, also computer use/thinking about shapes. Astra is able to hear waveform images which is pretty nuts.
For just doing things or coding things up, Sol all the way.
Wow, that's the opposite of what I see. I always simply use Astra (high), despite the cost and latency overhead -- it's just a lot more reliable. I find that on average it uses fewer tokens for a task (and I lose less hair per task), simply because I don't have to steer it as precisely .
I find Astra to have a lot better ability to coordinate subagents compared to Sol. I haven’t found a distinction in the complexity of work they can handle.
EE/circuit design is way better with astra.
Surprising that their top model would be still be so cheap, and yet not surpass Fable in the high cost part of the curve
Website pegs my GPU at 100%. Are they mining for crypto with this announcement?
Trying new ways to create distributed systems to train their LLM
The announcement video is really cringe imo.
Like do we really want people to be making last minute unreviewed AI generated slides and reports for work?
> frontends that look and work better
The website doesn't event show the graphs properly.
The graphs look fine on my computer. Can you elaborate on what exactly isn't "shown properly"?
I wonder if these will have the same crazed rotwiler feel that previous OpenAI models have had.
> we’re launching GPT‑6.2 Astra and Sol with tighter safeguards for higher-risk cybersecurity assistance
Seems they really want these "safeguards" to be as ridiculous as those of Claude..
Benchmarks look good, it most often is the case with Astra and Sol.
And then i start doing some actual work and everything breaks down, and i have to go back to Claude to fix the mess.
I'll probably give it a try but i guess i should be prepared for some serious over engineering once again.
You ask for a 'hello world' and you get load bearing sealed artifacts with hashes and p99 performance testing for 3 days without having the actual 'hello world' in the end
I beseech the incoming Democratic politicians to not curtail progress in LLMs.
Waiting for the pelicans benchmark.
With the red scarf and fish in the front basket...
Is this the Bel model?
Nice 3D effect. Runs surprisingly well even on on brave on my linux pc without hardware acceleration
Those AI labs got to chill. almost every day there’s a new model! Lol
Another wave of Openai just killed ....
Also, is it website very laggy or is it just me?
I thought OpenAI would drop an awesome model on the day of Anthropic IPO, just to troll them
It's actually in OpenAI's interests that Anthropic's IPO is successful.
explain?
If Anthropic is a valuable company, then OpenAI is just as valuable, if not more.
"Agents' Last Exam v1.1" did a human really look at this and say "nah, a linear scale is definitely the right way to show this"
Called it out 7 days ago: https://news.ycombinator.com/item?id=50431144#50433568
Actually I was really wrong. The model is not expensive!
Dead on arrival, judging by the FrontierBench scores. Astra 6.2 is five percentage points below Fable 5.2 xhigh.
Whatever you’re doing, it’s unlikely to be what most users are doing. Fable and Astra aren’t in my price range so I’m paying attention to Sol.
The point is that it's a fraction of the cost.
I think "second place but also cheaper" isn't what OpenAI needs right now.
"Close second, and also cheaper" is however a pretty solid framing when your competitor is trying to IPO