It's still clearly not a part of any model's training set in the sense of pelicanmaxxing in particular. It's either a capability to generate arbitrary SVGs (or even something broader), or no specific training at all.
> How many pelican riding bicycle SVGs were there before this test existed? What if the training data is being polluted with all these wonky results...
But Simon has demonstrated before that asking for any animal doing something silly will still get similar results from the same model.
Certainly true but I think the question is “does targeting the SVG pelican mean that the quality of all SVG output improves?” If so it’s a worthwhile thing to target.
Why are all these AI's making the pelican's pouch so large though? Unless they have some huge fish in there or something they don't tend to look like that... Is it that it's drawing a "cartoon" pelican or something?
It starts to be so good that the model doesn't really matter. Each model starts to have its own personality, and it's the personality of the drawing, like a real artist, that defines the style of the pelican, and that makes it harder and harder to decide if it's good or bad.
Is it me or is there maybe some carryover happening where the pelicans are portrayed as expending different degrees of effort too? Like, the high effort pelican has a determined look on his face.
Is the prompt context telling the models what their effort level is, and the ambient presence of that concept of ‘high effort’ in the context window is leaking through and affecting the portrayal?
simonw, your pelicans are so famous now that the frontier labs can just RLVR for SVG images explicitly.
It's a simple manner of prompt...generate concept art with nanobanana, measure, llm creates svg with text, measure the distance between rasterized svg embedding to nanobanana art....do post-training RL on a 3 million such images and matching embeddings.
I swear this will be a canonical example of Goodhardt's Law in action.
You have to switch to a new benchmark that isn't so easy to run a verifiable rewards pipeline on...Perhaps a longer horizon task where it's easy to mess up midway.
Why? Don’t you think the steady improvement in pelican bicycling SVG art is itself a worthy field of endeavor for LLMs to be optimized for?
Yes, Simon has made pelicans a benchmark and now models have grown very good at rendering them. Is there a reason to believe that this isn’t manifesting as useful skill in general in SVG generation?
It's obvious OAI hillclimbs the image generation -> SVG loop. I'd put money down on that being why OAI's pelicans have a yellowish tint that matches the yellow tint of OAI's earlier image models.
The models that are really good at 3D modeling tend to also be really good at pelicans. I think the skills are strongly related - it's about having good 2D (and now 3D) spatial awareness.
The test is always going to be hard though, because it's an absurd image to ask for.
"Pelican on bicycle in SVG" is an adversarial example on multiple levels - out of distribution, ill choose of tool for the task, and anatomical impossibility of real pelicans using typical bycicles.
A lot depends on what the models were specifically trained for using reinforcement learning. Modern models are generally OK and everything, but they're only really good at the things they have extensive RL for.
Honestly surprising if other labs aren’t training for svg generation. Feel like SVG’s are going to be really useful for dynamic UI generation stuff that goes beyond cobbling Html together
The models are solving Millennium Prize questions and making progress on the Riemann Hypothesis, yet the top comment is still looking at pelicans somehow. To me, it feels even more superficial and pointless than arguing about the presidential candidates' tie choices in a debate.
Buddy, there's a Pelican riding it. No real cloud is like that, no real sea is like that, no real gulls are like that. And that's fine because it's a cartoon.
what is even the point of these now? labs are most definitely training on this question now. I feel like this test scratches the surface of a "real" test for these new models
It attracts at least one comment like yours on every single thread, so I guess the main purpose is to kick off a race to see who can post that comment first.
Medium and high are fixies apparently, (or they have a worm gear, although the hub doesn't look big enough for that), since neither has a derailleur. That would also explain the lack of brakes at least.
Low is still obviously failing the "test". It's pedals are not actually attached to the bike either.
Medium has a handlebar that goes through the pelican's beak. So I'd say also failing.
For the high, the only nitpick would be that the bell isn't quite on the handlebars and the stripes on his scarf don't line up correctly.
> The model launches today at an introductory price of $2 per million input tokens and $10 per million output tokens (with cached input tokens at 95% off).
Couldn't you just say what the price will be after the introductory period and when it'll take effect? Do they have some SKILL.md for these blog posts that requires the most important information to always be buried ten levels deep and hidden behind a login page?
With the pace they're delivering models, there's gonna be 1-2 new Gemini 4 versions by December 1st, presumably with their own introductory price windows. We just got Gemini 3.8 on 2 September and Gemini 3.7 on 13 August.
I'm using this model via API. I am most excited about its ability to write the english language (or any language). My least favorite trend on twitter has been people posting 2D illustrations/3D renders as a proxy for model capability. Well-written prose is far less likely to go viral but far outstrips the importance rendering a fantasy world in one shot.
I've been trying it out via the API and the model has been a delight so far. Clear in the head, not RL-fried, distinct writing style, and big model smell. That said, the em-dash is back w/ this model.
That said, I would be so frustrated with Sundar Pichai if I was on the Google team. The botched rollout of this model seems to be entirely a matter of capacity, and Pichai has been more than happy to give it to anybody but his own team.
Announced the model two weeks ago to a ton of hype, model was unavailable for anyone to try. On twitter, Google folks went on a great run taking shots at Muse ("Muse who?"). The hype died down bc nobody could touch the models and meta/muse came out with a similarly capable model in the meanwhile.
At the same time, OpenAI had a disastrous dev day that led a lot of users to jump ship to Claude.
Couple weeks later, Gemini-4-argon is still unavailable on anything less than the ultra tier, which means there's no natural funnel to get ppl to pay for the more expensive ultra plan other than "trust us the model is good".
If Google had argon broadly available earlier this month they would have picked up a ton of new users.
This reflects pretty much every result of mine from Google these days. Not just navigation, but search too.
The only thing I can figure is that Google's consumer-facing free-to-use models are set to "reason as fast as possible with only the most basic plausibility checks." I have to correct them like this every single time.
Often it seems like the problem with the Gemini apps is the harness, not the model itself. For example if I upload an image PDF it recognizes text worse than if I use the API.
The auto version is extra horrible because what it reads to you is just a summary of what the model actually said. I don't ask it anything when driving because it's too frustrating.
Yeah, whatever they have running on Google Maps is probably a pretty low tier model, but it is total trash. I’ve only used it twice though and that’s when it first came out. I don’t know if there have been any changes since then. The main problem is I felt like it was not keeping track of the context of the session.
It's a great model. I have had it trying to decrypt the Voynich manuscript for the past couple of days without being biased by other research/other people's attempts. If anyone is interested: https://github.com/d2kx/voynich-research
Good God, man. Google has the worst frontier lab rollout of them all. And they're delivering products that are behind the curve constantly.
Now, I happen to believe that's an excellent strategy. I bet they actually resolve more LLM requests than any other company at this point. The lay person still uses Google exclusively.
Are you suggesting it’s BETTER than their competitors?
Because if it’s not, and it’s not available to all paying users (Right now, it’s only available to Ultra users), then they are behind the curve if they’re treating it like Fable.
Moreover, you need a Google account in order to use it. I wouldn't bother paying 200 USD for it but I just don't want an entire google tracking package just for using one product.
Only available on Ultra, so I thought maybe I'll try it for one month to see if it's good. Well, I have Google One and for some reason this means I cannot sign up for AI Ultra? This is just one of the worst product rollouts I've ever seen
Interesting! Do you have webgl disabled for incognito or for chrome in general? I wonder if it's trying to do software rendering.
FWIW, I have literally the exact same configuration (M3 air, google chrome). Here's a recording on mine where I'm seeing 60fps: https://non.io/video/video-3587018401.mp4
I'm constantly blown away by it. I do think Astra is slightly more intelligent (although worse at this particular benchmark), but Opus 5.5 has become my default model for 99% of what I do. It's so good.
Of all models I like to write with Gemini the most. The language appears much more natural and elegant than what even Opus 5.5 or Astra do. Also, they give 5TB of cloud storage and discounts, it’s a steal for what you get.
To everyone saying not available yet: it looks from the page like Pro and Ultra subscribers have it today and given the play links, it'll probably show up in the API in a few hours.
I'm on Pro and it's trying to get me to upgrade to Ultra in the AI Studio UI when using Argon. In Antigravity CLI and the Gemini website it's not showing up at all despite being logged in
I like this phrasing and am curious what new word or phrase we might use to describe big companies over pushing Ai internally at the expense of their product quality. This pattern is so ubiquitous across google cloud products I have taken to calling them the 21st century IBM, having been a IBM baby and witnessing first hand their fall from exceptionalism.
Gemini isn't perfect but I find it a lot less annoying than other model families (personal preference or stockholm syndrome). Interested to read the reviews.
Seconded. Anthropic feel the need to limit how abusive you can be to Claude because god, I want to throttle that snotty little shit. Google don't have to do that with Gemini because it just isn't as irritating. It does what I ask and doesn't try to second-guess my requests, or do "the quick fix".
It's been really bad at lyrics but gotten better. You look something up and it will just make something up and it takes a bit of digging to discover it's because of copyright
It passed the test of using latest crystal-lang concurrent feature (with web search) when gemini 3.8 flash failed (again, with web search enabled). I think it is at least useful for coding, for sometime.
Right now there are two top posts one about GLM-Legal and one about Argon. Both score about the same in the Harvey Lab test, but the prices are $0.15 versus $10.55, the $0.15 is Vals AI's estimate using the base model since GLM-Legal has not published prices.
Yeah, but one is a general purpose model, and the other one is specifically geared towards the legal field. Considering Opus 5.5 only gets a 4% on the test compared to Gemini’s 20%, that’s actually pretty incredible.
"Argon autonomously turns raw video to a physics-accurate simulation, estimating depth and physical boundaries, and writing the exact simulation code required to replicate those real-world environments."
Geometry could be understood today with the model but what also really matters is the sensing, friction, mass how smooth the subjects are, dexterity is a very difficult problem so solve, not sure how simulations could handle such factors
Game engines have many of these features. At one point, iirc, Unity was positioning to be the simulation environment for world model agents before we spoke about it in these words.
Because you have an android phone and don’t want to mess around with anything else(i.e the same reason people just use internet explorer on windows) or you hold Google stock so you want to support your investment :)
I'm open to Google winning me over again, but there isn't enough hype in the world to do it. They will need to earn it after these disappointing 6 months.
I appreciate how it's being integrated into the Google ecosystem, but the user experience still feels unpolished when actually using the system. The marketing, however, is impressive.
I hope all the people very concerned about AI safety are cheering this one. The best guard against one AI swarm taking over is having multiple adversarial orgs all with similarly advanced AI swarms.
I wonder if any of the AI safety people will begin encouraging AI researchers to swap jobs to work at a failing lab in order to keep them all extremely competitive with one another.
I have the Free AI Pro plan from when I purchased my Pixel 11 Pro. I still don't have Argon in AGY nor web nor Gemini app. I have a free product and still feel cheated.
I'm still waiting for a model which I can tell to "take these 3 screenshots and make a product launch video", where it doesn't randomly modify the screenshots or garble/rewrite the text as it moves the screenshots around the video.
Every time I try to generate one it's like a slot machine, where a different part will be messed up after fixing the first.
I dont have that problem with opus5.5. I dont even take the screenshots anymore (it does all screen recordings)... I tell it a simple narrative and it follows through. e.g. https://x.com/plannotator/status/2108589088139886769?s=20
I would love to try it out, and the Gemini models had previously been some of the ones I'd liked best in an overall sense, but unfortunately their updated inference endpoint is no longer ZDR on OpenRouter (change takes effect on the 18th). I think this is one of the more unfortunate turns in AI development in 2026, as it seemed like labs offering it was becoming the norm last year, and Google had previously done so consistently.
Really? Reminds me of Gemini 3. It took so long before being able to get a subscription for it, and then I would run out of their equivalent of the 5-hour usage window in a heartbeat.
Digging in the model card, "Gemini 4 is the next generation in the Gemini series, a suite of highly-capable, natively multimodal, reasoning models. Gemini 4 Argon delivers frontier performance..." If they're using noble gases there are 2 obvious lighter naming candidates and 3 larger (or 4 if you count Oganesson). I'm not seeing them claiming Argon is the top/biggest/bestest model among the Gemini 4 series, just that it has frontier performance.
Edit: noble gases not noble taxes, though the latter would be a funnier naming scheme.
> Across Google and our early access partners, Argon is quickly becoming the preferred model for engineers tackling long-horizon software engineering.
"Across Google", wonder how much that is because of an effort by Google to curb usage of Anthropic models. MSFT and Meta similarly are pushing for in-house models. It doesn't mean the models are better when you push your engineers to dogfood and restrict access to other LLMs.
(Not that I know for sure this is the case, it's more of a hypothesis).
Pricing is interesting. At launch Argon is the same price as GPT-6 Sol, half the price of Claude Opus 5.5 and 1/5th the price of Claude Fable 5.1 and GPT-6 Astra.
But on December 1st Argon doubles in price - so it will be the same price as Opus 5.5.
All AI pages now include canvas animations. There isn't a single page without one. This is neither good nor bad, just an observation. I think OpenAI has the best animations so far, with Anthropic in second place. This one looks a bit off.
When you are on "gemini.google.com" there is a little lock icon next to Gemini 3 and Gemini 4. When you click it you get the prompt to subscribe (pay). Once you have successfully done so, the Gemini 4 option disappears!
It seems the model is a little verbose, using more tokens to complete a task that other similar models as stated more precisely in the linked article about Artificial Analysis's work on this model.
Page says it is now available to Ultra and Pro subscribers. Although in AI Studio it currently says you need Ultra. But it's technically released either way.
Yeah, I think this is it! Google tried to hype their model but they have no chance against queen Dario. Perhaps they should just delete Gmail and say that Gemini did it. That way they would get some attention …
The slides they show in the catering video demo are so bad (in addition to having all the usual signs of AI design), I would genuinely laugh if a supplier sent me this.
Finally, gemini 3.8 has been falling way behind opus and sol in my experience. It's showing up in my antigravity. My work account though still has 3.6, 3.8, 3.1.
The previous gen pro used to have this weird tick where if you ask it a reasonably open ended question, it would number things in the response and assign each one some weird label starting with 'The...'
e.g.
X: The Parameters
Y: The "System"
Z: The "Ongoing Grind"
The first two prompts I tested in AI Studio does the same, unfortunately.
Another quite unpleasant tick was that if you specify a peculiar term in your prompt as a conversational shortcut, it would keep repeating that throughout the conversation. e.g. "...What is a bitter lesson pilled way to do this?", then you'd see it will quote "bitter lesson pilled" at like 7 places in its response. This one seems to have eased off a little.
Glad to see them abandon the absolute train wreck of their model naming "strategy".
Gemini Live vs. Gemini Lite vs. Gemini Pro vs. Gemini Flash vs. Nano Banana Lite (that one apparently has no Gemini prefix?)... Please, just pick a metaphor and stick with it without wildly mixing reasoning modes, model sizes, and input/output modalities in the model name!
Just switched, and it seems to be much better. I'm hoping it stops lying. I have Pro max subs for Claude, Codex, Grok, and Google AI... I was forced into using Google by running out of credits elsewhere.
I have found that Gemini often claims it did something, but didn't actually complete the task.
I just had 3.8 Flash do a major refactor -- that never should have needed to be done in the first place -- and switched to Argon 4, and had it review it.
Almost everything had been stubbed out by 3.8, and none of it was properly integrated into the codebase. :facepalm:
>"Argon helped the Chrome team separate core webpage handling from a ~6 million line legacy folder of browser features held within Chromium’s 40 million-line codebase. Piping multiple clean code changes at a time, Argon helped the team finish an architectural refactor across hundreds of files, completing months of work in weeks."
Now that needed to be done for some time!
Regardless of if an AI helped or not, regardless of whether that was Google's or some other AI or not (though Argon does sound pretty cool!), regardless of whether an AI cut the time from months to weeks or not, refactoring Chrome -- decomposing it into individual clearly demarcated non-tightly coupled software parts with core webpage handling at the center -- that needed to be done for some time!
Now all of that being said, Gemini 4 Argon does sound pretty cool, and we congratulate both teams at Google -- the team that created Gemini 4 Argon, and the team that refactored Chrome with it!
But all of that being said, while the advancements made for Gemini 4 Argon are important and impressive (and they are!)... refactoring Chrome properly (IMHO) is slightly more important and impressive!
(You know, one man's humble opinion! That is, I am not the guy who is responsible for awarding performance-based bonuses at Google at the end of the year, so take everything I say with the proverbial "grain of salt"! :-) I am probably biased, one way or the other! :-) )
Unlike most people it seems, I like all the progress in AI but I really do wish they would reign in the over-the-top marketing. I mean, can we just get this stuff to the point where I can reliably work with it to design a well-architected software project without having to constantly struggle against bloat and nonsense directions no experienced or sane developer would ever take.
fyi: argon is an interesting choice of name, coming from the Greek argos meaning "lazy or living w/out labor".
Why can't the Gemini model code?
I'd guess Gemini probably used the best dataset.
Even Google is the place where the world's best coders are gathered, isn't it?
Yet the GPT code from OpenAI, which collaborated with Microsoft, is beautiful, while the Gemini model's coding is trash.
I have no problem with Gemini Flash 3.8 for coding, though I only use Elixir which is one of the better languages for LLMs. Where does it fall short elsewhere?
Meanwhile I paid a subscription and they only provide access to 3.6 Flash. Consequently, I have moved on to open-weight models. It is what happens when you screw your customers over.
3.1 Gemini was amazing for anything non-agentic. I switched to it for my primary chatbot purposes.
I never touch 3.8, it's awful. Curious to try this, but I've yet to use any Gemini that is good for agentic work.
Also, unrelated but this made me look at Artificial Analysis to see where it lands on Cost per Task, and it ranks Opus 5.5 at almost double Astra 6. That does not align with my experience.
Blatantly obvious that models will not slow down. For any task, if we can build a dataset or environment of verifications, it can be automated. This will eat all of human tasks.
The fact that multiple companies can breach any benchmark means there is no ambiguity in terms of how to build AI, its just a matter of time.
Certainly Meta and Google are benchmaxxing to please public markets, but it still proves the above
but they have slowed down. the delta in capability between model releases now is smaller than ever before and i expect it to continue to diminish until we reach some kind of asymptotic maximum, which hopefully will mean turn towards optimization and local model improvements. The current frontier models running at 10x the tok/s would be incredible for my use cases
The delta isn't really slower, we're just unlocking capability bundles in batches. The capabilities bundles you care about might have mostly been unlocked already, and just need refinement, which is coming slower than the initial capabilities came, but don't discount the breadth of advances being made across all possible tasks.
We're not living in the same universe. Look at capability year-over-year. The asymptotic maximum is wishful thinking. Someone needs to step in before this gets wildly out of control.
A little over a year ago they were winning their first gold in an elite high school math competition. Have you not seen what they are doing in math now?
Models just solved a Millennium price and you think their progress is slowing down? I will concede that their capabilities vary a lot from one field to another, but still.
Unsurprisingly, the pelicans are excellent. Google are the only lab who have publicly shared that they have specifically trained for SVG output - https://twitter.com/sunjiao123sun_/status/202455551655137292... - and they've had great pelican results in the past too (which they have shown off themselves - https://twitter.com/JeffDean/status/2024525132266688757).
Here's low, medium, high. Even low is better than pelicans from any other lab: https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
low cost 10.185 cents, medium was 11.098 cents, high was 17.166 cents.
The costs are similar because it used more output tokens than reasoning tokens for all three pelicans, even for high.
Me in 2025 https://news.ycombinator.com/item?id=44759781
> If it's on HN and is a meme at this point, it will end up in the training set.
> It's kind of fun to imagine that there is an intern in every AI company furiously trying to get nice looking svg pelicans on bicycles.
Google listened and assigned a few interns, I guess, for the task
It's still clearly not a part of any model's training set in the sense of pelicanmaxxing in particular. It's either a capability to generate arbitrary SVGs (or even something broader), or no specific training at all.
We all said it: https://news.ycombinator.com/item?id=46978726 (8 months ago)
> How many pelican riding bicycle SVGs were there before this test existed? What if the training data is being polluted with all these wonky results...
But Simon has demonstrated before that asking for any animal doing something silly will still get similar results from the same model.
SVG is useful for actual tasks, not just looking good in a silly "benchmark"
Certainly true but I think the question is “does targeting the SVG pelican mean that the quality of all SVG output improves?” If so it’s a worthwhile thing to target.
They said they are training for svg not pelican svg specifically
Maybe so, and yet it's still not enough -- the pelicans all have flaws.
They look remarkably good, sure -- but not perfect.
I end up getting SVG output for all kinds of things, and never once have I asked for, or gotten, a pelican.
Why are all these AI's making the pelican's pouch so large though? Unless they have some huge fish in there or something they don't tend to look like that... Is it that it's drawing a "cartoon" pelican or something?
https://en.wikipedia.org/wiki/Pelican#/media/File:Pelikan_Wa...
Well, a photorealistic pelican would generally be in a raster-graphics format rather than an SVG, so yes, cartooniness is expected.
The SVG is cartoonish, and to be honest, most of human-made SVG art is cartoonish or stylized. It's not a format with realistic expectations.
It starts to be so good that the model doesn't really matter. Each model starts to have its own personality, and it's the personality of the drawing, like a real artist, that defines the style of the pelican, and that makes it harder and harder to decide if it's good or bad.
You can really see the effort in the details.
Subtle highlights on the metal frame, sweat beads, jaunty little hat…it has style. At least compared to the rather literal ChatGPT outputs. https://static.simonwillison.net/static/2026/gpt-pelicans-gr...
Is it me or is there maybe some carryover happening where the pelicans are portrayed as expending different degrees of effort too? Like, the high effort pelican has a determined look on his face.
Is the prompt context telling the models what their effort level is, and the ambient presence of that concept of ‘high effort’ in the context window is leaking through and affecting the portrayal?
The fish in the basket on high is a lovely touch.
Simon’s headstone is going to have a pelican on a bicycle, isn’t it
He deserves a full parliament
Excellent, but not perfect -- if google really is training to pelicans, then clearly they are not capable of perfect pelicans.
As one point, the high pelican's arm is see-through/messed up.
They do look worlds better than previously, to be fair.
Wow, those are I think the best looking pelicans ever so far
simonw, your pelicans are so famous now that the frontier labs can just RLVR for SVG images explicitly.
It's a simple manner of prompt...generate concept art with nanobanana, measure, llm creates svg with text, measure the distance between rasterized svg embedding to nanobanana art....do post-training RL on a 3 million such images and matching embeddings.
I swear this will be a canonical example of Goodhardt's Law in action.
You have to switch to a new benchmark that isn't so easy to run a verifiable rewards pipeline on...Perhaps a longer horizon task where it's easy to mess up midway.
Why? Don’t you think the steady improvement in pelican bicycling SVG art is itself a worthy field of endeavor for LLMs to be optimized for?
Yes, Simon has made pelicans a benchmark and now models have grown very good at rendering them. Is there a reason to believe that this isn’t manifesting as useful skill in general in SVG generation?
It's obvious OAI hillclimbs the image generation -> SVG loop. I'd put money down on that being why OAI's pelicans have a yellowish tint that matches the yellow tint of OAI's earlier image models.
Regarding the pelican test. Maybe someone will know this.
Modern models can make nearly photorealistic games in threejs. So why do they struggle with these pelicans?
The models that are really good at 3D modeling tend to also be really good at pelicans. I think the skills are strongly related - it's about having good 2D (and now 3D) spatial awareness.
The test is always going to be hard though, because it's an absurd image to ask for.
UPDATE: I like this theory about using an agentic harness that can render and iterate better: https://news.ycombinator.com/item?id=50073879#50074395
"Pelican on bicycle in SVG" is an adversarial example on multiple levels - out of distribution, ill choose of tool for the task, and anatomical impossibility of real pelicans using typical bycicles.
The pelicans are one-shot with not tools: Think a bunch, write a bunch of SVG, done.
The pretty 3d stuff you see are agents controlling a Blender MCP in a loop where they look at the scene, think, adjust the scene, repeat.
A lot depends on what the models were specifically trained for using reinforcement learning. Modern models are generally OK and everything, but they're only really good at the things they have extensive RL for.
Have you tried editing SVG by hand?
Honestly surprising if other labs aren’t training for svg generation. Feel like SVG’s are going to be really useful for dynamic UI generation stuff that goes beyond cobbling Html together
doesn't this mean that you should change the pelican test to something else?
If a result is suspiciously good I run a spot check with one of my many other SVG things.
Here's 'Generate an SVG of an armadillo in fishnet tights jaywalking on Mars' for Gemini 4 Argon: https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
Some other models for comparison: https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
The models are solving Millennium Prize questions and making progress on the Riemann Hypothesis, yet the top comment is still looking at pelicans somehow. To me, it feels even more superficial and pointless than arguing about the presidential candidates' tie choices in a debate.
still radial spokes on both wheels of the bike which is disappointing. no real bicycle is built this way
Buddy, there's a Pelican riding it. No real cloud is like that, no real sea is like that, no real gulls are like that. And that's fine because it's a cartoon.
Holy shit, these are like art. Especially the bikes.
what is even the point of these now? labs are most definitely training on this question now. I feel like this test scratches the surface of a "real" test for these new models
It attracts at least one comment like yours on every single thread, so I guess the main purpose is to kick off a race to see who can post that comment first.
I enjoy it. Many readers here look specifically for Simon's pelican comment when a new model launches
It seems to be a deadpan inside joke around here that everyone pretends to care about the pelican test.
It's merely become part of the culture of new model drops even as it's become a bit less useful as a diagnostic.
low version doesn't have a chain.
Medium and high are fixies apparently, (or they have a worm gear, although the hub doesn't look big enough for that), since neither has a derailleur. That would also explain the lack of brakes at least.
Low is still obviously failing the "test". It's pedals are not actually attached to the bike either.
Medium has a handlebar that goes through the pelican's beak. So I'd say also failing.
For the high, the only nitpick would be that the bell isn't quite on the handlebars and the stripes on his scarf don't line up correctly.
Memes and self-promotion.
> The model launches today at an introductory price of $2 per million input tokens and $10 per million output tokens (with cached input tokens at 95% off).
Couldn't you just say what the price will be after the introductory period and when it'll take effect? Do they have some SKILL.md for these blog posts that requires the most important information to always be buried ten levels deep and hidden behind a login page?
They're doubling the price on December 1st: https://ai.google.dev/gemini-api/docs/pricing#gemini-4-argon
$2/million input and $10/million output today. $4/$20 after December 1st.
With the pace they're delivering models, there's gonna be 1-2 new Gemini 4 versions by December 1st, presumably with their own introductory price windows. We just got Gemini 3.8 on 2 September and Gemini 3.7 on 13 August.
Opus 5.5 is also priced at $4/$20, but are they on the same level?
>$4/$20
Is Elon Musk doing their pricing?
It's irrelevant information, they know the reader switches models every week.
They don’t really. Most normies which are the vast majority of people just use ChatGPT. All tech folk just use Claude.
Obfuscating token cost is the new great marketing frontier
I'm using this model via API. I am most excited about its ability to write the english language (or any language). My least favorite trend on twitter has been people posting 2D illustrations/3D renders as a proxy for model capability. Well-written prose is far less likely to go viral but far outstrips the importance rendering a fantasy world in one shot.
I've been trying it out via the API and the model has been a delight so far. Clear in the head, not RL-fried, distinct writing style, and big model smell. That said, the em-dash is back w/ this model.
That said, I would be so frustrated with Sundar Pichai if I was on the Google team. The botched rollout of this model seems to be entirely a matter of capacity, and Pichai has been more than happy to give it to anybody but his own team.
> far outstrips the importance rendering a fantasy world in one shot.
Given FelonyBench, do we also have a FantasyBench?
What was botched about it?
Announced the model two weeks ago to a ton of hype, model was unavailable for anyone to try. On twitter, Google folks went on a great run taking shots at Muse ("Muse who?"). The hype died down bc nobody could touch the models and meta/muse came out with a similarly capable model in the meanwhile.
At the same time, OpenAI had a disastrous dev day that led a lot of users to jump ship to Claude.
Couple weeks later, Gemini-4-argon is still unavailable on anything less than the ultra tier, which means there's no natural funnel to get ppl to pay for the more expensive ultra plan other than "trust us the model is good".
If Google had argon broadly available earlier this month they would have picked up a ton of new users.
I missed the em dash — ngl
Gemini 4 Argon is a new leader in Pareto-frontier. Looks like, Google is back in the race.
https://arena.ai/leaderboard/agent/chat/pareto
For almost same intelligence (+15.56%, +15.18%), drastic difference in cost/task ($3.51 vs $0.41)
Meanwhile, here's Gemini in practice:
Me: Gemini, navigate to nearest McDonalds.
Gemini: [Lists 10 McDonalds locations wnd asks which I want.]
Me: What were the priginal instructions just now?
Gemini: [Recites imperatives about tone and content from its memory system.]
Me: No. I told you to navigate to the NEAREST McDonalds. There's only one NEAREST McDonalds. Do that one.
Gemini: My mistake, navigating to the nearest McDonalds. [Opens Maps, navigating to a McDonalds across town.]
Pretty sure your phone assistant isn't using their best model. It's probably using one of the old flash models no?
When you ask stuff in your car on android auto it uses 3.5 flash-lite. The response details would include it in the logs.
Flash lite is very noticeably more idiotic than 3.8 Flash. It's why the quick AI overview is often much worse than the AI mode one.
This reflects pretty much every result of mine from Google these days. Not just navigation, but search too.
The only thing I can figure is that Google's consumer-facing free-to-use models are set to "reason as fast as possible with only the most basic plausibility checks." I have to correct them like this every single time.
Often it seems like the problem with the Gemini apps is the harness, not the model itself. For example if I upload an image PDF it recognizes text worse than if I use the API.
The first law of robotics prevents it from facilitating regular ingestion of low-quality nutrients.
The auto version is extra horrible because what it reads to you is just a summary of what the model actually said. I don't ask it anything when driving because it's too frustrating.
Gemini models are in fact very good. Google's harness(es) for Gemini are nightmares, terrible
Yeah, whatever they have running on Google Maps is probably a pretty low tier model, but it is total trash. I’ve only used it twice though and that’s when it first came out. I don’t know if there have been any changes since then. The main problem is I felt like it was not keeping track of the context of the session.
It sounds like the tool call is probably not grounded with your precise position.
I hate to be that guy, but Tesla's Grok doesn't seem to have that problem.
Sounds like it might not have your precise location in context?
It's a great model. I have had it trying to decrypt the Voynich manuscript for the past couple of days without being biased by other research/other people's attempts. If anyone is interested: https://github.com/d2kx/voynich-research
I saw 'jargon-free' and my eyes immediately glazed over
hahaha. I also build a site (also using Gemini 4 Argon) that makes the data more accessible: https://www.dmh.dev/voynich
Good God, man. Google has the worst frontier lab rollout of them all. And they're delivering products that are behind the curve constantly.
Now, I happen to believe that's an excellent strategy. I bet they actually resolve more LLM requests than any other company at this point. The lay person still uses Google exclusively.
I pay Google a reasonable price for 5TB storage/backup across GMail/Docs/etc. Any AI they throw in with that is icing on top of the cake.
What plan is this? How much a month?
How is this behind the curve? Seriously
Are you suggesting it’s BETTER than their competitors?
Because if it’s not, and it’s not available to all paying users (Right now, it’s only available to Ultra users), then they are behind the curve if they’re treating it like Fable.
It’s tone deaf.
Isn't OPUS much cheaper and generally regarded as more intelligent?
> I bet they actually resolve more LLM requests than any other company at this point. The lay person still uses Google exclusively.
Unique users, maybe. But a simple query that's answered in a paragraph is negligible compared to devs running agents continuously for tens of hours.
Moreover, you need a Google account in order to use it. I wouldn't bother paying 200 USD for it but I just don't want an entire google tracking package just for using one product.
You can use an alt account. Or buy an old one for couple dollars, without security measures.
Yeah, Gemini 1.5 might be the most used (certainly by consumers) ai model. That is the one used by Google Search AI Answers.
False. Last information was that Google Search uses Gemini 3 Flash-lite. Gemini 1.5 is totally outdated and used nowhere.
Only available on Ultra, so I thought maybe I'll try it for one month to see if it's good. Well, I have Google One and for some reason this means I cannot sign up for AI Ultra? This is just one of the worst product rollouts I've ever seen
Image->html test for Argon. I'd put it at B tier, just under gpt 6.1 sol. It isn't A tier like Opus 5.5 or Astra.
Argon build: https://html.non.io/lcars-argon/
GPT 6.1 sol: https://html.non.io/lcars-gpt-6.1-sol/
Opus 5.5 (currently SOTA for this): https://html.non.io/lcars-opus-5.5/
Original designs: https://diffui.ai/app/canvas/5093e689-1e74-4f26-b632-2a4500f...
I don't know how you rate, but the Opus 5.5 is literally stuttering and lagging when scrolling on a Macbook Air M3 (for me).
(Chrome, also incognito)
Interesting! Do you have webgl disabled for incognito or for chrome in general? I wonder if it's trying to do software rendering.
FWIW, I have literally the exact same configuration (M3 air, google chrome). Here's a recording on mine where I'm seeing 60fps: https://non.io/video/video-3587018401.mp4
Maybe a Chrome bug? It's fine in Safari on an M1.
Mac Air M1 in Safari here, buttery smooth.
One interesting observation - it's the only one I've seen make an ASCII representation of the layout of each of the pages before building: https://image.non.io/46e0cce3-34d9-4fcb-8775-bfc2c70bc4bb.we...
Damn opus 5.5 is just such a good model - what kind of moloch sacrifices are they doing in there
I'm constantly blown away by it. I do think Astra is slightly more intelligent (although worse at this particular benchmark), but Opus 5.5 has become my default model for 99% of what I do. It's so good.
> built to solve your hardest problems
Like a working scroll on a webpage.
God damn this is impossible to use of safari mobile.
If you can't afford a Google engineer's high-end device that doesn't lag you don't matter to them :)
I have a high-end device and the page still behaves atrociously for me. The scrolling is totally out of whack
Clicked on "Performance". Scrolled to "Safety".
It literally can’t load in my mobile web browser without running out of memory and force-refreshing.
I have a more than 10 years old laptop, same feeling here!
Coding is solved, scrolling not yet :)
worked ok for me on ios safari, but probably because I have the “reduce motion” setting enabled. Every day that setting feels more justified
also a mess on desktop firefox
Of all models I like to write with Gemini the most. The language appears much more natural and elegant than what even Opus 5.5 or Astra do. Also, they give 5TB of cloud storage and discounts, it’s a steal for what you get.
The code is bad though.
Same, gemini for writing is the strongest. Qwen was also surprisingly good last time I did a shootout
Yeah rumor ahead of release was also that some Googlers were anxious about the known subpar coding given modern trends.
To everyone saying not available yet: it looks from the page like Pro and Ultra subscribers have it today and given the play links, it'll probably show up in the API in a few hours.
OpenRouter seems to have it: https://openrouter.ai/google/gemini-4-argon
In agy 1.3.4
> Gemini 4 Argon (Medium)
It looks like Pro can use argon in the Gemini app, but not in antigravity:
https://one.google.com/about/google-ai-plans/?icid=microsite...
Im on pro, installed gemini app, tried argon 4, keeps failing with 404, resource now found. Switched to gemini-3.8-flash in same session... worked.
It seems to be rolling out gradually.. not available at once to everyone.
It's available for me (pro sub) in Antigravity after I ran an update.
Antigravity tells me specifically "Gemini 4 Argon is currently available to Ultra plan subscribers"
Interesting, Google's own article says it's rolling out today to Ultra subscribers. No mention of Pro.
https://antigravity.google/blog/gemini-4-argon-in-google-ant...
Huh ? Am also on Pro sub but Antigravity IDE say's that it's only available for Ultra users.
EDIT:
Apparently Ultra subscribers gets Argon everywhere including in Google Antigravity
Pro subscribers only get them in Gemini app and Search
It's showing up in Antigravity on mobile UI as "unavailable" for Pro user (you can use Remote Control to access its web UI).
It doesn't show in the desktop UI.
I'm on Pro and it's trying to get me to upgrade to Ultra in the AI Studio UI when using Argon. In Antigravity CLI and the Gemini website it's not showing up at all despite being logged in
Pro subscriber in the EU (via Indian discount code) and don't have it.
Not available to me on gemini.google.com (Pro, USA).
Nope only Ultra. I went into Antigravity to check
As far as I can tell, I can successfully call Gemini 4 via API now.
Pro user here, don't see it in the Gemini app
Pro user - I can't see it.
I'm a Pro user and I do not have it. Seems scuffed.
I'm getting popup that I need to be Ultra :D
I'm Ultra and seeing it now (US)
> Seems scuffed.
I like this phrasing and am curious what new word or phrase we might use to describe big companies over pushing Ai internally at the expense of their product quality. This pattern is so ubiquitous across google cloud products I have taken to calling them the 21st century IBM, having been a IBM baby and witnessing first hand their fall from exceptionalism.
Not really this is how they always release stuff in batches so can roll back if something is wrong so people get access over time
Checked, and already in Antigravity (as the page said).
Gemini isn't perfect but I find it a lot less annoying than other model families (personal preference or stockholm syndrome). Interested to read the reviews.
Is it actually available?
Seconded. Anthropic feel the need to limit how abusive you can be to Claude because god, I want to throttle that snotty little shit. Google don't have to do that with Gemini because it just isn't as irritating. It does what I ask and doesn't try to second-guess my requests, or do "the quick fix".
I'm the opposite, I found Gemini so annoying that I had to stop using it, Claude was a breath of fresh air afterwards.
It's been really bad at lyrics but gotten better. You look something up and it will just make something up and it takes a bit of digging to discover it's because of copyright
It passed the test of using latest crystal-lang concurrent feature (with web search) when gemini 3.8 flash failed (again, with web search enabled). I think it is at least useful for coding, for sometime.
It still thinks Charlie Kirk is alive, though.
Right now there are two top posts one about GLM-Legal and one about Argon. Both score about the same in the Harvey Lab test, but the prices are $0.15 versus $10.55, the $0.15 is Vals AI's estimate using the base model since GLM-Legal has not published prices.
Yeah, but one is a general purpose model, and the other one is specifically geared towards the legal field. Considering Opus 5.5 only gets a 4% on the test compared to Gemini’s 20%, that’s actually pretty incredible.
"Argon autonomously turns raw video to a physics-accurate simulation, estimating depth and physical boundaries, and writing the exact simulation code required to replicate those real-world environments."
This is crazy if it can work accurately.
Geometry could be understood today with the model but what also really matters is the sensing, friction, mass how smooth the subjects are, dexterity is a very difficult problem so solve, not sure how simulations could handle such factors
Game engines have many of these features. At one point, iirc, Unity was positioning to be the simulation environment for world model agents before we spoke about it in these words.
Only available in the $100 plan, and it's worse than Opus 5.5? Why would you ever buy this?
> Only available in the $100 plan
Source? Their documentation says it's available on the $20 plan and I can confirm firsthand.
https://one.google.com/intl/en_us/about/google-ai-plans/
Not for coding (antigravity).
https://x.com/OfficialLoganK/status/2110038940841545827
Because you have an android phone and don’t want to mess around with anything else(i.e the same reason people just use internet explorer on windows) or you hold Google stock so you want to support your investment :)
I'm open to Google winning me over again, but there isn't enough hype in the world to do it. They will need to earn it after these disappointing 6 months.
I appreciate how it's being integrated into the Google ecosystem, but the user experience still feels unpolished when actually using the system. The marketing, however, is impressive.
I hope all the people very concerned about AI safety are cheering this one. The best guard against one AI swarm taking over is having multiple adversarial orgs all with similarly advanced AI swarms.
I wonder if any of the AI safety people will begin encouraging AI researchers to swap jobs to work at a failing lab in order to keep them all extremely competitive with one another.
I think we’re past the point where making the best model guarantees company success.
At this point a lot of stuff happens at the product level.
OpenAI and anthropic have distribution and data advantages now that are hard to replicate.
I have the Free AI Pro plan from when I purchased my Pixel 11 Pro. I still don't have Argon in AGY nor web nor Gemini app. I have a free product and still feel cheated.
I'm still waiting for a model which I can tell to "take these 3 screenshots and make a product launch video", where it doesn't randomly modify the screenshots or garble/rewrite the text as it moves the screenshots around the video.
Every time I try to generate one it's like a slot machine, where a different part will be messed up after fixing the first.
I dont have that problem with opus5.5. I dont even take the screenshots anymore (it does all screen recordings)... I tell it a simple narrative and it follows through. e.g. https://x.com/plannotator/status/2108589088139886769?s=20
I literally did that with Opus today and it did a great job.
I told it to strip screenshots from our old basic explainer video to generate modern 3D video.
Pretty happy with the result. I told it to keep the audio and match the video and it did.
Pretty sure Opus 5.5 can do this with code only now. Seen a lot of strong motion design skills floating around
I would love to try it out, and the Gemini models had previously been some of the ones I'd liked best in an overall sense, but unfortunately their updated inference endpoint is no longer ZDR on OpenRouter (change takes effect on the 18th). I think this is one of the more unfortunate turns in AI development in 2026, as it seemed like labs offering it was becoming the norm last year, and Google had previously done so consistently.
Excited to finally see this out. When new big models come out I like to make them audit any other code I generated to see how things are.
I'm pro and I see that I need to be Ultra to use Argon :/
Honestly not even that is enough, I exhausted the limit within 5 minutes of trying to use it. I'll stick to Flash I think.
Really? Reminds me of Gemini 3. It took so long before being able to get a subscription for it, and then I would run out of their equivalent of the 5-hour usage window in a heartbeat.
Digging in the model card, "Gemini 4 is the next generation in the Gemini series, a suite of highly-capable, natively multimodal, reasoning models. Gemini 4 Argon delivers frontier performance..." If they're using noble gases there are 2 obvious lighter naming candidates and 3 larger (or 4 if you count Oganesson). I'm not seeing them claiming Argon is the top/biggest/bestest model among the Gemini 4 series, just that it has frontier performance.
Edit: noble gases not noble taxes, though the latter would be a funnier naming scheme.
> Across Google and our early access partners, Argon is quickly becoming the preferred model for engineers tackling long-horizon software engineering.
"Across Google", wonder how much that is because of an effort by Google to curb usage of Anthropic models. MSFT and Meta similarly are pushing for in-house models. It doesn't mean the models are better when you push your engineers to dogfood and restrict access to other LLMs.
(Not that I know for sure this is the case, it's more of a hypothesis).
I mean, don't we want engineers to dogfood their own product so that they know it's strengths and weaknesses and be in position to improve it?
Pricing is interesting. At launch Argon is the same price as GPT-6 Sol, half the price of Claude Opus 5.5 and 1/5th the price of Claude Fable 5.1 and GPT-6 Astra.
But on December 1st Argon doubles in price - so it will be the same price as Opus 5.5.
Here I was thinking Google had given up on frontier models...
All AI pages now include canvas animations. There isn't a single page without one. This is neither good nor bad, just an observation. I think OpenAI has the best animations so far, with Anthropic in second place. This one looks a bit off.
Here's how the 1M token output (long decode) works: https://ai.google.dev/gemini-api/docs/long-decode
can't use in agy with Pro subscription:
Unavailable · Gemini 4 Argon is currently available to Ultra plan subscribers.
When you are on "gemini.google.com" there is a little lock icon next to Gemini 3 and Gemini 4. When you click it you get the prompt to subscribe (pay). Once you have successfully done so, the Gemini 4 option disappears!
Pretty misleading.
Maybe it’s just an incomplete rollout not fraud lol
don't attribute to malice that which can equally be due to incompetence
Curious what this means for the anthropic ipo. Between this and meta, seems like there’s zero moat
Gemini 3.1 is still SOTA at weird handwriting recognition. If Argon is better it will be amazing
I bet that's one of those areas where a finetune of a 4B model will wipe the floor with anything else.
Google's models are by far the best for readable text but also for nuanced understanding of the input.
Even after using the free model in Google search talking to Anthropic or OpenAI models feels like talking to an ox.
It seems the model is a little verbose, using more tokens to complete a task that other similar models as stated more precisely in the linked article about Artificial Analysis's work on this model.
not available yet. why to launch something that can never be used even after weeks
Page says it is now available to Ultra and Pro subscribers. Although in AI Studio it currently says you need Ultra. But it's technically released either way.
Because Mythos started an arms-race for "models that are too smart for you" and other labs are jealous.
Yeah, I think this is it! Google tried to hype their model but they have no chance against queen Dario. Perhaps they should just delete Gmail and say that Gemini did it. That way they would get some attention …
not available. classic google.
The slides they show in the catering video demo are so bad (in addition to having all the usual signs of AI design), I would genuinely laugh if a supplier sent me this.
I laughed for the first half of the greatest "slop deck" their cloud sales team sent us, until other emotions like disappointment took over.
Finally, gemini 3.8 has been falling way behind opus and sol in my experience. It's showing up in my antigravity. My work account though still has 3.6, 3.8, 3.1.
The previous gen pro used to have this weird tick where if you ask it a reasonably open ended question, it would number things in the response and assign each one some weird label starting with 'The...'
e.g.
The first two prompts I tested in AI Studio does the same, unfortunately.Another quite unpleasant tick was that if you specify a peculiar term in your prompt as a conversational shortcut, it would keep repeating that throughout the conversation. e.g. "...What is a bitter lesson pilled way to do this?", then you'd see it will quote "bitter lesson pilled" at like 7 places in its response. This one seems to have eased off a little.
I feel like this is a quirk of LMSYS-style optimization where humans in SxS preference evals will pick based on formatting more than actual content.
It's available for gemini pro users.
Glad to see them abandon the absolute train wreck of their model naming "strategy".
Gemini Live vs. Gemini Lite vs. Gemini Pro vs. Gemini Flash vs. Nano Banana Lite (that one apparently has no Gemini prefix?)... Please, just pick a metaphor and stick with it without wildly mixing reasoning modes, model sizes, and input/output modalities in the model name!
Not available in my browser/app yet (pro), but I'm confident it'll roll out soon.
That was a long wait.
Can’t even get it on pro tier in antigravity?
What’s the point lol. I’m not subbing to ultra to have it
Just switched, and it seems to be much better. I'm hoping it stops lying. I have Pro max subs for Claude, Codex, Grok, and Google AI... I was forced into using Google by running out of credits elsewhere.
I have found that Gemini often claims it did something, but didn't actually complete the task.
I just had 3.8 Flash do a major refactor -- that never should have needed to be done in the first place -- and switched to Argon 4, and had it review it.
Almost everything had been stubbed out by 3.8, and none of it was properly integrated into the codebase. :facepalm:
>"Argon helped the Chrome team separate core webpage handling from a ~6 million line legacy folder of browser features held within Chromium’s 40 million-line codebase. Piping multiple clean code changes at a time, Argon helped the team finish an architectural refactor across hundreds of files, completing months of work in weeks."
Now that needed to be done for some time!
Regardless of if an AI helped or not, regardless of whether that was Google's or some other AI or not (though Argon does sound pretty cool!), regardless of whether an AI cut the time from months to weeks or not, refactoring Chrome -- decomposing it into individual clearly demarcated non-tightly coupled software parts with core webpage handling at the center -- that needed to be done for some time!
Now all of that being said, Gemini 4 Argon does sound pretty cool, and we congratulate both teams at Google -- the team that created Gemini 4 Argon, and the team that refactored Chrome with it!
But all of that being said, while the advancements made for Gemini 4 Argon are important and impressive (and they are!)... refactoring Chrome properly (IMHO) is slightly more important and impressive!
(You know, one man's humble opinion! That is, I am not the guy who is responsible for awarding performance-based bonuses at Google at the end of the year, so take everything I say with the proverbial "grain of salt"! :-) I am probably biased, one way or the other! :-) )
at this rate we're gonna run out of stupid names for these things
"Hans, we're doing a new point release! What isn't crossed out on the periodic table yet?"
Not available for me on Pro in the UK.
As with the previous Gemini releases, rollout is batched.
Unlike most people it seems, I like all the progress in AI but I really do wish they would reign in the over-the-top marketing. I mean, can we just get this stuff to the point where I can reliably work with it to design a well-architected software project without having to constantly struggle against bloat and nonsense directions no experienced or sane developer would ever take.
fyi: argon is an interesting choice of name, coming from the Greek argos meaning "lazy or living w/out labor".
I think in this case it’s from the element argon
Yes probably but that's still from Greek (see: https://en.wikipedia.org/wiki/Argon#History)
Why can't the Gemini model code? I'd guess Gemini probably used the best dataset. Even Google is the place where the world's best coders are gathered, isn't it? Yet the GPT code from OpenAI, which collaborated with Microsoft, is beautiful, while the Gemini model's coding is trash.
I have no problem with Gemini Flash 3.8 for coding, though I only use Elixir which is one of the better languages for LLMs. Where does it fall short elsewhere?
[dead]
why they wait so long ?????
Are they gonna go to Krypton or Neon next? Kinda started in the middle there.
Meanwhile I paid a subscription and they only provide access to 3.6 Flash. Consequently, I have moved on to open-weight models. It is what happens when you screw your customers over.
You can probably get 3.8 via ai studio interface.
God knows why the server different things to the same people on different interfaces
3.1 Gemini was amazing for anything non-agentic. I switched to it for my primary chatbot purposes.
I never touch 3.8, it's awful. Curious to try this, but I've yet to use any Gemini that is good for agentic work.
Also, unrelated but this made me look at Artificial Analysis to see where it lands on Cost per Task, and it ranks Opus 5.5 at almost double Astra 6. That does not align with my experience.
not available in our org (EU)
People use Antigravity still?
It's alright.
Fable 5.5 about to drop the hammer on them and then another Astra version goes out lol.
And so the game of cat and mouse goes on. Your point being?
so this is prob why antigravity is not working well at the moment lol
Blatantly obvious that models will not slow down. For any task, if we can build a dataset or environment of verifications, it can be automated. This will eat all of human tasks.
The fact that multiple companies can breach any benchmark means there is no ambiguity in terms of how to build AI, its just a matter of time.
Certainly Meta and Google are benchmaxxing to please public markets, but it still proves the above
but they have slowed down. the delta in capability between model releases now is smaller than ever before and i expect it to continue to diminish until we reach some kind of asymptotic maximum, which hopefully will mean turn towards optimization and local model improvements. The current frontier models running at 10x the tok/s would be incredible for my use cases
The delta isn't really slower, we're just unlocking capability bundles in batches. The capabilities bundles you care about might have mostly been unlocked already, and just need refinement, which is coming slower than the initial capabilities came, but don't discount the breadth of advances being made across all possible tasks.
We're not living in the same universe. Look at capability year-over-year. The asymptotic maximum is wishful thinking. Someone needs to step in before this gets wildly out of control.
A little over a year ago they were winning their first gold in an elite high school math competition. Have you not seen what they are doing in math now?
opus 5.5 is a step change, you arent using the models enough if you think there is a slowdown
Models just solved a Millennium price and you think their progress is slowing down? I will concede that their capabilities vary a lot from one field to another, but still.