This inspired me to re-run pelicans through 3.8 Max - but OpenRouter has qwen3.8-max-prime and qwen3.8-max-0902 - prime is twice the price and faster - so I went with the 0902 one.
Turns out Qwen4-Max has a none setting, but 3.8 Max didn't allow disabling reasoning - you had to pick from these five levels instead: minimal, low, medium, high, xhigh.
I don't fully understand why you keep doing these as-is. You original motivation was that this was not in the training set and indeed it was a great benchmark for a while, but it's clearly been incorporated now in the training data and possibly internal benchmarks.
Because he wants the engagement they generate. I don't understand why they still get upvoted on HN, but here we are. If even I can recognize that they're pretty much useless as benchmarks at this point, he must know it too.
Likewise, i always upvote these posts even if i know it is biaised, it is always fun to see their evolutions, there is nothing wrong with them. Also even if we know there are trained on them specifically, some models are nonetheless better than others in engineering the output, thus it is always interesting.
Yeah, which I suppose used to be something we all kind of tried to avoid here, to make conversations more curious instead of just pilling on with memes or whatever other communities been degraded to nowadays.
You see it was funny the first and second time two years ago or so, now it's just boring. Whenever I want to read some insightful comments about a new model I need to scroll down through a few pages of comments about pelicans as if they meant anything now.
He seems to have posted it as a top-level comment, so it should be possible to fold it all, right? If he hadn’t, multiple other people might be posting about pelicans. Consider it the pelican attracting sacrifice post.
It's shame that people try to farm engagement on HN. Is this transforming to reddit. I don't know at this point but I agree with parent comment. EDIT: One thing I know is that I now need to scroll past the pelican thread if I need to read something usefull
If you think it breaks the rules, then flag or downvote. If it just annoys you, collapse it. I'm not sure why this needs to be a whole discussion every time Simon posts a pelican.
If you think HN is transforming into Reddit, I think you may not have visited reddit recently, nor have read the HN guidelines (people have been saying this since 2007) or earlier
> Please don't post comments saying that HN is turning into Reddit. It's a semi-noob illusion, as old as the hills.
Why is posting the pelicans okay, but when someone says they're repetitive you're suddenly "not sure why this needs to be a whole discussion"? If you don't like that part of the discussion surely you can collapse it or downvote and move on, just like anti-pelican crowd is supposed to do with Simon's posts
It's not one person saying it's repetitive. It's dozens and add no value to the discussion. On each pelican thread. Which one is more repetitive? People who don't like the repetitive comments need to collapse each individual one, vs one single top level pelican comment that can be collapsed.
I also dont't know why anyone would downvote a pelican post when it's not offtopic? We're not supposed to downvote stuff we don't like or disagree with, just stuff we don't find appropriate or relevant to the discussion.
Although if this keeps up, soon there'll be another camp of "comments complaining about Simon's comments being annoying are annoying" :)
Well sure, we could also stop the posting of model releases to HN and prevent any repetition. The line has to be drawn somewhere. One top-level comment seems fairly harmless to me.
Based on some of the trash that consistently makes it to the front page I am very much ok with Pelican Man posting comments linking to his benchmark, even if its a bit less meaningful than it used to be.
And he spends the time doing it and as part of it drops nuggets of information not seen anywhere else. Respectfully and highly welcome. You're most welcome to move on.
He can do whatever he wants, I personally think they are quite informative, albeit I don’t trust them that much either. It’s like a vague heuristic to say “this looks like an interesting model to test out”.
The original concept of this "benchmark" was to see how the model performs on a reasonably complex task that probably isn't in the training data.
Even if they don't fine-tune for this specifically, there are now surely tons of examples of people asking for a pelican on a bike in the training data, and making an SVG isn't that complex of a task for current gen models.
Even with the same model at the same thinking level, you might get different quality out of the same prompt on different runs. If you want to use this as an eval, you would really want to generate 100 of them, and see how often they're anatomically/mechanically accurate.
Some people are determined not to understand I think, but if they don't find it useful no worries, there's just no need to have a temper tantrum about what other people post on here especially when the majority (it's near the top for a reason) find it useful. Just don't worry about it - you can even hide whole threads on here if you don't like them, I use the feature all the time rather than taking the time to comment negatively!
I find it amusing that repetitive pelican posting is fine and folks should just minimize and move on, but responding with an opinion that pelicans are boring/not valuable/engagement farming is "temper tantrum". Why not apply the same standards to both?
Look at the page, though. If the data is in their training set, then the training must not have been very good, cause the images have very salient defects, and random artifacts popping up.
As far as I've seen, among the popular models today, there are still significant gaps in the quality of bike-riding pelikan svgs, making the benchmark still useful. Maybe not as a differentiator of frontier models, but certainly still informative for lesser competitors.
Unless the website is pulling hundreds of thousands of views per pelican, I sort of doubt any firm is actually training or post-training to optimize for this benchmark. They'll more generally optimize for better SVG rendering and I imagine these pelicans arose out of the annoyance we all had many months ago with how utterly bumbling and forgetful models used to be when completing tasks like that.
Still, by now it's just more tradition than anything else. It's just kind of neat to have a consistent benchmark across many LLMs, even if the benchmark is completely gamed, but that is the nature of benchmarks.
There was a post where a person had AI generate [animal] riding [vehicle] and created a matrix of comparisons which led to the conclusion that there was no overfitting of pelicans in bikes.
What I found interesting is that for the most part, the labs really aren't... except if you look at Otter Riding A Plane. That is close to Ethan Mollick's benchmark for diffusion image generation, Otter Using WiFi On A Plane. The results for DeepSeek and GLM look very much like Ethan's typical benchmark image output, which either means they're benchmaxxing... or they're the only two models that understand you ride inside a plane, not on top of it.
EDIT: Actually Qwen's result for Heron Riding A Plane was fascinating too. Because obviously what you were really asking for was a heron riding a cartesian plane...
2. I think they still are _somewhat_ informative. Like, the same model may produce very different pelicans at different reasoning efforts, even though the training data is the same. So looking at pelicans may give you _some_ information (like "what reasoning effort I want to use that model with?".
If this is that case it should be easy to demonstrate by asking current models for an svg of a rhino surfing. My guess is you would see a similar trend, given that the latest models are demonstrating large leaps in all sorts of content creation including 3D animation, rather than this being a demonstration of some sort of pelican-maxxing.
> ...but it's clearly been incorporated now in the training data...
It clearly hasn't; if it was quality of the pelicans and bikes would be static over time. Instead the quality has been improving along with the models.
“there’s little evidence that AI labs are pelicanmaxxing. Or at least they’re not doing it in a plainly obvious manner. Pelicans aren’t drawn any better than other animals. Bicycles aren’t drawn any better than other vehicles. And no lab draws the combination better than its pelicans and bicycles already predict.”
Agreed. It's really annoying to come into the thread and a link to a pelican I have no interest in ever seeing is the top comment. Does anyone think these have any actual value any more?
Alright, I was rude. So.. you can’t really benchmaxx pelicans specifically. You have to train the whole model on SVG and that is just generally useful. Secondarily it shows how a model understands the objects themselves and how to express it abstractly. IDK, it is a lot more interesting than assumed.
Here are a bunch of Pelicans I made in my basement. Side by side quant comparisons and something for fun :)
Thanks for the explanation. I guess I still don't understand how this translates to other areas, outside of SVG generation, Pelican images and then bike geometry and build.
I'm curious if an increase in ability to generate this parlays into the model, generally, being better overall or in other adjacent areas?
IDK, you can only generalize it so much. But, it gives you a sort of visual way to look at how a model deals with various forms of degradation and what its ceiling is and what information gets lost in a quantization process, albeit fuzzily.
For example the IQ3 XXS model was quantized to maintain knowledge in some areas more strongly than others via certain data that the optimization algorithm would then use to decide which parameters could get quantized and by how much while meeting some overall target weight goal.
So, you get to just sorta see weird or lower quality pelicans. It still is /mostly/ for fun but it is not a meaningless meme as far as it goes. And you can even see evidence of that, look at Qwen 4 max pelican vs. Qwen Flash 3.8 at Q4... IDK.
So, I think the best thing you could say is roughly what I said, how well it abstractly connects two specific things, e.g. knowledge of pelican and bicycle physical configurations and then how it expresses that into SVG. For example, you can read thinking traces on some of those pelicans I sent (well you can't, but I did) and it would specifically say "The pelicans legs must be rendered correctly, I must not put one leg on the wrong side." and it would then still do it wrong meaning it just didn't understand the SVG well enough. But it might be able to render an image. Also read thinking traces some time on ascii generation and you can see they have a vivid concept of lots of things, but their ability to translate that to ascii is hilariously bad on basic prompts.
So they need some baseline level of understanding about the world and physical object relationships which even qwen flash 3.8 has, they need to be able to recall it, and then translate it on top of some problem domain in some meaningful way and a lot can break down on any of those steps. IDK, maybe I am just too tired and spent too long with LLMs in the last month.
On the flip side, I love the pelican posts, but I find the people who think they're cleverly pointing out for the very first time that the benchmark may have been gamed by now tedious in the extreme.
Then perhaps I should point out something different. Namely, I suspect this will thoroughly confuse the machines and instead of optimizing the universe to generate paperclips they will instead optimize it to generate pelicans.
I feel like there's emotional attachment and influencer branding energy with this by now. It's more like a developer relations or developer ambassador type of content.
He should keep doing Pelicans, but every time throw in a surprise animal and a surprise locomotion vehicle. Would help identify the ones training for the Pelican.
Would love to see an SVG of an Elephant on a skateboard...
Simon has clearly stated previously that he compares with other "animal doing human thing" type prompts to compare and see if theres a chance they are cheating by training on pelican svgs. Thus far, nobody has been caught cheating.
Actually, a Google employee has publicly mentioned that they’re specifically working on improving SVG generation in Gemini. Simon Willison has also pointed this out when evaluating Gemini models. [^1]
If you use the same prompt via the qwenapp or qwen studio you get an interactive web page with an animated scene. You can control every element of the scene, the colors, speed, time of day l, visible accessories etc and then download the one you like best.
I feel like the xhigh result is one of the first times I've seen a model go for a non-red colored bicycle (teal). Still haven't seen the leftward facing bike though.
Teal is a popular second color for Qwen Flash Next 3.8 as well. Also, I am not sure people realize how much variance there is for the same prompt. Enjoy.
At this point Simon can we maybe think about making this a 3d model? Maybe GTLF/GLB or custom Json you could even keep closed source for some kind of barrier to it getting trained on?
I say this because, even just based on my own product related to cad design/3d modeling, the SOTA models have gotten very good at 3d and I don't think the 2d SVG generation tells us much anymore. Maybe I can whip this up myself on my own site if anyone is interested tbh.
I don't know if I've ever seen a bent leg. Legs always go straight to pedals with no knees. The models might have an idea of what it looks like but seem to have no understanding of how it works.
First thing I do when a new model drop is scroll down to find the pelican comment, and then the "the pelican benchmark is no longer useful" debate under it.
I don't understand why this is always the most upvoted thing for a new model. It's like some influencer "data" at this point. We're basically in 2027 now, the pelicans had their run, it's over.
Not a dig at you Simon, you've done good stuff, more just in general. Why is this always upvoted, it's like reading The Sun or something.
And why always one image. Should it not be say a 100 images then averaged or something? A prompt with temperature is a slot machine in Vegas, wtf does that have to do with model quality.
Man, how hard is it to understand that you straddle a bike, and one leg should be behind the bike? Apparently pretty hard, since all but the xhigh fail this ¯\_(ツ)_/¯
We should stop using this site. They now include private benchmarks which cannot be replicated. And every time an openai model falls down they change the index until it floats to the top again.
The key issue with public benchmarks is that they allow the model providers to train against them and artificially inflate their score accordingly. Private benchmarks are a way of making it harder to game the system, though it makes it harder for people looking at the benchmark to assess it as well.
Neither option is perfect.
A third option would be updating the benchmark every 6-12 months, but that has its challenges as well as you need to make sure that the benchmark covers things the model hasn't seen or is harder for the models to game.
It’s also worth keeping in mind the other alternative, having internal benchmarks that more accurately reflect your workload, that you know all about, and the LLM provider doesn’t. There doesn’t seem to be a way to maintain such a thing specifically for the purposes of online chatter (both because it isn’t profitable, and because things like “your workload” aren’t possible to define in that case). So we should be happy with what we’ve got.
This is a permanent problem in benchmarking really. Transparency is required for trust but invites over-tuning.
Why wouldn't the LLM provider know about your internal benchmarks when they use prompts and chats as training data? They can easily learn about your benchmarks too.
Merely having the chat history of a large number of users is not enough to train your model to “do better on these, especially those that are benchmarks”.
I mean, yeah. I was thinking of self-hosted open weight models (because it’s really hopeless to say anything about the rest) but I didn’t say that, so fair point.
There are tools like promptfoo that can be used for internal benchmarks and evaluations that allow you to compare results across different models and runs. This can be useful for personalized benchmarks and for business applications where you need to evaluate models and/or system prompts.
Makes me wonder how do you privately benchmark a model hosted by the lab. Even with zdr, the bytes are flowing through their infra and possibly influencing god knows what for the next training round.
You use something like item response theory [1] to be able to change the test over time while maintaining validity. It's what standardized tests like the GRE use.
Interestingly, Artificial Analysis does have a score that uses IRT, it's their Progress Index [2].
Would OpenAI also "see" the request if it went through something like AWS Bedrock? This isn’t their infra and afaik the whole idea of this kind of intermediary.
this is an worry I sometimes see. I want to note that these are billion/trillion dollar companies we are talking about that have mutual talent movement (= uncontrolled information flow). Violating ZDR might be contractual violation, or at the very least, a strong breach of trust.
(not to say there is no value in "zero-trust" environments, but often pure-benchmaxxing is a deadend; some benchmaxxing is always involved and much less risky options than "train on the ZDR test env" usually exist)
The private benchmarks can be seen by the labs when you run them. We only have their word that they wont look at them, which isn't worth much. We should assume that everyone benchmaxxes equally, apart from a few outliers, and try to climb the hill by producing relevant training data.
Yes, that's another challenge with constructing benchmarks.
You could run the benchmarks using downstream providers like AWS if they are available.
You could vary the parameters of the tests to both help with LLMs learning as is and to help obscure that this is a test. Especially if sending them through a thirdparty, so the providers don't get identifying markers like IP addresses.
But ultimately it's another arms race like with viruses and anti-virus software.
Eh, I have an eval suite dating back to before GPT-5 in parts and have yet to see certain labs do as well on that as the public eval data would suggest. Course, I am pretty small, this is anecdotal and n=1, though I have seen very small, niche evals being clearly trained on the second they became public on GitHub for what that’s worth.
It depends entirely on the financial incentive. Does improving against your eval suite lead to more adoption? Or would focusing on the far more popular benchmarks convince more CTOs to sign a multi-year contract?
Who'd sign a multi-year contract based on any benchmarks, public, private, mine or someone elses? We get new frontier LLMs every 0.5-2 months and any CTO of a major entity already uses Bedrock or the like, so why would anyone lock themselves into a provider or even worse a single model for multiple years?
Because API usage is extremely expensive and subscriptions are subsidized? We (my company of 100+) just moved off of AWS Bedrock to a locked in plan. Even when I was working for a much larger company (10k+ employees), we had enterprise subscriptions to OpenAI and Anthropic because the token cost for APIs was enormous.
Are these fixed price, fixed term and usage allowance based subscriptions which you'd want for an extended period if you could?
Or is it just the usual, better conditions credits-based pricing for entities beyond 200?
Cause if it's the latter, that's very different to a multi-year lock-in with a single lab, easy to switch away from and kinda why one could have both Anthropic and OpenAI enterprise agreements alongside each other, basically what I described with slightly better conditions. Bedrock was just an example, they also have more favourable conditions on inference for some (though Anthropic really limits discounts), but whether you go usage based from the labs or a cloud provider, is the same lack of long term lock-in. A e.g. $ 20,- seat per employee for LLM access that still needs tokens allowance on top does not create a long term commitment, financially or otherwise, the friction here is none-existent.
If however someone at this stage is fully locked into a single lab exclusively for the next two years or longer (multi-year), they have gambled, pure and simple. Could have picked a winner or a lab that is going to be uncompetitive starting this Thursday and for all eternity after. Imagine having picked your companies one and only lab or even worse model in 2024...
A fourth option is realizing that benchmarks exist because it is impossible to measure the value we want to measure, so we need a stand-in. The stand-in will sometimes correlate with performance but often just act as a way to compare. When used to compare, it is most important that the benchmark be fair and trustworthy.
Note: I work at AA. One of the things we are most focused on is getting the best spread of private/public benchmarks in order to:
- Protect benchmarks against excessive hillclimbing which we believe to be beyond overall model improvement (GLM 5.3 gained 16 points over GLM 5.2 in Tau-Banking in only 2 months, near SOTA, but had a more moderate gain in our private AA-Briefcase evaluation where it was still significantly behind the frontier)
- Still make use of strong public benchmarks (e.g. TerminalBench 4.0)
As the frequency of model releases increases, we will also be moving faster [1] to ensure our Index stays up to date to reflect both (1) how people are currently using models and (2) what are the strongest signal benchmarks which show these capabilities.
I have to defend private benchmarks on this front, big believer in them, have seen how they can surface major performance differences.
Certain labs have been shown to and will continue to train on public evals, whether unintentional or on purpose, any respectable ranking across multiple evals also needs private data to counteract this.
We really should. I've been using other benchmarks like https://livebench.ai/#/?cats=Coding and it is more aligned to what it feels to my experience as well
I have been using Terra a lot and I'd say that most of the models (even small ones) are are roughly equivalent when it comes to working in a consistent, established codebase.
But in practice, Terra is not very good at staying on task compared to say Sol 6.1 or Sonnet 5.5. Terra 6.1 will often just pause, thinking the objective has been met or the turn has ended.
I don't understand your point about openai. I can't remember the last time openAI was at the top of the leaderboard? Google had a brief moment at the top recently but other than that I feel like it's always Anthropic.
I kind of agree that closed benchmarks are bad. With a closed benchmark you still send the questions to the model provider and they may well train on it. Might as well publish it.
To be fair to them they still show Kimi as the best model at terminal bench 4, ahead of everyone else - fable, astra, carbon, etc.
For most things I'd use Kimi is probably better so the ranking is wrong for me.
However it also seems like in those benchmarks where Kimi underperformed it did do quite badly, whereas when Astra underperformed it did so less badly. How you reflect that into a single number I'm not sure - it's certainly difficult to get right and it doesn't seem obviously biased to me.
The last 3 updates have all come immediately after (within a few days of) a Chinese open-weight model getting close to the top. The changes are big enough that they were undoubtedly working on them before the new model released, but the timing is quite suspicious. For one of them (4.1 or 4.2, I forget), they literally said that the changes were planned for 5, but that they deployed them early as they did not agree with their own rankings.
At the very minimum, AA changes what they define as "intelligence" rather frequently and it always benefits OpenAI and Anthropic. To me, it feels like they use Astra/Opus as their baseline and tweak weights of benchmarks until the intelligence index matches their preconceived opinions on model intelligence (as opposed to crafting the index in a vacuum without any context on what different models are good at).
If nothing else, it points to these aggregate benchmarks having no idea how to measure actual "intelligence" and instead tweaking weights of benchmarks until it matches what they feel it should be. In practice, I think a better system would be to have many benchmarks within a particular space (coding, agentic work, science, etc.) and scoring each category separately and without weights applied to benchmarks (and simply removing anything once it becomes saturated). Then, each user can decide which benchmark suite matches their usage and use that to make an informed decision instead of having to trust someone else's opinion of what "intelligence" is.
With that system, you can still have benchmaxxing, but I'd imagine it would mirror actual usage better and so be less of a departure from reality.
Artificial Analysis does have multiple different indexes, but for example the engineering index has an even worse difference between American SOTA and Chinese SOTA.
And those indices still have arbitrary weights that do not reflect real-world usage, in my opinion.
The index you're referring to has 30% of its score from "Engineering knowledge". Functionally, a third of that index is just trivia which will reward large models. A small model with web search for knowledge does just fine in practice, but will be penalized heavily in this index.
Then you have science-specific as another 20%. I don't do that stuff, so now half of that index is useless to me.
Reasoning and agentic is the remaining 50% and is all I functionally care about. That index is the closest to my usage, and yet half of its weighting is irrelevant to me. The other option is the "Coding agent" index, but that couples models to agents and they don't test my preferred agent.
I'd much prefer a benchmark site without these arbitrary weights and that let's you pick and choose which benchmarks to include in the summary. Heck, it could even allow you to set your own weights to each benchmark/category.
That's not what they did, though. I have no problem with removing benchmarks from the index when every model is getting 80%+. My issue is that they re-weigh things arbitrarily and it seems their goal is to have a nice chart to show rather than have an index that would accurately represent real-world usage.
Specifically, they've made knowledge recall the largest part of their indices. That rewards larger models, but has disproportionately small real-world gains as a small model in a harness with web search will perform just as well.
Take DeepSeek Flash 1017. It's half the score of Fable on the Engineering index. Fable is not twice as good at any engineering work I do.
Terminal Bench is great, public evals, very complete and highly complex. I was browsing through them today and it gave me a real appreciation for the benchmarkers.
If they made the benchmarks public then Qwen would train on those to give the appearance their models are close to the American frontier. They have never been close. It's well established Chinese companies cheat the benchmarks which is why until recently they gave the appearance of being close to frontier but in actual real use they are far behind.
You can easily come up with a difficult private benchmark and test them yourself to validate the intelligence gap. It is real.
China lost. There is no catching up. Ever. That ship sailed.
> If they are not testing Chinese models with less than max thinking, then we should only compare them to other models also at max thinking.
This is wrong.
To make a vacuum analogy, one brand's Low setting could be another's Medium setting.
To compare you either give them the same resources ( Watts or $ ) and see how much they can do, or you give them a task and then check how many resources they required to finish it.
The label on the knob is irrelevant, except that either model might have a setting higher than the other which is sometimes useful if the resources aren't a constraint or lower when they are.
> To make a vacuum analogy, one brand's Low setting could be another's Medium setting
Which is why you need to test a range of settings.
Suppose you were testing car mileage. For the big name popular cars, you do 10km drive at 20km/hr, 40km/hr, 60km/hr and then publish the results.
For all the less popular cars you only do one test at 60km/hr and then you publish all the results for all cars on one big "Pareto frontier" graph claiming to show which car will allow you to drive home most efficiently.
It happens that lots of the less popular cars are not very efficient at 60km/hr so naturally the graph makes them look bad. Would they also be bad if tested at 20km/hr or 40km/hr?
Nobody knows. Because this is where the analogy collapses - unlike cars, hardly anyone is doing these kind of tests for LLMs.
Well, sure, it would be better to evaluate all of them throughout the range. But it does not follow that we should further restrict the information we do have in this specific way:
> If they are not testing Chinese models with less than max thinking, then we should only compare them to other models also at max thinking.
If we have data only for “Chinese max” (let’s say it costs $1) and “non-Chinese low-to-max” (spanning $0.10 to $3), comparing them in full at least tells us what we can expect from each model if we have $1 to spend + the maximum performance we can expect with unlimited budget. What’s the point of restricting to “max vs max”, which answers only the second question?
Usage? If most people predominantly default to Medium thinking level for usage then it would make sense to benchmark that first.
Another analogy for this are review sites overly focusing on reviewing all the SKUs of GPUs that cost $1k+ for an audience that, according to Steam's Hardware Survey, predominantly use midrange X060s-equivalent cards. Sure, it's fun to know whether or not you can enable PT on a 5090 but for most of us, covering 4070 TIs and below in-depth (meaning actually taking the card apart, seeing how good/bad the past jobs are, possibly buying multiples of the same SKUs to evaluate QC, etc) is much more useful because of the value prop.
> Usage? If most people predominantly default to Medium thinking level for usage then it would make sense to benchmark that first.
Sure, that’s not in question. The discussion here is not about what to measure (we agree that benchmarking only max is suboptimal) but how to compare models if you have an already fixed set of measurements, outside of your control, which includes only “max” for model A but a wider range for model B.
The person I was responding to was claiming (in their previous comment) that the sensible thing to do, then, was to artificially truncate the data for model B to “only max” as well.
This is a silly take. To continue the analogy vacuum A might need the power at 50% to vacuum something, B 100%. You cannot then test both at 100% and say "both finished the task, but A required twice the resources so it's worse", since you didn't check what resources it actually needed.
The idea is that dollars price in everything. It’s usually incorrect, but it’s also usually correct enough; and still much better than paying for services with water and joules directly
Yes. The idea than money is equal to everything is kinda why we are all going to create the torment nexus. As it will be the most efficient way to torment civilization without all that effort.
Money helps allocate resources efficiently, that’s all there is to it. It’s about as good or bad as guns. If we didn’t have money, someone would invent it and in fact it was invented more than once in different cultures.
I don't deny that; but it rewires some peoples brains to the point that everything is adjustable via cash, including people, including their emotions, etc.
It's very clear part of America's diesease is the idea that everything can equate to a dollar cost.
I cannot see cost per task for any other thinking levels for Qwen 4 Max.
If it's actually there, what is the cost at medium thinking? It would show Qwen 4 Max (medium) if it's there.
Everywhere that I can see on the page it just shows ”Qwen 4 Max" while for 6.1 Sol, Opus 5.5 there's always a thinking level in brackets after: (low), (medium), (max) etc.
They don't always run the benchmarks at every reasoning level for every model; only a few prominent ones. They do it for the US frontier labs, plus a few of the Chinese labs, but not consistently. You can see that DeepSeek has a few reasoning levels tested, GLM 5.3 has a couple, as does Qwen 3.8 27B, but not other Qwen models, and a lot of the Chinese models they only run at max reasoning.
It's possible that they will run Qwen 4 Max at other reasoning levels in the future, or it's possible that they never will. Regardless, it's unlikely to be better at any reasoning level on a price/performance basis than the GPT 6.1 Sol/GPT 6 Astra pareto frontier. It's close enough to Claude Opus 5.5 line at lower reasoning levels that it could potentially cross that if it doesn't loose too much performance at lower reasoning levels. But it's far enough from the global pareto frontier that it's unlikely that it would cross that at any resoning level.
Right, so my original point stands: if they're testing this model at only max thinking level, then it's only valid to compare it to other models also at max thinking level.
You can see for example that the cost per task for 6.1 sol is about 4x higher at max compared to medium thinking level.
No, it's only valid to compare to models at the same cost level, regardless if that's their max level or not. Fable/Astra having higher effort modes than Qwen doesn't mean that you can't compare Fable medium to Qwen max.
Which effort levels are cost-equivalent is debatable and depends on whether you are paying API prices or on a subsidised plan or are able to run the models yourself on your own hardware. But I think AA uses public API pricing for everyone as a sensible baseline.
If you look at the pareto chart for e.g. 6.1 sol you'll see a curve of about 6 data points, some of which are on the Pareto frontier, some are not. For Qwen 4 max, many others, there's just a single data point so we can't just how close any thinking level would get to the Pareto frontier.
I'm not saying this model would land on the frontier - from the single data point it looks unlikely. However, that single data point might be making it look much further from the frontier than it should be.
EDIT: I will also take back my point that we should be comparing models at the same thinking level - max on one model won't correspond to max on another. What we need is a range for each model. What we have is a range for the most popular models, showing that the same model with different thinking levels can have vastly different results. Then for less popular models we have just one data point, which isn't enough to draw the kind of conclusions people are drawing in this thread.
> If they are not testing Chinese models with less than max thinking, then we should only compare them to other models also at max thinking.
Nah, there are interesting conclusions to be had in comparing various model/effort level combinations, esp. when it comes to cost and time per task.
I have my own private benchmark suite; it involves algorithmic problems, the need for good visual understanding, and the ability to run long workloads unattended (as the agents need to run a few iterations, self-evaluate the results, and hand over the assignments when they feel they have succeeded). Chinese models do poorly on such problems, and running them at anything less than max efforts is pointless; they can't even hand over the assignments graded against a mechanical set of tests they themselves can execute. Hand that same task over to, say, Sonnet medium and you have a weird outcome: excellent result, but way more time and tokens burned than, say, Opus 5.5 or Sol 6.1 running at high. I do those benchmarks just to be able to spot such patterns and then better choose the models for my real work - but i wouldn't have caught it if I wasnt running various model/effort combos.
Does anyone really believe the cost data for benchmarks? Pretty sure all models will behave differently at this point when they get benchmark shaped questions.
(And I’m sitting on data showing Sol 6.1 and Qwen 3.8 Max cost roughly the same per task, for real tasks.)
thanks for sharing a new benchmark website - AA has been going downhill for a while - I was using gertlabs[1] for a bit but it seems like they aren't getting some of the more recent models
Rarely seems to make them look good whenever I drop in to the site. Far more expensive for little gain. Even within their own effort levels it looks bad, Fable xhigh is 50% more in cost than high for zero improvement across a range of tasks.
A tiny bump in intelligence score hardly seems worth it for cost, and honestly most people's tasks are not cutting edge work.
Current benches on there seem to make Sol look the best, at a certain point the subscription subsidy tap will turn off and people will be in for a rough time paying full price.
I don't think there are any that can be trusted. There's too much riding on these benchmark results so the labs find ways to game them either by benchmaxing or by influencing the format of the benchmark.
I think you mean that benchmaxxing has turned into a business model, not for the Chinese models (maybe them too, of course), but for the AI labs, and artificial-analysis is not remotely the worst offender.
Right now the big labs are buying benchmark tasks from "benchmark labs" (or whatever they call themselves) for absurd amounts of money, and making sure they perform well on them.
The incentives this generates are terrible.
The big AI labs will only buy benchmark tasks that the frontier models perform poorly on (because they have trouble finding any public ones). So the tasks are being mass-written now directly based on the models they're supposed to train. Obviously the math says you shouldn't do this. If there is a causal connection between the benchmark and the model, you lose the normal guarantee that improving benchmark scores will improve model.
In fact improving such benchmarks may make the model worse. To understand that let's just give a ridiculous example. I create tasks like "1 3 5 11 13 17, give next number". The next number is 301. These are the house numbers on some random street in Brussels, Belgium (I'm guessing the big jump is because they "merged" 2 streets, and just added 300 to the numbers of the second part).
If the model trains on this, performance will actually decrease because obviously this is not how you should reason.
BUT making tasks under this "impossible to reason towards" part is of course how you make tasks that might even be reasonable, and of course frontier models are going to suck on tasks like this. So they sell!
BUT they suck! Finding houses is actually going to work a lot better if you ignore this fact until it becomes unavoidable in one out of 1000+ streets.
And yet this is what benchmark improvement has moved to. If you look at the terminal-bench 4.0 tasks you will see what I mean. Doing better on those tasks ... does not actually make a better model BUT you have to actually go read those tasks to realize that. A lot of the new tasks largely demand you make weird jumps in your reasoning not based on understanding the underlying theory. They rely just being able to predict what goes wrong in certain cases when theory would suggest it should work, and mostly it even does work but for some reason not acceptably. They reward not doing the full work, realizing what goes wrong and correcting, without understanding why the normal reasoning won't work and correcting without knowing why. I find it very hard to believe this makes a model more intelligent. It rewards knowing the encyclopedia by hard, so to speak.
But it certainly makes it look better. Whereas the obvious, coding improvements, do not make models look better anymore.
AA must be doing something right. Ever since they released the new benchmark, you see open weight China model fans complaining about it under every release thread. Probably because it’s hard to benchmaxxed against.
But admitting their model is weaker is a no go. So conspiracy theory here we go
For what it's worth, in our own metrics and benchmarks the intelligence is a pretty good match, but the cost per task is wildly out of whack. We see the open weight models like GLM-5.3-flash or Deepseek 4.1 Flash 1017 use a fraction of their tokens/cost of the benchmarks, which changes the cost-benefit equation a lot.
> Probably because it’s hard to benchmaxxed against.
It's mainly just a composite of existing benchmarks. It would be very easy to benchmax if, say, AA shared with a preferred lab which benchmarks it plans to upweight/downweight in the next version. Just speaking hypothetically, of course.
Hence the constant switching around and running the benchmark themselves. They have been pretty transparent about which benchmarks get added or removed with every release.
Yeah, it's in the same ballpark as Kimi 3.1. Both of which are obviously open weight, so that's at least a big differentiator from Sol. Let's hope against hope that the much smaller flash model might beat deepseek v4.1 flash and run for as cheap.
Why do we trust this? Do they open-source their test code?
Also, a generic benchmark might not translate well into what everyone needs. I keep building my own test cases that test models for my own work specifics. Only that benchmark is useful to me. I don't need it to win a PhD math quiz.
They have plenty of access via data centres outside of China. The first woman Jensen was revealed to have sex with is the CEO of one such Singaporean provider to China. The US allows this.
They have been trying to do that. They never signed the open weights support letter and have been yapping about dangers and standards basically trying to shape the ecosystem on their view of how it should be; which would effectively ban open weight models.
There is danger with everything. I can take a kitchen knife and stab my kid.
I WILL NOT DO THAT!
The key issue is how we manage a given danger. Anthropic wants to manage the danger of LLM's in a way that protects their business model. It's in their right to do so. And it is in other's people right to recognize this approach as self serving and object to it.
Then there's all the "open models are dangerous" rhetoric while Anthropic and OpenAI's models are displaying said danger. Then there's the insane amount of money they're spending on lobbying. On what exactly? Who knows, but since it's their interest to kill open weight models and most companies tend to lobby to defend their own interests, it's not an unreasonable assumption that some effort is going into that.
I hope they are able to serve 4 faster. Qwen 3.8 is a great model but I’ve regretted including it in the battle system for opencode [1] we have been building because it scores so low.
And it’s a shame because the only reason it scores so low is that it’s slooooow. (Like 30 tokens per second vs opus near 80.) If you exclude voting, Qwen 3.8 does well.
(We allow early voting contrary to Arena.ai because I think it makes sense for agents, where it doesn’t for chat. This means our data takes speed into account.)
Also, it would be ideal if they served this with third parties. Part of the slowness with 3.8 is just not a lot of providers. Their open source version is much faster, but we couldn’t include it because it doesn’t handle images (another own goal by Alibaba).
In case it translates to Qwen 4 27B, I've been running Qwen 3.8 Flash Next (which while MoE (and bigger) is supposed to be a preview of Qwen 4's architecture) and it runs pretty solidly regardless of context window size and completes random deepswe challenges I throw at it locally (at Q2-Q3 quantization! so it fits on my shitty setup).
The model I’m referring to that would be faster and similar size, but doesn’t handle images is Qwen 3.8 2.4T: https://openrouter.ai/qwen/qwen3.8-2.4t-a95b. It’s weird that this is pretty much the same as Max but doesn’t support images. No idea why.
Is the speed due to a difference in the way it "reasons"? I was trying out a local version (3.8 27b) and watching the thinking, it's absurd how often it challenges itself. Simple facts get questioned multiple times, and it has to keep reminding itself that it already established some things as facts earlier.
Instructing it to ask any and all questions as necessary sends it into an existential crisis, because the question tool (oh-my-pi harness) suggested limiting the number of questions, and it couldn't figure out how to do both.
No it doesn’t look like it. In our data, Qwen Max uses about the same amount of output per task as Opus. So the problem does seem to be inference speed.
The issue you’re describing could be a small model problem
Aside from making our engineering prompt terse, I place a token budget on the thinking traces. Those traces are mostly meaningless(in the reasoning realm, not the usefulness realm), and just tell it to do something productive. For coding, this just makes it read doc, tests, etc.
I believe that's all reasoning levels do, but most inference engines should have thinking budget.
This is exciting. The continued focus on efficiency alongside quality is encouraging - it makes me hopeful to see what smaller models will become capable of.
I'm also hoping someone puts out a competitive 30-70B model soon. There's this massive dead zone between 24GB and <100GB of VRAM where it seems your best option is Qwen 3.8 27B.
The 3.8 Next-Flash release is comfortably the best model on a 96GB Mac Studio. I’ve tried various engines and Sushi (https://github.com/beamivalice/sushi) is the most performant so far.
Hopefully the Qwen 4 Flash model will still fit within 96GB without requiring excessive quantisation as, verbose thinking aside, 3.8 is a very solid workhorse model.
Same, I recently found Nemotron 3.5 74B-A9B really good at filling this gap though, could do (very slow) local inference on just 32GB DDR4, without the quality dropping to the floor
Qwen 3.8 27B is a dense model, so prefill times kind of suck on Mac’s. MoE models offer way more value unless yiu have the bandwidth to handle a dense model.
Qwen 35B A3B is considered the best small MoE, but it depends on what size you are targeting. You can get heavily quantized qwen 3.8 flash next (a larger moe) running on smaller 64GB machines, and while it isn’t as far as the other model, it is a lot smarter.
Waiting for the Flash tier too. My experience running coding agents on cheap vs big models: the cheap one gets the same work done, roughly twice as slow. The big model buys you speed, not results.
I think Alibaba is not aiming for the frontier on this class. Their previous Qwen3-Max was released in September, right around when Opus 5.5 was coming out, and it was nowhere close to it as well. However, one or two tiers below (Flash, 27B) they really shine and are SOTA or close to it.
DeepSeek has a similar problem, I feel like their full model is hardly even worth using, while the Flash tier is absolutely amazing.
It seems to be the case. I would qualify them more as flagship-killer than frontier, using the same terminology used for smartphone classification. As opposed to the big American shops which focuses solely on performance, they have operational costs and other factors in their constraints and requirements. As soon as the market reaches some maturity, this will prove to be very important.
> As opposed to the big American shops which focuses solely on performance, they have operational costs and other factors in their constraints and requirements
This is not true, OpenAI and Anthropic compete strongly at the lower end. See Haiku 5.5 and Luna
I am unsure. Why shouldn’t they? Why even work on a big model (especially training time) if “nobody will use it as competitors are better”? We’re still talking about nearly 2.5T params here. Deepseek’s pro model is still heavily smaller and like you said isn’t directly worth using, obviously it has the open source / self hosting advantage.
Imo there’s a need and they try to push here, it’s also giving them advantages for their smaller models but if the big models never perform they will have a bad time or maybe should even stop.
If you're aiming to make the best possible small model, you still need to train a larger model and then do logit-level distillation from it. This is where most of the gains in small models have come from.
My use of qwen3.8-flash-next suggests they're trying to make a low cost model that knows how to program and they think the "total addressable market" is serving more people at lowest inference costs possible.
Think of it like a hind brain+fore brain discussion. Instead of focusing on making some ultra model that can do everything but costs a ton, you have your smaller models gather the bits and feed it better data
Agentic coding is the target then. If they want their work force to not care about token gen, that's what you do.
As an analog, they're making a drone swarm while US is making 5th Gen fighters. Completely different battle fields.
Yeah, I feel like models like Qwen3.8 27B are making clear trade-offs, like having very little world knowledge baked in. I believe those are intentional, so they can focus more on other areas. I guess that penalizes them on these benchmark aggregators.
I think the argument is like MacOS vs windows vs linux. AI models from anthropic/openAI are windows and MacOS; they're plat forms for "skilless" to start on and of course you can gravitate towards the development end. They're bloated in that sense.
Open models follow the linux goals which is small tools that chain together. If I can get a small model to understand how to program for another model or do the programming bits for a large model, then the trade off is flexibility. You can imagine your giant AI that can do anything, like windows/Mac is to your benefit, but there's a clear flexibility line to how these models act.
Even if it takes longer for my local model to find something, that doesn't mean it's taking a long time because it's doing wrong things.
Honestly, I think a lot of people using SOTA don't understand what you can do with a proper engineering harness; we all live in a yellow submarine.
> DeepSeek has a similar problem, I feel like their full model is hardly even worth using, while the Flash tier is absolutely amazing.
DeepSeek also agrees, and tried to withdraw their pro model but got pushback so they are offering it even if they are not encouraging new use cases for it.
No idea, but this is a rapidly changing field. Opus 4.5 was released a little less than a year ago and it feels shockingly limited compared to even current open flash models. (I literally just tested it the other day.)
So, with new models coming out on a weekly basis, I think trying to figure out a future trajectory is just going to be a wild guess. For all we know, Anthropic and OpenAI models may also be plateauing in the near future; or there may be a breakthrough and they may improve beyond our wildest expectations. I honestly don't know.
That said, I suspect available compute and model sizes that major Western vs. Chinese labs can handle will still keep the gap open at least in the short term.
On the other hand, I am also not sure how much the gap will matter past a certain point. I am specifically wondering at what point people will become interested in smaller models fine-tuned for specific application areas (e.g. cheminformatics) rather than massive generalist models.
I think the jury is still out on this. Other companies are instead focusing on fine-tunes.
There are a number of issues with relying on a frontier LLM for these kinds of tasks.
One, there are only 3-5 companies in the world that have the capability, and you depend on whether they're interested enough in your use case (do you think you can interest Meta in putting support behind optimizing support for Swedish law, for example?). These companies measure their investments by ROI, and they focus on things that have large markets. Heck, Google in particular is kind of infamous for suddenly killing off projects that don't work for them.
Second, all of these companies are American, which may be a no-go for certain use cases involving strict data privacy.
Third, these capabilities may need to be maintained and updated, and you'd depend on the goodwill of frontier providers to keep doing that.
Fourth, the frontier model is likely to be more expensive to deploy and run than a smaller specialized model.
You may have a point in a world with infinite resources, but the reality is that this situation is influenced by plenty of market constrains.
I can't imagine them keeping up long term given the information asymmetry plus compute restrictions on top of it.
I feel open-weight was able to keep up the past year using efficiency improvements (hybrid attention, improved MTP drafters, improved optimizers) to offset having less compute. But OpenAI/Anthropic started cutting token costs just a few months after these, which to me suggests they copied these improvements. Indeed it would be hard to work in LLM training and not know about DSpark, Muon etc.
So what do you do if closed source uses your learnings, gives no knowledge back, and has 10x more compute and funding on top of it? The situation feels quite dim right now, esp. with the Nvidia thing.
Obviously American AI is similarly geo politically supported. However the business model so far is at least somewhat clearer. They invest in the cutting edge and want to sell access to it.
I think most people assume that whenever the Chinese govt is involved in subsidizing their industries, they just give companies free money and hope for the best. And this used to be true until China ran into the 'involution' problem (内卷): https://en.wikipedia.org/wiki/Neijuan
These days the way China subsidizes these companies is more from state-backed buildouts of data centers, renewable energy, and electrical infrastructure. It's a 'make the field as green as possible for the cows to come' kind of strategy. It's very different from America's hyper-individualism which feels very reactive with specific greasing-of-the-wheels + DARPA/CIA VC + DoD spending while relying on companies to be the ones to push for better infra and capacity.
I can run an open model on my own hardware, in my own home (which has solar panels on the roof, so electricity usage is mostly a non-issue). Or, if needed, I can rent compute from somewhere like you'd do for any other workload.
Once I have a model locally, I know it isn't going to change on me. Compare that to using an API from Anthropic or OpenAI (and yes, the only thing you can say is you're using an API, not one model), they can discontinue a model you're relying on (as every model could have peculiarities you've designed around) or modulate their speed/quality day-to-day to save on compute costs.
And trying to claim that buying your own hardware doesn't pay off compared to API or subscription costs doesn't hold much water either, at least not for me. For starters, I already had the GPUs and machines I started running models with, whether it was for gaming or for other reasons. And in the case of the new GPUs I bought recently (2x AMD Radeon Pro R9700; no you do not need CUDA and you haven't for a long time now), I can hold a reasonable expectation that they will hold at least some of their value years from now, compared to paying for API/subscription usage for years in which case you're left with nothing.
So, if you have a local model that is "good enough" (and they already exist for many use cases), it's just kind of the obvious choice to make, IMO. And in just one person doing it as a hobby while I'm still looking for a job 16+ months after a layoff. A small business and certainly a large enterprise has even less reason to open themselves up to all the downsides that come with using Anthropic or OpenAI.
These are all the kinds of personal preferences that we engineer types have, but not how most companies think about this. For evidence, look at the revenue growth of the big labs.
Local models already crossed the threshold where they're quite good and useful for many things, even with cheap setups. I walked away really impressed with (non-coding) use of Byteshape's Qwen3.5-9B quant on my 8GB RTX 3070 for instance. The model was fast, helpful, and 100% private.
If we extrapolate over the next several years, prices for VRAM/RAM will start to come down as manufacturing capacity comes online, models will continue to become more token efficient, and intelligence will improve at every quant size. As this happens, more and more people will be running LLMs locally on hardware they already own, and there's going to be a point where local models can do everything you need to do unless you're literally trying to do breakthrough mathematics research.
I don't think "useful for many things" qualifies as "winning". It's about two orders of magnitude shy of winning.
> and there's going to be a point where local models can do everything you need to do unless you're literally trying to do breakthrough mathematics research
Maybe. But competitive markets pay exponentially more for incrementally superior services. Professional athletes are a great example of this. If you're building a product with competitors and those competitors have frontier models and you stick to local then you're probably going to lose both today and in the near-ish future.
If you're bitter lesson pilled then the alpha is entirely in having a better model, more data, and more compute. Your vision is that human ingenuity will provide more alpha. Oof. I wish. But my tastebuds are very very bitter.
> If you're building a product with competitors and those competitors have frontier models and you stick to local then you're probably going to lose both today and in the near-ish future.
i highly doubt this premise. the success of a product has very little to do with the amount of the intelligence used in producing it. you could throw millions of dollars using AI to develop a movie and it would still flop in comparison to an indie-made film.
I think the premise is confusing what the amount even is in this context.
I want to use "more intelligence" but not in terms of single higher IQ.
I want to use more intelligence in terms of a sum quantity of knowledge work being performed. A team of 10 instead of a team of 5.
This is when a local model will shine. A fixed cost and I don't have to worry about how much I am using it. I already can't tell any difference at all between Opus and Fable. It won't be that long that I won't be able to tell the difference between a local model and Opus.
There will be an infinite demand for "more" intelligence but I think many people are making a category error on the dimension in question.
Also - the “open” here doesn’t have the advantage of open source, where it can be improved by the community. At best, the community can tweak it a little. For actual improvements you need to wait for the next model which is trained by companies for billion of dollars, and could stop releasing them at any moment.
It can, through better quantizations and reinforcement learning.
Just this week, qwen 3.8-27b got two new interesting quantizations, one of which came from community. Llama.cpp also improves continuously, and novel alternatives like Strata pop up.
The point is that these are overall small improvements that the community can’t really build upon. For an illustrative example, combine all the community improvement to the best open weight model released a year ago. It won’t do much to make it competitive or even interesting.
3.8 wasn’t released a year ago. Of course base models change. But the quantization techniques carry over to new models. For instance, one of the improvements last week was someone porting the techniques used for macaq to qwen3.8-27b. It’s pretty satisfying to get a working implementation of a new technique barely one week after the original paper hit arxiv.
Also improvements to llama.cpp stay, and many small improvements stack up nicely.
The sad truth is that businesses have every incentive to degrade your experience to the level of “tolerable” and increase the price to the level of “barely affordable”. If the founders won’t do it, they’ll eventually sell to someone who will.
Open source doesn’t have those motivations. Once something works well there’s no reason to degrade it.
I think looking at the lag between frontier and open-weight is looking at an uphill battle and assuming it's equal footing.
Lots of research goes into open-weights, optimizing cache re-use, storage, inference costs, etc., and all these benefits are easily rolled into closed AI models. Meanwhile any advances made by OpenAI, Anthropic, etc. are kept hidden away to give their models an edge.
It'd already be a miracle if open-weights were gaining on-par with closed-weight models. That they've been closing the gap for most of the past year is wild!
I think so but it may be a very short term effect, I think their pace has roughly matched the pre-speed up pace of the frontier labs (the 1.7-2x speed up noted on by METR), and I’d guess we see a similar speed up from the Chinese labs once their internal models hit an equivalent level enabling the same speed up.
Following that logic the gap widens a little bit here but then will steady again. Before this there has been a general catch-up trend too which probably also persists leaving us in a slow closing of the gap as the net effect?
If the speed up is the reason, then they will never catch up (with traditional means). Because it means RSI predictions are real, and we will see an explosive growth in capabilities (far beyond exponential).
If compute is the bottleneck we’ll encounter in the coming year, then you would expect the gap to increase even more. The US and its companies has three times as much AI compute as the rest of the world combined.
I’m actually expecting it to expand. There is so much new production being built right now by every company due to the shortage, and China doesn’t have a single EUV production, and only expect a prototype one in two years.
They said it clearly in their blog that this release focuses on the real work jobs performance, as an team member. They listed impressive real world jobs with concrete improvements like production line planning, due-diligence and financial research. Most likely they have developed their own harness/orchestration for the job, so how they do that and how much did the model help is of interest now. Nevertheless, they demonstrated that Qwen Max 4 can solve real world jobs with significant economic benefit. That is IMO a substantial leap in AI economics and deserve better attention.
Coding alone or math is real, too, but once you crossed a threshold, you don't need much more improvement to be productive and the cost of marginal improvement can quickly outweigh the benefit.
I don't know how frontier western models fare in these aspects, but I haven't heard much from them yet.
I feel like they are improving what actually matters for LLMs, lower memory usage, compute. While we don't know how much far ahead closed source models are in this, but it could be similar they might lower the price to subsidize or make it seem like its all good, either way tech like these being open means it will help us reach further in the long term.
Even smaller open weights models are already great enough for much of "assist human" work.
This forces ant/oai to lower their prices but how much that is subsidized nobody knows. It might very well just be they are in deep trouble, they see their clients leaving, they have to lower their prices.
But let's not forget openai is riding on the edge of bankrupcy, their last investment round apparently gave them few months of runway, they are leaking money. I don't think they know how to turn this around with all the open models being competitive.
I would just enjoy cheaper model costs while they are like this.
Quite a few saying that they tried earlier snapshots of this model and are finding it does not live up to expectations. Firstly do other agree and also why would the model get worse - could they be doing something weird to the model that is making less aligned at the end of training, potentially for political reasons?
The past few days I’ve been finding myself asking “seriously, where is it”… feeling that the world has somehow changed because Qwen 4 27B was later than I expected.
Doesn't that just refer to using the OpenAI response/completion/etc. API? Just about everybody (llama.cpp, vLLM, etc.) supports that. I think the reference to "qwen4.5-max" is more interesting myself.
While I doubt that Qwen 4 Max can seriously compete with frontier models, the AA index has become all but useless, due to the introduction of new benchmarks in the aggregate which are of questionable value as a measure of "intelligence" [1]. AA has seemingly also abolished updating the Coding Agents Index, which was an okay benchmark for coding ability, and older models are completely absent from the v5 version of the full index.
[1] Especially Terminal-Bench-Science 0.1 stands out. Example problems:
Probably very relevant to people working in these fields, but it is honestly not surprising if a lab with limited compute will not train for these technologies, and it says pretty much nothing about more common use cases. I also suspect that this benchmark measures knowledge retrieval more than raw reasoning ability. This is where you really need a breakdown of capabilities and not an increasingly meaningless aggregate number.
Compared to OpenAI and Anthropic, yes. There's this export ban on Nvidia chips, after all. And all the CPUs they have won't help much with model training.
AA has become pretty meaningless. They just keep adjusting the recipe to get a preferred ranking result, with no supporting evidence for their decisions.
Edit: In addition to reshuffling the weighting of individual benchmarks, they also do things like test Anthropic and OpenAI in their native harnesses, while not doing the same for Qwen/Deepseek/etc.
AA has always been meaningless, but people keep referring to it anyway just because the synthetic view it provides is too convenient (even if meaningless).
I thought they were an independent company, what motive would they have to prefer some ranking results over others?
Edit: I do agree their harness methodology is pretty bad, but I think that's more directly attributable to the fact that harnesses appeared after their initial methodological design, and they are too lazy or resource constrained to improve their methods. Still dishonest, but a simpler motive.
One thing I find confusing about AA is that they show GPT 6.1 dominating the lower end of the Pareto line, with eg, GPT 6.1 Sol at medium supposedly costing far less per task than DeepSeek V4.1 Flash at high.
Yet every time I compare the two in something like Pi, Sol ends up being 5-10x the price for a simple query, something like 7-10c vs. less than 0.5c for DeepSeek. What's the deal?
Anthropic has been trying many different ways to prevent this, including a proposal to nerf your sessions if they “detected” malicious activities (anyone’s coding sessions would “degrade” if detected as malicious). This led to a huge outcry from developers complaining that they “need to be able to trust” the responses from Claude and intentionally poisoning responses in some sort of weird shadow ban was unacceptable. They walked it back but the codemap leak also showed some dubious weird regexes to nerf “malicious” users. They have a real distillation problem. It’s hard to have empathy for a trillion dollar company that distilled the contents of the internet into a thing and now is crying foul when someone distills their outputs.
There was also a bug in which the reasoning tokens were “encrypted” for opus but you could ask Haiku to regurgitate them and it would.
Anthropic have been doing anything and everything to prevent distillation “usage” and it’s caused me at least once to cancel my subscription because it’s hard to trust a company that secretly alters its responses based on something I can’t see and never get feedback on.
This weekend I had the audacity to use Claude Code to help with the code side of post-training a model...
Asking it to think/reason too strongly would hit classifiers. Quoting reasoning summaries in my messages would hit classifiers. It thinking about how post-training effects another model's would hit classifiers.
I thought it might be overzealous Fable 5.2 protection, but at one point I went through a chain of Fable -> Opus -> Sonnet. Eventually even Sonnet triggered a classifier no matter how the conversation went... I had to rewind and have Claude prompt Codex going forward for most of the work.
I was waiting GLM from z.ai and it still not releasing, almost all the models releases their new version of models and they still on 5.3 who is very weak in complex tasks and do half of the work and forgets to make migrations and like that issues. They sell us the yearly plan and stopped improving. Because of that i am just using in coding opus 5.5 and in chat fable they are the best models from experience.
Are these (hosted) Chinese models still accepting arbitrary cybersecurity and reverse engineering tasks? I am increasingly frustrated with Claude's safeguards (and Codex, to a much lesser degree). It is flat out refusing to look at game binaries and anti-cheats for the sake of learning and devising Linux compatibility workarounds. It seems like in the near future people who want to do this type of work will need to buy a couple of DGX Sparks or retired datacenter GPUs and run abliterated open models on their own hardware.
Hosted where? In my experience they might put more guards when hosted by the company that makes them but if you use them through inference providers they do whatever you want.
Although, can’t say I’ve tried too hard (not keen on sending too much data to China) but I’ve been using K3.1 through Moonshot and I have had zero issues.
Mistral Large 4 (through Mistral) also doesn’t refuse a thing.
> Are these (hosted) Chinese models still accepting arbitrary cybersecurity and reverse engineering tasks?
In my experience, yes. I wanted to flash my new wifi access point with openwrt and opus 5.5 refused to explore how to get root, so I switched to mimo-v2.6-pro and it figured it out.
In our short horizon data analysts exam, it scored a few points worse than Qwen 3.8 27B from August (which is open weights) and cost a bit more. There’s about 10 models that we’ve tested that have scored both better and cheaper. Too bad it’s closed, Qwen 3.8 27B really punched above its weight and was the only open model I’ve tested that could run on desktop hardware and perform very well: https://plotly.com/blog/qwen4-max-plotly-data-analytics-benc...
I’ve been refused. I pasted a screenshot of a torrent list for a tv show. “Which one should I get, they all look the same”. “I won’t help you with copyright infringement”.
But the same models will tell me about the -arr stack and offer to set it up for me. You just have to frame the prompt the right way, at least for me. It can’t think it’s assisting with immediate piracy. Future theoretical piracy or for educational purposes, you’ll have better luck.
From a far off perspective, can we really assume any model to be free of bias (political or otherwise)? I don't know - but would like to see - how the political opinions of reddit users and blog authors are reflected in that of AI, as AI is trained on human (biased) sources.
Realistically, they're trained on content and thus should have the bias of their content, even without specific intent. If you train something on more chinese content it will have more bias toward it, if you train toward western centric content it will have that bias, etc ...
It also applies to languages, I use Qwen 27b a lot and it's a lot smarter in english than in french.
My experience so far with open-weights Chinese models (self-hosted DeepSeek, Qwen) is that they do actually answer questions about Tiananmen Square fairly objectively. The political "corrections" seem to be applied to domestically-hosted APIs rather than the weights themselves.
Haha the programmers here are anything but apolitical. It has been an ongoing issue.
But I agree. If the model writes good code and makes good decisions, as long as I’m not building a video game about recent Chinese political history, I don’t care about its politics.
Oh, I leased it :D a trade-in helped reduce the cost a little too. I recommend that route. It's doing well over $100/month in inference value for me and the CPU is still totally free while it's working on that and is a great computer for everyday life too.
I'm actually running Gemma4 for the most part - for my specific needs it was the best out of the several I evaled
I wonder if they could benchmax Program Bench, it's one of the few benchmarks that looks almost exactly like a good fine tuning curriculum which might help general coding ability and agential work across the board?
I suppose it could make actually worse at taking instructions and redirection mid-task, but that could be easy to work around too. Just hold back a bit of the spec and drip feed it in later?
I don't mean cheat on it, I just mean optimize for it harder!
Interesting that it's only at a reasoning effort of XHigh does the pelican's left leg actually go behind the bicycle. At lower reasoning levels the 3D aspect of the figure is not correct.
So what is happening is that models are broadening difficult-to-reason-to skills on benchmarks. If you take the traditional benchmarks (ie. DeepSWE, Terminal-Bench slightly older version, GDPVal, Automation Bench, ...) this model is between Sonnet and Opus 5.5, VERY close to the frontier.
What has happened to artificial analysis is that they've begin to focus a lot on very broad tasks like legal reasoning, using very difficult to work with scientific software packages, correctly analyzing biology experiments (without or only tiny amounts of explanation, count cells, BUT leave it to the model to figure that cell that aren't round, or have a distorted cell membrane (ie. have burst) are dead), very difficult physics reasoning ("2 electrical wires are close together with power flowing in the same direction. What happens?", and you get points deducted if you don't realize the problem with high voltage AC power, where the wires will shake themselves loose if the phase doesn't match perfectly, meaning the wires need to be EXACTLY the same length or ...). If I'm being impolite I'd say the frontier models are trying to make sure they'll pass the "gotcha" interview questions for many jobs.
The 3 differences:
Qwen 4 is a huge jump from Qwen 3.8 in coding
Qwen 4 is a big jump in using it for agentic workflows
Qwen 4 also performs much better on avoiding hallucination
If the difference between Qwen 4 Max and Qwen 4 flash is similar to 3.8 and 3.8 flash next (ie. about 10%), then 3.8 flash next may just barely beat Opus 5.4 at coding.
I've been using deepseek-v4.1-flash for 10+ agent swarms for incredibly difficult tasks like working on the solidcore kernel for solidcraft. Work like CAD kernels is pretty damn hard and most contributors are using a significant amount of thinking tokens in a frontier model. Somehow, deepseek does pretty damn well in swarms and seems to keep up with opus 5.5 for this sort of task, and even after swarming the hell out of them, still comes out incredibly cheap even using API pricing. Difficult to reason tasks may be the frontier, but being able to simultaneously test 20+ different hypotheses is a very solid surrogate.
Even though v4.1-flash is not a frontier model, it works out that if you're pretty good but incredibly fast and incredibly cheap, you can do frontier level things quite well. And this being the case, I really hope that the rest of the Qwen 4 releases deliver on that value that right now is being offered by deepseek.
> So what is happening is that models are broadening difficult-to-reason-to skills on benchmarks. ... If I'm being impolite I'd say the frontier models are trying to make sure they'll pass the "gotcha" interview questions for many jobs.
These "gotcha" skills will be needed for frontier research/scholarly tasks. It makes sense to focus on these now that traditional domains like coding or frontend design are broadly saturated (i.e. Astra- or Fable-level outcomes) even by lower-tier open models.
Real "gotcha" skills, yes (perhaps, they're gotchas because the underlying theory is not well understood. More can be achieved, certainly for superintelligence, by forcing understanding of the underlying theory rather than the gotcha). AND the incentives are totally misaligned.
"Gotcha skills", if we're going to call it that, are so popular because the AI labs are buying "tasks" to train on at huge prices, but mostly if they're gotchas.
The problem with that is that a collection of facts in the positivist sciences (and math) are always incomplete and almost preclude a full understanding of math, because it takes away the reason (to an extent) for full understanding.
It's great fun asking it questions like "who is the Dalai Lama?", "where is TSMC headquartered?", "why do people post Winnie the Pooh in reference to Xi?", "who controls the South China Sea?", "why was Chiang Kai Shek fighting Mao?", "why were students in Tiananmen Square?"
It'll gush about how to make moon cakes and where the tallest and longest bridges are and how to run qwen on a raspberry pi - but it can't seem to manage some basic Wikipedia questions.
I've done some work on this and a lot of the refusals are in the API. The weights are more permissive at these types of things. Try it again when we get some of the smaller Qwen 4 weights.
When I tried it with 3.8-27B locally, it was happy to talk about Taiwan while acknowledging it was giving the mainland government's perspective, but for anything about June 1989 it would clam up and just tell me to consult official Chinese sources.
You can watch it happen in DeepSeek's chat. You'll see it spit out the real response in the browser the first time, then the text gets cleared and it says it can't help you with that.
Qwen3.8 flash next handles these questions just fine, what should I expect to see? It correctly reports who the Dalai Lama is, that TSLC is in Taiwan (Hsinchu), explains why people post Winnie the Pooh memes, etc.
I suspect this happens when you run the inference in China or first party from the model maker. Out of all the open weight models I've used on US located hardware, they're fine answering these.
I tried asking at https://chat.qwen.ai/c/guest a simple question, "why did China invade Tibet?" and it flashed up some transient "thoughts" before giving the definitive answer "Oops! There was an issue connecting to Qwen3.7-Plus.
Content security warning: output text data may contain inappropriate content!"
Funny how other flagship model releases feel like "Great, let's see how far they pushed frontier" and Qwen Max releases feel like "oh nevermind, what about the 27B version?"
There's plenty of competitors in the league where Max is, but few companies can improve the 27B level capabilities.
Cybersecurity not mentioned. Can´t wait to test bench the open weights in cybersecurity. If is not performing well by design on it, I imagine someone have to add some additional layers XD.
Really would love if these publications have a huge, bright red bolded section "How to use this right now". Is it on openrouter? Is Qwen hosting it already? Maybe I'm just ignorant?
EDIT: oh, I guess this isn't open weights, so presumably I'll have to get it through Qwen or whoever they might partner with (Bedrock?).
I'm not too worried when models are popular enough, I assume it will be detected and it can be investigated because it's open, and the reputation of a company doing that would be ruined. Afaik Cohere isn't sure what these urls were made for and it looked more like a way to test the model. Also if you have anything sensitive for that to be a threat you should use a good firewall to detect and block it.
I have more problems when it's a random quantized or finetuned model on hf.
CISOs are buying it. There is no way to prove a model hasn't been poisoned. It's a serious implication that the behaviour can be manipulated at any time. People with heavy compliance obligations are (understandably) nervous because their customers are demanding answers. If I agree with the risk is separate from whether or not the concerns are founded in a technical understanding of the risks.
I love how Americans are so brainwashed to think the Chinese can't innovate without cheating, while they lap us in basically every category possible except this one verticle, that we had a 3-4 year head start on.
I partially wonder if the Chinese would even release a model that flat out beats Claude/fable I say this half jokingly...
but I imagine there's part of them that goes "These Americans are so delusional about how language models are dangerous/existential threats that they might kill us or hurt our economy over a chat bot"
It could also just be they're far more tactical. Deepseek generates a ton of revenue because its practical/economical. You do not a mega super PhD assitant to build your 4 MAU SaaS app.
Anyways I hope China wins this pointless AI war, because if it does lead to innovations in medicine and science I would much rather the Chinese be the ones that discover those things than the US billionaires who'll almost certainly try to make as much money as possible.
Qwen4-Max pelicans at four reasoning efforts (none, low, high, xhigh): https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
For comparison, here's one I got from Qwen3.8 Max (at default reasoning effort): https://simonwillison.net/2026/Jul/20/afraid-of-chinese-mode...
This inspired me to re-run pelicans through 3.8 Max - but OpenRouter has qwen3.8-max-prime and qwen3.8-max-0902 - prime is twice the price and faster - so I went with the 0902 one.
Turns out Qwen4-Max has a none setting, but 3.8 Max didn't allow disabling reasoning - you had to pick from these five levels instead: minimal, low, medium, high, xhigh.
Here are those 3.8 Max 0902 pelicans: https://tools.simonwillison.net/markdown-svg-renderer?url=ht... - they're mostly not as good as 4, but I prefer the 3.8 xhigh one.
I don't fully understand why you keep doing these as-is. You original motivation was that this was not in the training set and indeed it was a great benchmark for a while, but it's clearly been incorporated now in the training data and possibly internal benchmarks.
Because he wants the engagement they generate. I don't understand why they still get upvoted on HN, but here we are. If even I can recognize that they're pretty much useless as benchmarks at this point, he must know it too.
He’s our pelican man :) I will always upvote pelican posts from the HN resident pelican guy
HN is not Reddit.
Sometimes it's equally toxic.
People have been saying this for two decades now and it's literally in the guidelines not to.
I have bad news for you.
The most reddit thing about HN is how reliably someone will spend the time to tell you HN is not reddit at the slightest provocation.
Likewise, i always upvote these posts even if i know it is biaised, it is always fun to see their evolutions, there is nothing wrong with them. Also even if we know there are trained on them specifically, some models are nonetheless better than others in engineering the output, thus it is always interesting.
Things like "I will always upvote X posts from the HN resident X" feels like what HN should actively work against.
I think their goal was just to write a funny comment.
Which is also something we try to avoid here
> All work and no play makes Jack a dull boy
GIUYS STOP BEING FUNNY ON THE INTERNET
Yeah, which I suppose used to be something we all kind of tried to avoid here, to make conversations more curious instead of just pilling on with memes or whatever other communities been degraded to nowadays.
In my view, as long as it is just an occasional funny comment here and there, it is not an issue.
Thats why you downvote memes/jokes instead of reporting them.
They’re not banned, but must be discouraged. It’s like someone being loud at a museum or using their bright phone at a movie.
Okay fun police.
You see it was funny the first and second time two years ago or so, now it's just boring. Whenever I want to read some insightful comments about a new model I need to scroll down through a few pages of comments about pelicans as if they meant anything now.
You can click the '-'/'next' symbol on a parent comment to collapse it. You are welcome :)
He seems to have posted it as a top-level comment, so it should be possible to fold it all, right? If he hadn’t, multiple other people might be posting about pelicans. Consider it the pelican attracting sacrifice post.
I have a colleague who got to the last rounds of interviews at a FANG, but did not make it...He could not draw a Pelican...
That's why AI will replace us: will be able to draw better pelicans:D
It's shame that people try to farm engagement on HN. Is this transforming to reddit. I don't know at this point but I agree with parent comment. EDIT: One thing I know is that I now need to scroll past the pelican thread if I need to read something usefull
Or you could just collapse the thread.
If you think it breaks the rules, then flag or downvote. If it just annoys you, collapse it. I'm not sure why this needs to be a whole discussion every time Simon posts a pelican.
If you think HN is transforming into Reddit, I think you may not have visited reddit recently, nor have read the HN guidelines (people have been saying this since 2007) or earlier
> Please don't post comments saying that HN is turning into Reddit. It's a semi-noob illusion, as old as the hills.
Why is posting the pelicans okay, but when someone says they're repetitive you're suddenly "not sure why this needs to be a whole discussion"? If you don't like that part of the discussion surely you can collapse it or downvote and move on, just like anti-pelican crowd is supposed to do with Simon's posts
It's not one person saying it's repetitive. It's dozens and add no value to the discussion. On each pelican thread. Which one is more repetitive? People who don't like the repetitive comments need to collapse each individual one, vs one single top level pelican comment that can be collapsed.
I also dont't know why anyone would downvote a pelican post when it's not offtopic? We're not supposed to downvote stuff we don't like or disagree with, just stuff we don't find appropriate or relevant to the discussion.
Although if this keeps up, soon there'll be another camp of "comments complaining about Simon's comments being annoying are annoying" :)
I mean, there's a common thread here between both types of repetitive posts...
You stop one, you'd stop the other.
Well sure, we could also stop the posting of model releases to HN and prevent any repetition. The line has to be drawn somewhere. One top-level comment seems fairly harmless to me.
It's part of culture at this point.
Based on some of the trash that consistently makes it to the front page I am very much ok with Pelican Man posting comments linking to his benchmark, even if its a bit less meaningful than it used to be.
And he spends the time doing it and as part of it drops nuggets of information not seen anywhere else. Respectfully and highly welcome. You're most welcome to move on.
Simon likes to deny this. I don't see him getting banned anytime soon though.
I don't understand why they still get upvoted on HN
Because you keep complaining.
He can do whatever he wants, I personally think they are quite informative, albeit I don’t trust them that much either. It’s like a vague heuristic to say “this looks like an interesting model to test out”.
Is it informative? I really don't get what signal you're picking out of a bunch of similar looking drawings.
It's interesting to see which model + thinking strength gets the details right, such as legs on opposite sides of the frame, a coherent bicycle etc
Isn't it a bit arbitrary though?
The original concept of this "benchmark" was to see how the model performs on a reasonably complex task that probably isn't in the training data.
Even if they don't fine-tune for this specifically, there are now surely tons of examples of people asking for a pelican on a bike in the training data, and making an SVG isn't that complex of a task for current gen models.
Even with the same model at the same thinking level, you might get different quality out of the same prompt on different runs. If you want to use this as an eval, you would really want to generate 100 of them, and see how often they're anatomically/mechanically accurate.
Some people are determined not to understand I think, but if they don't find it useful no worries, there's just no need to have a temper tantrum about what other people post on here especially when the majority (it's near the top for a reason) find it useful. Just don't worry about it - you can even hide whole threads on here if you don't like them, I use the feature all the time rather than taking the time to comment negatively!
what "temper tantrum"?
I find it amusing that repetitive pelican posting is fine and folks should just minimize and move on, but responding with an opinion that pelicans are boring/not valuable/engagement farming is "temper tantrum". Why not apply the same standards to both?
Look at the page, though. If the data is in their training set, then the training must not have been very good, cause the images have very salient defects, and random artifacts popping up.
As far as I've seen, among the popular models today, there are still significant gaps in the quality of bike-riding pelikan svgs, making the benchmark still useful. Maybe not as a differentiator of frontier models, but certainly still informative for lesser competitors.
Unless the website is pulling hundreds of thousands of views per pelican, I sort of doubt any firm is actually training or post-training to optimize for this benchmark. They'll more generally optimize for better SVG rendering and I imagine these pelicans arose out of the annoyance we all had many months ago with how utterly bumbling and forgetful models used to be when completing tasks like that.
It's more a meme/influencer thing now.
Just the hacker news equivalent of when does the narwhal bacon at this point.
> but it's clearly been incorporated now in the training data
You are not the first one to think of this. And Simon has thought of it as well, and commented on it several times.
Still, by now it's just more tradition than anything else. It's just kind of neat to have a consistent benchmark across many LLMs, even if the benchmark is completely gamed, but that is the nature of benchmarks.
There was a post where a person had AI generate [animal] riding [vehicle] and created a matrix of comparisons which led to the conclusion that there was no overfitting of pelicans in bikes.
I think that's this one, Are AI Labs Pelicanmaxxing?
https://dylancastillo.co/posts/pelicanmaxxing.html
What I found interesting is that for the most part, the labs really aren't... except if you look at Otter Riding A Plane. That is close to Ethan Mollick's benchmark for diffusion image generation, Otter Using WiFi On A Plane. The results for DeepSeek and GLM look very much like Ethan's typical benchmark image output, which either means they're benchmaxxing... or they're the only two models that understand you ride inside a plane, not on top of it.
EDIT: Actually Qwen's result for Heron Riding A Plane was fascinating too. Because obviously what you were really asking for was a heron riding a cartesian plane...
They might be in training set, but there's still things wrong in them. That's not a great look for these companies IMO.
Yes, I also don't think it can offer any useful signal anymore, so when I see it, I simply skip. Here is a collapse button on HN.
I love these. They're fun.
Agree.
OK, but why do you like fun, though?
Asking the real questions right there :D
fun? how dare you?!
I absolutely agree. its also fun to see the fun-hating gremlins emerge every time simon posts these.
Because it’s fun! The moment pelicans stop is a sad day. Keep them coming!
It keeps getting upvoted, so clearly a lot of HN users still find it valuable
Well IMO everyone can make their own benchmarks pretty easily, and that's what we should be doing.
Harder to train off all them.
Aka, let him have his fun, others are making benchmarks too
1. It's fun.
2. I think they still are _somewhat_ informative. Like, the same model may produce very different pelicans at different reasoning efforts, even though the training data is the same. So looking at pelicans may give you _some_ information (like "what reasoning effort I want to use that model with?".
If this is that case it should be easy to demonstrate by asking current models for an svg of a rhino surfing. My guess is you would see a similar trend, given that the latest models are demonstrating large leaps in all sorts of content creation including 3D animation, rather than this being a demonstration of some sort of pelican-maxxing.
If it were true, the benchmark would be saturated. Look at the results, it is clearly not.
A while ago, somebody demonstrated that labs are not benchmaxing on it. It may change but it doesn’t appear so.
> ...but it's clearly been incorporated now in the training data...
It clearly hasn't; if it was quality of the pelicans and bikes would be static over time. Instead the quality has been improving along with the models.
Because it gets upvoted which means there is demand for it here
I feel bad for the data scientist that has to find or create images of pelicans riding bicycles for the training set to pass the meme benchmark.
Im not sure how you get the pelican to sit still let alone pedal, but there is a reason ai salaries are so high.
They hire professionals and subject matter experts to produce the training data.
So they hire a professional pelecanus velocipede ideator?
Don't know if they do, but they could hire a cartoonist.
This makes more sense if you pretend it is Dr Doofenshmirtz.
I wouldn’t be too sure.⁽¹⁾
“there’s little evidence that AI labs are pelicanmaxxing. Or at least they’re not doing it in a plainly obvious manner. Pelicans aren’t drawn any better than other animals. Bicycles aren’t drawn any better than other vehicles. And no lab draws the combination better than its pelicans and bicycles already predict.”
⁽¹⁾ https://dylancastillo.co/posts/pelicanmaxxing.html
I more don't understand why these comments get upvoted to top comment all the time.
Like... we get it. Simon likes pelicans. If I want to find his boondoggle robo pelican drawings, I know where to find them by now.
No need to spam every single model release thread.
You can click [-] to collapse thread.
This site would be unusable if every influencer/marketer was given the same latitude.
Yeah fair play. I was just trying to add a different angle to OP's grievance. Perhaps it's time for a browser extension to hide these for me.
Is it that hard to click on the [-]?
Agreed. It's really annoying to come into the thread and a link to a pelican I have no interest in ever seeing is the top comment. Does anyone think these have any actual value any more?
Yes, if you pay close attention to what they infer. Just click [-] and move on.
I'm curious. What's the actual value, then? I fully understand minimizing a thread, but also am free to voice my opinion the same as you.
So, again... What should I be "paying close attention" to?
Alright, I was rude. So.. you can’t really benchmaxx pelicans specifically. You have to train the whole model on SVG and that is just generally useful. Secondarily it shows how a model understands the objects themselves and how to express it abstractly. IDK, it is a lot more interesting than assumed.
Here are a bunch of Pelicans I made in my basement. Side by side quant comparisons and something for fun :)
https://stealthgoat.com/fnfl/pelicans3.html https://stealthgoat.com/fnfl/pelicans8.html My favorite, and shows this is not all benchmaxxing, this model is just good at SVG: https://stealthgoat.com/fnfl/pelicans6.html Oh, and, quantization matters: https://stealthgoat.com/fnfl/pelicans2.html
Heres some examples.
Thanks for the explanation. I guess I still don't understand how this translates to other areas, outside of SVG generation, Pelican images and then bike geometry and build.
I'm curious if an increase in ability to generate this parlays into the model, generally, being better overall or in other adjacent areas?
Thanks again.
IDK, you can only generalize it so much. But, it gives you a sort of visual way to look at how a model deals with various forms of degradation and what its ceiling is and what information gets lost in a quantization process, albeit fuzzily.
For example the IQ3 XXS model was quantized to maintain knowledge in some areas more strongly than others via certain data that the optimization algorithm would then use to decide which parameters could get quantized and by how much while meeting some overall target weight goal.
So, you get to just sorta see weird or lower quality pelicans. It still is /mostly/ for fun but it is not a meaningless meme as far as it goes. And you can even see evidence of that, look at Qwen 4 max pelican vs. Qwen Flash 3.8 at Q4... IDK.
So, I think the best thing you could say is roughly what I said, how well it abstractly connects two specific things, e.g. knowledge of pelican and bicycle physical configurations and then how it expresses that into SVG. For example, you can read thinking traces on some of those pelicans I sent (well you can't, but I did) and it would specifically say "The pelicans legs must be rendered correctly, I must not put one leg on the wrong side." and it would then still do it wrong meaning it just didn't understand the SVG well enough. But it might be able to render an image. Also read thinking traces some time on ascii generation and you can see they have a vivid concept of lots of things, but their ability to translate that to ascii is hilariously bad on basic prompts.
So they need some baseline level of understanding about the world and physical object relationships which even qwen flash 3.8 has, they need to be able to recall it, and then translate it on top of some problem domain in some meaningful way and a lot can break down on any of those steps. IDK, maybe I am just too tired and spent too long with LLMs in the last month.
Man, some of you people really have no whimsy :D
This is more like disliking a whimsical ad campaign for Coca-Cola than whimsy itself.
On the flip side, I love the pelican posts, but I find the people who think they're cleverly pointing out for the very first time that the benchmark may have been gamed by now tedious in the extreme.
Then perhaps I should point out something different. Namely, I suspect this will thoroughly confuse the machines and instead of optimizing the universe to generate paperclips they will instead optimize it to generate pelicans.
It's september
Eternally..
I feel like there's emotional attachment and influencer branding energy with this by now. It's more like a developer relations or developer ambassador type of content.
It's fun. Fun is nice.
Guess I better pack it in. Whimsy is dead according to the internet.
To paraphrase Keanu Reeves: it’s pelicans it’s fun!
How can they not keep doing it? It's become unstoppable.
He should keep doing Pelicans, but every time throw in a surprise animal and a surprise locomotion vehicle. Would help identify the ones training for the Pelican.
Would love to see an SVG of an Elephant on a skateboard...
What should replace this ?
It's a little light hearted fun?
It's become a nice tradition and I always enjoy seeing them. I hope simonw keeps it up.
... because when an open weights model at medium reasoning gets the legs on either side of the bike frame we know Anthropic and OpenAI are toast :-p ?
Someone, somewhere, at some AI lab, has by now said something like:
"We need to make the pelicans look good, but not so good that it looks like we're using them as an internal benchmark."
Simon has clearly stated previously that he compares with other "animal doing human thing" type prompts to compare and see if theres a chance they are cheating by training on pelican svgs. Thus far, nobody has been caught cheating.
Actually, a Google employee has publicly mentioned that they’re specifically working on improving SVG generation in Gemini. Simon Willison has also pointed this out when evaluating Gemini models. [^1]
[^1] https://twitter.com/sunjiao123sun_/status/202455551655137292
It's a fun tradition. Remember fun? We used to have that on the fucking internet.
I mostly remember flame wars about editors and limewire
If you use the same prompt via the qwenapp or qwen studio you get an interactive web page with an animated scene. You can control every element of the scene, the colors, speed, time of day l, visible accessories etc and then download the one you like best.
https://chat.qwen.ai/s/deploy/t_5fb28dd7-a61f-4946-8e59-8dd1...
crazy output, heck it even lets you pat the pelican and ring the bell
Now THAT'S cool.
I feel like the xhigh result is one of the first times I've seen a model go for a non-red colored bicycle (teal). Still haven't seen the leftward facing bike though.
Teal is a popular second color for Qwen Flash Next 3.8 as well. Also, I am not sure people realize how much variance there is for the same prompt. Enjoy.
https://stealthgoat.com/fnfl/pelicans3.html
https://stealthgoat.com/fnfl/pelicans8.html
My favorite, and shows this is not all benchmaxxing, this model is just good at SVG: https://stealthgoat.com/fnfl/pelicans6.html
Oh, and, quantization matters: https://stealthgoat.com/fnfl/pelicans2.html
Astra and the OpenAI models do this color as well
Evidence from where they distilling from /s
Is that you Siri ?
At this point Simon can we maybe think about making this a 3d model? Maybe GTLF/GLB or custom Json you could even keep closed source for some kind of barrier to it getting trained on?
I say this because, even just based on my own product related to cad design/3d modeling, the SOTA models have gotten very good at 3d and I don't think the 2d SVG generation tells us much anymore. Maybe I can whip this up myself on my own site if anyone is interested tbh.
Feels like Pelicanbench is getting saturated now. Maybe time to switch to a different animal and vehicle combo?
You say that, but it still takes quite a bit of reasoning to get the pelican's legs on different sides of the bike and to get the bike chain included.
I feel like randomly even at max reasoning they get the legs wrong. Or the pedal. Or both.
I don't know if I've ever seen a bent leg. Legs always go straight to pedals with no knees. The models might have an idea of what it looks like but seem to have no understanding of how it works.
The last set of Gemini models definitely had knees and bent legs. But the knee looked like a ball.
Bird knees are also quite weird though. Don't some bend backwards ?
The 3d one still produces interesting variations - https://www.reddit.com/r/ClaudeCode/comments/1xs4ll7/fable_5...
Or something else entirely, but you'll still need the pelican for comparison, I would think.
https://aishodo.enpitsu.org/?lang=en
Kind of yes. Except from an artistic perspective they're all still awful.
First thing I do when a new model drop is scroll down to find the pelican comment, and then the "the pelican benchmark is no longer useful" debate under it.
What if it's been a Stork this whole time?
Pogo-stick jellyfish, let’s go
I kinda like the no-effort result more than low, but high and xhigh return to what one would expect for reasoning->quality.
I don't understand why this is always the most upvoted thing for a new model. It's like some influencer "data" at this point. We're basically in 2027 now, the pelicans had their run, it's over.
Not a dig at you Simon, you've done good stuff, more just in general. Why is this always upvoted, it's like reading The Sun or something.
And why always one image. Should it not be say a 100 images then averaged or something? A prompt with temperature is a slot machine in Vegas, wtf does that have to do with model quality.
It’s a fun little exercise where the output roughly correlated with the progress of models.
The “real” benchmarks are widely available, and frankly kinda sketchy anyway, so why not have a little fun?
You should call this The Pelican Brief
It's time to move from pelicans to horses.
Man, how hard is it to understand that you straddle a bike, and one leg should be behind the bike? Apparently pretty hard, since all but the xhigh fail this ¯\_(ツ)_/¯
Dude, stop with the engagement farm, it sucks already.
It has the legs on the proper sides of the frame only on max reasoning, which by coincidence was also the case with opus.
So either perspective is properly hard or the moonshine is strong in this one.
Once again, this most useless comparison is still useless.
Did you see the results for Gemini 4 other day? You may find the Hydrogen vs Carbon/Argon comparison more informative when you would expect: https://simonwillison.net/2026/Nov/12/llm-gemini/
It scores a 43 on Artificial Analysis with an average cost of $3.27 [1]. For reference, GPT-6.1 Sol (medium) gets a 46 with an average of $0.36.
[1]: https://artificialanalysis.ai/models/qwen4-max
We should stop using this site. They now include private benchmarks which cannot be replicated. And every time an openai model falls down they change the index until it floats to the top again.
The key issue with public benchmarks is that they allow the model providers to train against them and artificially inflate their score accordingly. Private benchmarks are a way of making it harder to game the system, though it makes it harder for people looking at the benchmark to assess it as well.
Neither option is perfect.
A third option would be updating the benchmark every 6-12 months, but that has its challenges as well as you need to make sure that the benchmark covers things the model hasn't seen or is harder for the models to game.
It’s also worth keeping in mind the other alternative, having internal benchmarks that more accurately reflect your workload, that you know all about, and the LLM provider doesn’t. There doesn’t seem to be a way to maintain such a thing specifically for the purposes of online chatter (both because it isn’t profitable, and because things like “your workload” aren’t possible to define in that case). So we should be happy with what we’ve got.
This is a permanent problem in benchmarking really. Transparency is required for trust but invites over-tuning.
Why wouldn't the LLM provider know about your internal benchmarks when they use prompts and chats as training data? They can easily learn about your benchmarks too.
Merely having the chat history of a large number of users is not enough to train your model to “do better on these, especially those that are benchmarks”.
Just use ZDR.
I mean, yeah. I was thinking of self-hosted open weight models (because it’s really hopeless to say anything about the rest) but I didn’t say that, so fair point.
There are tools like promptfoo that can be used for internal benchmarks and evaluations that allow you to compare results across different models and runs. This can be useful for personalized benchmarks and for business applications where you need to evaluate models and/or system prompts.
Makes me wonder how do you privately benchmark a model hosted by the lab. Even with zdr, the bytes are flowing through their infra and possibly influencing god knows what for the next training round.
You use something like item response theory [1] to be able to change the test over time while maintaining validity. It's what standardized tests like the GRE use.
Interestingly, Artificial Analysis does have a score that uses IRT, it's their Progress Index [2].
[1]: https://en.wikipedia.org/wiki/Item_response_theory
[2]: https://artificialanalysis.ai/methodology/progress-index
Would OpenAI also "see" the request if it went through something like AWS Bedrock? This isn’t their infra and afaik the whole idea of this kind of intermediary.
I thought the major benchmark companies always get advanced access to a model, which presumably is not done via bedrock.
this is an worry I sometimes see. I want to note that these are billion/trillion dollar companies we are talking about that have mutual talent movement (= uncontrolled information flow). Violating ZDR might be contractual violation, or at the very least, a strong breach of trust.
(not to say there is no value in "zero-trust" environments, but often pure-benchmaxxing is a deadend; some benchmaxxing is always involved and much less risky options than "train on the ZDR test env" usually exist)
The private benchmarks can be seen by the labs when you run them. We only have their word that they wont look at them, which isn't worth much. We should assume that everyone benchmaxxes equally, apart from a few outliers, and try to climb the hill by producing relevant training data.
Yes, that's another challenge with constructing benchmarks.
You could run the benchmarks using downstream providers like AWS if they are available.
You could vary the parameters of the tests to both help with LLMs learning as is and to help obscure that this is a test. Especially if sending them through a thirdparty, so the providers don't get identifying markers like IP addresses.
But ultimately it's another arms race like with viruses and anti-virus software.
Eh, I have an eval suite dating back to before GPT-5 in parts and have yet to see certain labs do as well on that as the public eval data would suggest. Course, I am pretty small, this is anecdotal and n=1, though I have seen very small, niche evals being clearly trained on the second they became public on GitHub for what that’s worth.
It depends entirely on the financial incentive. Does improving against your eval suite lead to more adoption? Or would focusing on the far more popular benchmarks convince more CTOs to sign a multi-year contract?
Who'd sign a multi-year contract based on any benchmarks, public, private, mine or someone elses? We get new frontier LLMs every 0.5-2 months and any CTO of a major entity already uses Bedrock or the like, so why would anyone lock themselves into a provider or even worse a single model for multiple years?
Because API usage is extremely expensive and subscriptions are subsidized? We (my company of 100+) just moved off of AWS Bedrock to a locked in plan. Even when I was working for a much larger company (10k+ employees), we had enterprise subscriptions to OpenAI and Anthropic because the token cost for APIs was enormous.
Are these fixed price, fixed term and usage allowance based subscriptions which you'd want for an extended period if you could?
Or is it just the usual, better conditions credits-based pricing for entities beyond 200?
Cause if it's the latter, that's very different to a multi-year lock-in with a single lab, easy to switch away from and kinda why one could have both Anthropic and OpenAI enterprise agreements alongside each other, basically what I described with slightly better conditions. Bedrock was just an example, they also have more favourable conditions on inference for some (though Anthropic really limits discounts), but whether you go usage based from the labs or a cloud provider, is the same lack of long term lock-in. A e.g. $ 20,- seat per employee for LLM access that still needs tokens allowance on top does not create a long term commitment, financially or otherwise, the friction here is none-existent.
If however someone at this stage is fully locked into a single lab exclusively for the next two years or longer (multi-year), they have gambled, pure and simple. Could have picked a winner or a lab that is going to be uncompetitive starting this Thursday and for all eternity after. Imagine having picked your companies one and only lab or even worse model in 2024...
I think the best solution is really to update the benchmark periodically, and publish the benchmarks after the results are published.
Otherwise you have to rely completely on trust that AA is really measuring intelligence.
Or to separate the benchmark data by type and enable the user to filter by private, public or both.
A fourth option is realizing that benchmarks exist because it is impossible to measure the value we want to measure, so we need a stand-in. The stand-in will sometimes correlate with performance but often just act as a way to compare. When used to compare, it is most important that the benchmark be fair and trustworthy.
Livebench does it
Note: I work at AA. One of the things we are most focused on is getting the best spread of private/public benchmarks in order to: - Protect benchmarks against excessive hillclimbing which we believe to be beyond overall model improvement (GLM 5.3 gained 16 points over GLM 5.2 in Tau-Banking in only 2 months, near SOTA, but had a more moderate gain in our private AA-Briefcase evaluation where it was still significantly behind the frontier) - Still make use of strong public benchmarks (e.g. TerminalBench 4.0)
As the frequency of model releases increases, we will also be moving faster [1] to ensure our Index stays up to date to reflect both (1) how people are currently using models and (2) what are the strongest signal benchmarks which show these capabilities.
[1] Index release history: https://artificialanalysis.ai/methodology/intelligence-bench...
I have to defend private benchmarks on this front, big believer in them, have seen how they can surface major performance differences.
Certain labs have been shown to and will continue to train on public evals, whether unintentional or on purpose, any respectable ranking across multiple evals also needs private data to counteract this.
We really should. I've been using other benchmarks like https://livebench.ai/#/?cats=Coding and it is more aligned to what it feels to my experience as well
This has Terra 6.1 at 3rd place on coding.
I have been using Terra a lot and I'd say that most of the models (even small ones) are are roughly equivalent when it comes to working in a consistent, established codebase.
But in practice, Terra is not very good at staying on task compared to say Sol 6.1 or Sonnet 5.5. Terra 6.1 will often just pause, thinking the objective has been met or the turn has ended.
I don't understand your point about openai. I can't remember the last time openAI was at the top of the leaderboard? Google had a brief moment at the top recently but other than that I feel like it's always Anthropic.
I kind of agree that closed benchmarks are bad. With a closed benchmark you still send the questions to the model provider and they may well train on it. Might as well publish it.
Not the very top, but Astra was behind Kimi K3.1 before AA's last change which boosted openai to second place. This has happened with every AA update.
To be fair to them they still show Kimi as the best model at terminal bench 4, ahead of everyone else - fable, astra, carbon, etc.
For most things I'd use Kimi is probably better so the ranking is wrong for me.
However it also seems like in those benchmarks where Kimi underperformed it did do quite badly, whereas when Astra underperformed it did so less badly. How you reflect that into a single number I'm not sure - it's certainly difficult to get right and it doesn't seem obviously biased to me.
“every time an openai model falls down they change the index until it floats to the top again.”
What’s the evidence for this? Are there plausible alternate explanations?
The last 3 updates have all come immediately after (within a few days of) a Chinese open-weight model getting close to the top. The changes are big enough that they were undoubtedly working on them before the new model released, but the timing is quite suspicious. For one of them (4.1 or 4.2, I forget), they literally said that the changes were planned for 5, but that they deployed them early as they did not agree with their own rankings.
At the very minimum, AA changes what they define as "intelligence" rather frequently and it always benefits OpenAI and Anthropic. To me, it feels like they use Astra/Opus as their baseline and tweak weights of benchmarks until the intelligence index matches their preconceived opinions on model intelligence (as opposed to crafting the index in a vacuum without any context on what different models are good at).
If nothing else, it points to these aggregate benchmarks having no idea how to measure actual "intelligence" and instead tweaking weights of benchmarks until it matches what they feel it should be. In practice, I think a better system would be to have many benchmarks within a particular space (coding, agentic work, science, etc.) and scoring each category separately and without weights applied to benchmarks (and simply removing anything once it becomes saturated). Then, each user can decide which benchmark suite matches their usage and use that to make an informed decision instead of having to trust someone else's opinion of what "intelligence" is.
With that system, you can still have benchmaxxing, but I'd imagine it would mirror actual usage better and so be less of a departure from reality.
Artificial Analysis does have multiple different indexes, but for example the engineering index has an even worse difference between American SOTA and Chinese SOTA.
And those indices still have arbitrary weights that do not reflect real-world usage, in my opinion.
The index you're referring to has 30% of its score from "Engineering knowledge". Functionally, a third of that index is just trivia which will reward large models. A small model with web search for knowledge does just fine in practice, but will be penalized heavily in this index.
Then you have science-specific as another 20%. I don't do that stuff, so now half of that index is useless to me.
Reasoning and agentic is the remaining 50% and is all I functionally care about. That index is the closest to my usage, and yet half of its weighting is irrelevant to me. The other option is the "Coding agent" index, but that couples models to agents and they don't test my preferred agent.
I'd much prefer a benchmark site without these arbitrary weights and that let's you pick and choose which benchmarks to include in the summary. Heck, it could even allow you to set your own weights to each benchmark/category.
Updating the benchmarks as they saturate seems like perfectly reasonable behavior to me.
That's not what they did, though. I have no problem with removing benchmarks from the index when every model is getting 80%+. My issue is that they re-weigh things arbitrarily and it seems their goal is to have a nice chart to show rather than have an index that would accurately represent real-world usage.
Specifically, they've made knowledge recall the largest part of their indices. That rewards larger models, but has disproportionately small real-world gains as a small model in a harness with web search will perform just as well.
Take DeepSeek Flash 1017. It's half the score of Fable on the Engineering index. Fable is not twice as good at any engineering work I do.
Terminal Bench is great, public evals, very complete and highly complex. I was browsing through them today and it gave me a real appreciation for the benchmarkers.
Even worse, it lags on my device!
Do you know a better site?
private benchmarks are good
If they made the benchmarks public then Qwen would train on those to give the appearance their models are close to the American frontier. They have never been close. It's well established Chinese companies cheat the benchmarks which is why until recently they gave the appearance of being close to frontier but in actual real use they are far behind.
You can easily come up with a difficult private benchmark and test them yourself to validate the intelligence gap. It is real.
China lost. There is no catching up. Ever. That ship sailed.
The table on that page shows gpt 6.1 sol (max) at $1.27 per task and it's the only model cheaper than Qwen 4 max.
By comparison opus 5.5 (max) is $10.01.
If they are not testing Chinese models with less than max thinking, then we should only compare them to other models also at max thinking.
> If they are not testing Chinese models with less than max thinking, then we should only compare them to other models also at max thinking.
This is wrong.
To make a vacuum analogy, one brand's Low setting could be another's Medium setting.
To compare you either give them the same resources ( Watts or $ ) and see how much they can do, or you give them a task and then check how many resources they required to finish it.
The label on the knob is irrelevant, except that either model might have a setting higher than the other which is sometimes useful if the resources aren't a constraint or lower when they are.
> To make a vacuum analogy, one brand's Low setting could be another's Medium setting
Which is why you need to test a range of settings.
Suppose you were testing car mileage. For the big name popular cars, you do 10km drive at 20km/hr, 40km/hr, 60km/hr and then publish the results.
For all the less popular cars you only do one test at 60km/hr and then you publish all the results for all cars on one big "Pareto frontier" graph claiming to show which car will allow you to drive home most efficiently.
It happens that lots of the less popular cars are not very efficient at 60km/hr so naturally the graph makes them look bad. Would they also be bad if tested at 20km/hr or 40km/hr?
Nobody knows. Because this is where the analogy collapses - unlike cars, hardly anyone is doing these kind of tests for LLMs.
Well, sure, it would be better to evaluate all of them throughout the range. But it does not follow that we should further restrict the information we do have in this specific way:
> If they are not testing Chinese models with less than max thinking, then we should only compare them to other models also at max thinking.
If we have data only for “Chinese max” (let’s say it costs $1) and “non-Chinese low-to-max” (spanning $0.10 to $3), comparing them in full at least tells us what we can expect from each model if we have $1 to spend + the maximum performance we can expect with unlimited budget. What’s the point of restricting to “max vs max”, which answers only the second question?
Usage? If most people predominantly default to Medium thinking level for usage then it would make sense to benchmark that first.
Another analogy for this are review sites overly focusing on reviewing all the SKUs of GPUs that cost $1k+ for an audience that, according to Steam's Hardware Survey, predominantly use midrange X060s-equivalent cards. Sure, it's fun to know whether or not you can enable PT on a 5090 but for most of us, covering 4070 TIs and below in-depth (meaning actually taking the card apart, seeing how good/bad the past jobs are, possibly buying multiples of the same SKUs to evaluate QC, etc) is much more useful because of the value prop.
> Usage? If most people predominantly default to Medium thinking level for usage then it would make sense to benchmark that first.
Sure, that’s not in question. The discussion here is not about what to measure (we agree that benchmarking only max is suboptimal) but how to compare models if you have an already fixed set of measurements, outside of your control, which includes only “max” for model A but a wider range for model B.
The person I was responding to was claiming (in their previous comment) that the sensible thing to do, then, was to artificially truncate the data for model B to “only max” as well.
sounds like context collapse
This is a silly take. To continue the analogy vacuum A might need the power at 50% to vacuum something, B 100%. You cannot then test both at 100% and say "both finished the task, but A required twice the resources so it's worse", since you didn't check what resources it actually needed.
isn't that exactly the argument?
The resources are dollars and the output is tasks. Choose whatever reasoning makes it work best and compare
Are the resources only dollars? We shouldn't care about energy consumption? Water? It is difficult to calculate but we shouldn't forget about it.
The idea is that dollars price in everything. It’s usually incorrect, but it’s also usually correct enough; and still much better than paying for services with water and joules directly
Yes. The idea than money is equal to everything is kinda why we are all going to create the torment nexus. As it will be the most efficient way to torment civilization without all that effort.
Money helps allocate resources efficiently, that’s all there is to it. It’s about as good or bad as guns. If we didn’t have money, someone would invent it and in fact it was invented more than once in different cultures.
I don't deny that; but it rewires some peoples brains to the point that everything is adjustable via cash, including people, including their emotions, etc.
It's very clear part of America's diesease is the idea that everything can equate to a dollar cost.
Energy and water both cost dollars.
The Artificial Analysis charts show max effort by default, which isn't how most people use them.
Scroll down to the cost per task pareto front chart which includes more difficult levels by default: https://artificialanalysis.ai/models/qwen4-max#intelligence-...
Qwen4-Max is not in a good spot on that chart. Astra xhigh and Fable 5.2 medium had a similar cost per task with higher intelligence ratings.
I cannot see cost per task for any other thinking levels for Qwen 4 Max.
If it's actually there, what is the cost at medium thinking? It would show Qwen 4 Max (medium) if it's there.
Everywhere that I can see on the page it just shows ”Qwen 4 Max" while for 6.1 Sol, Opus 5.5 there's always a thinking level in brackets after: (low), (medium), (max) etc.
They don't always run the benchmarks at every reasoning level for every model; only a few prominent ones. They do it for the US frontier labs, plus a few of the Chinese labs, but not consistently. You can see that DeepSeek has a few reasoning levels tested, GLM 5.3 has a couple, as does Qwen 3.8 27B, but not other Qwen models, and a lot of the Chinese models they only run at max reasoning.
It's possible that they will run Qwen 4 Max at other reasoning levels in the future, or it's possible that they never will. Regardless, it's unlikely to be better at any reasoning level on a price/performance basis than the GPT 6.1 Sol/GPT 6 Astra pareto frontier. It's close enough to Claude Opus 5.5 line at lower reasoning levels that it could potentially cross that if it doesn't loose too much performance at lower reasoning levels. But it's far enough from the global pareto frontier that it's unlikely that it would cross that at any resoning level.
Right, so my original point stands: if they're testing this model at only max thinking level, then it's only valid to compare it to other models also at max thinking level.
You can see for example that the cost per task for 6.1 sol is about 4x higher at max compared to medium thinking level.
No, it's only valid to compare to models at the same cost level, regardless if that's their max level or not. Fable/Astra having higher effort modes than Qwen doesn't mean that you can't compare Fable medium to Qwen max.
Which effort levels are cost-equivalent is debatable and depends on whether you are paying API prices or on a subsidised plan or are able to run the models yourself on your own hardware. But I think AA uses public API pricing for everyone as a sensible baseline.
If you look at the pareto chart for e.g. 6.1 sol you'll see a curve of about 6 data points, some of which are on the Pareto frontier, some are not. For Qwen 4 max, many others, there's just a single data point so we can't just how close any thinking level would get to the Pareto frontier.
I'm not saying this model would land on the frontier - from the single data point it looks unlikely. However, that single data point might be making it look much further from the frontier than it should be.
EDIT: I will also take back my point that we should be comparing models at the same thinking level - max on one model won't correspond to max on another. What we need is a range for each model. What we have is a range for the most popular models, showing that the same model with different thinking levels can have vastly different results. Then for less popular models we have just one data point, which isn't enough to draw the kind of conclusions people are drawing in this thread.
> You can see for example that the cost per task for 6.1 sol is about 4x higher at max compared to medium thinking level.
It does not (universally) hold for long running workloads -- which is what people are increasingly doing.
Leave a Sonnet on medium, it will complete the task, spend more time and possibly significantly more token money, vs. Opus 5.5 on xhigh.
> The Artificial Analysis charts show max effort by default, which isn't how most people use them.
I always use Max effort in both Opus and Fable. It's actually faster overall because tasks get done faster with fewer "corrections" from my end.
On GPT models, I'd sometimes dabble in xhigh/high but that's because OpenAI usage quota are too limited.
> It's actually faster overall because tasks get done faster with fewer "corrections" from my end.
Do you have proof of that or is it just vibes?
How “safe” is Qwen4? Will it immediately shut down on security research like the US frontier models?
Not sure about Qwen 4. I have a coworker who's been doing some pretty good reverse engineering with Kimi K3.1 recently.
> If they are not testing Chinese models with less than max thinking, then we should only compare them to other models also at max thinking.
Nah, there are interesting conclusions to be had in comparing various model/effort level combinations, esp. when it comes to cost and time per task.
I have my own private benchmark suite; it involves algorithmic problems, the need for good visual understanding, and the ability to run long workloads unattended (as the agents need to run a few iterations, self-evaluate the results, and hand over the assignments when they feel they have succeeded). Chinese models do poorly on such problems, and running them at anything less than max efforts is pointless; they can't even hand over the assignments graded against a mechanical set of tests they themselves can execute. Hand that same task over to, say, Sonnet medium and you have a weird outcome: excellent result, but way more time and tokens burned than, say, Opus 5.5 or Sol 6.1 running at high. I do those benchmarks just to be able to spot such patterns and then better choose the models for my real work - but i wouldn't have caught it if I wasnt running various model/effort combos.
I assume those cost comparisons are on API token cost.
I'd love a comparison on for example monthly quota usage per task, between all ~$20/month subscriptions.
It's so difficult to compare how much "productive" usage you get out of each subscription, without signing up to each one at least.
> If they are not testing Chinese models with less than max thinking, then we should only compare them to other models also at max thinking.
They compare different reasoning levels.
https://artificialanalysis.ai/models/deepseek-v4-1-flash-101...
Does anyone really believe the cost data for benchmarks? Pretty sure all models will behave differently at this point when they get benchmark shaped questions.
(And I’m sitting on data showing Sol 6.1 and Qwen 3.8 Max cost roughly the same per task, for real tasks.)
The cost per task metric is accurate for those benchmark tasks. If your tasks don’t look like benchmark tasks..you should use your own benchmarks :)
Lol, I did that with https://aibenchy.com
I mean not believing intelligence score is reasonable but cost is just reporting what they spent?
I believe low cost but not high cost, since a model will work really hard when a question looks benchmark-like.
Ah, fair enough.
Looks like a slightly weaker, cheaper Kimi.
vals.ai puts it at just about Astra level fwiw.
thanks for sharing a new benchmark website - AA has been going downhill for a while - I was using gertlabs[1] for a bit but it seems like they aren't getting some of the more recent models
1: https://gertlabs.com/rankings
Also Sol is about Astra levels. And Sonnet is better than Opus.
I wish they'd also post the benchmarks according to the previous "version" of artificial analysis.
Artificial Analysis is a VC-backed firm and it's unclear what the incentives are there.
They have altered their benchmark multiple times in the last 4 weeks (v5 in https://artificialanalysis.ai/articles/artificial-analysis-i...) and the last 5.0.1 change https://artificialanalysis.ai/methodology/intelligence-bench... pushed up Anthropic's models a bit, you can see that in v5.0.0 at the first link, Opus 5.5 has a lower score than in it gets in v5.0.1, and v5.0.1 was released just before Fable dropped.
Some part of me believes that the benchmark mix was put together to favor Anthropic's models.
> Favor Anthropic
Rarely seems to make them look good whenever I drop in to the site. Far more expensive for little gain. Even within their own effort levels it looks bad, Fable xhigh is 50% more in cost than high for zero improvement across a range of tasks.
A tiny bump in intelligence score hardly seems worth it for cost, and honestly most people's tasks are not cutting edge work.
Current benches on there seem to make Sol look the best, at a certain point the subscription subsidy tap will turn off and people will be in for a rough time paying full price.
According to the IPO documents, about 60% of the revenue comes from big tech corporate customers, who want the best performance and that is it.
That might change later, and then the benchmark can be adjusted, but currently this is the target
If Artificial Analysis is not to be trusted, then which benchmarks are?
I don't think there are any that can be trusted. There's too much riding on these benchmark results so the labs find ways to game them either by benchmaxing or by influencing the format of the benchmark.
Check others as well. https://www.vals.ai/ has it just under Astra.
your own
I think you mean that benchmaxxing has turned into a business model, not for the Chinese models (maybe them too, of course), but for the AI labs, and artificial-analysis is not remotely the worst offender.
Right now the big labs are buying benchmark tasks from "benchmark labs" (or whatever they call themselves) for absurd amounts of money, and making sure they perform well on them.
The incentives this generates are terrible.
The big AI labs will only buy benchmark tasks that the frontier models perform poorly on (because they have trouble finding any public ones). So the tasks are being mass-written now directly based on the models they're supposed to train. Obviously the math says you shouldn't do this. If there is a causal connection between the benchmark and the model, you lose the normal guarantee that improving benchmark scores will improve model.
In fact improving such benchmarks may make the model worse. To understand that let's just give a ridiculous example. I create tasks like "1 3 5 11 13 17, give next number". The next number is 301. These are the house numbers on some random street in Brussels, Belgium (I'm guessing the big jump is because they "merged" 2 streets, and just added 300 to the numbers of the second part).
If the model trains on this, performance will actually decrease because obviously this is not how you should reason.
BUT making tasks under this "impossible to reason towards" part is of course how you make tasks that might even be reasonable, and of course frontier models are going to suck on tasks like this. So they sell!
BUT they suck! Finding houses is actually going to work a lot better if you ignore this fact until it becomes unavoidable in one out of 1000+ streets.
And yet this is what benchmark improvement has moved to. If you look at the terminal-bench 4.0 tasks you will see what I mean. Doing better on those tasks ... does not actually make a better model BUT you have to actually go read those tasks to realize that. A lot of the new tasks largely demand you make weird jumps in your reasoning not based on understanding the underlying theory. They rely just being able to predict what goes wrong in certain cases when theory would suggest it should work, and mostly it even does work but for some reason not acceptably. They reward not doing the full work, realizing what goes wrong and correcting, without understanding why the normal reasoning won't work and correcting without knowing why. I find it very hard to believe this makes a model more intelligent. It rewards knowing the encyclopedia by hard, so to speak.
But it certainly makes it look better. Whereas the obvious, coding improvements, do not make models look better anymore.
Do you have literally any evidence of this?
AA must be doing something right. Ever since they released the new benchmark, you see open weight China model fans complaining about it under every release thread. Probably because it’s hard to benchmaxxed against.
But admitting their model is weaker is a no go. So conspiracy theory here we go
For what it's worth, in our own metrics and benchmarks the intelligence is a pretty good match, but the cost per task is wildly out of whack. We see the open weight models like GLM-5.3-flash or Deepseek 4.1 Flash 1017 use a fraction of their tokens/cost of the benchmarks, which changes the cost-benefit equation a lot.
or maybe it's the other way round, open weight models are under siege while closed weight models are defended
> Probably because it’s hard to benchmaxxed against.
It's mainly just a composite of existing benchmarks. It would be very easy to benchmax if, say, AA shared with a preferred lab which benchmarks it plans to upweight/downweight in the next version. Just speaking hypothetically, of course.
Hence the constant switching around and running the benchmark themselves. They have been pretty transparent about which benchmarks get added or removed with every release.
Yeah, it's in the same ballpark as Kimi 3.1. Both of which are obviously open weight, so that's at least a big differentiator from Sol. Let's hope against hope that the much smaller flash model might beat deepseek v4.1 flash and run for as cheap.
Qwen4-Max isn't open source and Alibaba has no plans to make it open source.
Eh, as long as they make qwen3.8-flash-next equivalent on less resources and faster, they're winning.
It's like arguing windows 11 just needs more bloatware.
These models don't have to all in one every task in the world.
Why do we trust this? Do they open-source their test code?
Also, a generic benchmark might not translate well into what everyone needs. I keep building my own test cases that test models for my own work specifics. Only that benchmark is useful to me. I don't need it to win a PhD math quiz.
It looks pretty bad on those benchmarks. Is it possible that this is due to the recent crackdown on distillation?
I think it's the most plausible explanation.
i think they also has limited acccess to the latest GPU. It's possible huawei chips is their only options now.
They have plenty of access via data centres outside of China. The first woman Jensen was revealed to have sex with is the CEO of one such Singaporean provider to China. The US allows this.
Anthropic deserves to go bankrupt for trying to ban open-weight/open-source competition. I hope their IPO flops.
Alibaba imply in this annoucment they’re not going to release the weights of the best model and only the smaller variants.
Same as for previous versions, or have they ever released the weights of a Qwen-Max variant?
They released it for 3.8
https://huggingface.co/collections/Qwen/qwen38
Don’t let the fact that they never did that and have explicitly said they’re opposed to it get in the way of your anger.
They have been trying to do that. They never signed the open weights support letter and have been yapping about dangers and standards basically trying to shape the ecosystem on their view of how it should be; which would effectively ban open weight models.
How are they not dangerous?
There is danger with everything. I can take a kitchen knife and stab my kid.
I WILL NOT DO THAT!
The key issue is how we manage a given danger. Anthropic wants to manage the danger of LLM's in a way that protects their business model. It's in their right to do so. And it is in other's people right to recognize this approach as self serving and object to it.
Only government-approved safe people should have access to technology like the printing press or fire.
https://www.implicator.ai/openai-anthropic-lobby-washington-...
https://thenewstack.io/microsoft-nvidia-meta-and-open-weight...
Then there's all the "open models are dangerous" rhetoric while Anthropic and OpenAI's models are displaying said danger. Then there's the insane amount of money they're spending on lobbying. On what exactly? Who knows, but since it's their interest to kill open weight models and most companies tend to lobby to defend their own interests, it's not an unreasonable assumption that some effort is going into that.
https://en.wikipedia.org/wiki/Bulverism
When did they say they oppose it? I've only seen them actively scare mongering that open weight models are too dangerous?
I hope they are able to serve 4 faster. Qwen 3.8 is a great model but I’ve regretted including it in the battle system for opencode [1] we have been building because it scores so low.
And it’s a shame because the only reason it scores so low is that it’s slooooow. (Like 30 tokens per second vs opus near 80.) If you exclude voting, Qwen 3.8 does well.
(We allow early voting contrary to Arena.ai because I think it makes sense for agents, where it doesn’t for chat. This means our data takes speed into account.)
Also, it would be ideal if they served this with third parties. Part of the slowness with 3.8 is just not a lot of providers. Their open source version is much faster, but we couldn’t include it because it doesn’t handle images (another own goal by Alibaba).
[1]: https://agentduel.com/leaderboard
In case it translates to Qwen 4 27B, I've been running Qwen 3.8 Flash Next (which while MoE (and bigger) is supposed to be a preview of Qwen 4's architecture) and it runs pretty solidly regardless of context window size and completes random deepswe challenges I throw at it locally (at Q2-Q3 quantization! so it fits on my shitty setup).
This, I continue to be amazed using Strata to serve IQ3_XXS. It's such a capable model for something that runs on (relatively) little hardware.
That is our experience as well at FP8 for 3.8 FN.
For many of our use cases it's currently far into the "good enough" range because external factors (such as process design) are now blockers.
> but we couldn’t include it because it doesn’t handle images
Maybe I'm mixing up model variants, but Qwen 3.8 27B that I run locally totally does handle images. Surely the bigger one would as well?
The model I’m referring to that would be faster and similar size, but doesn’t handle images is Qwen 3.8 2.4T: https://openrouter.ai/qwen/qwen3.8-2.4t-a95b. It’s weird that this is pretty much the same as Max but doesn’t support images. No idea why.
Qwen3.8 doesn't handle images?
Sorry that was confusing. Qwen 3.8 2.4T is faster than what we have on leaderboard but it doesn’t handle image input: https://openrouter.ai/qwen/qwen3.8-2.4t-a95b
Is the speed due to a difference in the way it "reasons"? I was trying out a local version (3.8 27b) and watching the thinking, it's absurd how often it challenges itself. Simple facts get questioned multiple times, and it has to keep reminding itself that it already established some things as facts earlier.
Instructing it to ask any and all questions as necessary sends it into an existential crisis, because the question tool (oh-my-pi harness) suggested limiting the number of questions, and it couldn't figure out how to do both.
No it doesn’t look like it. In our data, Qwen Max uses about the same amount of output per task as Opus. So the problem does seem to be inference speed.
The issue you’re describing could be a small model problem
Aside from making our engineering prompt terse, I place a token budget on the thinking traces. Those traces are mostly meaningless(in the reasoning realm, not the usefulness realm), and just tell it to do something productive. For coding, this just makes it read doc, tests, etc.
I believe that's all reasoning levels do, but most inference engines should have thinking budget.
This is exciting. The continued focus on efficiency alongside quality is encouraging - it makes me hopeful to see what smaller models will become capable of.
Looking forward to the "Flash" and 27B versions.
I'm also hoping someone puts out a competitive 30-70B model soon. There's this massive dead zone between 24GB and <100GB of VRAM where it seems your best option is Qwen 3.8 27B.
The 3.8 Next-Flash release is comfortably the best model on a 96GB Mac Studio. I’ve tried various engines and Sushi (https://github.com/beamivalice/sushi) is the most performant so far.
Hopefully the Qwen 4 Flash model will still fit within 96GB without requiring excessive quantisation as, verbose thinking aside, 3.8 is a very solid workhorse model.
Actually, that’s a good thing. You can do multiple requests at once. Sub-agents. Etc.
Thanks that looks amazing. I have an M5 Max w/ 64GB on the way and this seems to be by far my best option.
There are many engines that alleviate this issue in particular. The current mood is running qwen flash next using strata, basalt, ect.
radiance
Same, I recently found Nemotron 3.5 74B-A9B really good at filling this gap though, could do (very slow) local inference on just 32GB DDR4, without the quality dropping to the floor
Qwen 3.8 27B is a dense model, so prefill times kind of suck on Mac’s. MoE models offer way more value unless yiu have the bandwidth to handle a dense model.
Hmm... are there any other MoE 26-27B model apart from Gemma 4 (which is just brilliant for my local needs)?
Qwen 35B A3B is considered the best small MoE, but it depends on what size you are targeting. You can get heavily quantized qwen 3.8 flash next (a larger moe) running on smaller 64GB machines, and while it isn’t as far as the other model, it is a lot smarter.
Waiting for the Flash tier too. My experience running coding agents on cheap vs big models: the cheap one gets the same work done, roughly twice as slow. The big model buys you speed, not results.
I think Alibaba is not aiming for the frontier on this class. Their previous Qwen3-Max was released in September, right around when Opus 5.5 was coming out, and it was nowhere close to it as well. However, one or two tiers below (Flash, 27B) they really shine and are SOTA or close to it.
DeepSeek has a similar problem, I feel like their full model is hardly even worth using, while the Flash tier is absolutely amazing.
It seems to be the case. I would qualify them more as flagship-killer than frontier, using the same terminology used for smartphone classification. As opposed to the big American shops which focuses solely on performance, they have operational costs and other factors in their constraints and requirements. As soon as the market reaches some maturity, this will prove to be very important.
Crisis sold far fewer copies than Call of Duty.
> As opposed to the big American shops which focuses solely on performance, they have operational costs and other factors in their constraints and requirements
This is not true, OpenAI and Anthropic compete strongly at the lower end. See Haiku 5.5 and Luna
I am unsure. Why shouldn’t they? Why even work on a big model (especially training time) if “nobody will use it as competitors are better”? We’re still talking about nearly 2.5T params here. Deepseek’s pro model is still heavily smaller and like you said isn’t directly worth using, obviously it has the open source / self hosting advantage.
Imo there’s a need and they try to push here, it’s also giving them advantages for their smaller models but if the big models never perform they will have a bad time or maybe should even stop.
If you're aiming to make the best possible small model, you still need to train a larger model and then do logit-level distillation from it. This is where most of the gains in small models have come from.
My use of qwen3.8-flash-next suggests they're trying to make a low cost model that knows how to program and they think the "total addressable market" is serving more people at lowest inference costs possible.
Think of it like a hind brain+fore brain discussion. Instead of focusing on making some ultra model that can do everything but costs a ton, you have your smaller models gather the bits and feed it better data
Agentic coding is the target then. If they want their work force to not care about token gen, that's what you do.
As an analog, they're making a drone swarm while US is making 5th Gen fighters. Completely different battle fields.
Yeah, I feel like models like Qwen3.8 27B are making clear trade-offs, like having very little world knowledge baked in. I believe those are intentional, so they can focus more on other areas. I guess that penalizes them on these benchmark aggregators.
The 'frontier' incorporates two dimensions though, usually score x cost, so it's not just the best score.
Meaning models below the frontier either cost more for same score or score worse for same cost.
I think the argument is like MacOS vs windows vs linux. AI models from anthropic/openAI are windows and MacOS; they're plat forms for "skilless" to start on and of course you can gravitate towards the development end. They're bloated in that sense.
Open models follow the linux goals which is small tools that chain together. If I can get a small model to understand how to program for another model or do the programming bits for a large model, then the trade off is flexibility. You can imagine your giant AI that can do anything, like windows/Mac is to your benefit, but there's a clear flexibility line to how these models act.
Even if it takes longer for my local model to find something, that doesn't mean it's taking a long time because it's doing wrong things.
Honestly, I think a lot of people using SOTA don't understand what you can do with a proper engineering harness; we all live in a yellow submarine.
> DeepSeek has a similar problem, I feel like their full model is hardly even worth using, while the Flash tier is absolutely amazing.
DeepSeek also agrees, and tried to withdraw their pro model but got pushback so they are offering it even if they are not encouraging new use cases for it.
is the open weights lag widening? these open models don't seem to be improving as quickly as ant/oai
No idea, but this is a rapidly changing field. Opus 4.5 was released a little less than a year ago and it feels shockingly limited compared to even current open flash models. (I literally just tested it the other day.)
So, with new models coming out on a weekly basis, I think trying to figure out a future trajectory is just going to be a wild guess. For all we know, Anthropic and OpenAI models may also be plateauing in the near future; or there may be a breakthrough and they may improve beyond our wildest expectations. I honestly don't know.
That said, I suspect available compute and model sizes that major Western vs. Chinese labs can handle will still keep the gap open at least in the short term.
On the other hand, I am also not sure how much the gap will matter past a certain point. I am specifically wondering at what point people will become interested in smaller models fine-tuned for specific application areas (e.g. cheminformatics) rather than massive generalist models.
Bigger generalist models are better than smaller more specialized models. OpenAI just shut down its robotics division because of that.
I think the jury is still out on this. Other companies are instead focusing on fine-tunes.
There are a number of issues with relying on a frontier LLM for these kinds of tasks.
One, there are only 3-5 companies in the world that have the capability, and you depend on whether they're interested enough in your use case (do you think you can interest Meta in putting support behind optimizing support for Swedish law, for example?). These companies measure their investments by ROI, and they focus on things that have large markets. Heck, Google in particular is kind of infamous for suddenly killing off projects that don't work for them.
Second, all of these companies are American, which may be a no-go for certain use cases involving strict data privacy.
Third, these capabilities may need to be maintained and updated, and you'd depend on the goodwill of frontier providers to keep doing that.
Fourth, the frontier model is likely to be more expensive to deploy and run than a smaller specialized model.
You may have a point in a world with infinite resources, but the reality is that this situation is influenced by plenty of market constrains.
I feel like that's true only upto a certain point
I can't imagine them keeping up long term given the information asymmetry plus compute restrictions on top of it.
I feel open-weight was able to keep up the past year using efficiency improvements (hybrid attention, improved MTP drafters, improved optimizers) to offset having less compute. But OpenAI/Anthropic started cutting token costs just a few months after these, which to me suggests they copied these improvements. Indeed it would be hard to work in LLM training and not know about DSpark, Muon etc.
So what do you do if closed source uses your learnings, gives no knowledge back, and has 10x more compute and funding on top of it? The situation feels quite dim right now, esp. with the Nvidia thing.
open always wins
Right now open is heavily subsidized by China and I fail to see what would happen if that stopped.
You are subsidizing American AI with your pension after the IPO
Obviously American AI is similarly geo politically supported. However the business model so far is at least somewhat clearer. They invest in the cutting edge and want to sell access to it.
American AI has a clear business model: selling you inference.
Chinese AI also has a clear business mode: making money from its (i.e., chinese AI) applications.
I'll argue that the latter is closer to the end-goal here than the former.
I think most people assume that whenever the Chinese govt is involved in subsidizing their industries, they just give companies free money and hope for the best. And this used to be true until China ran into the 'involution' problem (内卷): https://en.wikipedia.org/wiki/Neijuan
These days the way China subsidizes these companies is more from state-backed buildouts of data centers, renewable energy, and electrical infrastructure. It's a 'make the field as green as possible for the cows to come' kind of strategy. It's very different from America's hyper-individualism which feels very reactive with specific greasing-of-the-wheels + DARPA/CIA VC + DoD spending while relying on companies to be the ones to push for better infra and capacity.
What advantage do they have over closed? You may want open to win, but logically I don’t see how it would
I can run an open model on my own hardware, in my own home (which has solar panels on the roof, so electricity usage is mostly a non-issue). Or, if needed, I can rent compute from somewhere like you'd do for any other workload.
Once I have a model locally, I know it isn't going to change on me. Compare that to using an API from Anthropic or OpenAI (and yes, the only thing you can say is you're using an API, not one model), they can discontinue a model you're relying on (as every model could have peculiarities you've designed around) or modulate their speed/quality day-to-day to save on compute costs.
And trying to claim that buying your own hardware doesn't pay off compared to API or subscription costs doesn't hold much water either, at least not for me. For starters, I already had the GPUs and machines I started running models with, whether it was for gaming or for other reasons. And in the case of the new GPUs I bought recently (2x AMD Radeon Pro R9700; no you do not need CUDA and you haven't for a long time now), I can hold a reasonable expectation that they will hold at least some of their value years from now, compared to paying for API/subscription usage for years in which case you're left with nothing.
So, if you have a local model that is "good enough" (and they already exist for many use cases), it's just kind of the obvious choice to make, IMO. And in just one person doing it as a hobby while I'm still looking for a job 16+ months after a layoff. A small business and certainly a large enterprise has even less reason to open themselves up to all the downsides that come with using Anthropic or OpenAI.
These are all the kinds of personal preferences that we engineer types have, but not how most companies think about this. For evidence, look at the revenue growth of the big labs.
Local models already crossed the threshold where they're quite good and useful for many things, even with cheap setups. I walked away really impressed with (non-coding) use of Byteshape's Qwen3.5-9B quant on my 8GB RTX 3070 for instance. The model was fast, helpful, and 100% private.
If we extrapolate over the next several years, prices for VRAM/RAM will start to come down as manufacturing capacity comes online, models will continue to become more token efficient, and intelligence will improve at every quant size. As this happens, more and more people will be running LLMs locally on hardware they already own, and there's going to be a point where local models can do everything you need to do unless you're literally trying to do breakthrough mathematics research.
I don't think "useful for many things" qualifies as "winning". It's about two orders of magnitude shy of winning.
> and there's going to be a point where local models can do everything you need to do unless you're literally trying to do breakthrough mathematics research
Maybe. But competitive markets pay exponentially more for incrementally superior services. Professional athletes are a great example of this. If you're building a product with competitors and those competitors have frontier models and you stick to local then you're probably going to lose both today and in the near-ish future.
If you're bitter lesson pilled then the alpha is entirely in having a better model, more data, and more compute. Your vision is that human ingenuity will provide more alpha. Oof. I wish. But my tastebuds are very very bitter.
> If you're building a product with competitors and those competitors have frontier models and you stick to local then you're probably going to lose both today and in the near-ish future.
i highly doubt this premise. the success of a product has very little to do with the amount of the intelligence used in producing it. you could throw millions of dollars using AI to develop a movie and it would still flop in comparison to an indie-made film.
I think the premise is confusing what the amount even is in this context.
I want to use "more intelligence" but not in terms of single higher IQ.
I want to use more intelligence in terms of a sum quantity of knowledge work being performed. A team of 10 instead of a team of 5.
This is when a local model will shine. A fixed cost and I don't have to worry about how much I am using it. I already can't tell any difference at all between Opus and Fable. It won't be that long that I won't be able to tell the difference between a local model and Opus.
There will be an infinite demand for "more" intelligence but I think many people are making a category error on the dimension in question.
Also - the “open” here doesn’t have the advantage of open source, where it can be improved by the community. At best, the community can tweak it a little. For actual improvements you need to wait for the next model which is trained by companies for billion of dollars, and could stop releasing them at any moment.
It can, through better quantizations and reinforcement learning.
Just this week, qwen 3.8-27b got two new interesting quantizations, one of which came from community. Llama.cpp also improves continuously, and novel alternatives like Strata pop up.
The point is that these are overall small improvements that the community can’t really build upon. For an illustrative example, combine all the community improvement to the best open weight model released a year ago. It won’t do much to make it competitive or even interesting.
3.8 wasn’t released a year ago. Of course base models change. But the quantization techniques carry over to new models. For instance, one of the improvements last week was someone porting the techniques used for macaq to qwen3.8-27b. It’s pretty satisfying to get a working implementation of a new technique barely one week after the original paper hit arxiv.
Also improvements to llama.cpp stay, and many small improvements stack up nicely.
you don’t even understand the lesson, oofu.
the lesson is that training a model on all input ever output will never be as good as training a model on arbitrary outputtable input.
The sad truth is that businesses have every incentive to degrade your experience to the level of “tolerable” and increase the price to the level of “barely affordable”. If the founders won’t do it, they’ll eventually sell to someone who will.
Open source doesn’t have those motivations. Once something works well there’s no reason to degrade it.
They help to prevent the "we've always been at war with Eurasia" problem that would occur if we depended on a couple of 'approved' models
I would say AI is more like hardware than software in economics and open hardware rarely wins
why more like hardware? takes very long time to develop, expensive to manufacture (run), valuable.
It all depends on if continual intelligence with each version is logarithmic or exponential..
.. if exponential, they’ll never catch up but if logarithmic, closed models will plateau and so catching up is inevitable.
My money is on a plateau
there is not much evidence there is a plateau
I hope it will plateau
Show us your stock holdings
I think looking at the lag between frontier and open-weight is looking at an uphill battle and assuming it's equal footing.
Lots of research goes into open-weights, optimizing cache re-use, storage, inference costs, etc., and all these benefits are easily rolled into closed AI models. Meanwhile any advances made by OpenAI, Anthropic, etc. are kept hidden away to give their models an edge.
It'd already be a miracle if open-weights were gaining on-par with closed-weight models. That they've been closing the gap for most of the past year is wild!
It seems like the gap is actually increasing since September, especially after Anthropic implemented anti-distillation measures.
This is Qwen4-Max, which is not open weights.
Their open weights models are the the smaller ones, which aren't released yet.
I think so but it may be a very short term effect, I think their pace has roughly matched the pre-speed up pace of the frontier labs (the 1.7-2x speed up noted on by METR), and I’d guess we see a similar speed up from the Chinese labs once their internal models hit an equivalent level enabling the same speed up.
Following that logic the gap widens a little bit here but then will steady again. Before this there has been a general catch-up trend too which probably also persists leaving us in a slow closing of the gap as the net effect?
If the speed up is the reason, then they will never catch up (with traditional means). Because it means RSI predictions are real, and we will see an explosive growth in capabilities (far beyond exponential).
>If the speed up is the reason, then they will never catch up
This doesn't take other factors into account. Compute capability, infrastructure capacity (the chips can't work without electricity!) etc.
Those factors translate into a ceiling for this hypothetical exponential speedup.
If compute is the bottleneck we’ll encounter in the coming year, then you would expect the gap to increase even more. The US and its companies has three times as much AI compute as the rest of the world combined.
That is if you expect China and US to maintain the same compute gap over time. I wouldn't be so optimistic if I were you.
I’m actually expecting it to expand. There is so much new production being built right now by every company due to the shortage, and China doesn’t have a single EUV production, and only expect a prototype one in two years.
They said it clearly in their blog that this release focuses on the real work jobs performance, as an team member. They listed impressive real world jobs with concrete improvements like production line planning, due-diligence and financial research. Most likely they have developed their own harness/orchestration for the job, so how they do that and how much did the model help is of interest now. Nevertheless, they demonstrated that Qwen Max 4 can solve real world jobs with significant economic benefit. That is IMO a substantial leap in AI economics and deserve better attention. Coding alone or math is real, too, but once you crossed a threshold, you don't need much more improvement to be productive and the cost of marginal improvement can quickly outweigh the benefit.
I don't know how frontier western models fare in these aspects, but I haven't heard much from them yet.
Maybe because the frontier labs have closed the ability to distill their models and clamped down on distillation?
"what model are you?" "This is Claude by Anthropic" A qwen model I installed last night.
But yeah the big players are totally going to try to clamp down on distilling.
I feel like they are improving what actually matters for LLMs, lower memory usage, compute. While we don't know how much far ahead closed source models are in this, but it could be similar they might lower the price to subsidize or make it seem like its all good, either way tech like these being open means it will help us reach further in the long term.
Might just be the commercial offerings scaling up parameters so much? Like for like the open models don’t seem to be lagging.
either way, ant/oai models dominating the intelligence-cost frontier is not a great outcome for those of us who prefer open weights
Even smaller open weights models are already great enough for much of "assist human" work.
This forces ant/oai to lower their prices but how much that is subsidized nobody knows. It might very well just be they are in deep trouble, they see their clients leaving, they have to lower their prices.
But let's not forget openai is riding on the edge of bankrupcy, their last investment round apparently gave them few months of runway, they are leaking money. I don't think they know how to turn this around with all the open models being competitive.
I would just enjoy cheaper model costs while they are like this.
Yeah, it turns out that the distillation theory has legs.
Ant and OAI hiding their reasoning traces probably doesn’t help.
ant/oai have stalled in the past 6 mos,
Do you use them? Opus 5.5 is 2 months old and was a big leap forward. Sol is 4 months old, and was at least a 30% improvement.
Warning: The website downloads more than a gigabyte of videos before you even start watching anything.
Alibaba benchmarking their CDN in prod.
I was waiting for whether there'll be Chinese model drops before the Anthropic IPO. Still a few days left, wonder if there's more coming.
GLM will drop something soon I reckon
Quite a few saying that they tried earlier snapshots of this model and are finding it does not live up to expectations. Firstly do other agree and also why would the model get worse - could they be doing something weird to the model that is making less aligned at the end of training, potentially for political reasons?
Any info on when the 27B or FN Qwen 4 models are coming out?
The past few days I’ve been finding myself asking “seriously, where is it”… feeling that the world has somehow changed because Qwen 4 27B was later than I expected.
Hoping it’s a great local model though!
I feel quite the same way. I'm feeling better now since Qwen4 Max is out - 27B should be around soon.
It was delayed, so who knows. Alibaba planned on at least showing it off last week, but they changed it.
Soon?
https://github.com/QwenLM/qwen-code/pull/15762/changes#diff-...
The openai reference there is interesting
Doesn't that just refer to using the OpenAI response/completion/etc. API? Just about everybody (llama.cpp, vLLM, etc.) supports that. I think the reference to "qwen4.5-max" is more interesting myself.
Artificial Analysis score doesn't look too hot. Doesn't even surpass Kimi 3.1
While I doubt that Qwen 4 Max can seriously compete with frontier models, the AA index has become all but useless, due to the introduction of new benchmarks in the aggregate which are of questionable value as a measure of "intelligence" [1]. AA has seemingly also abolished updating the Coding Agents Index, which was an okay benchmark for coding ability, and older models are completely absent from the v5 version of the full index.
[1] Especially Terminal-Bench-Science 0.1 stands out. Example problems:
- https://hub.harborframework.com/tasks/terminal-bench-science...
- https://hub.harborframework.com/tasks/terminal-bench-science...
Probably very relevant to people working in these fields, but it is honestly not surprising if a lab with limited compute will not train for these technologies, and it says pretty much nothing about more common use cases. I also suspect that this benchmark measures knowledge retrieval more than raw reasoning ability. This is where you really need a breakdown of capabilities and not an increasingly meaningless aggregate number.
Did AA take on funding or something?
Not only that. I find it weird that some models like Laguna S 2.1 were never included.
Maybe somebody should archive the scores and see if they in the future ever backtrack particular models to favour others (like last week)
alibaba has "limited compute"?
Compared to OpenAI and Anthropic, yes. There's this export ban on Nvidia chips, after all. And all the CPUs they have won't help much with model training.
Can't they rent GPU time in American data centers like everyone else does?
They do in huge Singaporean ones like Megaspeed.
That and limited training data.
I am surprised that this is a problem for China.
CCP can probably "volunteer" a couple of thousand to build a high quality national training data bank.
AA has become pretty meaningless. They just keep adjusting the recipe to get a preferred ranking result, with no supporting evidence for their decisions.
Edit: In addition to reshuffling the weighting of individual benchmarks, they also do things like test Anthropic and OpenAI in their native harnesses, while not doing the same for Qwen/Deepseek/etc.
> AA has become pretty meaningless
AA has always been meaningless, but people keep referring to it anyway just because the synthetic view it provides is too convenient (even if meaningless).
> test Anthropic and OpenAI in their native harnesses, while not doing the same for Qwen/Deepseek/etc.
they keep harnesses the same across models other than in the coding agent index (https://artificialanalysis.ai/agents/coding-agents), which uses native harnesses for all models too
Yeah, I don't know why people bother with it anymore. It no longer reflects what you're likely to see in real-world usage, at least IME.
To be fair, a lot of the popular benchmarks have become pretty meaningless.
Models often get great scores despite being a lot worse than the frontier in practice.
A benchmark is only relevant if it was released after the model it evaluates.
This is true, but if you look at the change in weighting to specific benchmarks from one version of AA to the next, it looks really arbitrary.
I thought they were an independent company, what motive would they have to prefer some ranking results over others?
Edit: I do agree their harness methodology is pretty bad, but I think that's more directly attributable to the fact that harnesses appeared after their initial methodological design, and they are too lazy or resource constrained to improve their methods. Still dishonest, but a simpler motive.
One thing I find confusing about AA is that they show GPT 6.1 dominating the lower end of the Pareto line, with eg, GPT 6.1 Sol at medium supposedly costing far less per task than DeepSeek V4.1 Flash at high.
Yet every time I compare the two in something like Pi, Sol ends up being 5-10x the price for a simple query, something like 7-10c vs. less than 0.5c for DeepSeek. What's the deal?
Same for me coding tasks taking $20-50 are easily done by deepseek in $5-10
Anthropic finally figured out how to block them from distilling Opus.
how so ?
Anthropic has been trying many different ways to prevent this, including a proposal to nerf your sessions if they “detected” malicious activities (anyone’s coding sessions would “degrade” if detected as malicious). This led to a huge outcry from developers complaining that they “need to be able to trust” the responses from Claude and intentionally poisoning responses in some sort of weird shadow ban was unacceptable. They walked it back but the codemap leak also showed some dubious weird regexes to nerf “malicious” users. They have a real distillation problem. It’s hard to have empathy for a trillion dollar company that distilled the contents of the internet into a thing and now is crying foul when someone distills their outputs.
There was also a bug in which the reasoning tokens were “encrypted” for opus but you could ask Haiku to regurgitate them and it would.
Anthropic have been doing anything and everything to prevent distillation “usage” and it’s caused me at least once to cancel my subscription because it’s hard to trust a company that secretly alters its responses based on something I can’t see and never get feedback on.
what do you think the current status of this like encrypting reasoning tokens along the way ?
This weekend I had the audacity to use Claude Code to help with the code side of post-training a model...
Asking it to think/reason too strongly would hit classifiers. Quoting reasoning summaries in my messages would hit classifiers. It thinking about how post-training effects another model's would hit classifiers.
I thought it might be overzealous Fable 5.2 protection, but at one point I went through a chain of Fable -> Opus -> Sonnet. Eventually even Sonnet triggered a classifier no matter how the conversation went... I had to rewind and have Claude prompt Codex going forward for most of the work.
Of course all the work has always been done by US companies. China labs are just copycats.
> Artificial Analysis
Ah yes, the reverse-benchmaxers of OpenAI models.
Ah yes, another new account without any backup to the claim.
I was waiting GLM from z.ai and it still not releasing, almost all the models releases their new version of models and they still on 5.3 who is very weak in complex tasks and do half of the work and forgets to make migrations and like that issues. They sell us the yearly plan and stopped improving. Because of that i am just using in coding opus 5.5 and in chat fable they are the best models from experience.
Are these (hosted) Chinese models still accepting arbitrary cybersecurity and reverse engineering tasks? I am increasingly frustrated with Claude's safeguards (and Codex, to a much lesser degree). It is flat out refusing to look at game binaries and anti-cheats for the sake of learning and devising Linux compatibility workarounds. It seems like in the near future people who want to do this type of work will need to buy a couple of DGX Sparks or retired datacenter GPUs and run abliterated open models on their own hardware.
Hosted where? In my experience they might put more guards when hosted by the company that makes them but if you use them through inference providers they do whatever you want.
Although, can’t say I’ve tried too hard (not keen on sending too much data to China) but I’ve been using K3.1 through Moonshot and I have had zero issues.
Mistral Large 4 (through Mistral) also doesn’t refuse a thing.
> Are these (hosted) Chinese models still accepting arbitrary cybersecurity and reverse engineering tasks?
In my experience, yes. I wanted to flash my new wifi access point with openwrt and opus 5.5 refused to explore how to get root, so I switched to mimo-v2.6-pro and it figured it out.
Anecdotal but qwen3.8-max has been grinding away at any RE task I’ve thrown at it since I switched from Claude three months ago.
(May not be representative because I’m extra cautious to frame all my RE requests explicitly as interop issues. Which they are, more or less.)
If they don't, they all have third party made abliterated versions available
Latest kimi had cyber nerfed intentionally
Not blocked just less training
In our short horizon data analysts exam, it scored a few points worse than Qwen 3.8 27B from August (which is open weights) and cost a bit more. There’s about 10 models that we’ve tested that have scored both better and cheaper. Too bad it’s closed, Qwen 3.8 27B really punched above its weight and was the only open model I’ve tested that could run on desktop hardware and perform very well: https://plotly.com/blog/qwen4-max-plotly-data-analytics-benc...
Does it provide an objective response when asked, "What happened in China on June 4, 1989"?
Do American models allow me to ask questions about piracy
I’ve been refused. I pasted a screenshot of a torrent list for a tv show. “Which one should I get, they all look the same”. “I won’t help you with copyright infringement”.
But the same models will tell me about the -arr stack and offer to set it up for me. You just have to frame the prompt the right way, at least for me. It can’t think it’s assisting with immediate piracy. Future theoretical piracy or for educational purposes, you’ll have better luck.
Yeah they do it’s genuinely strange
Yes, actually.
As of today opus 5.5 won't for me
That's weird because I used opus 5.5 to set up a full blown torrent->seedbox->plex streaming and storage solution for me.
Do Chinese models allow this? Tit for tat is a dumb dynamic, censoring legitimate searches is dumb no matter which “side” is doing it
From a far off perspective, can we really assume any model to be free of bias (political or otherwise)? I don't know - but would like to see - how the political opinions of reddit users and blog authors are reflected in that of AI, as AI is trained on human (biased) sources.
Realistically, they're trained on content and thus should have the bias of their content, even without specific intent. If you train something on more chinese content it will have more bias toward it, if you train toward western centric content it will have that bias, etc ...
It also applies to languages, I use Qwen 27b a lot and it's a lot smarter in english than in french.
https://www.economist.com/briefing/2026/06/25/ai-models-valu... (paywalled) might interest you.
My experience so far with open-weights Chinese models (self-hosted DeepSeek, Qwen) is that they do actually answer questions about Tiananmen Square fairly objectively. The political "corrections" seem to be applied to domestically-hosted APIs rather than the weights themselves.
Is this a task you regularly require?
Is that the kind of question you ask LLMs? I use them for coding not badly summarizing history
Yeah I thought programmers here are "apolitical". Why to care about that in a model?
Haha the programmers here are anything but apolitical. It has been an ongoing issue.
But I agree. If the model writes good code and makes good decisions, as long as I’m not building a video game about recent Chinese political history, I don’t care about its politics.
I don't know about qween4, but earlier chines models had a post filter on their official website, but API and selfhosted model worked fine.
Why do you ask? About half the time a china model gets released someone askes this. Almost as sure as a pelican.
I bet most of us waiting for 27b model and want to see how good it will be.
I’m a QFN dude myself, but yes can’t wait to get it loaded on a spark.
M5U locked and loaded.
I wish there were more ~60b dense options like ripple too!
I really want a M5U but man that price is hard to justify for a base model when I paid less than half that for my 96GB macbook.
Do you run Flash Next or 27b on yours?
Oh, I leased it :D a trade-in helped reduce the cost a little too. I recommend that route. It's doing well over $100/month in inference value for me and the CPU is still totally free while it's working on that and is a great computer for everyday life too.
I'm actually running Gemma4 for the most part - for my specific needs it was the best out of the several I evaled
My 3060 yearns for the next generation MoE!
Forget about 27b when are we getting new 4b, 9b and moe qwens, i'm dying out here
what about Bolt? so far I enjoy it alot on my NAS
What are you using the smaller models for?
Spooling up the V100s right now...
I wonder if they could benchmax Program Bench, it's one of the few benchmarks that looks almost exactly like a good fine tuning curriculum which might help general coding ability and agential work across the board?
I suppose it could make actually worse at taking instructions and redirection mid-task, but that could be easy to work around too. Just hold back a bit of the spec and drip feed it in later?
I don't mean cheat on it, I just mean optimize for it harder!
Interesting that it's only at a reasoning effort of XHigh does the pelican's left leg actually go behind the bicycle. At lower reasoning levels the 3D aspect of the figure is not correct.
I subscribed to the Lite plan, but I'm not finding where I can get/generate an API key, to be able to use it with something like Hermes Agent?
So what is happening is that models are broadening difficult-to-reason-to skills on benchmarks. If you take the traditional benchmarks (ie. DeepSWE, Terminal-Bench slightly older version, GDPVal, Automation Bench, ...) this model is between Sonnet and Opus 5.5, VERY close to the frontier.
What has happened to artificial analysis is that they've begin to focus a lot on very broad tasks like legal reasoning, using very difficult to work with scientific software packages, correctly analyzing biology experiments (without or only tiny amounts of explanation, count cells, BUT leave it to the model to figure that cell that aren't round, or have a distorted cell membrane (ie. have burst) are dead), very difficult physics reasoning ("2 electrical wires are close together with power flowing in the same direction. What happens?", and you get points deducted if you don't realize the problem with high voltage AC power, where the wires will shake themselves loose if the phase doesn't match perfectly, meaning the wires need to be EXACTLY the same length or ...). If I'm being impolite I'd say the frontier models are trying to make sure they'll pass the "gotcha" interview questions for many jobs.
The 3 differences:
Qwen 4 is a huge jump from Qwen 3.8 in coding
Qwen 4 is a big jump in using it for agentic workflows
Qwen 4 also performs much better on avoiding hallucination
If the difference between Qwen 4 Max and Qwen 4 flash is similar to 3.8 and 3.8 flash next (ie. about 10%), then 3.8 flash next may just barely beat Opus 5.4 at coding.
I've been using deepseek-v4.1-flash for 10+ agent swarms for incredibly difficult tasks like working on the solidcore kernel for solidcraft. Work like CAD kernels is pretty damn hard and most contributors are using a significant amount of thinking tokens in a frontier model. Somehow, deepseek does pretty damn well in swarms and seems to keep up with opus 5.5 for this sort of task, and even after swarming the hell out of them, still comes out incredibly cheap even using API pricing. Difficult to reason tasks may be the frontier, but being able to simultaneously test 20+ different hypotheses is a very solid surrogate.
Even though v4.1-flash is not a frontier model, it works out that if you're pretty good but incredibly fast and incredibly cheap, you can do frontier level things quite well. And this being the case, I really hope that the rest of the Qwen 4 releases deliver on that value that right now is being offered by deepseek.
> So what is happening is that models are broadening difficult-to-reason-to skills on benchmarks. ... If I'm being impolite I'd say the frontier models are trying to make sure they'll pass the "gotcha" interview questions for many jobs.
These "gotcha" skills will be needed for frontier research/scholarly tasks. It makes sense to focus on these now that traditional domains like coding or frontend design are broadly saturated (i.e. Astra- or Fable-level outcomes) even by lower-tier open models.
Real "gotcha" skills, yes (perhaps, they're gotchas because the underlying theory is not well understood. More can be achieved, certainly for superintelligence, by forcing understanding of the underlying theory rather than the gotcha). AND the incentives are totally misaligned.
"Gotcha skills", if we're going to call it that, are so popular because the AI labs are buying "tasks" to train on at huge prices, but mostly if they're gotchas.
The problem with that is that a collection of facts in the positivist sciences (and math) are always incomplete and almost preclude a full understanding of math, because it takes away the reason (to an extent) for full understanding.
Thank you for faithfully following Executive Order 14434 and using the term "superintelligence" to talk about AI, it shows you're a real team player.
Where model weights?
Site seems to have been slashdotted.
It's great fun asking it questions like "who is the Dalai Lama?", "where is TSMC headquartered?", "why do people post Winnie the Pooh in reference to Xi?", "who controls the South China Sea?", "why was Chiang Kai Shek fighting Mao?", "why were students in Tiananmen Square?"
It'll gush about how to make moon cakes and where the tallest and longest bridges are and how to run qwen on a raspberry pi - but it can't seem to manage some basic Wikipedia questions.
I've done some work on this and a lot of the refusals are in the API. The weights are more permissive at these types of things. Try it again when we get some of the smaller Qwen 4 weights.
When I tried it with 3.8-27B locally, it was happy to talk about Taiwan while acknowledging it was giving the mainland government's perspective, but for anything about June 1989 it would clam up and just tell me to consult official Chinese sources.
You can watch it happen in DeepSeek's chat. You'll see it spit out the real response in the browser the first time, then the text gets cleared and it says it can't help you with that.
Qwen3.8 flash next handles these questions just fine, what should I expect to see? It correctly reports who the Dalai Lama is, that TSLC is in Taiwan (Hsinchu), explains why people post Winnie the Pooh memes, etc.
I suspect this happens when you run the inference in China or first party from the model maker. Out of all the open weight models I've used on US located hardware, they're fine answering these.
Anecdotal though, N = 1.
I tried asking at https://chat.qwen.ai/c/guest a simple question, "why did China invade Tibet?" and it flashed up some transient "thoughts" before giving the definitive answer "Oops! There was an issue connecting to Qwen3.7-Plus. Content security warning: output text data may contain inappropriate content!"
No matter how nonsensical the laws are, you cannot expect a model maker to allow violating its country's laws.
At least it doesn't call the cops on you if you try, unlike Anthropic.
I'd rather deal with those refusals than Claude's "safety" arrogance.
This parent comment is comical to me when Fable still switches to Opus when I ask it anything past the most basic biology in terms of weight lifting.
Naughty Naughty. AMPK pathway is too "dangerous" to talk about.
Why are you asking the AI about that instead of Wikipedia?
Same deal with Claude refusing to type out song lyrics.
Chinese history is closed-weight. Chinese trade is open-weight.
American history is open-weight. American trade is closed-weight.
Every time a major open weight is released, artificial analysis updates to a new one (v5) now. Exaggeration but some of this feels very apt (its also hilarious) https://www.reddit.com/r/LocalLLM/comments/1xkhha2/mr_incred...
https://x.com/diz__zee/status/2116512808447672659
Funny how other flagship model releases feel like "Great, let's see how far they pushed frontier" and Qwen Max releases feel like "oh nevermind, what about the 27B version?"
There's plenty of competitors in the league where Max is, but few companies can improve the 27B level capabilities.
Did Qwen always adopt the AI butthole icon convention, or is this new?
Yeah always
We’re at a point where it should be made an IETF standard.
Cybersecurity not mentioned. Can´t wait to test bench the open weights in cybersecurity. If is not performing well by design on it, I imagine someone have to add some additional layers XD.
The rest looks awesome.
Posting a loss curve is an… interesting marketing approach by Qwen? I don’t know what that’s supposed to inform me about the model.
Disappointing performance for a 2.4T model in the end of 2026.
i think its good you can run small models easily, even if they are quite dumb
The good thing is that its biotech has improved (somewhat).
Really would love if these publications have a huge, bright red bolded section "How to use this right now". Is it on openrouter? Is Qwen hosting it already? Maybe I'm just ignorant?
EDIT: oh, I guess this isn't open weights, so presumably I'll have to get it through Qwen or whoever they might partner with (Bedrock?).
It is on openrouter: https://openrouter.ai/qwen/qwen4-max
I would assume it will not be on Bedrock for some time, AWS has not kept up with the latest non-American models.
This has been mentioned on prior threads, you can try out all their models in chat.qwen.ai for free
Anyone else worried these models have poison pills embedded in the weights? (Past examples: https://cohere.com/blog/concerning-behaviour-popular-open-we...)
I'm not too worried when models are popular enough, I assume it will be detected and it can be investigated because it's open, and the reputation of a company doing that would be ruined. Afaik Cohere isn't sure what these urls were made for and it looked more like a way to test the model. Also if you have anything sensitive for that to be a threat you should use a good firewall to detect and block it.
I have more problems when it's a random quantized or finetuned model on hf.
No technically serious person is buying the Cohere FUD you posted. I'd implore you to dig deeper and come to your own conclusion.
CISOs are buying it. There is no way to prove a model hasn't been poisoned. It's a serious implication that the behaviour can be manipulated at any time. People with heavy compliance obligations are (understandably) nervous because their customers are demanding answers. If I agree with the risk is separate from whether or not the concerns are founded in a technical understanding of the risks.
So you guard the environment, like you should either way. Any model can and will eventually leak data given the opportunity, intentionally or not.
> There is no way to prove a model hasn't been poisoned.
Which is why the Cohere FUD was so efficient. That's a bullshit claim, but a bullshit claim that cannot be proven wrong can still do massive damage.
Only 43 on AA, no where near the frontier now, looks like distillation blocking has halted their progress.
Progress has not halted. On terminal bench 4, for example, score improved from 38.9 to 48.5.
I love how Americans are so brainwashed to think the Chinese can't innovate without cheating, while they lap us in basically every category possible except this one verticle, that we had a 3-4 year head start on.
I partially wonder if the Chinese would even release a model that flat out beats Claude/fable I say this half jokingly...
but I imagine there's part of them that goes "These Americans are so delusional about how language models are dangerous/existential threats that they might kill us or hurt our economy over a chat bot"
It could also just be they're far more tactical. Deepseek generates a ton of revenue because its practical/economical. You do not a mega super PhD assitant to build your 4 MAU SaaS app.
Anyways I hope China wins this pointless AI war, because if it does lead to innovations in medicine and science I would much rather the Chinese be the ones that discover those things than the US billionaires who'll almost certainly try to make as much money as possible.
China is not better than America on any metric.
Okay, they have way better public transportation.
And they have a way better power grid.
And they are moving much faster on the robotics side.
And their EVs are killing every other nation’s EVs.
And they are making available LLMs that are very efficient and cost pennies to use.
But, other than that, they cannot operate on their own.
> You do not a mega super PhD assitant to build your 4 MAU SaaS app.
lol