Am I wrong or are these evaluations, while interesting, not really meaningful for anyone doing serious development work?
I'm asking because I personally only use AI with specific and detailed instructions, building my projects piece-by-piece.
I mostly don't look at the low level code and some of it I don't understand as much as I'd like, but I very much give much more technical instructions than a simple, two sentence prompt.
So, it seems to me this "oneshot from a simple prompt" eval is fairly meaningless when it comes to model evaluation itself, as it is in no way representative of real world application.
This would be more something for "vibe coders", people with little to no programming background wanting a website?
I checked out the development benchmarks for agents, and was suprized to find that they're not really representative of many of my own day-to-day development workflows.
It appears that they're mostly testing the ability to make business tasks autonomous, with ~20% associated to development tasks (ssh here, install this, etc.), but not actual programming.
1. Long form task based examinations like this that test the ability of the model+harness to remain on task, tool calling, overall effectiveness and taste.
2. More direct 1:1 and qualitative comparisons that you might get with a tool like https://evvl.ai/ - which also uses OpenRouter and does similar one off model comparisons (or lets you use it as a MCP from your dev env to be like: "take the prompt from this loop and try it against these other models")
I've been using code review as my benchmark. Launched a massive parallel Fable/max code review on my lone lisp codebase and recorded all the results and transcripts. Switched to OpenAI and am now running the exact same code review with Sol/max.
It's still a work in progress but preliminary results reveal that Sol is able to reproduce 70%-90% of Fable's performance. This is a very meaningful result for me because code review is what I use AI for.
As someone who's done web development for 20+ years, I find the model personalities pretty dang fascinating, especially how they develop (and evolve) design sensibilities.
I'm interested to know how the classic AI "purple preference" emerged (organically?) and if the beige wave came out of specific training efforts to combat it?
To your point on development work (the code itself), I was talking to some friends on the Google Chrome team about any research understanding the model's preferences around framework ergonomics and abilities to properly implement core web standards for given tasks. I think that would be super fascinating.
I'm repeatedly noticing that people working at big ai and tech companies are surprisingly not that... good... at using ai? It's like theyre doing a plausible thing to get something done and calling it a day
> Build a one-page site for a neighbourhood coffee shop: opening hours, the address, a short menu and a photo. Nothing on it changes unless I edit it myself.
If that's the entire prompt, it's quite depressing how much alike these all look. I appreciate some of the details from the Opus 5 version, but I can't help but strongly feel the AI vibes emanating from that design.
I trialed Claude for a sorely needed redesign on my website. I was quite impressed with the results, but decided to hang fire on deploying it. Cue my surprise when I just found the same design applied to a coffee shop!
The similarities are beyond coincidence, to the point I'll be scrapping Claude's version of the redesign.
A user, even without much design background, could ask the AI in a follow up prompt to change a few key parameters like overall color and differentiate it more. Even taking a sample image from a design magazine or blog and saying "incorporate some of the design elements here" would probably work great to diverge it.
Without more knowledge about the technical aspect, it could be a good thing that they're all so similar. If you tell humans to go from their kitchen to their bedroom, they all stand up and walk the same way. Nobody decides to crab walk. Maybe you'd have to cripple the model in some way to do that.
On the other hand, if you want something different with LLMs, all it takes is a few more words of creative flair in the prompt.
This is an interesting point. But even then everyone has their own style. Some might walk with a bit of a swagger, some with a limp, some might have to get in a wheelchair to go over. Would model temperature be another knob to turn beyond a more creative prompt?
The models are giving you very average, middle of the road responses. The average human gait doesn't have swagger, limps, or extensive accessibility needs. If you want the swagger, you need to steer it that way.
It’s always been interesting to see how similar output is across ostensibly very different models. I remember testing short story writing in the early days and having the models all choose the same niche topic across e.g GPT, Llama, Phi, Claude, etc
It's also kind of pointless. Why does a coffee shop need a website anyway? Nobody is saying, "man I would love to go to this coffee shop but I can't find their website".
If all you want is "opening hours, the address, a short menu and a photo" there are easier and cheaper ways to do that.
> Nobody is saying, "man I would love to go to this coffee shop but I can't find their website".
Me, I'm saying that, and I've skipped going to coffee shops and restaurants because they don't have a website, just a fucking Facebook page. I don't use Meta products and can't see their page if I'm not logged into an account I don't have, so I do what the business owner intended: I go fuck myself and get coffee somewhere else.
It amazes me how close minded people have gotten. Do you really have that hard of a time believing there is a use case for a human looking up information on a coffee shop, on the internet? Oh, and god forbid the shop owner actually wants to provide it!
To see the hours, get a sense of the menu. There are many matcha shops by me for instance and my wife likes to check the specials before choosing which one to go to.
Where do you think Google and co. source that data from? Having an actual, searchable, up to date menu is, just for a11y and accuracy far more valuable than someone’s poorly lit, low res upload of an photo taken from the menu circa 2019, just to name one advantage. Have found data from restaurants without their own webpages on Maps utterly inaccurate, even suggesting some that have been shut down for months to where there is clear value. Then there is restaurants closing for holidays, illness, etc. and the ability to make reservations without calling.
Whether these need their own over an Astro template is another topic, but having a webpage at all is incredibly valuable.
Why does anyone look up anything before going there? Maybe no one should have a website, everyone should go check everything out themselves in person like the telegraph hasn't been invented.
Why does your brain instantly go to "whats the cheapest and easiest way to do things", instead of "whats creative, fun, interactive, usable, accessible, etc...".
This is the reason the world is slowly becoming a boring ass place and the internet basically consists of 3 websites that are all trying to suck your soul dry.
If I ran a business, a website that stands out from the crowd and provides a good experience would be in my top 3 things to do.
Finally some really useful apples-to-apples comparisons rather than endless benchmarkmaxxing and anecdata. I want to see this test run again for the open-weights models that fit into 128GB combined RAM. Like how does Muse-glimmer compare with qwen3.6-35B?
I like the output of the cheaper/older models better, surprisingly, say Gemini 3.1 or DeepSeek V4. They're mostly no frills and just text, and ironically look less AI-generated to me because of the lack of hip slogans and graphics.
Definitely closer to what I'd want for my own site, but I don't claim to know what people want from a website for a coffeeshop.
Building ad-hoc evals is trivial these days. And you can place an LLM judge in front to disambiguate between output. e.g. I have Frigate gating a video feed so that an LLM can watch our home cameras and the labeling task takes a few minutes and then the evals run at fairly low cost https://wiki.roshangeorge.dev/w/images/4/4e/Screenshot_-_Eva...
This means that generic benchmarks and evals are sort of passé. There's no need to have them write "make a coffee shop page" or whatever. Instead, simply focus on the real problem you have and evaluate whether they solve that problem. You want to move along the price-performance frontier alone and the SOTA models are decidedly at the top of the performance curve but very expensive so they serve very well to produce the gold standard and to judge.
The change from prior to today is that benchmaxxing and per-token pricing allowing for evaluation point to the same direction: do not proxy results. Instead deploy the highest end model you have as a judge for others until you have statistical confidence in discrimination and move along the task-specific frontier.
I highly doubt that you have solved it. Writing proper evals and benchmarks for real-world scenarios is far from trivial.
Benchmarking an agent essentially means freezing, at the very minimum:
- the model
- the model's configuration (e.g. effort, permissions, provider)
- the dataset (e.g. a git repository at a specific sha)
- the code running the agent itself (you can build your own harness, trivial, but you still need to ship it as a single executable, froze in time. benchmarking against closed source runtime like claude code is quite useless, they change too frequently and in ways you cannot directly inspect).
- the tools at agent's disposal. Even a slightly different implementation of tool X (e.g. grep or readfile or sed) has an impact. In general this implies also freezing a very specific container image. In my personal benchmarks I provide a specific list of tools that come with the executable, there's no possibility of interacting with the outside world besides the provided apis, the agent bundles its own tools.
And even then: there's significant noise coming from the LLM providers themselves which noticeably change the models behaviour, I don't know whether this is because they optimize some settings or change the inference over time, etc.
And, last but not least, the output of LLMs is non deterministic.
Also, the LLM as judge presents essentially the same non-deterministic problems, has to be benchmarked itself thoroughly, and writing quality rubrics or "golden answers/outputs" is just difficult. One of the metrics I consistently try to emphasize is to avoid the "shotgun vomit dump" of information. So answers that get right to the point in plain terms avoiding dumps of information filled of jargon on top of the user are rated differently.
In short: its far from trivial to benchmark models on real-world agentic work taken from your personal or professional projects.
And even creating the test cases themselves is hard. No: you cannot take the output of some "sota" and use it as gold standard. This is a very crap approach. It's the sloppiest solution to the problem, in the very sense of slop: plausible, average, lacking any creativity or out of the box thinking, the things that make the real difference in complex software development.
The very point of creating these benchmarks is to find which configuration/model/tools/harness (skills/mcps/documentation/agents.md, etc) works better.
And it only works if you create these benchmarks yourself from genuinely difficult non-trivial work and find a solution that is better by most metrics implementation-wise, albeit you could settle on the implementation solving a series of cases and edge cases.
I've been doing a lot of benchmarking for a long time now with a number of local models for the purposes of spam filtering. The major observation is that there is a lot of variance in model performance. This should not be surprising, as these are probabilistic machines based on random numbers, so your performance will vary from run to run. But this also means that any sort of evaluation of benchmark with a sample size of 1 is essentially worthless for the purposes of model comparison.
In my benchmarks, I started insisting on having at least 5 runs.
This also makes me very suspicious of some of the posted benchmarks that compare models. If the results don't say how many benchmark runs were performed and what the variance was, there is really no basis for comparison. You are just guessing.
Yeah the difference in token usage across difference models they found in the article was broader than I expected, but I was sort of floored by the variance in token usage for the same prompt (run multiple times) with the same model.
Benchmarks are so difficult with ai because as soon as one gets popular it enters the dataset so the next iteration of the model is trained on the solution. I’m not sure if there’s any potential work around here or anyone doing interesting work but would love to hear about it if so
But if this effect were really that strong, the models should be getting nearly 100% on the common benchmarks. But for many of the benchmarks, even after being public for more than a year, the new models only get 60-80%.
In my opinion, this kind of a benchmark doesn't tell much about the models' capabilities on normal software development tasks. It's fun to look at the differences in the output of course, but how often would anyone prompt with very brief instructions, without even hinting the model about caring about any of the details in the outcome nor the implementation?
When there are implicit boundaries, negotiable tradeoffs, taste, whatever, in the mix, then the differences in the model capabilities become way more interesting.
I'd be curious to see Terra xhigh vs Sol low, only in that the visual languages are actually kinda different, that Terra work was a lot less "llm" feeling than a lot of the other designs, wonder if pumping up the effort would result in a more in-depth design but within that style.FWIW I enjoyed reading this way more than any usual benchmark posts we see.
I think we should establish a professional evaluation organization for models; otherwise, it will be difficult for informal evaluations to form standards.
Approaching this from the perspective of a potential customer and not a designer - I find that I actually like the smaller/open model output quite a bit more.
Kimi, GLM, and Deepseek all absolutely run away with the "Can I quickly read the menu and find the address" challenge.
Most of the rest of the pages are stylistic, but hard to parse.
If I were in a car on a mobile phone trying to find the address of the place to meet a friend for coffee... I don't want a bunch of fluff and stylistic design that makes it hard to parse the information on the site.
Interesting exploratory comparison, but I be cautious about treating it as a model benchmark
With only three runs per model, the results are highly sensitive to randomness
Am I wrong or are these evaluations, while interesting, not really meaningful for anyone doing serious development work?
I'm asking because I personally only use AI with specific and detailed instructions, building my projects piece-by-piece.
I mostly don't look at the low level code and some of it I don't understand as much as I'd like, but I very much give much more technical instructions than a simple, two sentence prompt.
So, it seems to me this "oneshot from a simple prompt" eval is fairly meaningless when it comes to model evaluation itself, as it is in no way representative of real world application.
This would be more something for "vibe coders", people with little to no programming background wanting a website?
I checked out the development benchmarks for agents, and was suprized to find that they're not really representative of many of my own day-to-day development workflows.
It appears that they're mostly testing the ability to make business tasks autonomous, with ~20% associated to development tasks (ssh here, install this, etc.), but not actual programming.
The eval world is split into:
1. Long form task based examinations like this that test the ability of the model+harness to remain on task, tool calling, overall effectiveness and taste.
2. More direct 1:1 and qualitative comparisons that you might get with a tool like https://evvl.ai/ - which also uses OpenRouter and does similar one off model comparisons (or lets you use it as a MCP from your dev env to be like: "take the prompt from this loop and try it against these other models")
I've been using code review as my benchmark. Launched a massive parallel Fable/max code review on my lone lisp codebase and recorded all the results and transcripts. Switched to OpenAI and am now running the exact same code review with Sol/max.
It's still a work in progress but preliminary results reveal that Sol is able to reproduce 70%-90% of Fable's performance. This is a very meaningful result for me because code review is what I use AI for.
As someone who's done web development for 20+ years, I find the model personalities pretty dang fascinating, especially how they develop (and evolve) design sensibilities.
I'm interested to know how the classic AI "purple preference" emerged (organically?) and if the beige wave came out of specific training efforts to combat it?
To your point on development work (the code itself), I was talking to some friends on the Google Chrome team about any research understanding the model's preferences around framework ergonomics and abilities to properly implement core web standards for given tasks. I think that would be super fascinating.
I'd assume it comes from Tailwind boilerplate/template sites as it seemed almost all of them were purple at the time
Definitely. It was already a trend before LLMs exploded. And the whole "all websites look the same" has been a thing since at least Bootstrap times.
I'm repeatedly noticing that people working at big ai and tech companies are surprisingly not that... good... at using ai? It's like theyre doing a plausible thing to get something done and calling it a day
You touch a point I quickly skimmed in another comment.
Yes, the most valuable benchmarks and evaluations you can write are those that resemble your work.
The evaluations are extremely hard to write and test.
And yes, virtually all benchmarks are E2E one shots, they do not reflect multi turn processes or how most people interact with LLMs.
Which is why every Opus after 4.6 looks better on benchmarks, but is hard to work with interactively.
> Build a one-page site for a neighbourhood coffee shop: opening hours, the address, a short menu and a photo. Nothing on it changes unless I edit it myself.
If that's the entire prompt, it's quite depressing how much alike these all look. I appreciate some of the details from the Opus 5 version, but I can't help but strongly feel the AI vibes emanating from that design.
I trialed Claude for a sorely needed redesign on my website. I was quite impressed with the results, but decided to hang fire on deploying it. Cue my surprise when I just found the same design applied to a coffee shop!
The similarities are beyond coincidence, to the point I'll be scrapping Claude's version of the redesign.
A user, even without much design background, could ask the AI in a follow up prompt to change a few key parameters like overall color and differentiate it more. Even taking a sample image from a design magazine or blog and saying "incorporate some of the design elements here" would probably work great to diverge it.
Without more knowledge about the technical aspect, it could be a good thing that they're all so similar. If you tell humans to go from their kitchen to their bedroom, they all stand up and walk the same way. Nobody decides to crab walk. Maybe you'd have to cripple the model in some way to do that.
On the other hand, if you want something different with LLMs, all it takes is a few more words of creative flair in the prompt.
This is an interesting point. But even then everyone has their own style. Some might walk with a bit of a swagger, some with a limp, some might have to get in a wheelchair to go over. Would model temperature be another knob to turn beyond a more creative prompt?
The models are giving you very average, middle of the road responses. The average human gait doesn't have swagger, limps, or extensive accessibility needs. If you want the swagger, you need to steer it that way.
> with a limp
> wheelchair
You really took "cripple the model" to heart!
It’s always been interesting to see how similar output is across ostensibly very different models. I remember testing short story writing in the early days and having the models all choose the same niche topic across e.g GPT, Llama, Phi, Claude, etc
It's also kind of pointless. Why does a coffee shop need a website anyway? Nobody is saying, "man I would love to go to this coffee shop but I can't find their website".
If all you want is "opening hours, the address, a short menu and a photo" there are easier and cheaper ways to do that.
> Nobody is saying, "man I would love to go to this coffee shop but I can't find their website".
Me, I'm saying that, and I've skipped going to coffee shops and restaurants because they don't have a website, just a fucking Facebook page. I don't use Meta products and can't see their page if I'm not logged into an account I don't have, so I do what the business owner intended: I go fuck myself and get coffee somewhere else.
But... why do you want to look up a coffee shop online before going there? Honestly have never heard anyone say this in my life before
It amazes me how close minded people have gotten. Do you really have that hard of a time believing there is a use case for a human looking up information on a coffee shop, on the internet? Oh, and god forbid the shop owner actually wants to provide it!
To see the hours, get a sense of the menu. There are many matcha shops by me for instance and my wife likes to check the specials before choosing which one to go to.
Doesn't most of this show up on google anyway?
It sounds like you know everything about the internet and people's preferences, why ask?
Where do you think Google and co. source that data from? Having an actual, searchable, up to date menu is, just for a11y and accuracy far more valuable than someone’s poorly lit, low res upload of an photo taken from the menu circa 2019, just to name one advantage. Have found data from restaurants without their own webpages on Maps utterly inaccurate, even suggesting some that have been shut down for months to where there is clear value. Then there is restaurants closing for holidays, illness, etc. and the ability to make reservations without calling. Whether these need their own over an Astro template is another topic, but having a webpage at all is incredibly valuable.
Why does anyone look up anything before going there? Maybe no one should have a website, everyone should go check everything out themselves in person like the telegraph hasn't been invented.
Why does your brain instantly go to "whats the cheapest and easiest way to do things", instead of "whats creative, fun, interactive, usable, accessible, etc...".
This is the reason the world is slowly becoming a boring ass place and the internet basically consists of 3 websites that are all trying to suck your soul dry.
If I ran a business, a website that stands out from the crowd and provides a good experience would be in my top 3 things to do.
Finally some really useful apples-to-apples comparisons rather than endless benchmarkmaxxing and anecdata. I want to see this test run again for the open-weights models that fit into 128GB combined RAM. Like how does Muse-glimmer compare with qwen3.6-35B?
I like the output of the cheaper/older models better, surprisingly, say Gemini 3.1 or DeepSeek V4. They're mostly no frills and just text, and ironically look less AI-generated to me because of the lack of hip slogans and graphics. Definitely closer to what I'd want for my own site, but I don't claim to know what people want from a website for a coffeeshop.
Building ad-hoc evals is trivial these days. And you can place an LLM judge in front to disambiguate between output. e.g. I have Frigate gating a video feed so that an LLM can watch our home cameras and the labeling task takes a few minutes and then the evals run at fairly low cost https://wiki.roshangeorge.dev/w/images/4/4e/Screenshot_-_Eva...
This means that generic benchmarks and evals are sort of passé. There's no need to have them write "make a coffee shop page" or whatever. Instead, simply focus on the real problem you have and evaluate whether they solve that problem. You want to move along the price-performance frontier alone and the SOTA models are decidedly at the top of the performance curve but very expensive so they serve very well to produce the gold standard and to judge.
The change from prior to today is that benchmaxxing and per-token pricing allowing for evaluation point to the same direction: do not proxy results. Instead deploy the highest end model you have as a judge for others until you have statistical confidence in discrimination and move along the task-specific frontier.
> Building ad-hoc evals is trivial these days.
I highly doubt that you have solved it. Writing proper evals and benchmarks for real-world scenarios is far from trivial.
Benchmarking an agent essentially means freezing, at the very minimum:
- the model
- the model's configuration (e.g. effort, permissions, provider)
- the dataset (e.g. a git repository at a specific sha)
- the code running the agent itself (you can build your own harness, trivial, but you still need to ship it as a single executable, froze in time. benchmarking against closed source runtime like claude code is quite useless, they change too frequently and in ways you cannot directly inspect).
- the tools at agent's disposal. Even a slightly different implementation of tool X (e.g. grep or readfile or sed) has an impact. In general this implies also freezing a very specific container image. In my personal benchmarks I provide a specific list of tools that come with the executable, there's no possibility of interacting with the outside world besides the provided apis, the agent bundles its own tools.
And even then: there's significant noise coming from the LLM providers themselves which noticeably change the models behaviour, I don't know whether this is because they optimize some settings or change the inference over time, etc.
And, last but not least, the output of LLMs is non deterministic.
Also, the LLM as judge presents essentially the same non-deterministic problems, has to be benchmarked itself thoroughly, and writing quality rubrics or "golden answers/outputs" is just difficult. One of the metrics I consistently try to emphasize is to avoid the "shotgun vomit dump" of information. So answers that get right to the point in plain terms avoiding dumps of information filled of jargon on top of the user are rated differently.
In short: its far from trivial to benchmark models on real-world agentic work taken from your personal or professional projects.
And even creating the test cases themselves is hard. No: you cannot take the output of some "sota" and use it as gold standard. This is a very crap approach. It's the sloppiest solution to the problem, in the very sense of slop: plausible, average, lacking any creativity or out of the box thinking, the things that make the real difference in complex software development.
The very point of creating these benchmarks is to find which configuration/model/tools/harness (skills/mcps/documentation/agents.md, etc) works better.
And it only works if you create these benchmarks yourself from genuinely difficult non-trivial work and find a solution that is better by most metrics implementation-wise, albeit you could settle on the implementation solving a series of cases and edge cases.
I've been doing a lot of benchmarking for a long time now with a number of local models for the purposes of spam filtering. The major observation is that there is a lot of variance in model performance. This should not be surprising, as these are probabilistic machines based on random numbers, so your performance will vary from run to run. But this also means that any sort of evaluation of benchmark with a sample size of 1 is essentially worthless for the purposes of model comparison.
In my benchmarks, I started insisting on having at least 5 runs.
This also makes me very suspicious of some of the posted benchmarks that compare models. If the results don't say how many benchmark runs were performed and what the variance was, there is really no basis for comparison. You are just guessing.
Yeah the difference in token usage across difference models they found in the article was broader than I expected, but I was sort of floored by the variance in token usage for the same prompt (run multiple times) with the same model.
You betta lose yoself in the context
It’s not theft, you own it
You betta neva use Go (go) (go)
You only get 1 prompt
Do not use canvas (No!)
Cause opportunity comes once in a lifetime
Benchmarks are so difficult with ai because as soon as one gets popular it enters the dataset so the next iteration of the model is trained on the solution. I’m not sure if there’s any potential work around here or anyone doing interesting work but would love to hear about it if so
But if this effect were really that strong, the models should be getting nearly 100% on the common benchmarks. But for many of the benchmarks, even after being public for more than a year, the new models only get 60-80%.
In my opinion, this kind of a benchmark doesn't tell much about the models' capabilities on normal software development tasks. It's fun to look at the differences in the output of course, but how often would anyone prompt with very brief instructions, without even hinting the model about caring about any of the details in the outcome nor the implementation?
When there are implicit boundaries, negotiable tradeoffs, taste, whatever, in the mix, then the differences in the model capabilities become way more interesting.
My "favourite" site was <https://6a6fa3376288679d094a8437--ar-testing-coffee-2b15c6a0...> with footer image captions that are amazing. Precision Late Art being the best one.
This seems to be the easiest way to get on HN front page:
Come up with an arbitrary test, let a bunch of LLMs work on it. Make some very subjective judgement about the result...
Pretty interesting how DeepSeek V4 Flash 0731 has such disparate (and cheap!) results. I would never guess they come from the same model and prompt.
I'd be curious to see Terra xhigh vs Sol low, only in that the visual languages are actually kinda different, that Terra work was a lot less "llm" feeling than a lot of the other designs, wonder if pumping up the effort would result in a more in-depth design but within that style.FWIW I enjoyed reading this way more than any usual benchmark posts we see.
I think we should establish a professional evaluation organization for models; otherwise, it will be difficult for informal evaluations to form standards.
Approaching this from the perspective of a potential customer and not a designer - I find that I actually like the smaller/open model output quite a bit more.
Kimi, GLM, and Deepseek all absolutely run away with the "Can I quickly read the menu and find the address" challenge.
Most of the rest of the pages are stylistic, but hard to parse.
If I were in a car on a mobile phone trying to find the address of the place to meet a friend for coffee... I don't want a bunch of fluff and stylistic design that makes it hard to parse the information on the site.
I always enjoy these comparisons between models, especially when they demonstrate the actual costs in addition to the outputs.
this is fun!
Interesting exploratory comparison, but I be cautious about treating it as a model benchmark With only three runs per model, the results are highly sensitive to randomness
> Vector graphics actually require a lot of work from the models
How so? Surely they can just steal such generic graphics off existing web sites.