This looks great! I also think speed should be part of the metric (i.e. how long does the model take to actually solve a task). For me, I prefer to run expensive models such as Sol on light reasoning, which usually gives me good answers with quick responses.
For my style of coding (quick back-and-forths and corrections) it makes a big difference if a model comes back in 1-2 minutes compared to 5-10, and I am happy to pay a bit extra for that.
to be fair, you don't need to start the y-axis at zero [0] but for some of the graphs where the lowest value is close to 0, you probably should which would include a few charts in this report
this could also just be an Excel artifact where it auto-selects the 'relevant' range with no adjustment for how proportionally close to 0 the values are - a professional data sci publishing to a journal should really know better but for a personal blog by an SWE, this isn't the worst sin in the world
Yeah, this doesn't include older models, some of which were already saturating many common tasks a year ago. I'll often add them to the AA graph for reference.
I think the key there is “hardware you already own.”
If you own a graphics card you bought for gaming or a laptop you bought for doing schoolwork there is $0 in cost of local AI tokens, because 100% of the cost was assigned to doing other things.
Well done. The inability to switch between log and linear always bothered me.
Another thing is if you're using the subscriptions with OpenAI or Anthropic you get an order of magnitude discount relative to the per-token price. So you need to move their models ~10x to the left on the plots to get a fair comparison.
> The first issue I have with it is that it uses a logarithmic scale on the cost axis. Using a log scale is the only way to make you spot the difference between a model that costs $0.015 per task and one that costs $0.032, while the same plot contains a model that costs $3.69 — almost 250 times as expensive. However, the net result is that the viewers can no longer appreciate the immensity of the price difference between the cheap models and the heavy ones; nor can they realize how inconsequential the price differences are between the cheap models.
This is an asinine complaint, and nobody can seriously tell me that the last plot on their page [0] is more readable than the AA one [1]. If I'm using a model at the lower range of the cost scale for whatever list of tasks, and i switch to another model at the lower end of the cost scale, my spending might double anyways! This should be reflected in the plot, and linear scale doesn't do it justice.
It's also much easier to see the mentioned pareto frontier in the log plot than in the linear one.
I can see why they disagree with the pricing determination for open/local models, but I don't think there is one clear right way to do it. So how do they do it instead?
>Hardware is priced at zero, on the basis that both an RTX 3090 PC and a 64GB Strix Halo are desirable gaming/work machines anyways.
...oh
Would have been nice to mention explicitly how the pareto frontier changes with those new calculations.
It’s funny they state log plot is “the only way” to keep the cheap area readable, say they hate it, and then immediately have to zoom into their non-log plot cheap area because it’s unreadable.
Open Teams originally wanted to rent out open source developers to sponsors with Oliphant controlling everything. Now they pivoted to installing local LLMs (on what hardware exactly?).
What will happen is that this will be the third consultancy with a lofty narrative after Enthought and Anaconda that Oliphant established. It is always bait-and-switch.
This looks great! I also think speed should be part of the metric (i.e. how long does the model take to actually solve a task). For me, I prefer to run expensive models such as Sol on light reasoning, which usually gives me good answers with quick responses.
For my style of coding (quick back-and-forths and corrections) it makes a big difference if a model comes back in 1-2 minutes compared to 5-10, and I am happy to pay a bit extra for that.
Complaining about "bad charting" and posting a chart with y-axis that doesn't start at 0 is kinda weird.
to be fair, you don't need to start the y-axis at zero [0] but for some of the graphs where the lowest value is close to 0, you probably should which would include a few charts in this report
this could also just be an Excel artifact where it auto-selects the 'relevant' range with no adjustment for how proportionally close to 0 the values are - a professional data sci publishing to a journal should really know better but for a personal blog by an SWE, this isn't the worst sin in the world
[0] https://digitalblog.ons.gov.uk/2016/06/27/does-the-axis-have...
Yeah, this doesn't include older models, some of which were already saturating many common tasks a year ago. I'll often add them to the AA graph for reference.
Why use electricity alone for open models? Surely you’d want to spread the cost of hardware over the period too?
I think the key there is “hardware you already own.”
If you own a graphics card you bought for gaming or a laptop you bought for doing schoolwork there is $0 in cost of local AI tokens, because 100% of the cost was assigned to doing other things.
Well done. The inability to switch between log and linear always bothered me.
Another thing is if you're using the subscriptions with OpenAI or Anthropic you get an order of magnitude discount relative to the per-token price. So you need to move their models ~10x to the left on the plots to get a fair comparison.
When I accessed the site, it showed the FBI badge says that the site was blocked and redirect to fbi.gov !! WTH?
Not seeing that, that's weird... Maybe a site sharing the same IP is blocked by the FBI through your ISP?
to choose a model, you have to consider intelligence, price, and tokens per second. would be nice to see the 3 dimensional plot.
> Why AA’s plot is misleading
> The first issue I have with it is that it uses a logarithmic scale on the cost axis. Using a log scale is the only way to make you spot the difference between a model that costs $0.015 per task and one that costs $0.032, while the same plot contains a model that costs $3.69 — almost 250 times as expensive. However, the net result is that the viewers can no longer appreciate the immensity of the price difference between the cheap models and the heavy ones; nor can they realize how inconsequential the price differences are between the cheap models.
This is an asinine complaint, and nobody can seriously tell me that the last plot on their page [0] is more readable than the AA one [1]. If I'm using a model at the lower range of the cost scale for whatever list of tasks, and i switch to another model at the lower end of the cost scale, my spending might double anyways! This should be reflected in the plot, and linear scale doesn't do it justice.
It's also much easier to see the mentioned pareto frontier in the log plot than in the linear one.
I can see why they disagree with the pricing determination for open/local models, but I don't think there is one clear right way to do it. So how do they do it instead?
>Hardware is priced at zero, on the basis that both an RTX 3090 PC and a 64GB Strix Halo are desirable gaming/work machines anyways.
...oh
Would have been nice to mention explicitly how the pareto frontier changes with those new calculations.
[0] https://openteams.com/wp-content/uploads/2026/09/all_models-... [1] https://artificialanalysis.ai/#intelligence-comparison-tabs
It’s funny they state log plot is “the only way” to keep the cheap area readable, say they hate it, and then immediately have to zoom into their non-log plot cheap area because it’s unreadable.
Open Teams originally wanted to rent out open source developers to sponsors with Oliphant controlling everything. Now they pivoted to installing local LLMs (on what hardware exactly?).
What will happen is that this will be the third consultancy with a lofty narrative after Enthought and Anaconda that Oliphant established. It is always bait-and-switch.