Hi HN, I’m Guanlan, founder of Runta. The eval runs on our runtime, so weigh this accordingly.
The short version: when you change the model, think about changing the harness too. On any one model, cost per pass varies by up to 4.4x
What we ran: 22 harness configurations × 5 open-weight models × 62 terminal tasks × 4 attempts. 27,280 runs, 83B tokens. Each harness ran with default settings at maximum reasoning effort, each trial in a fresh sandbox. Costs are Baseten list prices times reported tokens.
What we found:
The lowest-cost harness changes with the model. Codex CLI ranks 1st on cost with GLM-5.3 and 15th with Kimi-K3.
Paying more doesn’t buy passes. The harness with the highest pass rate ranked anywhere from 4th to 19th on cost.
The model still sets most of the bill (93% of cost variation). The harness is the second lever.
Starting prompt size doesn’t predict cost. Output tokens do.
64% of inference spend went to runs that didn’t pass.
Hi Guanlan, I vouched for your comment since you involved in this post, but you should be aware that all of your posts have been flagged, likely for AI patterns in your writing.
Hi HN, I’m Guanlan, founder of Runta. The eval runs on our runtime, so weigh this accordingly.
The short version: when you change the model, think about changing the harness too. On any one model, cost per pass varies by up to 4.4x
What we ran: 22 harness configurations × 5 open-weight models × 62 terminal tasks × 4 attempts. 27,280 runs, 83B tokens. Each harness ran with default settings at maximum reasoning effort, each trial in a fresh sandbox. Costs are Baseten list prices times reported tokens.
What we found: The lowest-cost harness changes with the model. Codex CLI ranks 1st on cost with GLM-5.3 and 15th with Kimi-K3. Paying more doesn’t buy passes. The harness with the highest pass rate ranked anywhere from 4th to 19th on cost. The model still sets most of the bill (93% of cost variation). The harness is the second lever. Starting prompt size doesn’t predict cost. Output tokens do. 64% of inference spend went to runs that didn’t pass.
Hi Guanlan, I vouched for your comment since you involved in this post, but you should be aware that all of your posts have been flagged, likely for AI patterns in your writing.