"Almost every model finds the same winning ideas. What separates the best traces is what an experiment leaves behind. They preserve weak signals long enough to validate them, but they also have a better understanding of the results."
Curious if a harness that helped preserve signals in some history log would change the outcome.
Also curious if different goal prompts would have changed the outcome. Not a bunch of prompt engineering; small diffs like "consider novel solutions, keep track of weak signals".
IMO they allocated quite a bit of GPU time to the same goal prompt.
I might’ve missed it, but why was Fable 5 tested on high while Opus 5 was tested on max? Seems like quite a few of them aren’t on the same effort setting as well. Although effort doesn’t really matter anymore since they can change it dynamically, seems like that might be viewed as an experimental error to some.
Basically they do 8 runs trying to optimize to under 3.28 loss in the fewest training steps possible under time/token constraint. I dunno why 18 * 8 != 153 (it's 144)
I misread the title and thought it would be about the (for lack of a better term) NanoGPT speedrun[1]. Which previous to the article was meant to be the world speed records for Andrej Karpathy's GPT-2 (small) reproduction.
"Almost every model finds the same winning ideas. What separates the best traces is what an experiment leaves behind. They preserve weak signals long enough to validate them, but they also have a better understanding of the results."
Curious if a harness that helped preserve signals in some history log would change the outcome.
Also curious if different goal prompts would have changed the outcome. Not a bunch of prompt engineering; small diffs like "consider novel solutions, keep track of weak signals".
IMO they allocated quite a bit of GPU time to the same goal prompt.
I might’ve missed it, but why was Fable 5 tested on high while Opus 5 was tested on max? Seems like quite a few of them aren’t on the same effort setting as well. Although effort doesn’t really matter anymore since they can change it dynamically, seems like that might be viewed as an experimental error to some.
Neat!
The graphs show the "best validated result" for each model. I wonder how much variation there is between runs for a model?
“We ran 153 autonomous runs across 18 frontier models on the nanoGPT optimizer speedrun.”
Uh.. okay.. but whats a run… read blog
“We want to measure how well frontier models can conduct research….””we ran 153 autonomous runs on the nanoGPT optimizer speedrun across”
Okay but what is a optimiser run and what connection does it have to being good at research?
“For comparison, Anthropic's internal automated AI R&D evaluation optimizes a model on a CPU node,”
So I should go look what Anthropic was doing to understand?
Why not just explain what it means in their blog..
I think that's explained here:
https://www.primeintellect.ai/blog/measuring-autonomous-rese...
Basically they do 8 runs trying to optimize to under 3.28 loss in the fewest training steps possible under time/token constraint. I dunno why 18 * 8 != 153 (it's 144)
I misread the graph and genuinely thought you put NanoGPT where Fable is.
Lol.
I misread the title and thought it would be about the (for lack of a better term) NanoGPT speedrun[1]. Which previous to the article was meant to be the world speed records for Andrej Karpathy's GPT-2 (small) reproduction.
1. https://github.com/KellerJordan/modded-nanogpt#world-record-...
You're not the only one. I thought so too.
I just ran it the last couple days extensively to verify my data training pipeline I'm building for my gonano SIMD port.
Given that the speed records and the runs are sponsored by the same company I was confused a bit.