I'm a co-author on a recent blog post from METR about the NanoGPT speed-run here [1]. I think it'd be of interest to anyone who enjoyed the original post. Appreciate the good beefy runs and spend here, it's a (from my experience) not super easy to do!
(Also: just to label this comment clearly: it's written hastily from a car, and based on lighter skim of the original blog post [2] than would be ideal. Please correct any mistakes or misinterpretations I have here!)
A few callouts:
1. If I understand the experiment correctly, they start the models at the original baseline. If this is true, I have some worries about contamination. Appendix C [3] has some notes on likely contamination we observed in recent models. This makes interpretation a bit harder.
2. If you look at the token scaling plots in the original post: not all models are hitting a performance plateau. This is an important point: we shouldn't treat these results as a full upper-bound on capabilities, but rather some bound on model performance @ cost (assuming good scaffolding, etc).
3. Our post is mostly about how to _interpret_ the results given here. Quoting from our post: "If we can estimate performance as a function of cost for both humans and agents, we can measure the “expenditure horizon” as the point at which those curves cross: the budget at which humans become more cost-effective than AIs. "
Feedback appreciated. I think you can see expenditure horizon as a sibling methodology (that is much less validated) to METR's time horizon work [4] - roughly, instead of baselining against the time it takes humans to complete tasks, you baseline against cost. This may be better suited to some types of problems similar to NanoGPT.
"Almost every model finds the same winning ideas. What separates the best traces is what an experiment leaves behind. They preserve weak signals long enough to validate them, but they also have a better understanding of the results."
Curious if a harness that helped preserve signals in some history log would change the outcome.
Also curious if different goal prompts would have changed the outcome. Not a bunch of prompt engineering; small diffs like "consider novel solutions, keep track of weak signals".
IMO they allocated quite a bit of GPU time to the same goal prompt.
I might’ve missed it, but why was Fable 5 tested on high while Opus 5 was tested on max? Seems like quite a few of them aren’t on the same effort setting as well. Although effort doesn’t really matter anymore since they can change it dynamically, seems like that might be viewed as an experimental error to some.
Basically they do 8 runs trying to optimize to under 3.28 loss in the fewest training steps possible under time/token constraint. I dunno why 18 * 8 != 153 (it's 144)
They gave 18 frontier models the task of “researching” how to improve a lab-rat nano model’s training. Stopping when it met a quality goal of a target loss rate. During each autonomous research session, the AI repeatedly tried changes, tested them, and used the results to decide what to try next. They repeated the whole research session many times with different seeds to average out variance.
This seems very cool, but I'm not sure I understand exactly what it's doing. Are they making a new speculative drafter for Qwen 3.8 27B? Maybe they're optimizing the MLX code for the decoder itself? Thank you in advance.
I'm highly interested in their Grok 4.6 run which is currently running. I think it is a very good fit for doing really well on this benchmark. Will have to check this page again in a couple of days.
I misread the title and thought it would be about the (for lack of a better term) NanoGPT speedrun[1]. Which, previous to the article, was meant to be the world speed records for Andrej Karpathy's GPT-2 (small) reproduction.
I'm a co-author on a recent blog post from METR about the NanoGPT speed-run here [1]. I think it'd be of interest to anyone who enjoyed the original post. Appreciate the good beefy runs and spend here, it's a (from my experience) not super easy to do!
(Also: just to label this comment clearly: it's written hastily from a car, and based on lighter skim of the original blog post [2] than would be ideal. Please correct any mistakes or misinterpretations I have here!)
A few callouts:
1. If I understand the experiment correctly, they start the models at the original baseline. If this is true, I have some worries about contamination. Appendix C [3] has some notes on likely contamination we observed in recent models. This makes interpretation a bit harder.
2. If you look at the token scaling plots in the original post: not all models are hitting a performance plateau. This is an important point: we shouldn't treat these results as a full upper-bound on capabilities, but rather some bound on model performance @ cost (assuming good scaffolding, etc).
3. Our post is mostly about how to _interpret_ the results given here. Quoting from our post: "If we can estimate performance as a function of cost for both humans and agents, we can measure the “expenditure horizon” as the point at which those curves cross: the budget at which humans become more cost-effective than AIs. "
Feedback appreciated. I think you can see expenditure horizon as a sibling methodology (that is much less validated) to METR's time horizon work [4] - roughly, instead of baselining against the time it takes humans to complete tasks, you baseline against cost. This may be better suited to some types of problems similar to NanoGPT.
[1] https://metr.org/blog/2026-07-21-expenditure-horizon/ (Most of this work was my coauthors listed on the post, not me). [2] https://www.primeintellect.ai/blog/measuring-autonomous-rese... [3] https://metr.org/blog/2026-07-21-expenditure-horizon/#append... [4]https://metr.org/time-horizons/
Curious if a harness that helped preserve signals in some history log would change the outcome.
Also curious if different goal prompts would have changed the outcome. Not a bunch of prompt engineering; small diffs like "consider novel solutions, keep track of weak signals".
IMO they allocated quite a bit of GPU time to the same goal prompt.
Uh.. okay.. but whats a run… read blog
“We want to measure how well frontier models can conduct research….””we ran 153 autonomous runs on the nanoGPT optimizer speedrun across”
Okay but what is a optimiser run and what connection does it have to being good at research?
“For comparison, Anthropic's internal automated AI R&D evaluation optimizes a model on a CPU node,”
So I should go look what Anthropic was doing to understand?
Why not just explain what it means in their blog..
https://www.primeintellect.ai/blog/measuring-autonomous-rese...
Basically they do 8 runs trying to optimize to under 3.28 loss in the fewest training steps possible under time/token constraint. I dunno why 18 * 8 != 153 (it's 144)
They gave 18 frontier models the task of “researching” how to improve a lab-rat nano model’s training. Stopping when it met a quality goal of a target loss rate. During each autonomous research session, the AI repeatedly tried changes, tested them, and used the results to decide what to try next. They repeated the whole research session many times with different seeds to average out variance.
The graphs show the "best validated result" for each model. I wonder how much variation there is between runs for a model?
I wrote a quick review of Grok 4.6 here: https://taonexus.com/publicfiles/aug2026/grok-4-6-review/
Lol.
1. https://github.com/KellerJordan/modded-nanogpt#world-record-...
I just ran it the last couple days extensively to verify my data training pipeline I'm building for my gonano SIMD port.
Given that the speed records and the runs are sponsored by the same company I was confused a bit.