The big trend in that graph is that smaller models collapse the most. Everything with "-flash" in the name, plus OpenAI Luna.
Look at GPT 5.6-Sol vs. GPT 5.6-Luna in particular.
(The GLM pair shouldn't be expected follow this pattern, because they're misnamed: GLM-5.3 and GLM-5.3-flash are distinct lineages. The -flash is an early release of a new pretrain on a new architecture).
If this *is* the main factor, I'd expect Kimi K3.1 and Gemini Carbon would also do well on this test (unlike Gemini-flash); while Haiku 6 would collapse (breaking the trend of the other recent Claude models). It's a falsifiable prediction.
Another hypothesis is that it's distilled models which collapse the most (small models from big labs are distilled versions of their large models).
It's not unreasonable to assume Kimi 3.1 is also a distilled model even though it's large (the difference being that it's a distilled from US models). I wouldn't be surprised if it also collapsed on this benchmark.
Academically a - very - interesting article, which confirms my ongoing suspicion of LLM Benchmarks in general ( probably why I like it so much). Another thing that caught my eye is DeepSWE having a bunch of bugs in them. The more I learn about these benchmarks the more sceptical I become. I would expect such a prolific benchmark to have these bugs ironed out by now.
The practical takeaway, however, seems to be an interesting "don't use max mode". The passage about deepseeks weird API behavior dropping to pretty much zero the moment you used high says a lot.
Also, personally, even though Opus is clearly so much ahead of the Open Source models after the changes, I think the adjusted score is still very much competitive. I see other commenters using this as proof to justify why their opinions of the open source models - but a drop from 74% to ~50% is, while a big drop, still about a ~50% pass rate afterwards. That's far from useless for coding, especially since DeepSWE seems to be such a difficult benchmark. It simply seems to more accurately reflect the capabilities of these models.
All in all, I'd love to see more of such articles, and will probably keep an eye on this one - if they continue "benchmarking" these models this way.
AI Labs actually. They would love the perfect benchmar representing real usecase. I'd even dare and guess that they don't share much publically because it is a huge competitive advantage.
This DEFINITELY squares with my anecdata that Gemini 3.8 Flash is not even remotely close to Opus 5.5 on the types of problems DeepSWE CLAIMS to be benchmarking.
Astra does seem close.
I'd be curious to see Argon on there, since my experience is that Argon is also significantly behind Opus 5.5 and Astra despite the benchmarks (obviously much closer than Flash, though).
I've been meaning to take a closer look at what's behind the curtain of DeepSWE - since I built what I thought should be the easiest possible coding benchmark - and the models I tested it on scored roughly the same as they do on DeepSWE... Which is supposed to be hard...
One thing to consider is that even when some model has big gap between those two benchmark it doesn't mean it is much worse. It could be worse becasue he has assuptions based on training but on real scenario it wouldn't have it so it still could achive great results
DeepSWE was released on May and Deepseek v4 flash was released on April. By definition it couldn't be benchmaxxed on this particular benchmark. My guess is the mutation is not like real world and it is tripping small models, not benchmaxed model.
It's not clear which v4 flash they used. There are 3 different releases for that one.
Edit: I think that's the answer. This page lists the score as 53.3% which is close to the 54.4% reported here https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731 for the 0731 update. The original from April was called preview, doesn't list the score, but in the comparison with the update it scores 7.3%.
Funny, I was just talking about this on HN a couple of days ago[0..2] and had a thread full of anklebiters telling me I'm using Luna wrong. It's clear that the vast majority of people never privately benchmark, because anyone who does knows models like Luna aren't remotely close to the performance the published benchmarks claim, immediately failing when even a single unexpected constraint is introduced, no matter how trivial it is, exactly like Maxxbench's "filename must be lowercase". It's nice to have a public one to point to, but unfortunately every time you publish your private benchmarks they become targets for the next round, and as soon as the next generation of models releases you're back to arguing with people who take benchmaxxed results at face value.
you completely missed the point that the "ankle biters" were making - smaller models can still be useful despite being less intelligent. nobody makes the argument that luna is just as smart as opus, but they do make the argument that someone sufficiently knowledgeable and skilled can use smaller models to great effect. just because you and others require the maximally powerful models to make sense of vague, underspecified prompts, to hold your hand throughout the entire development lifecycle and make decisions for you, does not mean we all do.
This is the same stupid argument trotted out repeatedly in the thread, and it doesn't hold up to reality. Would you care to explain how adding specifications helps a model that demonstrably ignores specifications? A model that cannot even handle the trivial specification of "the filename must be lowercase" is not going to get better when you add more specifications it ignores. I would actually guess that you're underspecifying relative to me. Perhaps you're happy with whatever slop you get and don't notice Luna's overwhelming incapability to actually provide what one requires, because you don't have requirements.
> Most models tried pleasing an imagined grader over the prompt
>When we read the trajectories, we saw that many models had figured out they were in a benchmark and started second-guessing the prompt. Instead of just building what we asked for, they started coding to the tests they thought the grader would run.
Different expectations maybe. In my experience, DSF 4.1 high/max is basically what Terra should be for coding, and is very acceptable (Terra is hot garbage, yes even the latest 6.1). Luna at xhigh+ stands out as a cheap implementor/reviewer of very very well defined tasks, but goes off the rails if you're not very specific, Sol 6.1 and Sonnet 5.5 at medium are basically powerhouses for implementation with a half decent plan (slightly prefer sonnet), and Opus 5.5 high is an excellent planner (and implementor). And Astra is amazing at burning money.
I was beginning to worry I had overlooked the inner beauty of DeepSeek. When I have tried it, it was totally useless. Never understood HN cheering for it.
Don't know what you're doing, but it's good enough for the reasonably sized tasks I give LLMs. It's also a much better price. My browser stopped rendering the UI frame recently because I messed up permissions on /usr/share/icons. Took DS 4.1 flash about 3 minutes and a few cents to diagnose it. Same harness with opus 5 took twice as long and several dollars.
Moreover, the months I've used anthropic/OAI exclusively my API token costs are in the hundreds (or sometimes thousands). With DS flash as the first agent I reach for, I rarely exceed $40/mo.
I figured it was assumed I was including that? It's been fine as long as I break tasks up into reasonably sized chunks, which is fine because I think you should only be generating as much as you can comfortably review at a time.
I've had significant success using DS 4 Pro and now 4.1 flash for fairly complex coding tasks on C++, Rust and Python projects at home.
I have access to Claude Code at work (and use it quite a bit), so I feel I can honestly compare them.
You do have to plan things first a bit more with DS (than say Opus can (Opus will have more chance of one-shotting something, although I generally don't think that's a good idea), although it often over-engineers and tries to be too helpful in my experience) and be a bit more explicit at times, but I feel it's quite close to Sonnet in performance for implementing plans, debugging issues and explaining how codebases work.
This has always been a problem with the Internet. When the iPad was released, anti-Apple subreddits talked about what a garbage device it was. Wishful thinking dominates a lot of human content generators. Extracting information from humanslop is harder than avoiding it entirely.
Benchmarks they share are on the much older v4 of DSF. DSF V4.1 1017 is a big step up in benchmarks and is also actually twice as many parameters.
Anyway, I use DSF v4.1 on max as much as I use Opus 5.5 on xhigh, Opus is definitely a more versatile model, but I’m not getting any value using it to implement well defined tasks that DSF v4.1 does just as well at for a fraction of the cost.
You see where GLM-5.3 is in this test down but respectable, but this model benches below DSF 4.1. TLDR basically nothing in this benchmark invalidates the Deepseek that everyone is using today.
Assume about 50% of internet writing has zero tether to reality and is written out of pure motivation to shape a preferred narrative (and usually the same one).
When you see this your worldview will instantly become much clearer.
Not everyone is operating with the same constraints. As for me, once the models got good enough (~opus 4.5 levels of intelligence) that they could solve most problems that I was dealing with, I stopped following the frontier intelligence trajectory for daily use and instead followed constant intelligence and decreasing cost. Nowadays I primarily use GPT 6.1 luna, and DeepSeek 4.1 flash.
I've also been very happy playing around with qwen 3.8 27b running locally for personal projects. It sometimes can't quite get there, and that's when I'll reach for GPT.
Hard to say if you don't share what kind of project, workflow and prompt you're trying tbh. I've been driving it mainly to write a quite big golang and android app and most of the tasks are all handled fine by DeepSeek.
> There is always a huge discrepancy between benchmarks and real world capabilities.
I mean of course, the real world is messy, the models are only good for what they've been trained on, and post-training frequently takes benchmark scores into account too, leading to this sort of "Great score, sucks in the real world" syndrome that seemingly most models are affected by.
Look at GPT 5.6-Sol vs. GPT 5.6-Luna in particular.
(The GLM pair shouldn't be expected follow this pattern, because they're misnamed: GLM-5.3 and GLM-5.3-flash are distinct lineages. The -flash is an early release of a new pretrain on a new architecture).
If this *is* the main factor, I'd expect Kimi K3.1 and Gemini Carbon would also do well on this test (unlike Gemini-flash); while Haiku 6 would collapse (breaking the trend of the other recent Claude models). It's a falsifiable prediction.
It's not unreasonable to assume Kimi 3.1 is also a distilled model even though it's large (the difference being that it's a distilled from US models). I wouldn't be surprised if it also collapsed on this benchmark.
The practical takeaway, however, seems to be an interesting "don't use max mode". The passage about deepseeks weird API behavior dropping to pretty much zero the moment you used high says a lot.
Also, personally, even though Opus is clearly so much ahead of the Open Source models after the changes, I think the adjusted score is still very much competitive. I see other commenters using this as proof to justify why their opinions of the open source models - but a drop from 74% to ~50% is, while a big drop, still about a ~50% pass rate afterwards. That's far from useless for coding, especially since DeepSWE seems to be such a difficult benchmark. It simply seems to more accurately reflect the capabilities of these models.
All in all, I'd love to see more of such articles, and will probably keep an eye on this one - if they continue "benchmarking" these models this way.
Astra does seem close.
I'd be curious to see Argon on there, since my experience is that Argon is also significantly behind Opus 5.5 and Astra despite the benchmarks (obviously much closer than Flash, though).
I've been meaning to take a closer look at what's behind the curtain of DeepSWE - since I built what I thought should be the easiest possible coding benchmark - and the models I tested it on scored roughly the same as they do on DeepSWE... Which is supposed to be hard...
Something seems amiss.
Edit: I think that's the answer. This page lists the score as 53.3% which is close to the 54.4% reported here https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731 for the 0731 update. The original from April was called preview, doesn't list the score, but in the comparison with the update it scores 7.3%.
https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017
[0] https://news.ycombinator.com/item?id=50373961 [1] https://news.ycombinator.com/item?id=50374111
This is the same stupid argument trotted out repeatedly in the thread, and it doesn't hold up to reality. Would you care to explain how adding specifications helps a model that demonstrably ignores specifications? A model that cannot even handle the trivial specification of "the filename must be lowercase" is not going to get better when you add more specifications it ignores. I would actually guess that you're underspecifying relative to me. Perhaps you're happy with whatever slop you get and don't notice Luna's overwhelming incapability to actually provide what one requires, because you don't have requirements.
>When we read the trajectories, we saw that many models had figured out they were in a benchmark and started second-guessing the prompt. Instead of just building what we asked for, they started coding to the tests they thought the grader would run.
I ran into the same problem on my English exams.
For my use case deepseek 4.1 flash is almost as dumb as haiku 5.5. Both are fast garbage producers.
Moreover, the months I've used anthropic/OAI exclusively my API token costs are in the hundreds (or sometimes thousands). With DS flash as the first agent I reach for, I rarely exceed $40/mo.
I have access to Claude Code at work (and use it quite a bit), so I feel I can honestly compare them.
You do have to plan things first a bit more with DS (than say Opus can (Opus will have more chance of one-shotting something, although I generally don't think that's a good idea), although it often over-engineers and tries to be too helpful in my experience) and be a bit more explicit at times, but I feel it's quite close to Sonnet in performance for implementing plans, debugging issues and explaining how codebases work.
Anyway, I use DSF v4.1 on max as much as I use Opus 5.5 on xhigh, Opus is definitely a more versatile model, but I’m not getting any value using it to implement well defined tasks that DSF v4.1 does just as well at for a fraction of the cost.
You see where GLM-5.3 is in this test down but respectable, but this model benches below DSF 4.1. TLDR basically nothing in this benchmark invalidates the Deepseek that everyone is using today.
When you see this your worldview will instantly become much clearer.
I've also been very happy playing around with qwen 3.8 27b running locally for personal projects. It sometimes can't quite get there, and that's when I'll reach for GPT.
I mean of course, the real world is messy, the models are only good for what they've been trained on, and post-training frequently takes benchmark scores into account too, leading to this sort of "Great score, sucks in the real world" syndrome that seemingly most models are affected by.
Edit: another comment pointed out it's probably not benchmaxxing due to the chronology.
https://news.ycombinator.com/item?id=50406948
But why should minor changes to the task cause catastrophic performance drops? How is that not evidence of benchmaxxing?