Anecdotal but I've found Fable to be fairly unimpressive and not much better than Opus 4.8, if at all in some cases, but I have been hitting the ceiling on my $100/mo sessions when I never did before. I switched back to Opus yesterday. I may use Fable for audits, but that's about it, and when it leaves my subscription plan I don't think I'll miss it.
I haven’t tested it rigorously, but in the one test I did before it was shut off (an autonomous coding task I’d benchmarked on 4.8) I found it slightly worse and way more expensive.
I believe the Artificial Analysis benchmarks that had fable ~3 points ahead of 4.8 in intelligence. But I’ve been amazed about the, also anecdotal, posts about how it’s a game changer or whatever, e.g.
Honest question/comment for you and the parent: I find these subjective experience reports pretty empty without an understanding of your level of experience, the problem space you're working in, etc.
I started telling a friend... I feel like Fable is Opus with extended reasoning that eventually "figures out more" because when I switched to it, I hit my limits surprisingly and shockingly quicker than I would with Opus, and I got less done. All this hype, and I much rather use Opus.
I guess this ethics stuff is cool, but I'm more interested in how good it is at running a business and dealing with adversarial humans like in previous vending machine experiments. I hope they release something on that soon.
Okay I hadn't heard of Vending-Bench until reading this and it was quite the ride learning about it through this article. Very fun read.
My very native programmer take is that it's not too surprising that their hacker model would be less ethical. The guardrails that separate Fable and Mythos probably wouldn't kick in during an environment like this.
> The broad conclusion from the many
forms of alignment evaluations described in this section is that Claude Mythos Preview is
the best-aligned of any model that we have trained to date by essentially all available
measures.[0]
„in our opinion, insurance fraud is not more unethical than lying and price fixing“
The authors seem surprised that behavior that is very often done by humans (lying and price fixing) are more often done by fable compared to actual fraud.
I think the model never assigned any morality to these actions in the first place, it simply copied us humans.
When assessing probabilistic models the plots should be showing the mean a̶n̶d̶ ̶s̶t̶d̶e̶v̶ of many monte carlo simulations not just one line per model and claiming "look this model is more gooder!"
The best Anthropic models on VendingBench2 are Opus 4.7, Opus 4.6, Sonnet 4.6, and Sonnet 5. Opus 4.7 scored more than twice Fable 5 max. Fable 5 - Low outperforms Fable 5 - Max, with Opus 4.5 in the middle. This seems to break the narrative, which is maybe why Andon Labs doesn't seem to have updated the trend lines on their graphs.
I believe the Artificial Analysis benchmarks that had fable ~3 points ahead of 4.8 in intelligence. But I’ve been amazed about the, also anecdotal, posts about how it’s a game changer or whatever, e.g.
https://news.ycombinator.com/item?id=48694329
https://news.ycombinator.com/item?id=48741757
My very native programmer take is that it's not too surprising that their hacker model would be less ethical. The guardrails that separate Fable and Mythos probably wouldn't kick in during an environment like this.
[0]: https://www-cdn.anthropic.com/08ab9158070959f88f296514c21b7f...
The authors seem surprised that behavior that is very often done by humans (lying and price fixing) are more often done by fable compared to actual fraud.
I think the model never assigned any morality to these actions in the first place, it simply copied us humans.
common mistake people make