> Let’s say every company gets about three innovation tokens. You can spend these however you want, but the supply is fixed for a long while.
This is one of my favorite blog posts, and it can basically be encapsulated in the idea of "innovation tokens." It is one of the most useful concepts I have had as a PM / eng leader in my career. It helps actually make the the right tradeoffs, and helps even more in explaining those tradeoffs to colleague of all levels. Highly recommend.
There was another, much older, post similar to this about, I think, "beans" that engineers use to solve problems. If I remember correctly, it was something like, solving a problem costs "beans", and engineers will always use most all of their "beans" to solve a given problem, because it's somewhat "easy" until you run out of them.
Maybe it wasn't beans? But, I've been looking for it for years.
This is where efficiency can be applied in two distinct ways. On the one hand, if using more beans means you can solve the problem more quickly, then it is more efficient to the problem solver to do so.
But if beans are scarce, and that's where you can try to find ways to use less beans per function, then the goal is to make the function more efficient by using less beans.
I love this post. It’s also interesting to revisit in the age of agents.
Using the language of the article, I’d say “push all your innovation tokens into agents” is probably a good move. This means the tech your agents work with should all be boring tech.
Another way of saying this is “use in-distribution technology”. If agents are substantially better at Rust than Zig, probably you should use Rust, even if Zig is “better”. The amount that Zig is better is going to get swamped by the amount that in-distribution agents are better.
(This is not a claim that Rust actually is better, just a hypothetical fact pattern for discussion.)
It's a total tangent but AI being so good at developing rust means that my first impression of seeing a rust project has totally changed.
4 or 5 years ago if I had to weigh up two options and one was written in rust it was almost a sure thing that the program was snappy, fast, reliable and that the author was competant.
Now it's a sure fire sign that the project was probably vibe coded. Not saying it can't still be good, I just have a different first impression now
One thing I learned about AI is that, e.g., if you're making websites you're much better off going for php/ruby/elixir, even if you don't like the languages much.
Pipelines and deploys gets much easier and faster than the very common TypeScript monorepo, and so does communication between server and client.
And I say that as a TypeScript and Effect aficionado who has no particular love for neither php or ruby, but they are extremely solid choices to move fast, well, and get excellent performance and tooling out of the box.
Great point! I'd also add that Django's another solid choice for boring tech that LLM agents will know very well and will very likely continue to do so. A fair bit of SWE Bench and other Python benchmarks are Django related tests, which the LLM vendors care very deeply about keeping their scores up. Also, Django's docs are excellent, so strategically pointing an LLM to them in a prompt can often produce great results.
I disagree. We've been doing this for two decades already. A lot of monoliths were rewritten in a way that best fit the AWS pricing model of the time with the software itself seemingly being an after afterthought. (I'm not here to discuss whether monoliths or micro services are better, I'm just saying the choice of how they were rewritten was too often mostly for AWS pricing)
I think you're completely misreading what I wrote.
PHP (lets use this as an example) brings already many benefits over a common solution like a TypeScript monorepo mostly operational simplicity. That was already true before AI.
It's boring, very fast and easy to deploy, offers straightforward horizontal scaling, no need to orchestrate containers and/or multiple runtime processes, has excellent html rendering (nothing in JS-land really does), and has very solid framework solutions like Laravel where everything works out of the box.
Those merits existed before AI already.
The issue was that you had to buy into PHP as a language, which was a horrible experience.
But if AI writes most of the code? Suddenly PHP becomes an excellent candidate to choose for many use cases.
If anything, AI makes the choice of programming and languages and software about finding the right tool for the job. Somehow the industry instead keeps vomiting React/Tailwind slop which are the right tool for 1% of the jobs.
It's bad for building your resume. Managers and employees are rewarded for popular thing. Popularity is mostly a function of novelty. So the (social and therefore economic) incentive structure is inversely correlated with choosing boring technology.
i just did an AI hackathon and 90% of the submissions were written in TypeScript and Next.js which is mostly due to the training data. AI is skewed to use these tools by default vs the best for the job.
I'll push back against this, despite it being so popular. I dislike the arbitrary "innovation tokens" and I think this entire concept really blurs the lines and feels sort of unserious.
Engineers should understand requirements, risks, tradeoffs, and potential gains. New technology may be right for that. Novel approaches may be right for that. "Novel" or "New" are only proxies and they're weak.
For example, I may think "New" means untested, but is that true? What if a new project has Jepsen testing, a fuzzing suite, massive compute running tons of oracle tests, etc? I should just say "Choose well tested" instead of "Choose old" - lots of old software is very poorly tested.
Maybe I think that "Old" implies better documentation, but does it? Lots of older projects have insane cruft and weird edge cases that are undocumented and accumulated over years.
Why do we need a metaphor? Why is "innovation token" helpful?
If you're incapable of evaluating a technology in terms of these properties, you aren't a serious developer and "boring" will not save you.
Sit down, write our your requirements, determine candidate solutions, and choose them based on their fit. "Boring" means nothing, it's a vague proxy term. "Well tsted", "performant for our use case", "developers know it", etc mean something.
> MySQL is boring. Postgres is boring. PHP is boring. Python is boring. Memcached is boring. Squid is boring. Cron is boring.
Literally every one of these has caused hilarious and disastrous failures for me in my career. But yep, boring.
> If you choose to write your website in NodeJS, you just spent one of your innovation tokens. If you choose to use MongoDB, you just spent one of your innovation tokens.
What if you know NodeJS really well? Or MongoDb? What if you have empirical, verifiable reasons for why they fit better?
I'm a bit tired of "simple" and "boring" and other nonsense words in this field taking up the air in the room that should be spent evaluating solutions on their actual merits.
I don't think that context is relevant to my comment. I didn't say "in hindsight, those technologies are great!", I pointed out that "boring" is meaningless, and any meaning you attribute to it like "defined as the tech you know the sharp edges of" is better substituted in.
That is, if someone said two sentences, I would only care about the second one:
1. "We should use this because it is boring"
2. "We should use this because we understand the sharp edges"
I wouldn't care at all about (1) and I'd have a real conversation based on (2).
Any productive conversation that starts with (1) immediately has to follow "can you clarify what that means", so the term is useless at best and thought terminating at worst.
I see innovation as a guardrail against CV driven development. More, I think you need to consider the context of when this was written. It was a period of rapid innovation/evolution - I remember more than a handful of projects failing (either undelivered or rewritten well under their expected lifecycle) around this time because teams had taken bets on new tech either they didn't know how to use well or the tech didn't take off and was a dead end.
> What if you know NodeJS really well?
Then you consider it boring. I'm sure Node and MongoDB were singled out by the author because at the time of writing they were still relatively new and undergoing periods of rapid development and change.
CV driven development is just as well guarded against by asking someone to justify their technical decisions based on the requirements and how the solution meets them. So "boring" does nothing to further that.
> More, I think you need to consider the context of when this was written. It was a period of rapid innovation/evolution
It's linked today, people feel it's relevant today. This isn't a historic piece about how the tech industry used to be, people reference this post today.
> around this time because teams had taken bets on new tech either they didn't know how to use well or the tech didn't take off and was a dead end.
Yes, they should have had a discussion about their requirements and which technologies would have solved them.
> Then you consider it boring.
Then "boring" is useless and you should just say "I know this technology well and it maps to our use case well" and be able to justify that.
Well your points are making sense in isolation whereas this article is making more sense in general.
> I'm a bit tired of "simple" and "boring" and other nonsense words in this field...
This is hilarious in sense millions more will be tired and exhausted by evaluating new and exciting technology endlessly appearing all the time.
People go by these rule of thumbs which may not be perfect in every single case but they do increase success chances for even sub-par teams as opposed to "rigorously evaluating latest technology"
I don't think it's exhausting to determine if a solution fits your requirements and I don't think much about the people who would find it exhausting. I'm not suggesting some insane formal verification, but you really can't just answer basic questions about how technologies can address problems? Then what is your role? To choose mysql irrespective of requirements?
> People go by these rule of thumbs which may not be perfect in every single case but they do increase success chances for even sub-par teams as opposed to "rigorously evaluating latest technology"
Rule of thumb. And it's not a rule. It's a bias based on a vague term.
If it's understood then it's pointless. I also reject that it's understood.
I have no idea why you're talking about Prinicipia Mathematica as if I'm advocating for some sort of formal verification or extraordinary rigor as opposed to my suggestion that people just use their words and have reasons behind their decisions.
nawh. 50x node modules, typescript out the wazoo, all the state in the client (where you can't see it in prod), the most over-complicated UI, paired with async callback spaghetti is what you do these days.
We're "scalable" over here. It's a sexy problem to have.
I read a similar blog post years ago about restricting the technologies you use, and making do with a slightly worse option if it means reusing the stack you already have.
The example they gave was something like they wanted to use rabbitmq for a new side project (I might be misremembering) but they were forced to make it work with redis instead. The author said that years later he found out that that side project had exploded in popularity and it had coped with it fine because the infra team were already handling the stack and it wasn't some snowflake deployment.
Software that works year-after-year has never been a commodity. It's boring on the surface. It doesn't get the flashy posts. But I'll choose reliable over new in almost all cases.
The problem with this is that the list of tech that gets boring changes all time, faster than people's opinions. Kubernetes is very boring tech, but if you go through the old discussion threads on this (even from the last year or two), Kubernetes is still cited as some brand new wizbang thing you shouldn't spend tokens on.
This is one of my favorite blog posts, and it can basically be encapsulated in the idea of "innovation tokens." It is one of the most useful concepts I have had as a PM / eng leader in my career. It helps actually make the the right tradeoffs, and helps even more in explaining those tradeoffs to colleague of all levels. Highly recommend.
Maybe it wasn't beans? But, I've been looking for it for years.
But if beans are scarce, and that's where you can try to find ways to use less beans per function, then the goal is to make the function more efficient by using less beans.
[0] https://widgetsandshit.com/teddziuba/2010/10/taco-bell-progr...
Using the language of the article, I’d say “push all your innovation tokens into agents” is probably a good move. This means the tech your agents work with should all be boring tech.
Another way of saying this is “use in-distribution technology”. If agents are substantially better at Rust than Zig, probably you should use Rust, even if Zig is “better”. The amount that Zig is better is going to get swamped by the amount that in-distribution agents are better.
(This is not a claim that Rust actually is better, just a hypothetical fact pattern for discussion.)
4 or 5 years ago if I had to weigh up two options and one was written in rust it was almost a sure thing that the program was snappy, fast, reliable and that the author was competant.
Now it's a sure fire sign that the project was probably vibe coded. Not saying it can't still be good, I just have a different first impression now
Pipelines and deploys gets much easier and faster than the very common TypeScript monorepo, and so does communication between server and client.
And I say that as a TypeScript and Effect aficionado who has no particular love for neither php or ruby, but they are extremely solid choices to move fast, well, and get excellent performance and tooling out of the box.
PHP (lets use this as an example) brings already many benefits over a common solution like a TypeScript monorepo mostly operational simplicity. That was already true before AI.
It's boring, very fast and easy to deploy, offers straightforward horizontal scaling, no need to orchestrate containers and/or multiple runtime processes, has excellent html rendering (nothing in JS-land really does), and has very solid framework solutions like Laravel where everything works out of the box.
Those merits existed before AI already.
The issue was that you had to buy into PHP as a language, which was a horrible experience.
But if AI writes most of the code? Suddenly PHP becomes an excellent candidate to choose for many use cases.
If anything, AI makes the choice of programming and languages and software about finding the right tool for the job. Somehow the industry instead keeps vomiting React/Tailwind slop which are the right tool for 1% of the jobs.
I hope to have cleared the concept.
Engineers should understand requirements, risks, tradeoffs, and potential gains. New technology may be right for that. Novel approaches may be right for that. "Novel" or "New" are only proxies and they're weak.
For example, I may think "New" means untested, but is that true? What if a new project has Jepsen testing, a fuzzing suite, massive compute running tons of oracle tests, etc? I should just say "Choose well tested" instead of "Choose old" - lots of old software is very poorly tested.
Maybe I think that "Old" implies better documentation, but does it? Lots of older projects have insane cruft and weird edge cases that are undocumented and accumulated over years.
Why do we need a metaphor? Why is "innovation token" helpful?
If you're incapable of evaluating a technology in terms of these properties, you aren't a serious developer and "boring" will not save you.
Sit down, write our your requirements, determine candidate solutions, and choose them based on their fit. "Boring" means nothing, it's a vague proxy term. "Well tsted", "performant for our use case", "developers know it", etc mean something.
> MySQL is boring. Postgres is boring. PHP is boring. Python is boring. Memcached is boring. Squid is boring. Cron is boring.
Literally every one of these has caused hilarious and disastrous failures for me in my career. But yep, boring.
> If you choose to write your website in NodeJS, you just spent one of your innovation tokens. If you choose to use MongoDB, you just spent one of your innovation tokens.
What if you know NodeJS really well? Or MongoDb? What if you have empirical, verifiable reasons for why they fit better?
I'm a bit tired of "simple" and "boring" and other nonsense words in this field taking up the air in the room that should be spent evaluating solutions on their actual merits.
So yes, pick boring tech, defined as the tech you know the sharp edges of.
That is, if someone said two sentences, I would only care about the second one:
1. "We should use this because it is boring"
2. "We should use this because we understand the sharp edges"
I wouldn't care at all about (1) and I'd have a real conversation based on (2).
Any productive conversation that starts with (1) immediately has to follow "can you clarify what that means", so the term is useless at best and thought terminating at worst.
> What if you know NodeJS really well?
Then you consider it boring. I'm sure Node and MongoDB were singled out by the author because at the time of writing they were still relatively new and undergoing periods of rapid development and change.
> More, I think you need to consider the context of when this was written. It was a period of rapid innovation/evolution
It's linked today, people feel it's relevant today. This isn't a historic piece about how the tech industry used to be, people reference this post today.
> around this time because teams had taken bets on new tech either they didn't know how to use well or the tech didn't take off and was a dead end.
Yes, they should have had a discussion about their requirements and which technologies would have solved them.
> Then you consider it boring.
Then "boring" is useless and you should just say "I know this technology well and it maps to our use case well" and be able to justify that.
> I'm a bit tired of "simple" and "boring" and other nonsense words in this field...
This is hilarious in sense millions more will be tired and exhausted by evaluating new and exciting technology endlessly appearing all the time.
People go by these rule of thumbs which may not be perfect in every single case but they do increase success chances for even sub-par teams as opposed to "rigorously evaluating latest technology"
> People go by these rule of thumbs which may not be perfect in every single case but they do increase success chances for even sub-par teams as opposed to "rigorously evaluating latest technology"
Rule of thumb. And it's not a rule. It's a bias based on a vague term.
I have no idea why you're talking about Prinicipia Mathematica as if I'm advocating for some sort of formal verification or extraordinary rigor as opposed to my suggestion that people just use their words and have reasons behind their decisions.
We're "scalable" over here. It's a sexy problem to have.
The example they gave was something like they wanted to use rabbitmq for a new side project (I might be misremembering) but they were forced to make it work with redis instead. The author said that years later he found out that that side project had exploded in popularity and it had coped with it fine because the infra team were already handling the stack and it wasn't some snowflake deployment.
Does anyone remember the post I'm talking about?