Machine learning could need, and probably has needed, some unified math notation for the past 15 years IMO. With that said, it was worse back in the day - when ML papers were the products of researchers from all over, you'd see some wild notation.
Many will likely disagree with me, but inconsistent notation (across papers!) is to me friction. At least in this article the author explicitly explains the notation at the very start...that is not always the case. Rarely, even.
EDIT: Didn't even notice the notation switch, much appreciated.
You know its a doozy when the author writes a disclaimer at the top saying that bra-ket notation was chosen in order to make the algorithm and data structures clearer.
> The identity [...] is the whole trick. The outer product is a matrix; the inner product is a number. We no longer store every past key and value. We store their summed outer products in the fixed-size state S_t.
At first I felt bad about not having come up with this solution. But then I realized I have problems with writing binary search by myself in JS and immediately felt better.
Now way I could have come up with Kimi Delta Attention.
Lots of linear algebra codes are actually “easy to write” in a way. It isn’t like conventional CS where you are always going a bunch of recursive nonsense going on. There should be mathematical relationships between all of the variables, there are well implemented libraries for the common mathematical concepts, and it is rare to need to go more than a couple loops deep (anything more complex than that should get shunted off into a library anyway).
No, you couldn't have. There are plenty of ML innovations that when push comes to shove only depend on having access to more compute, but this is one of the worst examples I've ever seen.
I always thought that the jump from LSTM/GRU -> Attention wasn't a particularly big one. Instead of partial unroll, do a full unroll. Why not (because it's too expensive, that's why not). Every component was known, and everybody anywhere near ML knew perfectly well why NOT to try that: because you just don't have the compute to fully unroll an LSTM. From that point attention is optimized (they key-query mechanic). The big innovation is not so much the mechanism itself but realizing the parallelize-ability of it.
It's sort of like if one would today make the "improvement" to attention to replace they key-query-value mechanic by just dropping it while making the entire context the latent space. That will outperform attention, nearly guaranteed. It'll also make even Google's cluster networks meltdown. Attention is one of those innovations that came mostly from realizing you had better hardware than everybody else and asking yourself how to use it. It's still quite the accomplishment, they had to get it working. But nobody else was really capable of making this leap.
I agree 100%. This field is not amenable to progress from people with a pen sitting in a corner proving theorems. The math is mostly uncertain vibes and to test it you need millions of dollars of compute. Smart loners just can't.
He has a master's degree in physics from Oxford. Also there is a toggle to normal notation. Well, CS notation. I'm not a fan of transpose marks everywhere. I like an even more mathematics notation.
I could never get this about modern machine/deep learning or even the Transformers. Yes, it's not exactly rocket science, but when I see the data flow diagrams, it's not clear what is calculated in real time or multiple steps.
Each token prediction is one big function call. Then you just recursively generate more tokens until run out of context or the model predicts a next token indicating end of sequence. Technically the model outputs a matrix where the last row is a probability distribution, but I’m counting sampling from it as part of the chain. Hundreds of billions of dollars has gone into just making the function fatter and gradually changing pieces here and there.
Inference (the real time computation) and training (the computation you do when developing a model) are deeply tied by automatic differentiation.
In a way, you train by repeatedly inferring (forward), calculating a loss (how much your model sucks with its current predictions) and then improving the model by differentiating.
Take a look at Karpathy's micrograd repo to become more familiar with the process.
Many will likely disagree with me, but inconsistent notation (across papers!) is to me friction. At least in this article the author explicitly explains the notation at the very start...that is not always the case. Rarely, even.
EDIT: Didn't even notice the notation switch, much appreciated.
> The identity [...] is the whole trick. The outer product is a matrix; the inner product is a number. We no longer store every past key and value. We store their summed outer products in the fixed-size state S_t.
Now way I could have come up with Kimi Delta Attention.
I always thought that the jump from LSTM/GRU -> Attention wasn't a particularly big one. Instead of partial unroll, do a full unroll. Why not (because it's too expensive, that's why not). Every component was known, and everybody anywhere near ML knew perfectly well why NOT to try that: because you just don't have the compute to fully unroll an LSTM. From that point attention is optimized (they key-query mechanic). The big innovation is not so much the mechanism itself but realizing the parallelize-ability of it.
It's sort of like if one would today make the "improvement" to attention to replace they key-query-value mechanic by just dropping it while making the entire context the latent space. That will outperform attention, nearly guaranteed. It'll also make even Google's cluster networks meltdown. Attention is one of those innovations that came mostly from realizing you had better hardware than everybody else and asking yourself how to use it. It's still quite the accomplishment, they had to get it working. But nobody else was really capable of making this leap.
https://xkcd.com/2501/
Is it really one big computation f(g(h(x)))?
Each token prediction is one big function call. Then you just recursively generate more tokens until run out of context or the model predicts a next token indicating end of sequence. Technically the model outputs a matrix where the last row is a probability distribution, but I’m counting sampling from it as part of the chain. Hundreds of billions of dollars has gone into just making the function fatter and gradually changing pieces here and there.
In a way, you train by repeatedly inferring (forward), calculating a loss (how much your model sucks with its current predictions) and then improving the model by differentiating.
Take a look at Karpathy's micrograd repo to become more familiar with the process.
Is it all one big computation? Its turtles all the way down.