This paper shows a nice integration of older winner-take-all ideas for learning K-modal generative models (see e.g. https://arxiv.org/abs/1612.00197, https://arxiv.org/abs/2211.14286) into modern diffusion/flow pipelines. As-implemented, I think it has some downsides:
1. K-1 extra forward passes during training
2. Inaccurate sampling behavior (will sample all K modes with equal likelihood, rather than sampling them proportionally)
However, I think both of these downsides can be mitigated by adjusting the implementation a bit more (you can have the model predict K modes jointly in one forward pass, along with probabilities of each being the min-loss mode, which you can then use for properly-weighted mode sampling at inference).
That said, I'm not entirely sure if this strategy is as generally applicable as the authors are hoping. In particular:
1. For highly-conditional image generation (like modern commercial diffusion pipelines, which use a big LLM preprocessor), most of the low-frequency color/layout decisions are already made for you by the conditioning signal. The diffusion process mostly needs to generate high-frequency details, for which there are a huge number of equally-valid modes.
2. For LLMs themselves, the sequence-generation process is already factored into a discrete classification problem, and there's no mode smearing issue to fix.
In reference to the difficulties with applying this to autoregressive LLMs - I wonder if these type of hybrids might be a good candidate for this approach.
I agree, this seems like an important discovery. And it hits a sweetspot of being effective and very simple to implement. Have a look at the pseudocode comparison at the bottom of the accompanying github page: https://explorative-modeling.github.io/
Also, a very well-structured and easy to follow blog post by the author. I very much enjoyed reading it!
No mention of GANs? As far as I remember, their images felt much more like "sampled from training data" than "averaged from training data". The idea was that you train a "discriminator" that tells generated images apart from real ones, and make the "generator" try to fool it. But somehow it lost to diffusion models, and now all AI imagery looks like slop, when GAN outputs (though imperfect) didn't look nearly as slop. I don't really know what happened though.
1. K-1 extra forward passes during training
2. Inaccurate sampling behavior (will sample all K modes with equal likelihood, rather than sampling them proportionally)
However, I think both of these downsides can be mitigated by adjusting the implementation a bit more (you can have the model predict K modes jointly in one forward pass, along with probabilities of each being the min-loss mode, which you can then use for properly-weighted mode sampling at inference).
That said, I'm not entirely sure if this strategy is as generally applicable as the authors are hoping. In particular:
1. For highly-conditional image generation (like modern commercial diffusion pipelines, which use a big LLM preprocessor), most of the low-frequency color/layout decisions are already made for you by the conditioning signal. The diffusion process mostly needs to generate high-frequency details, for which there are a huge number of equally-valid modes.
2. For LLMs themselves, the sequence-generation process is already factored into a discrete classification problem, and there's no mode smearing issue to fix.
I was just reading this great breakdown of how diffusion Gemma works: https://newsletter.maartengrootendorst.com/p/a-visual-guide-...
In reference to the difficulties with applying this to autoregressive LLMs - I wonder if these type of hybrids might be a good candidate for this approach.
Also, a very well-structured and easy to follow blog post by the author. I very much enjoyed reading it!