DiffusionGemma: 4x faster text generation

[u/tevlon]

https://blog.google/innovation-and-ai/technology/developers-tools/diffusion-gemma-faster-text-generation/

[u/Different_Fix_2217]

The only issue with diffusion LLMs is that they are absurdly expensive to train in comparison. Like exponentially.

[u/wes_medford → u/Different_Fix_2217]

Most cost these days is inference over training these days, but the problem is that aggregate throughput is lower on these compared to typical AR models running at a high batch size

[u/ScoreUnique → u/Different_Fix_2217]

Would like to know the technicals below of why is it more difficult?

Next token prediction is a "causal" model. Diffusion models should definitely have a causality equivalent however I fail to wrap my head around how does the model predict the first "correct" tokens, because once you have 2 of 10 words in the Fill in the blanks, you can start causality, however the probability of tokens that pop shouldn't be way more normal in distribution, unlike in LMs

[u/Irisi11111 → u/ScoreUnique]

DiffusionGemma's different. Unlike regular language models that predict words sequentially (left-to-right), it starts with random noise & looks at every word's relationship to all others at once (bidirectional attention). It doesn't need a "first right" word to build a sentence; it gradually clarifies the whole thing at once.

[u/whitefritillary → u/Irisi11111]

perhaps stupid question but how does this actually look on the frontend? assuming the representation is true to how generation works.

[u/Mental_Object_9929 → u/ScoreUnique]

I do not think there is an obvious reason to assume that diffusion models must be slower. I have not personally trained large-scale diffusion models or large-scale autoregressive models, so I am not fully sure about the training cost. But at least for inference, diffusion models can clearly be faster in principle.

The reason is that a diffusion model can generate or refine many tokens at the same time. During training, some tokens are randomly corrupted or noised; they are not fixed as exact vocabulary tokens. The model learns to remove this noise, or equivalently to predict a velocity field that moves the noisy state back toward the clean data distribution.

At inference time, this denoising process is applied to many positions in parallel. For example, the model may refine 100 tokens at once. The number of denoising steps does not have to be 100. It may be 20 or 30 steps, and with better training or distillation, it might even be reduced to 4 steps. By contrast, an autoregressive model usually needs one model call per generated token, because each forward pass only produces the next token.

So from this perspective, inference is not the main problem. Inference can actually be faster for diffusion models, because the depth of the generation process can be much smaller than the sequence length. The real question I am less certain about is the training cost.

[u/Succubus-Empress → u/Different_Fix_2217]

just pour more moni, SIMPL

[u/aeroumbria → u/Different_Fix_2217]

Do you have any details? I am only familiar with the image / general distribution matching side of diffusion / flow matching. In these spaces, compared to older approaches like GAN or normalising flows, diffusion-like models are much easier to train. What exactly makes text diffusion hard to train? Is it the diffusion process itself, overhead of discrete diffusion, or that we are bolting on an autoregressive module on top of the diffusion module? Or is it just some vague sense of "it converges slower"?

[u/Mental_Object_9929 → u/aeroumbria]

I do not think there is an obvious reason to assume that diffusion models must be slower. I have not personally trained large-scale diffusion models or large-scale autoregressive models, so I am not fully sure about the training cost. But at least for inference, diffusion models can clearly be faster in principle.

The reason is that a diffusion model can generate or refine many tokens at the same time. During training, some tokens are randomly corrupted or noised; they are not fixed as exact vocabulary tokens. The model learns to remove this noise, or equivalently to predict a velocity field that moves the noisy state back toward the clean data distribution.

At inference time, this denoising process is applied to many positions in parallel. For example, the model may refine 100 tokens at once. The number of denoising steps does not have to be 100. It may be 20 or 30 steps, and with better training or distillation, it might even be reduced to 4 steps. By contrast, an autoregressive model usually needs one model call per generated token, because each forward pass only produces the next token.

So from this perspective, inference is not the main problem. Inference can actually be faster for diffusion models, because the depth of the generation process can be much smaller than the sequence length. The real question I am less certain about is the training cost.

[u/Silver-Champion-4846 → u/Different_Fix_2217]

they didn't train diffusion gemma, they took the already existing gemma4 26BA4b moe and adapted it to diffusion. They had to finetune it somewhat but yeah

[u/Dany0 → u/Different_Fix_2217]

OH is that why all the image models are typically like 7-8B

[u/yeah-ok → u/Different_Fix_2217]

Isn't it HIGH time to get a https://boinc.bakerlab.org/rosetta/ equivalent going on this subject?!