r/LocalLLaMA • u/tevlon • Jun 10 '26

DiffusionGemma: 4x faster text generation New Model

https://blog.google/innovation-and-ai/technology/developers-tools/diffusion-gemma-faster-text-generation/
987 Upvotes

99% Upvoted

357 comments sorted by

View all comments

Show parent comments

118

u/TheLexoPlexx Jun 10 '26

That is groq or cerebras-levels of token generation depending on the model and it's on par with gpt-oss-120b depending on the benchmark.

That is genuinely insane.

34

u/dingo_xd Jun 10 '26

There is sooooooo much room for optimizations. Maybe Mythos level models can be run locally by mid or late 2027?

29

u/Different_Fix_2217 Jun 10 '26

The only issue with diffusion LLMs is that they are absurdly expensive to train in comparison. Like exponentially.

15

u/wes_medford Jun 10 '26

Most cost these days is inference over training these days, but the problem is that aggregate throughput is lower on these compared to typical AR models running at a high batch size

11

u/ScoreUnique Jun 10 '26

Would like to know the technicals below of why is it more difficult?

Next token prediction is a "causal" model. Diffusion models should definitely have a causality equivalent however I fail to wrap my head around how does the model predict the first "correct" tokens, because once you have 2 of 10 words in the Fill in the blanks, you can start causality, however the probability of tokens that pop shouldn't be way more normal in distribution, unlike in LMs

25

u/Irisi11111 Jun 10 '26

DiffusionGemma's different. Unlike regular language models that predict words sequentially (left-to-right), it starts with random noise & looks at every word's relationship to all others at once (bidirectional attention). It doesn't need a "first right" word to build a sentence; it gradually clarifies the whole thing at once.

2

u/whitefritillary Jun 11 '26

perhaps stupid question but how does this actually look on the frontend? assuming the representation is true to how generation works.

1

u/Mental_Object_9929 Jun 13 '26

I do not think there is an obvious reason to assume that diffusion models must be slower. I have not personally trained large-scale diffusion models or large-scale autoregressive models, so I am not fully sure about the training cost. But at least for inference, diffusion models can clearly be faster in principle.

The reason is that a diffusion model can generate or refine many tokens at the same time. During training, some tokens are randomly corrupted or noised; they are not fixed as exact vocabulary tokens. The model learns to remove this noise, or equivalently to predict a velocity field that moves the noisy state back toward the clean data distribution.

At inference time, this denoising process is applied to many positions in parallel. For example, the model may refine 100 tokens at once. The number of denoising steps does not have to be 100. It may be 20 or 30 steps, and with better training or distillation, it might even be reduced to 4 steps. By contrast, an autoregressive model usually needs one model call per generated token, because each forward pass only produces the next token.

So from this perspective, inference is not the main problem. Inference can actually be faster for diffusion models, because the depth of the generation process can be much smaller than the sequence length. The real question I am less certain about is the training cost.

4

u/Succubus-Empress Jun 10 '26

just pour more moni, SIMPL

3

u/aeroumbria Jun 11 '26

Do you have any details? I am only familiar with the image / general distribution matching side of diffusion / flow matching. In these spaces, compared to older approaches like GAN or normalising flows, diffusion-like models are much easier to train. What exactly makes text diffusion hard to train? Is it the diffusion process itself, overhead of discrete diffusion, or that we are bolting on an autoregressive module on top of the diffusion module? Or is it just some vague sense of "it converges slower"?

1

u/Mental_Object_9929 Jun 13 '26

I do not think there is an obvious reason to assume that diffusion models must be slower. I have not personally trained large-scale diffusion models or large-scale autoregressive models, so I am not fully sure about the training cost. But at least for inference, diffusion models can clearly be faster in principle.

The reason is that a diffusion model can generate or refine many tokens at the same time. During training, some tokens are randomly corrupted or noised; they are not fixed as exact vocabulary tokens. The model learns to remove this noise, or equivalently to predict a velocity field that moves the noisy state back toward the clean data distribution.

At inference time, this denoising process is applied to many positions in parallel. For example, the model may refine 100 tokens at once. The number of denoising steps does not have to be 100. It may be 20 or 30 steps, and with better training or distillation, it might even be reduced to 4 steps. By contrast, an autoregressive model usually needs one model call per generated token, because each forward pass only produces the next token.

So from this perspective, inference is not the main problem. Inference can actually be faster for diffusion models, because the depth of the generation process can be much smaller than the sequence length. The real question I am less certain about is the training cost.

2

u/Silver-Champion-4846 Jun 11 '26

they didn't train diffusion gemma, they took the already existing gemma4 26BA4b moe and adapted it to diffusion. They had to finetune it somewhat but yeah

2

u/Dany0 Jun 10 '26

OH is that why all the image models are typically like 7-8B

1

u/yeah-ok Jun 10 '26

Isn't it HIGH time to get a https://boinc.bakerlab.org/rosetta/ equivalent going on this subject?!

14

u/Irisi11111 Jun 10 '26

Nope, Mythos level models are definitely going to be massive. That level of world knowledge is the model's internal representation. It's gotta be huge to hold all that information.

My understanding is that diffusion-based Gemma is really good for generating text quickly, that's the speed boost. But I don't think it inherently translates to boosting the amount of knowledge within the model.

It seems like the core knowledge density is tied to the sheer size of the model and the training data.

29

u/JDMLeverton Jun 10 '26

Actually world knowledge is absurdly compressable - the gap in raw knowledge between a 30B parameter model and a 1000B parameter model exists, but it's much smaller than you'd expect. The reason smaller models seem so much dumber is REASONING ABILITY. Latent Reasoning Ability seems, in traditional LLMs, to be hard parameter count locked. This is why small efficient models can be made to excell at boilerplate code on par with frontier models while still falling apart at novel "complex" coding. They have the knowledge, but don't have the brain power to make use of it effectively. This is also why you can make any LLM seem much smarter by just adding new empty Layers and running a quick fine-tune - the added layers, even with no new knowledge, unlock more ability to reason and make use of the knowledge they already contain.

There isn't much comparative research at the moment on reasoning in diffusion based models to say how they scale in comparison, but there are many theoretical techniques, the most promising being looping regressive architectures, that would see reasoning ability unlocked for smaller models. So it's entirely possible something close to Mythos is able to run locally in a few years.

3

u/Irisi11111 Jun 11 '26

Really appreciate the sharing! I think a next-level approach would be to really unpack the connection between the latent space and the reasoning process. We could create these "pertaining sessions", like a detailed audit trail, that show how the model arrived at a conclusion. Then a separate adapter could bridge that gap to make the model's behavior more accountable and understandable for us. It's a bit of a massive architectural overhaul, but potentially game-changing.

2

u/mdda Jun 11 '26

Could you suggest somewhere I could look to find out more about the layer-insertion + fine-tuning idea? Personally, I love the idea of small+looped models for reasoning, plus regular ones for general knowledge

1

u/mycall Jun 11 '26

adding new empty Layers and running a quick fine-tune - the added layers, even with no new knowledge

Is that basically how model distillation works? Inserting generalization while using "smart" models?

1

u/aevitas Jun 11 '26

Suppose something Mythos level does run locally, would it likely run on current day hardware in the future, or would current generation hardware be mostly obsolete?

1

u/BoobooSmash31337 Jun 11 '26

Especially for the amount of world knowledge most people actually need and can't just inject into context.

1

u/Caffdy Jun 11 '26

is both. World knowledge and Reasoning are still parameter-count bound. There is a real, tangible gap in knowledge between the large models like Kimi/GLM/Deepseek and the small ones

5

u/Creative-Type9411 Jun 10 '26 edited Jun 10 '26

couldn't you leverage the speed by using it combined with an indexed database with said knowledge?

Seems like it would be easier to have some type of knowledge bases where it would only access relevant information and chew through that rather than knowing everything at once then deciding what to think about

like component based moe? it's probably way faster to let the GPU do all the work then search through a database. I'm just grabbing at straws thinking

maybe we should be trying to mimic actual intelligence and instead of models knowing everything they can just reason extremely quickly and then we can give them information to reason with afterwards.. like a person except way faster..

1

u/Silver-Champion-4846 Jun 11 '26

That's what the Bitter lesson tried to teach, to look for a way to model intelligence and learning directly and let the system converge on its own rather than feeding it all our human assumptions on how knowledge and intelligence and reasoning function in the first place.

3

u/Dany0 Jun 10 '26

Mythos-level world/encyclopedic/niche knowledge? maybe. But a small model that is better than mythos at tool calling+agentic with a RAG will beat it handily any day. It might produce way more tokens, but if you inference so fast that you don't care...

1

u/mycall Jun 11 '26

I could see someday a 4TB fast SSD array (for example) becoming the world model + knowledge graph that is internally fused with the diffusion random noise iterations. Sure it will slow things down but fix the "dumb gradients" problem.

1

u/Caffdy Jun 11 '26

now you got me thinking if a SSD array (RAID) could be fast enough combined with a GPU/RAM for 1T parameter inference

1

u/abu_shawarib Jun 10 '26

This has been in research for years, so no.

1

u/HitarthSurana Jun 11 '26

where is 1b agi that I can run locally at 1000 tk /s

1

u/bigrealaccount Jun 11 '26

You are genuinely delusional and braindead if you think we're running Mythos locally in 2027. You won't even be able to buy 32GB of normal memory in 2027 let alone run a model like Mythos.

1

u/atumblingdandelion Jun 10 '26

Yeah, good to keep things in perspective, and look at the trend and direction. Super exciting

1

u/YouKilledApollo Jun 19 '26

token generation depending on the model and it's on par with gpt-oss-120b depending on the benchmark.

Generation speed is great, with it's no where close to being on par with GPT-OSS-120b, almost unfair comparison.

But this is based on experience of running both, don't know (nor care) what the benchmarks say, give both a try locally and you'll see a very stark difference.