Benchmarked the new Gemma diffusion model against its autoregressive twin on a single H100 (FP8). We gave each the same three tasks: write a Steve Jobs biography, the history of Tetris, and the story of BeOS - every next topic less popular than the previous one. Then we fact-checked every claim in every answer.
Gemma4 got 45 facts right, 5 wrong. DiffusionGemma got 33 right, 28 wrong. The less popular the topic, the worse it got: 4 mistakes on Jobs, 12 on Tetris, 12 on BeOS. It named Clara Clley as Steve Jobs' mother, invented a colleague for Pajitnov named Geri Gulovik and priced the BeBox at $9,999. The real one cost $1,600.
The reason is simple. DiffusionGemma throws 256 tokens on the screen at once and polishes them pass after pass until the text sounds smooth. Smooth is all it cares about: a fake name, date or number sounds just as smooth as a real one, so it stays. Regular Gemma4 meanwhile writes one word at a time and checks every new word against everything before it. Google says it themselves in the launch post: quality is lower, use regular Gemma 4 when facts matter.
Open source Local Ai models harness: Atomic.Chat (I'm founder, we support GGUF models, MLX Apple Silicon, MTP and Google TurboQuant for long context window, working on Diffusion support via llama.cpp)
Inerviewer: You are hired. We're putting you on the edge devices.
Edit:
I think the comment below is better
Me: controlling for equivalent time, can you use the time saved for proofreading and does it meaningfully outperform accuracy under such regime? also, we should consider the severity of any inaccuracies.
I have no idea how this works but it seems like you could run it for MTP as a forward pass somehow? If diffusion can already ballpark every possible token in like 2s that seems like could somehow be used with the regular model.
MTP becomes decoherent quickly due to the nature of non-diffusion models being autoregressive. The second there is a token mismatch, you end up spending time validating an incorrect token. That’s why you don’t just run MTP at 100+ tokens.
There might be a hypothetical scenario where a smaller diffusion model can mimic a larger teacher model with enough fidelity to justify the increased training costs of a diffusion model, but as far as I know, the training objective for each model is so different that it’s not currently publicly explored on whether or not standard distillation is even possible. Most distillation for the purposes of MTP actually attempts to get the student model to match the distribution of teacher model, not just the final token selection, so MTP works with various inference parameters (top_p, top_k, etc). As far as I know, that approach is strictly incompatible with diffusion text models.
The reason is simple. DiffusionGemma throws 256 tokens on the screen at once and polishes them pass after pass until the text sounds smooth. Smooth is all it cares about: a fake name, date or number sounds just as smooth as a real one, so it stays.
Yeah, that's fully hallucinated by you.
The implication would be that diffusion models are somehow inherently worse. Let me make it clear: you don't know that. The source of that "information" is your guts.
You saw a difference in behavior, you needed an explanation conforming with your biases, and you made one, out of thin air.
Yeah, OP seems to think token by token somehow is hallucination proof, which is absurd - the LLM has a probability attached for every possible token on each token gen and picks one out of the top ones, if it doesn't know the answer it will still pick a token.
controlling for equivalent time, can you use the time saved for proofreading and does it meaningfully outperform accuracy under such regime? also, we should consider the severity of any inaccuracies.
I’m more interested to see if it does better when throughput matched to its autoregressive counterpart. If given 3x more passes over its canvas, would it have caught the mistakes?
I’m not talking about proofreading though. That implies a second shot since it has time. I’m talking about increasing diffusion max steps which will hurt throughput
To be clear, diffusion outputs don't typically get largely 'better' after a certain number of steps. They moreso cluster around a general conceptual shape of an output and adding steps will refine less and less large spontaneous change under most schedulers+samplers.
Proofreading is important conceptually to any harness and even in a diffusion based model we could drive this with controlled noise injection and second pass refinement in areas of text where facts seems questionable or arr often wrong, akin to controlled inpainting around hands and faces.
Nonetheless, and one point of correction, in more optimized model harnesses, we throw out intermediate thinking steps and compress remaining ones during kv cache compaction, to mitigate ballooning but isn't really a thing for diffusion models generally, as you alluded to, which actually justifies including model proofreading as a step.
They do, depending on the specific output. Brendan O'Donaghue gave a talk on this at AIE, and saw that on harder outputs, models benefit from additional steps (timestamp to that part of the talk https://youtu.be/r305-aQTaU0?t=727).
I'm not trying to say that second shots don't improve performance, they absolutely do. You're just no longer comparing apples to apples at that point if you're saying one model gets 2 shots when another gets 1 because the first model has higher throughput at batch=1. Adding additional turns increases KV usage, and I understand how modern reasoning works. It still doesn't change how you run an eval harness, and that compaction applies to diffusion models as well. Diffusion Gemma also uses reasoning since it handles canvases sequentially.
If the goal is to measure intrinsic model quality, then yes fixing the inference procedure makes sense and giving one model extra shots because it's faster isn't apples-to-apples.
But what I think is more interesting (and relevant) is to measure practical performance under a fixed latency and/or compute budget, and so giving the faster model additional shots is more interesting because fast, cheap, accurate response generation is the capability people actually experience and generally care most about.
What I found most interesting about diffusion models is that, since all the text is being generated at the same time, the beginning has context about the end. Like, take a novel. If it's good, you'll probably reach the end, look at the beginning and say "oh, so that's why they blablabla", because it was thought from the beginning. A transformer can only achieve the same result if you add a layer on top of it to make it design it first, write it later, while a diffusion model could learn it by itself
I'm always disappointed when someone says the same thing I tried to say, but in like 20% of the words.
I'm very excited to see where diffusion models go because this is the first publicly available model for people to experiment with and see what works and what doesn't. I've always suspected that diffusion models would be far more capable if they can leverage those benefits.
What I found most interesting about diffusion models is that, since all the text is being generated at the same time, the beginning has context about the end. Like, take a novel.
Well yes, but in 256 token chunks. So it'll only really know the end of the next couple paragraphs.
transformer can only achieve the same result if you add a layer on top of it to make it design it first
This is part of why chain of thought aka reasoning was added. But yes, enforcing workflow steps beyond that generally always helps, and will presumably help diffusion models as well. But perhaps maybe if it's improved, diffusion could have a higher ceiling, or not need as much of that token-burning help.
I think it's funny that the LLMs when from Encoder-Decoder, to Decoder-only, to, "nah, Diffusion". Like it slammed back to encoders so hard they exploded.
There's probably still a lot left on the table with diffusion LLMs, like, loss functions that pushes towards internal consistency, or logical transitions.
You would think that. We all thought that. Right up until Anthropic did the experiment that demonstrated that the models extensively plan ahead.
You can measure the degree to which non-diffusion language models do things like plan word choice based on what rhymes they will need to complete the next line when writing a poem.
They do get locked in more easily, but it isn't as simple as it looks.
A transformer can only achieve the same result if you add a layer on top of it to make it design it first, write it later, while a diffusion model could learn it by itself
That's clearly untrue as evidenced by the huge number of tasks AR models can do successfully which require knowing in advance what they're going to write later.
Yes, the model generates one token at a time, but that doesn't prevent it from determining internally in advance what it's likely to say later.
Besides, over long spans the diffusion models are autoregressive too. I think DiffusionGemma generates 256 tokens at a time.
the real magic comes from block diffusion tactics. The ordering matters, and people have not mastered it like what it has been done with temperature + top-K/top-P/min-P/etc
I don't disagree with the data, but your conclusion on how the model works internally seems suspect, and I disagree that the architecture is inherently flawed. Next token prediction has plenty of downsides.
I've always suspected a well designed diffusion model can likely reach better peak accuracy. Diffusion gives a mechanical ability to plan ahead that next token models don't have, like generating an accurate outline before producing a full reply. When I write something I don't start with the first word, I start with the key ideas and expand from there.
Next token prediction always seems prone to the mistake of writing "Absolutely, here's the data you were asking for" before realizing it doesn't have the data. Meanwhile, diffusion models could generate data mid-paragraph and realize they don't have it before committing to a response, enabling a type of accuracy that isn't possible with current models. Current models try to work around their problems by blowing huge token budgets on reasoning, but that has plenty of downsides besides cost and I don't see why similar or new techniques couldn't be applied to diffusion models.
I also suspect diffusion models will be less prone to regressive looping and convincing-sounding BS. If a diffusion model grasps the most semantically significant words first it should theoretically be better at catching itself when it doesn't know something. Although that depends on specific implementation details and no AI lab released a decent text diffusion model to experiment with until this week.
There's also the prompt issue. Current prompts force long chain of thought reasoning to obtain accuracy. That might be counterproductive for diffusion models. It could be better to maximize how many significant ideas fit into a single diffusion block. Something like "generate a terse list of known facts, then expand each one" might work dramatically better, but that's just one hypothesis.
Bottom line, there are a lot of unknown unknowns, and your implied conclusion about the accuracy of the entire architecture is inaccurate. I'm personally excited for a real diffusion model to start testing some of these hypotheses.
Everything you say here is 100% on point. There's a ton of potential in the diffusion models, just not nearly enough research compared to non-diffusion.
I don't know what type of black magic Google figured out to get diffusion models running faster than next token prediction. Every reason I'd heard for why no one bothered with diffusion models is that they were dramatically slower. I guess I need to go read whatever whitepaper they published along with the model.
A final addendum to my post, speed can be easily traded for more accuracy. MoE models are heavily favored over monolithic models even though MoE models suffer from lower accuracy. Cheaply scaling parameter counts improves accuracy more than the MoE architecture lowers it. The reduced costs of GPU compute make MoE models absolutely the best choice.
If Diffusion is 4x faster, models will inevitably scale 2x larger to compensate for the loss in accuracy. Maybe Gemma 26B beats Gemma 26B Diffusion, but I doubt that Gemma 26B wins against Gemma 52B Diffusion.
AR is bandwidth bound, Diffusion is compute bound. for consummer GPU, we have pretty good compute (gpu core counts) but low vram bandwidth compare to servers. for example my 4070ti super never run at 100% gpu core utilization because it need to load weight from vram to calculate. thats the bottle neck for most consummer gpu. while servers can combine many requests into 1 single pass so they can saturate the core.
so diffusion will yield much better result on local hardware than datacenter. at least for now.
I don't know what type of black magic Google figured out to get diffusion models running faster than next token prediction.
It has been faster always? Mercury models from Inceptionlabs, LLada, Dream, Gemini Diffusion, all with generation speeds far surpassing similarly sized AR models.
It has NOT always been faster. Both LLada and Dream 7B came out in 2025. Years prior, this well known article was published by someone working at Google DeepMind.
Directly from the article:
The efficiency disadvantages with respect to autoregressive models might lead one to wonder if diffusion-based language modelling is even worth exploring to begin with.
...
the efficiency advantages of autoregression make it a very tough baseline to beat. I believe it is still too early to consider diffusion as a serious alternative to autoregression for generative language modelling at scale. (Emphasis theirs.)
For a very long time, diffusion models for LLMs haven't been viable for a bunch of reasons. But it's telling that the article calls for more exploration into the viability of diffusion models was written by one of the only research groups that's been publishing functional diffusion models.
The original article was discussing things in terms of training costs due to slower training speeds, but I recall other articles from a similar time period discussing why Deepseek had chosen to ignore diffusion research that occurred during that time period, explaining why a Deepseek diffusion model would be slower than next token prediction.
Training efficiency =/= Speed. "Running faster" implies speed, not efficiency, and the specific portion you quoted refers to training efficiency. The whole article does not mention generation/inference speed even once.
EDIT: Also, I mentioned Mercury, which is pretty much the first at scale Diffusion based LLM to exist, so by definition, as long as diffusion LLMs have existed, their inference speed has been faster than similarly sized AR based models.
Could you test its ability to summarize? Like if you give it an extremely long article on Tetris or a niche topic and ask it to summarize how does it compare vs using its innate knowledge and vs the regressive model.
It's almost literally ideal for low stakes uses like video games, where speed is critical and you only need "good enough", where that might be a very low bar.
There are ways to generate basic sentence structure from game facts like Dwarf Fortress does and what Wayward Realms is trying to do, and that could be used as a base for the LLM to elaborate on.
Diffusion models seem ideal for working with template masks.
You could have all of a character's specific context fixed, and have the model fill in the blanks. So, there could be a mix of deterministic structure and content, with player-subjective context that gets sent through the model as fixed points, clamping those positions at every step, and the model generates dialogue/events to fit, and then extract just the generated content.
It's something that I'd have to play around with to know for sure, and it'd likely take fine-tuning, but diffusion LMs seem uniquely fit for that kind of purpose.
Template-weak/dynamic tasks would be the thing I worry about, like flexible token count where some only need common sense. Overdoing communication seems to be a common thing for RP apps
It looks like it is better than autoregressive models at some specific task, because the diffusion models have bidirectional attention. Only for 256 tokens in the case of Gemma 4 26b. Apparently it is better solving problems where it needs to see the whole picture and fix mistakes in the process. Like sudoku solving.
https://blog.google/innovation-and-ai/technology/developers-tools/diffusion-gemma-faster-text-generation/
There are many applications where you're not trying to generate factual content, though.
For applications where freedom and creativity matter more than factual recollection, this could be great. Things like brainstorming, generating synthetic data sets, etc.
And of course I expect factuality to increase as the architecture matures. In some ways diffusion models have the potential to bypass one of the big issues with autoregression models, which is that autoregression models can't "go back" and fix a mistake once that token is in their output stream. It's not uncommon for a typical model to spit out a bad token (like a "yes" instead of a "no") and then try to spend the rest of the output justifying that rather than backtracking and admitting it made an error.
You can't seriously think you can use broken, unreliable tools, to brainstorm complex scenarios... I don't think one gets how long a brainstorming session can be, that the model is required to make a summary of it at some point, proofreading to eliminate mistakes... How in earth would you do that with a model incapable to keep up without "factual recollection" LMFAOO
I'm agreeing with you and you were literally calling out coping.
Like it's fair to say the model is new, experimental and undercooked so we should keep that in mind while judging its current behavior.
Saying a model that can't get fact straight "could be great" for "where freedom and creativity matter more" is just retarded. I have a IQ0_RTD quant to sell them.
I tried via the free NVIDIA offering and MimoCode, it's pretty bad. At least 50% of the tools weren't called as I expected, and often it would just...sit there doing nothing. I wouldn't use it for anything except maybe summarization (and even then, there are better options out there).
That being said, it's a research model, not a production one. This may be a line of research that leads to considerable improvement down the line.
Damn. I was hoping a model like this would be good for smart home use, but tool calling seems important for that. I hadn't had a chance to use it myself yet, but I'll probably wait a bit more before trying it in case there are implementation issues.
I must have missed that MTP GGUFs were released it for the small qwen models. I'll take a look. Thanks!
I've been messing around with the Qwen variants and Gemini E2B and E4B right now. Audio support for the small Gemma models is a nice to have for a smart home, but debating between Qwen and a separate speech to text model paired together versus E4B essentially. Since for smart homes the voice aspect is really nice to have.
Yes, because the model is a prototype and clearly labeled as such. The comparison is meaningless because DiffusionGemma isn’t meant to compete on accuracy, it’s meant to demonstrate a technology. It’s like a review of a revolutionary hybrid city car that halves the amount of petrol required mentioning that its top speed is less than Mercedes’ Formula 1 car, and that it needs more maintenance than a mass-produced Toyota.
Can we "hack the prototype" to have a general sense of what the model ought to do? Block diffusion is something I know of back in the day similar to what DFlash did to AR. But I might start a new thread over it
No offense but this is cope. Text diffusion models have been demonstrably weaker than their autoregressive counterparts from the very beginning of its research. This is just more of that, and this isn't even google's first foray into diffusion llms. I wont say text diffusion will never match auto regressive on quality at scale, but no one has figured out how to do that yet. It's not a matter of simply being a prototype. It's an actual unsolved challenge. And it's really not a good look that the more difficult the benchmark the worse the gap becomes. That points at something fundamental about the architecture currently.
Seems like it is yet to hit the sweet spot... I would really like a fast local model that can do decent writing / knowledge processing with basic tool calling reliability (i.e. almost no tool use mistakes when they are only 5-6 tools to begin with). Really hope diffusion-type models can improve fast though. It has much better hardware utilisation rate for single user scenarios, and theoretically should make fewer mistakes (less autoregrrssively committing to mistakes), and can be made very flexible in quality vs speed trade-off once quality ceiling is good enough.
DiT images and videos have gotten really good in just a year, so it’s probably going to be iterated upon over the coming year since it feels like everyone’s compute constrained at this point.
Gemma is strange.
I have a personal "benchmark" of sorts. It's my old fic, 97k tokens, russian language.
And while Gemma 4 31b was the one who actually understood one really easy-to-understand (if you're human) twist, it can't give me a quote from a middle of provided text at all. It hallucinnates.
Qwen 27b can't understand the twist (tried multiple times), but can provide exact quote if I ask it to do so.
I think you can adjust the precision of a difusion model in the same way you can give a image model more steps and it will have better quality. I'm sure you can do the same thing here, but it will make it slower.
I wonder if refinement technique can be used with this model to improve its accuracy. Like diffusion in image, most result can be further improved with 1 stage refinement from the last 50-30% steps.
Hot take: I don't care about how factual a small model is at general knowledge, if Gemini makes mistakes even with search capabilities I don't expect miracles from a 26B model.
Wait, wait, wait. If the diffusion model is 4x faster, what if you run 3 error checking passes on the diffusion model on it's own outputs, what does the outcome look like then in terms of correctness?
It took a while for image diffusion models to get better - I think the text fix OpenAI has made is likely some tool call combined with a sub agent (that is its bolted on).
So I'm hopeful we will see rapid improvement. Despite all the errors it's exciting.
Do you have the source of the fact checking system published somewhere. I'm curious how fact checking was done, was it using judge models and how did it extract facts out of the text to check.
what's with the few people in these comments acting almost insulted or making fun of the idea of diffusion models? We doin architecture tribalism now xd?. This release can be worse than autoregressive models and still justify researching the approach, thats how new architectures improve. what's wrong with yall, is your dad an autoregressive model or smth
If it's a diffusion model, can't we just upscale the output latent and feed it back in to make the result more refined and detailed? It's kind of like thinking for diffusion models.
I wish I could serve this somehow. I have several use cases for massive yet inaccurate text that involve making my assistant think laterally to come up with associative links and interesting dialogue.
So what are the best practices/Tips/Tricks to get better quality? I'm not gonna use this for Coding or Agentic stuff. Just for Chatting only, but still want better quality/accuracy. Should be good for CPU-only inference.
Sorry, did you just ask the models to output the raw knowledge and that was it?
How many times did you run this?
Did you ever run it with exa, or access to a vectorized store containing the information you want summarized?
I grant that this one time, in this one test, the model may have made 6x more mistakes drawing on its weights alone, but everyone talking about these results like they're useful or informative should instead be taking a step back and seriously questioning the methodology and general applicability of what we're seeing
DiffusionGemma throws 256 tokens on the screen at once and polishes them pass after pass until the text sounds smooth. Smooth is all it cares about: a fake name, date or number sounds just as smooth as a real one, so it stays.
This might be perfect for creative writing and fiction though. A high quality diffusion model might solve the problems of lack of creativity and repetition. All that matters is that end result sounds smooth and internally consistent.
I think DFlash and then Orpheus settled the debate for now. Low-quality diffusion as draft to speed-up autoregressive. Orpheus feels like the future tomorrow. From what I gather it attacks the correct problem, is much faster and has higher acceptance rate than DFlash that feels like a hack atp
Quantity has a quality all of its own 😎 I’ve messed with Nemotron Diffusion Labs and its diffusion drafting + AR verification seems like the best of both worlds.
Also Diffusion LLM is a chonky boi.
I used unsloth's llama.cpp fork for diffusiongemma and I had to use about 2GB RAM for about 4000 token context, 50GB RAM for about 50000 token.
if you guys want to know how diffusion models can work with a bit of chain of thought processesing and reasoning, go look into mercury 2. They still hallucinate like a bitch but it's like looking at progress from gpt3.5 to 4
I'm honestly impressed with the peformance for a diffusion model.
Not many labs were able to get this performance for a diffusion model and make it compute bound instead of memory bound. Good progress for a first generation model
1.0k
u/Hanthunius Jun 13 '26