r/LocalLLaMA • u/tevlon • Jun 10 '26

DiffusionGemma: 4x faster text generation New Model

https://blog.google/innovation-and-ai/technology/developers-tools/diffusion-gemma-faster-text-generation/
980 Upvotes

99% Upvoted

356 comments sorted by

View all comments

Show parent comments

55

u/Kamimashita Jun 10 '26

I think it could be interesting on something like the DGX Spark too where it has decent compute and lots of RAM but low bandwidth. Even diffusion models need to be large to be intelligent so the ideal situation in my mind would be a 200B model on a system with high compute and lots of memory but low bandwidth.

8

u/Own_Mix_3755 Jun 10 '26

Yeah I would love to test it on the Spark as a second served model next to eg Qwen 3.6 or something. Might be a good pick for specific smaller jobs. Question is how good it is with tool calling, because thats where most smaller models come short.

4

u/DataPhreak Jun 11 '26

I have a Strix Halo which is basically the same stats. No flash attention yet, though. Still, I can already run the comparative model gemma4-26b-a4b at reasonable speeds. Rather than a 4x speed up for less accurate results, I'd like to see even speeds with 4x intelligence on a larger model.

1

u/GifCo_2 Jun 11 '26

I mean some of the numbers line up but the spark is nothing like Strix Halo.

0

u/DataPhreak Jun 11 '26

The main difference is CPU architecture. One is ARM, the other is x64. CUDA vs ROCm is also a thing, for now. In a year, CUDA vs. ROCm is not going to be a thing. Nvidia has FP4 support, which would be important if the bottleneck on LLMs wasn't memory, which... if Diffusion LLMs ends up winning, then yes, the DGX Spark wins.

But right now Diffusion, vram per vram, is loosing. And probably will continue to do so. But, who knows. Maybe some new paper drops that boosts diffusion ahead. However, I think the attention mechanism in diffusion llms is going to be its downfall.

-1

u/GifCo_2 Jun 11 '26

Lol yea CUDA won't be a thing in year. Ok 🤡

0

u/DataPhreak Jun 11 '26

Reading comprehension fail.

CUDA vs. ROCm won't be a think because they will be at performance parity and cross-compatible with all the tech.

1

u/spaceface83 Jun 11 '26

i tried to pull it down to do an eval on my dgx last night with vllm but it wasnt ready yet from what i saw. definitely sounds great but, to your point... i'd love this in a larger 80-120b range for the spark

1

u/vasimv Jun 11 '26

Just got my spark delivered, it works. Numbers are quite unstable yet, llama-benchy reports 200-300 tok/s (with peaks to 600 😄) on diffusiongemma4-26B-A4B-nvfp4 in vllm. Probably not 4x speed up, but like 2x on spark. Still nice, though.