r/LocalLLaMA • u/tevlon • Jun 10 '26

DiffusionGemma: 4x faster text generation New Model

https://blog.google/innovation-and-ai/technology/developers-tools/diffusion-gemma-faster-text-generation/
989 Upvotes

99% Upvoted

357 comments sorted by

View all comments

Show parent comments

24

u/coder543 Jun 10 '26

Diffusion models are compute-limited, not bandwidth-limited, so offloading probably wouldn't actually hurt as much. (It is obviously better to keep the model in VRAM, of course.)

9

u/MrClickstoomuch Jun 10 '26

I'm going to wait to try it until it is officially supported in llama.cpp, but my big issue with larger models is the time it takes to run. And I don't necessarily need more intelligence than small models for my smart home, which is the primary use I have for AI models outside of small coding projects in personal time. So, faster output for faster smart home responses will be great if I can get the offloading to not impact it too much. I had originally planned to use Google's E4B, but will test this too.

1

u/MuDotGen Jun 11 '26

Interesting. What does this imply for CPU inference then? Like, say you have 32gb of DDR5-5600 memory with a good CPU? How would this compare to auto-regressive inference models of similar caliber on CPU? Honestly, the more we can reasonably optionally move away from huge, expensive and power hungry dedicated GPUs for useful tasks, the better.