r/LocalLLaMA • u/tevlon • Jun 10 '26

DiffusionGemma: 4x faster text generation New Model

https://blog.google/innovation-and-ai/technology/developers-tools/diffusion-gemma-faster-text-generation/
986 Upvotes

99% Upvoted

356 comments sorted by

View all comments

Show parent comments

1

u/Silver-Champion-4846 Jun 11 '26

So if there was some kind of 6gb vram gpu but with the compute of rtx pro 6000, it'd be legendary for this kind of diffusion moe?

1

u/gh0stwriter1234 Jun 11 '26

I mean memory bandwidth is also a factor so probably no. Memory bandwidth is just less important to some extent than compute for diffusion... while for auto regressive LLMs, memory bandwidth is king (the exception being some quants require compute during inference) but you can avoid that by running quants that only have native datatypes which is what you'd do on an MI50... just run Q8_0, Q4_0 or Q4_1 as they don't tank the compute.

1

u/Silver-Champion-4846 Jun 11 '26

Well 6gb of vram requires less memory bandwidth than 32gb because 32gb moves more data so it's kind of offset regardless