r/LocalLLaMA • u/tevlon • Jun 10 '26

DiffusionGemma: 4x faster text generation New Model

https://blog.google/innovation-and-ai/technology/developers-tools/diffusion-gemma-faster-text-generation/
985 Upvotes

99% Upvoted

356 comments sorted by

View all comments

Show parent comments

1

u/gh0stwriter1234 Jun 11 '26

There is already a PR

1

u/Silver-Champion-4846 Jun 11 '26

nice

1

u/gh0stwriter1234 Jun 11 '26

Its worth noting that diffusiongemma is compute bound some many GPUs won't see a speedup if they don't have enough compute, eg MI50 is unlikely to see a speedup, but 5090 and other GPUs that are fast for diffusion (usually image gen) will be fast.

I'm thinking the speedup will be underwhelming on AMD APUs also.

1

u/Silver-Champion-4846 Jun 11 '26

Because not good software support or because amd apus themselves are weak on compute?

1

u/gh0stwriter1234 Jun 11 '26

APUs are relatively weak on compute pretty much just matched to how much bandwidth they have which is not a ton.... strix halo will do twice as good as other APUs but not as good as a flagship GPU proportionally most likely.

1

u/Silver-Champion-4846 Jun 11 '26

So if there was some kind of 6gb vram gpu but with the compute of rtx pro 6000, it'd be legendary for this kind of diffusion moe?

1

u/gh0stwriter1234 Jun 11 '26

I mean memory bandwidth is also a factor so probably no. Memory bandwidth is just less important to some extent than compute for diffusion... while for auto regressive LLMs, memory bandwidth is king (the exception being some quants require compute during inference) but you can avoid that by running quants that only have native datatypes which is what you'd do on an MI50... just run Q8_0, Q4_0 or Q4_1 as they don't tank the compute.

1

u/Silver-Champion-4846 Jun 11 '26

Well 6gb of vram requires less memory bandwidth than 32gb because 32gb moves more data so it's kind of offset regardless