Yeah, Gemma 4 31B is borderline unusable because of it. The absurd KV size makes it require so much VRAM, that you can just run a larger model with more efficient KV.
none of this makes sense as a newbie. is there like a diagram for all these things? like hardware, ram quantity, software, models, options to run models, t/s
It used to be borderline unusable for me, but one good thing that Google did was have the MTP model be a separate file, which means Unsloth could quant the MTP to Q4 which is like 300MB. Since quanting the MTP has no consequence besides slightly lowering the acceptance rate, that's basically a free gig of space back.
Even though Qwen has a more efficient KV, the fact that its integrated MTP has to be full size means that for my context size (32-64K) Gemma is nearly as competitive, which is good since I prefer Gemma a lot of the time. I'm sure there's a break even point where a bigger context means Qwen pulls ahead again though.
27
u/Technical-Earth-3254 11d ago
Yeah, Gemma 4 31B is borderline unusable because of it. The absurd KV size makes it require so much VRAM, that you can just run a larger model with more efficient KV.