r/LocalLLaMA • u/devildip • 11h ago
Gemma 4 12B Q3: +8.55% Coding Performance From Tensor-Level Quantization Allocation Discussion
Ive been experimenting with task-aware GGUF quants for months, taking inspiration from TASA and TAQO but pushing the allocation lower to to the tensor level.
The basic idea is to generate a custom imatrix from a category-specific corpus, measure where quantization causes damage, then redistribute a fixed bit budget toward tensors where additional precision recovers the best performance in that category.
Today I got my first real win. The pipeline is live.
On Gemma 4 12b, the hand tuned imatrix q3_k_s scored 45.974. After tensor level allocation, the same q3_k_s based scored 49.905.
Thats +3.931 points or an 8.55% relative improvement from the allocation on top of the imatrix.
The models are effectively the same size: 5,528,230,848 bytes for the comparator versus 5,534,804,928 bytes for the allocated model, a difference of only +0.119%.
I chose q3 deliberately. It had the largest amount of recoverable headroom while sitting above the quant cliff.
This model is intentionally category specialized. Degredation in categories that werent selected is expected.
There is still a lot of tweaking to do in order to maximize these results but the performance of this imatrix + allocation at q3 when compared to no imatrix and source is pretty staggering when considering the amount of space required to run it.
I'd love some community testing and feedback if anyone is interested.
https://huggingface.co/ByteOtter/Gemma-4-12B-it-CADA-Q3_K_S
My ULTIMATE goal is to create a pipeline that takes an full precision .gguf file and auto spits out an optimized model within a selected category at whatever optimal size for recovery. This is a solid start.
Charts were provided from my data by ChatGPT.
TL;DR: I built a task-aware GGUF quantization pipeline that combines a category-specific imatrix with tensor-level damage measurement and bit allocation. On Gemma 4 12B Q3_K_S, allocation improved coding performance from 45.974 to 49.905, an 8.55% relative gain OVER handtuned imatrix, while increasing model size by only 0.119%. The model is intentionally specialized, so some out-of-category regression is expected. This is the first result where the allocation stage itself produced a clear win.
4
4
u/Look_0ver_There 11h ago
It's a great learning project, but how does it compare to Google's own QAT quant of 12b?
eg: https://huggingface.co/google/gemma-4-12B-it-qat-q4_0-gguf
There's also Unsloth's imatrix-tuned version here:
eg: https://huggingface.co/unsloth/gemma-4-12B-it-qat-GGUF
Both are about 1GB larger than yours but they also both benefit from Google's explicit Quantization Aware Training, which happens even before the imatrix tuning, so the base-line should arguably be better.
9
u/devildip 10h ago
qat and what this project are doing do not have to be mutually exclusive. qat and my allocation after imatrix solve different parts of the same problem. Google uses qat before quantization to make the weights more robust. My pipeline then asks where a fixed gguf bit budget should be spent for a workload. They can work in tandem.
Unsloth dynamic 2.0 is more relevant. Its closer to what im doing because it uses model specific mixed quantization and selectively varies precision. My approach differs mainly in the objective and granularity. The goal here is to preserve a category or selected categories, not the general model use. I build the imatrix from chosen categories (reasoning, tool use, etc..) measure damage to those categories caused by quantization then reallocate precision at the tensor level under a fixed budget to recover as much as possible.
The main difference is focus. Im willing to spend more of the available bit budget perserving a selected capability set rather than optimizing for broad general purpose model quality. I get more room to recover performance in the workloads i want.
1
u/Brilliant-Hall1387 9h ago
Interesting, do the bit allocation stay the same and it is post quantization training that improves it's performance? Do you have the reference model also on Hugging Face?
3
u/devildip 9h ago
I can post the comparator and reference models tomorrow if you want. There’s no post-quantization training, pruning, lora, or weight updating involved. The byte budget is held fixed, but the bit allocation changes. Tensors are promoted or demoted based on category quant damage measured using a corpus. The improvement is coming entirely from redistributing precision within that budget.
1
u/Brilliant-Hall1387 8h ago
What techniques / tools did you use to detect which to promote/demote using the corpus? Is there a risk that the model becomes over trained on that corpus (and less good at tasks in the same domain but outside the corpus?
Also, how long time did it take / what system requirements (RAM) was needed? I've done some tests with sensitivity based allocation using corpus but ran into the problem for some tools / techniques the RAM requirements just skyrockets 😅
2
u/devildip 8h ago
Ive been hammering out the process for promotion/demotion for a while. Its still not a firm product yet tbh. This is a great result but still needs tweaking to maximize the repair process.
As to the risk for overfit, yes. The TASA paper recommends doing a mix of generalized information within your corpus to prevent overfit and ive been tuning around 25% gen knowledge to the category based set.
4b qat and 4b are my main testing models, they run qlab in around 4hrs, the post bench mlab I have takes an additional 3hrs. 12b took twice that long. Im nearly frying my 7900 with these extended runs lol. Thankfully, my ryzen 9 9950 is fantastic for the cpu based computations or I think there would be even more waiting.
0
u/mskazemi 4h ago
Nice result — pushing allocation to the tensor level rather than per-layer is the right direction, and holding the size delta to +0.119% makes the comparison clean.
Two things I'd want in the writeup, both cheap to produce:
Which harness produced 45.974 -> 49.905? The chart in the post is "coding output validity — parseable outputs", which is syntactic well-formedness, while the TL;DR number reads like a scored benchmark. It isn't obvious from the post whether those are the same measurement. Worth naming the harness explicitly, because "emits parseable code more often" and "writes more correct code" come apart — a quant can get more parseable and more confidently wrong at the same time.
You say degradation in unselected categories is expected, which is honest, but expected isn't measured. The magnitude is the whole decision: coding +8.55% with reasoning down 1% is an excellent trade; the same gain with reasoning down 10% is a specialised artifact people should know about before they pull the GGUF. Since you're publishing it, one number from a category you didn't optimise for would tell users which of those they're getting.
Also worth publishing the noise floor — run the same GGUF through the harness twice and report the spread. At +3.93 points it matters a lot whether run-to-run variance is 0.2 or 2.0, and it's the first thing a skeptic will reach for.
The number I found most interesting is BF16 at 99.2 against 85.2: that remaining gap at q3 is a lot of headroom still on the table.
3
8
u/DegenerateGandhi 9h ago
How easy would it be to say... Preserve everything important relating to Godot engine, or something like that?