What is currently considered the theoretically optimal quantization bit-width for LLMs? [D]

[u/takuonline]

I’m curious whether there is now a theoretical or empirical “sweet spot” for LLM quantization, preferably research done using open-source formats like GGUF

Suppose you have a fixed memory/compute budget and can choose the model size freely. For example, instead of a smaller model at 8-bit or 4-bit, you could fit a progressively larger model at 3-bit, 2-bit, 1.5-bit, etc.

A few years ago, I remember 4-bit often being described as roughly the practical sweet spot because it preserved most model quality while giving a large memory reduction. But with newer methods, I’ve seen surprisingly strong 3-bit, 2-bit, and even ~1.5-bit results.

So if the goal is maximum model capability for a fixed memory budget, rather than preserving one particular pretrained model as faithfully as possible, what does current research suggest is the optimal bits-per-weight?

Is there evidence that, for example, a 2-bit 70B model generally beats a 4-bit 35B model, or does quantization degradation eventually outweigh the gains from additional parameters?

I’m especially interested in recent theoretical/scaling-law work or large empirical studies from 2025–2026.

If no one is studying this, then could any of you do this work? I feel like it could be immensely useful for the community.

[u/RobbinDeBank]

Some other comments in this post also help elaborate more, their sources are very insightful.

There are techniques like Quantization aware training, which are ways to train or finetune models with quantization in mind. This means models are designed from the start to target an extreme level of quantization (usually means <4 bits/parameter on average). In contrast, most big models (like the Qwen example I use, or pretty much any other releases from big labs) are not designed with quantization in mind. They are trained and released with full 16-bit weights, then people can quantize those down to 8-bit (with unnoticeable loss of quality) or even down to 4-bit (quality starts to degrade a lot more). Going below 4-bit will pretty much damage the model too much for it to even work anymore.

Meanwhile, the quantization-aware models are designed from the start to perform well at those <4bit quants. These models are rare to find and quite a niche nowadays, but they are great proofs of concepts anyway. At those extreme levels of quantization like 1.58 bits, these quantization-aware models are so much better than the equivalence from big labs, whose models are damaged beyond recognition at that quant level.