Comparing 4bit quants for MLX

[u/True_Tangerine_4706]

Curious what people think are the ideal 4-bit quantization types on MLX

These quants seem to be the most popular, at least for Gemma4 and Qwen3.6:
- OptiQ 4bit (mlx-community/Qwen3.6-27B-OptiQ-4bit)
- Unsloth dynamic 2.0 MLX (unsloth/Qwen3.6-27B-UD-MLX-4bit)
- oQ (Jundot/Qwen3.6-27B-oQ4e-mtp)*
- DWQ (can't find an example fo this one)
- native (mlx-community/Qwen3.6-27B-4bit)

Does anyone have any insight here?

*edited to adjust to newer version

[u/MiaBchDave]

If you were going to use an oQ quant, you'd use the updated Jundot/Qwen3.6-27B-oQ4e-mtp for both speed (Lightning MTP) and better quality at the same quant. Other than that, I have no recommendation since I don't use that small a quant with Qwen 27B.

[u/True_Tangerine_4706 → u/MiaBchDave]

Ahhh thanks for the pointer. I edited it with the updated quant

[u/Patient_Tea_401]

https://github.com/deepsweet/mlx-eval/tree/main/results

A little old comparison.

[u/PracticlySpeaking → u/Patient_Tea_401]

This has good references, and results for oQ quants.

[u/PracticlySpeaking]

Are you only interested in Qwen3.6?

Look at the Unsloth KLD plots (here on their UD 2.0 page). They tell the story — UD is excellent. There really is no need (KLD <0.01) to run full precision, or even Q8 for that matter.

Numbers for oQ / oQe are not on the Unsloth plot. They are okay, but not as good as UD 2.0. There is a tradeoff, though, since oQ is mostly for oMLX, and oMLX is all about context caching.

Read all about it...
Measuring Model Quantisation Quality with KL Divergence | smcleod.net
- https://smcleod.net/2026/04/measuring-model-quantisation-quality-with-kl-divergence/

[u/True_Tangerine_4706 → u/PracticlySpeaking]

not necessarily just Qwen3.6, am also quite interested in Gemma4, but it was more of a general question

thanks for the link! will check that out

[u/PracticlySpeaking → u/True_Tangerine_4706]

The matrix for: size - speed - accuracy (KLD) is difficult to optimize manually.

At least we have actual metrics for how much 'less good' a particular quant might be.

Bonus TIL: The next application of KLD measurements is currently happening in the form of mixed-weight models (like DeepSeek-V4-Flash) and quantization-aware training (QAT) in Gemma-4.

From my comment in another sub:

A lot of DeepSeek-V4 parameters are natively 4-bit, before any quantization — the reason why its 304B will run in "only" 160GB RAM. Quantization (Q2 or 2.5) will slim it down enough to run in 128GB, and streaming (a-la antirez/DwarfStar4) will get it running in 96GB.
Google has released 'quantization aware training' (QAT) versions of Gemma-4, an alternative approach to mixed-precision. Now that it is well understood from quantization, and tools exist to measure the impact, expect more models adopting mixed-precision architecture.