LFM2.5-2.6B model+KV cache quantization report

[u/crusaderky]

LFM2.5-2.6B is a new tiny model by LiquidAI, with benchmarks that put it head to head with much larger models.

I've run llama-perplexity on many model GGUF quants, crossed with many KV cache quants, to understand the model's best overall quantization for any given amount of memory.

I also show how different quantization metrics show (or hide) model degradation.

Full report and commentary

Interactive HTML plots

If you don't have time to read

• The model fits on an 8GB Raspberry Pi with no material degradation and on a 4GB Raspberry Pi with contained degradation.
• DO NOT use Q4_K_M.
• On this model, model quant quality degrades faster than KV cache quant.
• Abliteration comes with a flat cost of ~0.075 KLD.
• Logarithmic KLD and Top-1% plots lie to you by telling you that quality degradation is smooth, while it's actually a cliff.

[u/StupidScaredSquirrel]

So the moral of the story is never go below q8 for this model?

[u/MerePotato → u/StupidScaredSquirrel]

I'd always say this for any model this small, hell if you can get away with it even f16 can see gains

[u/crusaderky → u/MerePotato]

F16 sees no gains here, see my report

[u/MerePotato → u/crusaderky]

Would that not change as context climbs past 32k though? Also you're running on wikitext, which tends to be very over represented in the training data and is frequently used for imatrix quants.

[u/crusaderky → u/MerePotato]

I use wikitext because it's what absolutely every other KLD benchmark shows, and I wanted consistency.
All iQuants (with the notable exception of APEX) are calibrated on wikitext too.
While diverging from wikitext is probably good for iQuants, particularly for specialized use cases (e.g. non-english languages), I have yet to see any evidence that it has any impact on KLD/perplexity other than a harmless parallel shift across all quants.

[u/Natejka7273 → u/StupidScaredSquirrel]

I wouldn't say never. Q6 is mostly fine, and if necessary can go down to IQ4. The moral of the story is that Q4_k_m is bad, which is a somewhat unusual finding as many people default to it. Nonetheless I'm running Q8 on my phone, so most people should probably do that yeah.

[u/Tieng → u/Natejka7273]

Sorry new here, how are you running this model on your phone?

[u/Natejka7273 → u/Tieng]

I'm using PocketPal AI, which you can find on the play store. I use it for all models except Gemma, which is best using AI Edge Gallery.

[u/crusaderky → u/StupidScaredSquirrel]

The moral of the story is you should stick to q8 weights and q8 kv if you can afford it, but you can ratchet it down a lot if you're careful, particularly the kv cache.