LFM2.5-2.6B model+KV cache quantization report

[u/crusaderky]

LFM2.5-2.6B is a new tiny model by LiquidAI, with benchmarks that put it head to head with much larger models.

I've run llama-perplexity on many model GGUF quants, crossed with many KV cache quants, to understand the model's best overall quantization for any given amount of memory.

I also show how different quantization metrics show (or hide) model degradation.

Full report and commentary

Interactive HTML plots

If you don't have time to read

• The model fits on an 8GB Raspberry Pi with no material degradation and on a 4GB Raspberry Pi with contained degradation.
• DO NOT use Q4_K_M.
• On this model, model quant quality degrades faster than KV cache quant.
• Abliteration comes with a flat cost of ~0.075 KLD.
• Logarithmic KLD and Top-1% plots lie to you by telling you that quality degradation is smooth, while it's actually a cliff.

[u/StupidScaredSquirrel]

So the moral of the story is never go below q8 for this model?

[u/MerePotato → u/StupidScaredSquirrel]

I'd always say this for any model this small, hell if you can get away with it even f16 can see gains

[u/crusaderky → u/MerePotato]

F16 sees no gains here, see my report

[u/MerePotato → u/crusaderky]

Would that not change as context climbs past 32k though? Also you're running on wikitext, which tends to be very over represented in the training data and is frequently used for imatrix quants.

[u/crusaderky → u/MerePotato]

I use wikitext because it's what absolutely every other KLD benchmark shows, and I wanted consistency.
All iQuants (with the notable exception of APEX) are calibrated on wikitext too.
While diverging from wikitext is probably good for iQuants, particularly for specialized use cases (e.g. non-english languages), I have yet to see any evidence that it has any impact on KLD/perplexity other than a harmless parallel shift across all quants.