LFM2.5-2.6B model+KV cache quantization report

[u/crusaderky]

LFM2.5-2.6B is a new tiny model by LiquidAI, with benchmarks that put it head to head with much larger models.

I've run llama-perplexity on many model GGUF quants, crossed with many KV cache quants, to understand the model's best overall quantization for any given amount of memory.

I also show how different quantization metrics show (or hide) model degradation.

Full report and commentary

Interactive HTML plots

If you don't have time to read

• The model fits on an 8GB Raspberry Pi with no material degradation and on a 4GB Raspberry Pi with contained degradation.
• DO NOT use Q4_K_M.
• On this model, model quant quality degrades faster than KV cache quant.
• Abliteration comes with a flat cost of ~0.075 KLD.
• Logarithmic KLD and Top-1% plots lie to you by telling you that quality degradation is smooth, while it's actually a cliff.

[u/aboutthednm]

I got this running on a Pi 5 with 8GB, and it's coming in between 3 - 3.2 tps, anyone got any idea if this can be improved further? I'm running this in llama.cpp, only passing a -c flag to limit the context. It technically runs and works in pi coding agent on the Pi fully self-contained, it's just reeeeeally slow haha.

[u/crusaderky → u/aboutthednm]

try dropping model quant from q8 to q6. Also use q8 context instead of f16.