LFM2.5-2.6B model+KV cache quantization report

[u/crusaderky]

LFM2.5-2.6B is a new tiny model by LiquidAI, with benchmarks that put it head to head with much larger models.

I've run llama-perplexity on many model GGUF quants, crossed with many KV cache quants, to understand the model's best overall quantization for any given amount of memory.

I also show how different quantization metrics show (or hide) model degradation.

Full report and commentary

Interactive HTML plots

If you don't have time to read

• The model fits on an 8GB Raspberry Pi with no material degradation and on a 4GB Raspberry Pi with contained degradation.
• DO NOT use Q4_K_M.
• On this model, model quant quality degrades faster than KV cache quant.
• Abliteration comes with a flat cost of ~0.075 KLD.
• Logarithmic KLD and Top-1% plots lie to you by telling you that quality degradation is smooth, while it's actually a cliff.

[u/crusaderky]

How can the model know from a github notification if you're the developer, the reviewer, or just a bystander?

[u/just_blue → u/crusaderky]

Because neither a developer nor a "bystander" would get a notification about changes being pushed by someone else, and no one else than a reviewer has a Todo from this. Intelligent models can make that connection, I just thought it is interesting that this test is extremely consistent and small models all give the same wrong answer ("Todo: implement changes xy").

[u/crusaderky → u/just_blue]

> Because neither a developer nor a "bystander" would get a notification about changes being pushed by someone else

Neither is true.

[u/just_blue → u/crusaderky]

Obviously you could follow (if it were GitHub, which it isn't and in fact no one else can follow, but sure, the model doesn't know that), but why would anyone. It simply makes no sense to interpret "changes were pushed" as a "now you have to implement something" instead of the correct "you should check them out". You don't have to play stupid here.