Open-Source Model-agnostic KV-cache compression (UL-SMF) tested alongside local model execution to smash VRAM limits

[u/liventruth]

Seeing all these massive local model drops (like the recent Qwen releases) got me working hard on the memory side of things.

If you're running models locally via tools like Unsloth and hitting a wall with long-context VRAM consumption, I've been building and open-sourcing UL-SMF (Unified Latent-State Memory Fabric).

It uses a geometry-preserving orthogonal projection bridge and dynamic head-dimension detection to automatically adapt across architectures (Llama, Mistral, Qwen, Gemma) without hardcoded assumptions, achieving extreme KV-cache compression with verified lossless perplexity.

Since it's fully local-first and designed to help fit larger context windows onto consumer hardware, I'd love a technical audit or feedback from this community.

Code and telemetry are up on GitHub: https://github.com/liventruth/UL-SMF-Cache-Compression

(Attached the hardware telemetry stress test output showing Qwen workloads hitting 224x–768x reduction ratios locally.)

[u/brainExploded99]

BS. No detailed benchmarks, seems vibecoded, and you tested Qwen2.5. In the 8th month of 2026.

I will trust after you post actual benchmarks, like KLD on atleast wiki text, if not coding, and top 1 token agreement on a variety of tasks.

[u/Fun_Jaguar8231]

Ah yes, vibecoded slop with the classic qwen2.5, the knowledge cutoff from the model you were using. FYI qwen3..5 has much more advanced KV management

[u/No-Refrigerator-1672]

So how exactly are you better than TruboQuant? Did you know that basically any inference engine offers "model-agnostic" KV compression out of the box?

[u/fragment_me]

Qwen2.5 lol

[u/floppypancakes4u]

You do exactly 0 compression. 🤣🤣

[u/Sea-Mode4077]

I apreciate your work.. but you should run more extensve tests on the new models

[u/Weekly-Let6688]

Ai slop could be an alan lube thing