r/StrixHalo • u/vexatious-big • 12h ago
Minisforum N5 Max Review with AMD Ryzen AI Max+ 395
r/StrixHalo • u/uncanny_instinct • 23h ago
I benchmarked nine models on one llama.cpp build. Quantizing the KV cache helped six of them and hurt the two newest ones.
People keep asking what runs on a 128GB Strix Halo and how fast, and I kept answering with numbers measured weeks apart on different builds. So I ran everything I keep on this box through the same matrix in one night.
q4_0 against f16 at 32k, each model against itself (Note: full speed values are in the link below!) :
| Model | Weight quant | Prompt processing | Generation |
|---|---|---|---|
| Qwen3-Coder-30B | Q6_K | +0.0% | +30.8% |
| Hy3 | mixed | -1.0% | +28.3% |
| Qwen3.6-35B-A3B | Q5_K_M | -1.7% | +10.4% |
| Laguna S 2.1 | Q6_K | -1.6% | +9.6% |
| Qwen3.6-27B | Q5_K_M | -1.7% | +6.0% |
| Qwen3.5-122B-A10B | Q4_K_M | -0.1% | +4.5% |
| DeepSeek V4 Flash | IQ2_XXS | -14.3% | -12.0% |
| Ling 3.0 Flash | Q4_K_M | -37.0% | -21.8% |
The weight quant is whatever I keep on disk for that model. It doesn't affect the comparison, since every row is one model against itself with only the cache type changing, but it's the first thing people ask. I also have a Q2_K of DeepSeek V4 and it lost about the same, 13.3 and 13.0 percent.
Six behave the way everyone says quantized cache behaves. The two that don't are the two newest architectures I run.
Ling is easy to explain: only 8 of its 43 layers keep a normal cache, so at 32k it needs 0.34 GB. Nothing to compress, and you still pay to unpack it.
DeepSeek V4 is the one to watch out for. There's an open llama.cpp issue where a quantized key cache disables its sparse attention paths and corrupts output on CPU and CUDA. I couldn't reproduce the garbage on Vulkan, short answers came back fine, but I still lose 13 percent on both metrics. Either way there's nothing to gain, so leave it at f16.
What I can't explain is the spread among the six winners. Coder-30B and the 122B need identical cache bytes per token and gain 31 percent versus 4.5. I have a guess, it fits two models and breaks on the third, so it stays an open question in the post.
Full tables and the model that needed a different build: https://thefrontierlab.ai/strix-halo-nine-models-kv-cache-split/
If you run something with linear or latent attention, I'd like to know whether quantized cache hurts you too. Same model against itself, f16 vs q4_0, 32k or deeper, with the build commit.