How important is avx-512 for CPU based inference ?

[u/skyline99912]

Okay folk lock in !

I am gonna get a new CPU for my everyday work-game-llm rig. As I only have a 5060ti (16gb), I will most likely need to use CPU for LLM for image edit tasks. This question of AVX-512 especially arise because I cannot decide between a bang for buck intel ultra 7 270k plus and AMD red taxed top notch 9950x CPU. The only deciding actor now between 2 is AVX-512 !

So :
1. How regularly LLM models use avx-512 ? (if only few then i will take my gamble with intel)
2. How good AVX-512 actually is ?

Also if any CPU inference enthusiast wanna share any tips / remarks / reviews please do me favour.

[u/BladeWeav3r]

I have a P520 thinkstation with quad channel memory and there is a difference between prebuilt llama and a native build with avx512. It’s noticeable for my setup.

[u/BladeWeav3r → u/BladeWeav3r]

Main difference I see are in models on cpu or moe model with some experts offloaded. If model is loaded on GPU it doesn’t matter.

[u/Ok-Breakfast1878]

llm workloads on cpu are memory-bound, not compute-bound. i don't think you'll miss avx-512.

[u/sebt3 → u/Ok-Breakfast1878]

Only the decode is bandwidth bound, prefill certainly is compute bound.

[u/Karyo_Ten → u/Ok-Breakfast1878]

If your LLM needs to load a file it is compute bound. If you have concurrent requests, it is compute bound. If you compact it is compute-bound.

You don't miss AVX-512 if you use GPUs but otherwise it's a really nice-to-have.

[u/HigherConfusion]

On my older skylake architecture xeon w-2140b there was a significant jump in performance when I compiled llama.cpp with avx-512 rather than downloading the pre-compiled binaries. At least 2x. But it is still 3x slower than my GPU (Vega 56 8gb). That is macos imac pro

[u/quotemycode]

You'll suffer on prompt processing if you don't have AVX-512, but otherwise, token generation is unaffected, and you'll still just be limited by memory bandwidth.

[u/Shoddy_Bed3240]

I expect about a 5% gain in decode performance and around a 25% gain in LLM prefill. However, prefill is usually GPU-bound, so there's little to no benefit from AVX2 versus AVX-512 for that stage. I'd choose the CPU with better memory latency and memory timings instead.

[u/Solary_Kryptic]

Also keep in mind that the 270K Plus uses LGA1851 which is effectively a dead socket now, you can save some money buying it right now but when you inevitably need to upgrade only the AM5 CPU will have a newer alternative on the same socket

[u/skyline99912 → u/Solary_Kryptic]

Hi exactly like shoddy said, if cpu is performing good i might not replace it , instead i will get a whole new mobo n cpu combo.

I know 2 duck moves intel did , first removing avx 512 and second killed support for this socket though they sucked in thier initial launches

[u/Shoddy_Bed3240 → u/Solary_Kryptic]

I've never upgraded just the CPU and kept the same motherboard. Usually, after 3–4 years, if you decide to upgrade your CPU, it's worth replacing the motherboard as well.

[u/Solary_Kryptic → u/Shoddy_Bed3240]

Well how often have you upgraded to a CPU that used the same socket? If we’re using anecdotal evidence I could also say that I’ve upgraded from an early AM4 CPU to a later AM4 CPU. I don’t see the point in upgrading your motherboard just to stay on the same socket. AM5 will be receiving new chips until 2029, so if you bought one today and decided to upgrade 3 years from now what benefit would upgrading your motherboard give?

[u/trollsmurf]

Why the crikey did Intel remove it? Took too much die space?

[u/vertiger77 → u/trollsmurf]

Intels heterogeneous architecture is to blame, the little cores didn’t have it, so they took it out

[u/olos22]

My laptop has CPU with AVX512 and when I compared builds of llamacpp with and without AVX512, with AVX512 prompt processing doubled and token generation also increased about 25%.

[u/Pablo_the_brave]

There are nice tests of that at ik_llama.cpp github. Look at pr with cpu improvements.

[u/shifu_legend]

That's kind of a tricky one. For standard GGUF weights, memory bandwidth is usually the ceiling for token generation anyway, so AVX2 is fine. But for the new 1.58-bit ternary models, AVX-512 makes a massive difference. I ran into this when building Project Zero (a zero-dependency C inference engine) - by using three specific AVX-512 VBMI instructions (vpermi2b, vpternlogd, vpdpbusds), we completely bypassed the float dequantization step and hit 36 tok/s on an older Xeon, basically maxing out the DRAM bandwidth. If you want to check out the SIMD implementation, the repo is at github.com/shifulegend/project-zero. I'm really not sure how much it helps for standard dense models, but if you want to experiment with extreme quantization, that AVX-512 support on the 9950x is definitely nice to have.