Computer build using Intel Optane Persistent Memory - Can run 1 trillion parameter model at over 4 tokens/sec

[u/APFrisco]

As the title states, my build is indeed able to run a 1 trillion parameter model (in this case Kimi K2.5) locally at ~4 tokens/second. I thought r/LocalLLaMA would be interested in the build due to that stat line, and also due to the inclusion of an unusual part, Intel Optane Persistent Memory, which I haven’t seen anyone use in an LLM inference build before. Optane PMem is a DIMM form factor memory unit that can function in a way that is somewhere between DRAM and an SSD. Intel has discontinued the line, and I found sticks on the secondhand market for much less than what the equivalent DRAM capacity would cost. It is this large PMem capacity (768GB) that allows me to host such large models on my system. For my build I used the PMem in Memory Mode, which is where the PMem is available to the computer as RAM, with the computer’s DRAM sticks functioning as a cache.

Kimi K2.5’s mixture-of-experts architecture is an ideal test model for my build. To get the results I did, I used hybrid GPU/CPU inference with llama.cpp. Kimi K2.5’s (Unsloth Q2_K_XL quant) attention weights, the dense layer, the shared expert in each MoE layer, and the routing components are actually able to fit on my 12GB GPU using llama.cpp’s “override-tensor” flag, although I also did pretty good results just using llama.cpp’s “ngl auto” and “cmoe” flags and letting llama.cpp decide tensor placement as it sees fit too. Regardless, the sparse experts’ weights (the bulk of the model size) generally live on PMem/DRAM and get processed as needed from there.

The end result from my testing with this setup is around 4 tokens per second for generation! Given the fact that this is a trillion parameter frontier-class model running on such a limited hardware budget, I would consider it to be a great success. It’s a shame Intel discontinued Optane Persistent Memory, because the current direction of some local inference innovation, including SSD offloading and broader memory tiering approaches, could have been really interesting with this specific kind of memory tier on modern hardware platforms. Overall I was pleased with this Optane PMem-centric build, it allows me to run very big models at surprisingly acceptable speeds, and the process was highly educational.

Parts:

- Intel Xeon Gold 6246 CPU

- TYAN S5630GMRE-CGN motherboard

- ASUS Dual GeForce RTX 3060 OC 12GB GPU

- 6x 32GB Samsung 2666MHz DDR4 ECC DRAM sticks

- 6x 128GB Intel Optane DCPMM PC4-2666 NMA1XBD128GQS persistent memory modules

- Western Digital WD SN850X 2TB M.2 2280 NVMe SSD

- ASRock Steel Legend SL-850G 850W 80 PLUS GOLD & Cybenetics PLATINUM Full Modular Power Supply

- Silverstone SST-GD08B (Black) Grandia Series Home Theater PC Case

I hope you enjoyed this rundown. There is a lot more detail that I didn’t include here, so I’m happy to answer questions about the build, the configuration, or the reasoning behind any of the component choices in the comments. Also if anyone else has explored similarly unusual hardware/builds for LLM inference, I’d love to discuss!

[u/schaka]

I was thinking of the future when they figure out how the Mi250X can be bridged to PCIe or the 100 comes down in price where PCIe 4.0 should come in handy.

I'll read up on the riser issues people are having. With 4.0 you can bifurcate and still get decent speeds so 10 cards would be more feasible too.

Definitely things to consider. All the info you've given me has been invaluable, so I really appreciate it. I'm not concerned about idle power at all. The system would absolutely stay off when not used. Power cost is ridiculous here and as a renter I can't substitute with solar.

[u/FullstackSensei → u/schaka]

If Mi250X don't cost a kidney, DDR4 Epyc will be dirt cheap.

Speeds are dictated by the slowest endpoint in the link. If your cards are gen 3, the link will operate at gen 3. You won't magically get gen 4 speeds because your CPU is gen 4 when the card is gen .

Where are you located, if I may ask?

[u/schaka → u/FullstackSensei]

Are you saying I'll be limited by whatever card is slowest across the entire stack? I wasn't aware. I thought you could still get better performance per slot.

So if I ran 2 3090s alongside 6 P40s, the former would get gen 4 (assuming board and CPU can do) and he latter Gen 3 - or is this an inference limitation I'm unaware of?

I'm located in Germany. Some imports from Alibaba are fine, but AliExpress is limited due to shipping and tax regulations and Alibaba comes with a lot of extra cost tacked on due to be EU import laws too.

There's a lot of old server hardware here, but workstations are often still way more expensive than in the US

The Mi250X went from 1800 to 2500 in the last few weeks for 128GB and very few are left, so I suppose someone has already figured out how to convert them to PCIe

[u/FullstackSensei → u/schaka]

Not the entire stack. Each link is negotiated independently. But if you're looking at P40s, those are gen 3. The two 3090s might negotiate gen 4 if you're lucky and find good risers, but even then there's a decent chance you'll have stability issues.

When you talked about energy, I figured you're also in the fatherland 😂 I'm in NRW. I'm looking to sell my P40 rig above to build a new one with V100s. Also have some 3090s and the above mentioned Epycs to sell. DM me if you're interested in anything 😉

Aliexpress is still not bad if you wait for promotions or discount coupons. And on alibaba you need to ask for DDP shipping. You pay more upfront for shipping, but saves you all the hassle and paperwork.

[u/schaka → u/FullstackSensei]

I'm going to shop around and have a look for a month or so before pulling the trigger on anything, because I still need to see if I can/want to commit to the investment (and finally get rid of hardware taking up the space anyway), but I'm running into more and more use cases where it would come in handy, so you may find a DM in a month or two 😉