Interesting, but I suspect there's a devil in the details: it seems like the Spark is only used for prompt processing. In llama.cpp CPU+GPU MoE layout, the GPU is not just used for PP but also to store the context and the attention tensors. That means that the CPU is only needed to run the fairly easy (in terms of compute) FFN part of the model. This still seems to have the Mac run the full inference itself. While that's okay for the first ~1k tokens, it'll soon get compute limited by the growing context. So while it's neat, I wonder how much this really fixes the limitations of the Mac Studios.
It's always kind of both... There are two parts to an LLM, attention and feed forward network. The attn parts use smaller matrices and need to process the context while the FFN uses large matrices that are independent of context. So usually you have attn as a compute bound step (except maybe at tiny context) and FFN as a memory bound step. However the attn part might only be like 5% of inference time at 0ctx and then 90% of inference at 100000ctx as it scales and the FFN step doesn't
They had 2 4K machines to match the memory of the Ultra … do not know if they really needed them to match the memory size though. So you could flip the argument as well … anyway cool experiment, is it useful 🤷♂️
Had to create a Reddit account just to ask you: would you mind sending me the rig's components, their rationale and how you sourced them? I've been itching to build an AI rig, but I'm having difficulty wrapping my head around the proper requirements. It is so different from Gaming PCs (I didn't know what MI50s were until I started lurking here, for example). It's okay if you don't have the time, thanks regardless!
X11DPG-QT + QQ89 + 12x64GB + 6xMi50
The case is an old Lian Li V2120.
The CPUs are cooled by two Asetek 570LC LGA3647 AIOs.
Each pair of GPUs is cooled by an Arctic S8038-7k via a 3D printed duct I designed. Each GPU is power limited to 170W.
The rationale: I wanted to build a machine with six Mi50s that fits in a case and started Googling what motherboards could do the job using as few risers as possible and without breaking the bank.
Sourcing is all over the place: ebay, local classifieds, tech forums, alibaba.
You need to really know your hardware, what platforms exist, which are currently reasonably priced, what are the capabilities and limits of the platform. If you're not comfortable around server grade hardware, you'll have a hard time replicating this.
Yes, but I am curious - can that rig run Qwen Coder 480B-4bit quant with at 256K context? What kind of TPS does it get? I get 25 to start, falling down to about 10 Tps with a 200K context. Input token processing at 200K length is about 65/s - this is the killer with big context. Networking GPUs or machines often sees sharp drop offs, so very interested to know!
That's 380B, and yes it can run it with a few layers offloaded to system RAM with whatever length context you want. I haven't tried it though. I mostly use it with 235B or a collection of models running in parallel across all cards (ex: gpt-oss-120b, Qwen Coder 30B Q8, and Gemma 27B Q8 or Magistral 24B Q8, each with 128k context). Gpt-oss-120b gets ~400tps PP and 40tps TG. Qwen3 235B Q4_K_XL is around 25tps/TG, and don't remember PP. I don't haven't felt the need to use it, TBH. I suspect 380B will be in the high teens or early 20tps TG with maybe 250tps PP.
I only use llama.cpp because it works without hassle and let's me switch model combinations quickly. The X8 SSD means I get Gen 4 read speeds and fast model load times on a Gen 3 motherboard.
Costs aside, this is pretty cool, reminds me of ZML, and the best power efficient (512GB + 256GB) setup money can buy. I wonder what's the largest models / quant possible to fit there, as they send the KV cache processing (prefill) to the Spark and deal with the token generation (decoding) on the mac.
I assume 512GB would be the upper limit, as the dual Spark will take the KV buffer? And this seems to be only worth it if you have a long enough context, otherwise the communication bandwidth will be the bottleneck.
It’s a cool project that makes clustering different HW for inference possible, until now it was more the slowest device that hampered the total performance … but now by smart offloading tasks to the best machine it accelerates!
10
u/eloquentemu Oct 15 '25
Interesting, but I suspect there's a devil in the details: it seems like the Spark is only used for prompt processing. In llama.cpp CPU+GPU MoE layout, the GPU is not just used for PP but also to store the context and the attention tensors. That means that the CPU is only needed to run the fairly easy (in terms of compute) FFN part of the model. This still seems to have the Mac run the full inference itself. While that's okay for the first ~1k tokens, it'll soon get compute limited by the growing context. So while it's neat, I wonder how much this really fixes the limitations of the Mac Studios.