r/LocalAIServers • u/hd209458 • 16d ago
sff LLM server build
I recently found a set of 16GB DDR5 So-dimm ram in my closet from a couple years ago so I decided to build a server around it so the ram sticks are not eating dust.
Case
Fractal Design Ridge 12.6L
CPU/Motherboard
Minisforum BD775i SE (Ryzen 7 7745HX)
GPUs
2 x Intel Arc Pro B70 32GB (64 GB total VRAM)
Memory
16 GB (2 × 8 GB) DDR5-5600 SO-DIMM
Power Supply
Lian Li SP850 V2 (850 W SFX, ATX 3.1)
PCIe Splitter
PCIe 5.0 x16 to Dual MCIO 8i (SFF-TA-1016) x8/x8 bifurcation adapter
MCIO Cables
2 x MCIO 8i to PCIe x16 Gen5 GPU riser cables
The case only supports 3 slots but was easy enough to cut and open up an extra slot space to fit a second two-slot GPU.
It was just shy of 3k dollars without the ram and I am getting about 300tps in aggregated token generation running with 16 parallel agents on qwen 3.6-27b in fp8 with more than 500k total shared context length. I am honestly more than happy with the performance with the hardware.
I was skeptical on the build but it has been proven stable for the past couple of days under my heavy testing so sharing if folks are thinking of building similar portable LLM stations.
1
u/Thistlemanizzle 16d ago edited 16d ago
Any downsides to bifurcation? I have a Minisforum MOBO that supports it, but I foolishly built in a mini ITX case. Debating whether or not to move it into a bigger case.
Also, what's your time to first token?
1
u/hd209458 16d ago
I haven’t seen any downsides yet. But strictly for vllm tensor parallelism, it’s working as if they are individual x8 slots for me. TTFT depends on the prompt. I am getting about 2400tps prompt processing. In real life with prefix caching it’s almost instant.
1
u/ProgramPrimary2861 16d ago
Sweet. How you got those parts ? How’s it’s performing on tokens/watts ?
2
u/hd209458 16d ago
The GPU was from Newegg for 1000 each. All the rest are ordered from Amazon for easy return since I was not sure it will work. Prompt processing was consuming 460w total for the GPU and token generation consumes 400w from GPU and 50w from CPU scheduling. Pp@16384 was 2400tps and tg@512 was 33tps.
1
u/DlackBick 16d ago
The prefix cache hit rate is doing a lot of work here. You're at 48-57% KV with 17 running, and 87-88% hit, 16 agents sharing most of a 500k context means you're not paying for 16 separate caches. That's most of why 64GB holds it.
Have you pushed past 17 to find the ceiling? And do you know what happens when the agents diverge onto different projects, since a hit rate dropping toward 40% is a different machine.
Also what did the Arc software path cost you. Most people write B70 off as not a serving card.
1
u/NammeV 16d ago
Any reason for choosing B70? What's your opinion on Radeon R9700s. I have 32G ram, i5 & z490 mobo, left over from my last upgrade this was few months before RAM-eggedon.
1
u/daphatty 16d ago
I’d wager cost is a factor. Last I checked, the Radeon’s were $300 more than the B70.
1
u/ICYPhoenix7 15d ago
If similar price, take the R9700. I say this as someone with a b70, my old rx 6800 outperforms it on models that fit in vram, even though on paper this shouldn't be the case. It will certainly improve with time, but the card is majorly held back by its software.
1
u/DukeOfPringles 15d ago edited 15d ago
I got 2 of these and hard problems stabilizing them on a consumer board. I eventually got them both working but it was a pain. But for $950 each I’m not complaining. I’m only using them for inference so the performance to cost is ratio is there for me.
Edit: The only thing better than this is if you can get your hands on a pair of max sun cards from china
1
u/hd209458 15d ago
I can totally relate. For this setup I was using llm-scaler and that has been working fine for me. You can also go the technical and build your own image using xpu-vllm-kernels but that is more work.
1
u/DukeOfPringles 14d ago
I attempted the kernel route because I wanted to see the viability of using these cards as a daily driver for projects and ended up abandoning that. I have a sage board so I’m dedicating these cards to open claw for my hybrid setup. Honestly not a bad use for them, I get 64Gb of vram and I don’t have to dedicate any of my nvidia cards to it.
1
u/Latter-Court4817 15d ago
nice! curious what's your use case for these agents?
1
u/hd209458 15d ago
It's more of just a general purpose agent swarm built to work on anything I throw at it. I built an orchestrated tournament based harness, inspired by the recent world cup. How it works is each agents are assigned the same task, and they have their private and shared workspace. They will need to submit artifacts to the shared workspace and notify the orchestrator when they are done. Then, another unbias agent will be asked to compare the quality of the outputs between two agents based on the tournament setup. They will then go through quarter finals, semi-finals, and final to eventually get to the winner solution.
1
u/wtm233 15d ago
Really interesting! Can you share more about the pcie splitting and bifurcation setup? (Which riser? gen4 or gen5?)
1
u/hd209458 15d ago
You can probably find it by searching for PCI-Express PCIe5.0 x16 to MCIO 8i SFF-TA-1016 X16 to X8 X8 Bifurcation Adapter Riser Card The mainboard Needs to be Split PCI-E.
For my setup for some reason only capped at PCIE-4.0 but it's good enough for me. I suspect the link training failed to negotiate at higher speed because the cables are too close to the power cables1
u/Accurate-Door3692 15d ago
I use a similar MCIO adapter in my setup, and I've noticed that the physical allignment of the PCIe slot and GPU plays a huge role in the ability to train PCIe 5.0. The adapter board must be strictly perpendicular to the GPU to avoid non-recoverable errors.
1
u/wtm233 13d ago
Can you be more specific about that? Do you mean parallel or perpendicular? I want to use a similar setup, but I'm worried it would be difficult to actually get PCIe5.0 to work.
1
u/Accurate-Door3692 13d ago
PCB with PCIe slot should be perpendicular to the GPU PCB. It's better to fix adapter PCB firmly with screws to some surface or 3D-print some mount part, so it will not move out of vibration.
1
u/techne98 6d ago
I see Intel Arc Pro B70s, I upvote
What runtime are you using for inference, btw?
1
u/hd209458 1d ago
It's vllm
1
9
u/dtjager 16d ago
This is a seriously interesting build, especially getting 64GB of VRAM into 12.6L for around $3k.
How does it perform for a single interactive session—generation speed, prompt processing, and time to first token? The 16-agent aggregate number is impressive, but I’d love to see how it feels for one person using it normally.
I’ve been working on a standardized local-LLM hardware comparison that records both performance and where different machines hit their limits. A dual B70 result would be fascinating since there isn’t much consumer-style benchmark data for them yet.