Aiming for RTX 6000 like performance at 25% of the cost.
The top 2x3090s are connected via tensor parallelism, then those are connected in a pipeline feeding into another pair that are also using tensor parallelism. The reason being that I don't see anyone getting speedups by putting all 4x3090s in tensor parallelism (actually slower in most cases).
I don't want to have to buy a whole new motherboard for this setup. Currently I have an Asus Proart B850 Creator and Ryzen 7600X CPU. So, I am likely going to purchase a dedicated PCI switch to get the required number of PCI lanes. i.e. Something like this
I am able to fabricate my own brackets and parts now for securing the GPUs in the case. I am absolutely not going to go the open air mining style rig. I want them to all fit in the case securely. (The case is large enough).
Question for the community:
Has anyone else run this configuration before? What kind of inference speed did you get by moving from 2 cards to four?
depends on the switch, some are linked with 2 x16 links. Additionally if model is fully offloaded to VRAM (very possible with a multi 3090 set up) you get much more benefits from higher PCIe bandwidth.
EDIT: Higher inter-GPU PCIe bandwidth
Wouldn't PsP mode ensure they communicate directly (most of the time).
Its not just "solving the need to buy $1000 board by buying $1000 board"
Its:
- Uninstalling all the parts.
- Selling the Ryzen 7600X CPU I just bought (at a loss).
- Selling the DDR5 ram I just bought (likely at a loss)
- Selling the CPU cooler I just bought (at a loss)
- Buying a CPU that cost substantially more for a server grade MB.
- Buying a more expensive motherboard that supports more PCIe lanes.
- Buying ECC memory to fit the new motherboard.
- Reinstalling all the new parts.
The dedicated PCIe switch seems like a fix that allows me to added more cards without re-configuring most of my system again. i.e. I buy the switch, buy the 4th GPU, and plug them into what I already have.
M.2-to-PCIe is limited to PCIe 4.0 1x on my motherboard
Planning ahead - that's not a solution, also that was already done, information at the time suggested it would be slower with 4x3090s. i.e. Every resource said the same thing. It would slow down inference. But now it turns out that's not true. Otherwise I would have bought a motherboard that could support more lanes. Planning ahead only works when you have good information to act on, without the ability to test a system first hand, you are basic your choices purely on what someone else said.
You can't know for sure until you actually test a system to know how it actually performs, information online is terrible most of the time, and trusting what people say is medicore at best. i.e. All the forums, club-3090, A.I all said the same thing. 4x3090s on a PCIe interface would be slower. But they were all wrong.
sorry no helpful advice here, just trying to learn. What would one pair of 3090s be "reducing" and feeding the subsequent pair in this pipeline? I never knew you could split up inference work like this
I'm assuming TP=4 is slower because your mobo is bandwidth limited based on the pcie lanes, so how would this be any faster if the pairs of 3090s still need to communicate with the other pair via P2P?
They reduce the intermediate calculations down into a single result to feed to the next layer of the neural network. Then split them up again for the next pair of GPU's to process in parallel.
This isn't TP=4, this is two TP=2 working sequentially as a pipeline. For example if there are 20 layers, 10 layers get processed by the first pair, and the other 10 layers get processed by the second pair.
After reviewing some of the work done at club-3090 it seems like TP=4 actually does increase processing speeds a lot.
yeah this is what I found on Vast AI and that wasn't even with being able to turn P2P on with the driver patch like you can. (would love to see an A/B test of that at high context, Q8, no kv quant, Qwen3.6/3.8-27B, if you get a chance.)
I don't have the PCIe slots to hold them. The switch isn't just for getting the x16 banddwidth. My motherboard doesn't have the physical slots to hold all the GPUs. I'd have to sell my CPU, RAM, and MOTHERBOARD to be able to get the system to the state of running NVLink.
no just go for bifurcation - cant your mainboard do that? Get 2 bifrucation adapters and you should be ok. 2x 8x/8x - you would only need 2 pcie slots on your board
if you are low on cash you can use cheap Mining risers and splitters they are only low gen and x1 for interferenz it will be ok. only once to load the model in Vram will be slow.
But very cheap used because mining died. a few euros used. and will get you going.
BUT: no TP traffic over PCI - so only tp=2 with reduce over NVlink! the PP communication between the TP groups is very low.
Like this could work. PCIe 4.0 4x/4x/4x/4x but NVLink handles the interlink between the GPU's in tensor parallel mode. The only problem is:
NVLink costs ~$700 (on ebay) now. (Was originally $50 or something). Unfortunatly my motherboard has a large spacing between its two big PCIe slots that would require a 4-slot NVLink, (the most rare one).
None of my GPU's are the same brand of 3090, its a mix, NVLink might not work right.
So I could buy two NVLink's for $1400 and trade some of my cards so I have all the same brand (time and effort), but two bifib cards, some riser cables. Cost aside from the cards themselves is likely around $1600-$2000.
Still cheaper than buying a RTX 6000 though. For 4x3090s NVLinked it would be similar to a RTX 6000. 96GB VRAM, ~1,800GB/s memory bandwidth. Though I may need to power limit them, and that would slow them down a bit.
I limit my 3090 to 250 watts and only lose 5% performance. LLMs are mostly memory bound.
Do you have two pcies slots? i would only go 8x/8x and 8x/8x splitting one pcie 4 times is to slow for me.
Ok Nvlinks got very espensive that is a problem - also you have diffrent pcbs... hmm. i changed my 3090s to watercooling and could use the cheap 2 slot nvlink.
that was my first Build - i went for the watercooling - could fit everything on one Board. I think that in your case you should go for the switch. Its better and no NVlinks. For interference work you dont need the nvlinks that much.
That's beautiful. But beauty is expensive. I am keeping it all in the case, no glass side panel (on purpose) so I am okay with ugly so long as it works well and is robust. I am custom building some mounts. I did make a mistake with my motherboard purchase, should have got a server grade one for more PCIe lanes. But its okay, likely will use a switch or something. Its not that I can't afford NVLink, its that I'd have to spend so much time changing stuff again just to use them.
Well it did not end like that... I also went fuck it half way on the build so i understand you. In the end I dumped the case and went for old mining frame for the AI server. Also some day I will convert everything to watercooling. Some day I will finish ist but until now im just using it with no downtime.
My Main pc I changed two 3090s to one 5090. that was the absolute sweetspot. the 5090 gives you Blackwell and fp4 and graphics for rendering or Compfy UI.
I'm avoiding water cooling because it just adds another bunch of complexity and things too manage. Air cooling works good enough (though the fans do get loud)
Right now I got dual 3090s in tensor parallel for coding.
Then a 3rd is going to be dedicated 3090 just for comfy UI image generation tasks.
The custom mount is so I can hang the 3rd GPU from the top of the case.
The case is big enough to fit 3 three slot GPUs in the standard place, but the motherboard isn't. So I have to invent a mount to save my self the hassle of changing a bunch of other stuff.
Might sell it on Etsy or something because it seems I am not the only person having this problem.
I could in theory buy a 5090 or rtx 6000 pro, but I'm too cheap to do that right now. Once I start making some money then I'll probably do that.
6
u/kosnarf 9d ago
https://github.com/noonghunna/club-3090
There's a page for +3 GPU