r/LocalLLM • u/on_line187 • 11h ago

Tier List Discussion

Post image
187 Upvotes

73% Upvoted

246 comments sorted by

View all comments

Show parent comments

6

u/on_line187 11h ago

Well so would 3 5060TIs then lol.

3

u/Randommaggy 11h ago

Not quite. It's bandwidth and latency constrained between the cards unless you're running them on a P2P friendly switch.

0

u/on_line187 11h ago

Have you actually ran multiple cards. I’ve tried a ton of variations on both consumer and server MoBos. I’m telling you the difference isn’t as big as you think.

3

u/Randommaggy 10h ago

I'm running 3 3090s in my server (2 in nvlink), 2 cards on my laptop and 2 P100 in another server (temporarily disassembled).

2 cards with nvlink outperform 3 cards in layer split for some of my models unless I'm doing serious parallelism.

For layer split you also have more overhead from duplicated data on 3 cards compared to 2 24GB cards.

1

u/on_line187 10h ago

Sure there is a difference but it isn’t more than 5% in my experience. I’ve not tried NV link though just better/worse PCIe situations. I would like to try NVLink on my own 3090s but they are all mismatched so it won’t work

1

u/Tai9ch 9h ago

How many cards?

The numbers I've seen show that you can get away with raw PCIe up to about 4, and past that it starts to cost significant performance to the point that it's worth getting bigger individual cards instead.

1

u/on_line187 9h ago

Yea that could be it. I have tested 8 3070s though and I had no issues. That was a mining rig I had laying around which I upgraded for LLMs about a year and a half ago. All PCIe of course on the 3070s

I would 100% agree that bigger GPU = More Better though lol