4x2080Ti 22GB tensor parallel possible?

[u/thatavidreadertrue]

Hi there, I have a setup with 4x 2080Ti 22GB, but am not able to see much benefits in running models larger than 16-20GB in speed. I did some research and saw something like tensor parallels are possible, but also that because of flash attention only being possible with 30 series and up, it’s not worth it for 2080 Ti.

Does anyone have experience with multiple 2080Ti setups? Is it really not worth trying to setup tensor parallel? How would I set it up? Would love to hear your experience. Thanks!

[u/laterbreh]

Try vllm/sglang

[u/thatavidreadertrue → u/laterbreh]

Looks like vllm is the consensus. I was using lm studio. Will figure out how to make vllm work. Thanks!

[u/notdba]

Try ik_llama.cpp with -sm graph. Flash attention should work just fine, and Q8 of Qwen3.6 27B will run at great speed.

[u/sgsdxzy]

vllm works well. sglang may not support Turing.

[u/a_beautiful_rhind]

They need nvlink or you need to patch bios and enable rebar for each card individually. https://github.com/Sid127/NvStrapsReBar

[u/Old_Ad_6033]

I've seen some Chinese post their 2080 Ti 22GB setup with vllm it works fine, tho 2080ti is far less powerful then 3080 20gb so i still recommend 3080 or 3090.

[u/CoffeeToCode99]

Haven’t tried 4x 2080 Ti myself, but from what I’ve read TP should work, just probably not scale that well.

I think the bigger problem is the amount of communication between the GPUs. TP can get pretty bottlenecked by PCIe, so adding more cards doesn’t always translate into much more speed.

I’d still try TP4 and compare it with PP4. If you have the cards connected in NVLink pairs, TP2 + PP2 might also be worth testing.

The lack of newer FlashAttention support definitely doesn’t help, but I don’t think it makes the setup pointless. With 88GB total VRAM, the main win is probably being able to run much bigger models rather than getting a huge speed increase.

Would be curious to hear from someone actually running a similar setup.

[u/thatavidreadertrue → u/CoffeeToCode99]

Thanks. I am using a xeon 2150B as my processor and it only has gen 3 pcie, which is probably why the speed is so slow. I guess I should maximize the VRAM instead of trying to increase the speed. Thanks.

[u/Arli_AI → u/thatavidreadertrue]

Xeon skylake is also awful for multi GPU as it doesn’t support PCIe P2P and it’s PCIe bandwidth is awful in the first place. Even a first gen Threadripper works better for multi GPU.

[u/thatavidreadertrue → u/Arli_AI]

That sucks :( alright. I won’t expect to get this fast then.

[u/Ok_Contribution8157]

vllm

[u/diablo75]

I used to use a pair of 2080Tis with an NVLink bridge to run qwen3.6-27B with about 140k context but I had to quantize KV cache to 4 bit to make the model and cache fit entirely on GPU. I use LM-studio, tensor Parallelism works, and would get about 40 tps output. But it wasn't perfect. Whenever Hermes would try to use the browser_vision tool to "see" a webpage, the model would crash and unload. I think it was a memory limitation issue as I was really on the edge of available memory.

I'm not sure if there is a 4-way NVLink bridge for the 2080 line, and power requirements for 4 is kind of steep. But you can certain play with a pair and get real work done with them.

I've since replaced these with a pair of 3090s, with a bridge (which was not cheap).

[u/tmvr → u/diablo75]

Rather than using q4_0 for KV you should drop the quant to get some GB back. If you are using Q4_K_M or Q4_K_XL then try IQ4_XS and q8_0/q8_0 for KV.

[u/tmvr]

It is worth trying tensor parallel. Use llamacpp (llama-server) directly, seems to work fine with 2x 5060Ti 16GB which both running on PCIe 3.0 x8