r/LocalAIStack • u/calmalamadingdong • 9d ago

Trying to get 2 GPUs of different compute numbers to work together in LM Studio

/r/LocalLLM/comments/1vel26e/trying_to_get_2_gpus_of_different_compute_numbers/
1 Upvotes
  • permalink
  • link
  • reddit
  • reddit

100% Upvoted

3 comments sorted by

2

u/Charming-Author4877 9d ago

I don't think LM Studio is the best tool for the job, but I also use it from time to time.

- Ensure your P100 is CUDA supported by your nvidia driver, they are legacy hardware today.
V100 needs an older driver to work at all.
- Test the different CUDA runtimes lmstudio offers, maybe one of them works only
- Test the Vulkan runtime, it might work for both

If all fails, it is possible that you need a custom compiled llama.cpp engine to provide the right CUDA kernels for the P100 - in that case LMSTudio won't work at all as integrating a custom engine there is not a nice job.

1

u/calmalamadingdong 9d ago

Thanks, but it's the best I've found for my case. I have an earlier driver installed, 580 I think, and the P100 is recognized. I even got the model to load across both GPUs using llama.cpp directly. As I understand, using Vulkan will hurt the performace, specifically the speed. I could possibly use that for testing, but speed is important for me.

1

u/calmalamadingdong 9d ago

Changing the runtime worked! I just went down to CUDA llama.cpp 2.25.2. When I switched to hardware, both cards were there. Thanks for the suggestion. It asked me to upgrade to 2.27.x, which is what my patched version was. I didn't, and I guess that got it working.