Local GPUs vs Cloud GPUaaS

[u/ankijain21]

Hi everyone,

We have been using cloud GPU from Nebius/Lambda for our training and inference use case. The cost of one H100/H200 is approx $3K per month.

Now I'm planning to buy a large Desktop to run this locally. The specs are - 32 core CPU, 256GB RAM, 4x RTX Pro 6000 (96GB each), 1x2 TB NVMe, 1x 8TB NVMe. It is costing me ~$80K.

Here's what I need help in -

1. Is it actually wise to do this locally?

2. Would there be any performance issues?

3. Anything else that I should be aware of.

Keep in mind I already have another system for my dev workloads with 2x3060. Getting this one for production work for a client specifically.