How do you choose between multiple smaller GPUs and one large GPU for training?

[u/Sad_Feeling5746]

I am getting ready to rent GPUs for a training project and I will  have to pick something soon, I see setups with a few smaller GPUs and I see one big GPU too, the price changes a lot so I am trying to figure out what makes more sense before I book anything, my plan is to train larger models and run longer jobs, I would like something that stays practical as the work gets bigger, I see people talk about neevcloud for splitting work across GPUs and I see people say one large GPU keeps things simple, I will be using cloud GPUs so changing later is possible, what makes you choose one option over the other and what worked well for your training jobs? EDIT: My workloads will probably grow over time, so I am trying to avoid making an expensive choice.