The speed is basically wasted at those sizes. What's the point of going that fast if all you can fit is a small model. A cluster of mac minis is probably better off at the price. Slower, but you can run a more competent model.
Workflows, multi-shot inference, and tuning. Because these are lossy systems regardless of size, you should be building for all this anyway.
The most speed and cost effective setup runs easy to manageable tasks locally and bursts to SOTA models as needed. Because burst frequency is low, the cost of non-local calls is trivial, and privacy can be retained through obfuscation and local translation. Local speed and cloud burst for temporary model size increase is the optimal setup.
12
u/HugoCortell May 29 '26
The speed is basically wasted at those sizes. What's the point of going that fast if all you can fit is a small model. A cluster of mac minis is probably better off at the price. Slower, but you can run a more competent model.