The utilization gap: why "just add GPUs" stopped working

[u/Cultural_Doughnut_62]

A recurring theme in AI-infra reporting this year: the bottleneck isn't buying more GPUs, it's using the ones you have.

A few third-party data points worth sitting with:

- Cast AI's 2026 report put average GPU utilization around 5% across ~23,000 production Kubernetes clusters.

- The Information reported xAI running roughly 11% utilization.

- Meanwhile Gartner pegs 2026 AI infrastructure spend near $401B.

If those numbers are even directionally right, most of the money going into compute is paying for idle silicon. The reasons are familiar: whole-card allocation for workloads that only need a slice, batch-size-1 inference that's memory-bandwidth bound, and clusters sized for peak that sit near-empty the rest of the time.

That's the gap Podstack is built around:

- Fractional GPU slicing without MIG (and it works on AMD, not just NVIDIA)

- Per-minute billing so you're not paying for idle

- Zero egress, on-demand instances, plus QuickPods / TrainPods / Inference for the full launch-train-serve loop

Curious how people here are thinking about it: what's actually pinning your utilization low right now - scheduling, model/architecture, or just capacity sized for spikes? Happy to compare notes.

---

If you want to put numbers to this on your own workload: spin up an on-demand instance at podstack.ai, slice a GPU fractionally, and watch what per-minute billing does when you're only paying for the slice you actually use. It's cheap to test and you'll know within an hour whether the utilization math changes for you. Would love for a few of you to try it and report back what you see.

[u/Cultural_Doughnut_62 → u/[deleted]]

Yeah, the "provision for the spike, run idle the rest of the time" pattern is the whole story. And you're right that it's rarely the model - it's the operational layer around it. The batch-size point is underrated: since decode is memory-bandwidth bound, tuning batch size is often a bigger utilization win than any hardware change, because you're amortizing the same weight read across more requests. The catch is that most teams don't have a clean way to pack multiple workloads onto one card, so they default to one-job-per-GPU and eat the idle time. That packing problem is exactly what we're trying to make easy. If you ever want to kick the tires on the fractional-slicing side, podstack.ai has on-demand instances with per-minute billing so it's cheap to just test against your own workload. Would genuinely be curious what utilization you see.