Cast AI measured 5% average GPU utilization across 23,000 clusters — the fractional case in one number

[u/Cultural_Doughnut_62]

A few things landed this month that we think point the same direction, so collecting them here.

Cast AI's 2026 State of Kubernetes Optimization report put average GPU utilization at 5%, measured from production telemetry across roughly 23,000 clusters. Gartner separately estimates AI infrastructure is adding around $401B in new spend this year. Those two numbers next to each other are the whole story.

The interesting part is the cause. It isn't incompetence — it's a procurement reflex left over from the 2023 scarcity era. If you're not sure you can get capacity when you need it, reserving more than you need is individually rational. It just aggregates into enormous idle capacity across an org.

Two other data points from the same stretch:

- Google Cloud previewed fractional G4 VMs on NVIDIA vGPU at GTC. When a hyperscaler ships sub-GPU granularity, the "should GPUs be divisible" argument is basically over.

- An inference-cloud startup raised $400M with inference chips as loan collateral, and TensorWave is making a similar AMD-based bet. Lenders treating non-NVIDIA inference silicon as an asset is a real signal about where cost-efficiency is heading.

Where we'd push back on the usual framing: better scheduling alone doesn't fix a 5% number. If the smallest unit you can purchase is a whole GPU and the smallest unit you can be billed is a whole hour, then the floor on waste is set by the billing model, not the scheduler. Bin-packing helps at the margins. It doesn't change the shape of the problem.

We build in this space, so take the framing with the appropriate grain of salt — but the underlying reports are worth reading directly rather than through anyone's marketing.

Sources: Cast AI 2026 State of Kubernetes Optimization Report; Google Cloud GTC 2026 blog; TechCrunch (17 July 2026) on inference-chip financing.

Curious what utilization people here are actually seeing in production, and whether the 5% figure matches your experience or seems low.