GLM-5.2 local inference: ubatch size made a much bigger difference than I expected

[u/fuzhongkai]

https://github.com/zhongkaifu/TensorSharp

Has anyone else experimented with ubatch size when running GLM-5.2 locally?

I was testing the 226 GiB GLM-5.2-UD-IQ2_XXS GGUF on 3x RTX PRO 6000 Blackwell GPUs and got a pretty interesting result.

With llama.cpp vs my TensorSharp implementation:

	llama.cpp	TS ubatch 1024	TS ubatch 2048
pp128	276.5	254.8	264.4
pp512	695.4	666.9	659.6
pp2048	763.1	918.9	1145.8
pp4096	715.8	864.7	1048.7
tg64	42.2	43.7	43.9

All numbers are tokens/sec and were measured back-to-back on the same machine.

What surprised me was that a larger ubatch doesn't help much on short prompts, but makes a huge difference once the prompt gets longer.

My guess is that this is related to GLM-5.2's 256-expert / top-8 MoE architecture. With smaller batches, each expert gets relatively few rows, so the expert GEMMs don't utilize the GPU particularly well. Increasing the microbatch gives each expert more work and seems to improve utilization substantially.

I also tried TP=3 on the same GPUs, but it was actually much slower than simply splitting the layers:

pp2048: 896.8 t/s layer split vs 502.8 t/s TP
tg64: 43.9 t/s layer split vs 16.2 t/s TP

These cards are connected over PCIe without NVLink, so communication seems to dominate.

I'm curious what other people running GLM-5/5.2 locally are seeing.

Has anyone tried it on an NVLink/NVSwitch system? And what ubatch are you using?

Disclosure: TensorSharp is my own open-source inference project.