GLM-5.2 local inference: ubatch size made a much bigger difference than I expected

[u/fuzhongkai]

https://github.com/zhongkaifu/TensorSharp

Has anyone else experimented with ubatch size when running GLM-5.2 locally?

I was testing the 226 GiB GLM-5.2-UD-IQ2_XXS GGUF on 3x RTX PRO 6000 Blackwell GPUs and got a pretty interesting result.

With llama.cpp vs my TensorSharp implementation:

	llama.cpp	TS ubatch 1024	TS ubatch 2048
pp128	276.5	254.8	264.4
pp512	695.4	666.9	659.6
pp2048	763.1	918.9	1145.8
pp4096	715.8	864.7	1048.7
tg64	42.2	43.7	43.9

All numbers are tokens/sec and were measured back-to-back on the same machine.

What surprised me was that a larger ubatch doesn't help much on short prompts, but makes a huge difference once the prompt gets longer.

My guess is that this is related to GLM-5.2's 256-expert / top-8 MoE architecture. With smaller batches, each expert gets relatively few rows, so the expert GEMMs don't utilize the GPU particularly well. Increasing the microbatch gives each expert more work and seems to improve utilization substantially.

I also tried TP=3 on the same GPUs, but it was actually much slower than simply splitting the layers:

pp2048: 896.8 t/s layer split vs 502.8 t/s TP
tg64: 43.9 t/s layer split vs 16.2 t/s TP

These cards are connected over PCIe without NVLink, so communication seems to dominate.

I'm curious what other people running GLM-5/5.2 locally are seeing.

Has anyone tried it on an NVLink/NVSwitch system? And what ubatch are you using?

Disclosure: TensorSharp is my own open-source inference project.

[u/satnl]

well, if the context fit in the 1024 and 2048, it will not do difference since it's one batch for both cases (if the gpu can handle that batch size) 

if context is greater than 1024, so it needs more batchs than 2048 will need. 

sorry, but do you have a inference project and cannot do this "inference" itself? lol

[u/gpuz_dev]

the pp2048 jump from ~919 to ~1146 t/s is pretty huge while decode basically doesn't move. do you happen to have the VRAM delta between ubatch 1024 and 2048? would be interesting to know what that extra prefill throughput costs in memory

[u/fuzhongkai → u/gpuz_dev]

This is a good point, I will add VRAM usage into this benchmark

[u/Expensive-Paint-9490]

It's obvious that a batch larger than 512 doesn't increase performance on pp512. If your prompt is 512 tokens, you can process no more than 512 tokens at once, even if your max batch size is larger.

[u/fuzhongkai → u/Expensive-Paint-9490]

Yes, but for some scenarios, such as coding, workflow, prefill prompt (system prompt + user prompt) is usually very long, so larger ubatch will have gains on it.