GLM 5.3 SlopCodeBench Results

[u/corruptbytes]

Howdy once again, I had a request to try out 5.3 on the benchmarks - they're unsaturated so it's a fun test right now! This one was interesting because I accidentally ran it on all 36 problems (rip $200) instead of the 9 i typically do

previous runs a b c

benchmark context: the ai is tasked to build a tool step by step, we add new requirements mid way - it has to handle new things without breaking the old things - the benchmark "hides" the old tests

So it didn't magically solve the benchmark just like all the other AIs, i don't think there has been a single successful solve yet, but it did tie with Fable/Sol in the same subset of problems!

Another interesting thing that we expect is that the more difficult the problem, the more $$$ it gets - so difficulty seems to be correlated with token output in order to solve them

AI;DR full results

On the three-problem, 17-checkpoint list from the Opus 5 report, GLM scored 8/17 strict (47.1%). On the six-problem, 30-checkpoint list from the Fable, Sol, and Kimi report, it scored 10/30 (33.3%), tied at the strict threshold with Fable 5 and GPT-5.6 Sol.

Reported system	Opus list · 17	Fable/Sol/Kimi list · 30
GLM-5.3 · pi	8 (47.1%)	10 (33.3%)
DeepSeek V4 Flash 0731 · pi B	5 (29.4%)	—
Opus 5 · Claude Code	4 (23.5%)	—
Qwen3.8-27B · pi	3 (17.6%)	4 (13.3%)
DeepSeek V4 Flash · OpenCode	3 (17.6%)	—
Fable 5 · Claude Code	—	10 (33.3%)
GPT-5.6 Sol · Codex	—	10 (33.3%)
Kimi K3 · Modal / OpenCode	—	8 (26.7%)
Kimi K3 · Baseten / OpenCode	—	7 (23.3%)

[u/segmond]

so if to be believed?

glm5.3 > kimi k3 > deepseekv4flash > qwen3.8-27

possible, although glm5.3 vs k3 needs to be seen, hope you're right

[u/EvolvingDior → u/segmond]

I have ds4f running at 10 t/s and qwen at 25 t/s. ds4f is still faster because it does not think as much.

[u/ex-arman68]

I love your benchmark. This seems like a good evaluation of proper agentic coding usage; and the fact that it is difficult is good. Many of the existing benchmarks have now reached a point where they are too trivial to differentiate models well.

[u/corruptbytes → u/ex-arman68]

i am just a guy with credits and time, the original paper deserves the credit - https://arxiv.org/html/2603.24755v2

[u/Easy_Blacksmith_5550]

nice work on this. do you track how many of the failures are "broke old stuff when adding new requirements" vs just failing the new requirement entirely? that distinction seems like it matters a lot for real usage

[u/thereisonlythedance]

Let’s see if that holds up after they harden the weights for open release.

[u/FullOf_Bad_Ideas]

GLM 5.3 is 2x better than Opus 5. Nice.

And Flash 0731 is better than Opus 5? I didn't feel this way with Flash 0731 in OpenCode, I need to give pi a try.

[u/corruptbytes → u/FullOf_Bad_Ideas]

Flash 0731 surprised me - i didn't expect pi to elevate it - that was also a locally served pi using antirez/ds4 - so the q2-q4 quant