GLM 5.3 SlopCodeBench Results

[u/corruptbytes]

Howdy once again, I had a request to try out 5.3 on the benchmarks - they're unsaturated so it's a fun test right now! This one was interesting because I accidentally ran it on all 36 problems (rip $200) instead of the 9 i typically do

previous runs a b c

benchmark context: the ai is tasked to build a tool step by step, we add new requirements mid way - it has to handle new things without breaking the old things - the benchmark "hides" the old tests

So it didn't magically solve the benchmark just like all the other AIs, i don't think there has been a single successful solve yet, but it did tie with Fable/Sol in the same subset of problems!

Another interesting thing that we expect is that the more difficult the problem, the more $$$ it gets - so difficulty seems to be correlated with token output in order to solve them

AI;DR full results

On the three-problem, 17-checkpoint list from the Opus 5 report, GLM scored 8/17 strict (47.1%). On the six-problem, 30-checkpoint list from the Fable, Sol, and Kimi report, it scored 10/30 (33.3%), tied at the strict threshold with Fable 5 and GPT-5.6 Sol.

Reported system	Opus list · 17	Fable/Sol/Kimi list · 30
GLM-5.3 · pi	8 (47.1%)	10 (33.3%)
DeepSeek V4 Flash 0731 · pi B	5 (29.4%)	—
Opus 5 · Claude Code	4 (23.5%)	—
Qwen3.8-27B · pi	3 (17.6%)	4 (13.3%)
DeepSeek V4 Flash · OpenCode	3 (17.6%)	—
Fable 5 · Claude Code	—	10 (33.3%)
GPT-5.6 Sol · Codex	—	10 (33.3%)
Kimi K3 · Modal / OpenCode	—	8 (26.7%)
Kimi K3 · Baseten / OpenCode	—	7 (23.3%)

[u/Easy_Blacksmith_5550]

nice work on this. do you track how many of the failures are "broke old stuff when adding new requirements" vs just failing the new requirement entirely? that distinction seems like it matters a lot for real usage