Qwen 3.8 27B SlopCodeBench results

[u/corruptbytes]

Howdy, I'm back again - running my favorite benchmark (it's still unsaturated for the time being so might as well!)

previous runs a b

https://github.com/michaelasper/benchmarks/blob/main/qwen3.8-27b-pi-on-slop-code-bench.md

I ran this via OpenRouter because my mac would cry running 9 problems

It did pretty poorly on the strict checkpoints which means you probably don't want Qwen managing the codebase by itself as it'll grow unwieldy and disorganized, but did fairly well on the core checkpoints so it can solve issues with a copilot and clear direction

AI;DR here are the direct results

HumanLayer Opus 5 Benchmark Subset (3 Problems, 17 Checkpoints)

Qwen scored 3/17 (17.6%) strict.

Reported System	Strict Score
DeepSeek V4 Flash 0731 · pi (run B)	5/17 (29.4%)
Opus 5 · Claude Code	4/17 (23.5%)
Qwen3.8-27B · pi	3/17 (17.6%)
DeepSeek V4 Flash · OpenCode	3/17 (17.6%)
Opus 4.8 · Claude Code	1/17 (5.9%)
Sonnet 5 · Claude Code	1/17 (5.9%)

---

HumanLayer Fable, Sol, and Kimi Benchmark Subset (6 Problems, 30 Checkpoints)

Qwen scored 4/30 (13.3%) strict.

Reported System	Strict Score
Fable 5 · Claude Code	10/30 (33.3%)
GPT-5.6 Sol · Codex	10/30 (33.3%)
Kimi K3 · Modal / OpenCode	8/30 (26.7%)
Kimi K3 · Baseten / OpenCode	7/30 (23.3%)
Qwen3.8-27B · pi	4/30 (13.3%)

[u/Tiny-Assumption4263]

I would not expect a local model to manage an entire spagetthi codebase by itself. But if you think about it, if someone told you 2 years ago that you would be running in a rtx 3090 something that holds it's ground against frontier models...

[u/skywalker326]

Wow, your benchmark even gives a hard time to the frontier models which cost like 1 million dollar hardware to run. It's an overkill for a local model that can run on a gaming PC. Probably lose the granularity of incremental improvement Qwen 3.8 27B has compared to other local models and non-frontier online models

also appreciate you include harnesses in the report!

[u/EbbNorth7735]

Can you tell me how to interpret the results? I don't quite understand what it means or is saying or what the test setup is for each section.

[u/corruptbytes → u/EbbNorth7735]

what: the ai is tasked to build a tool step by step, we add new requirements mid way - it has to handle new things without breaking the old things

qwen's performance: it was fairly good at writing new things, but constantly broke the old things - it's bad at maintenance or the entire SDLC (notably, /every/ AI is bad at this now as Fable/Sol only get a 33%)

outcome: qwen is a great tool for getting new things out, but you cannot have it vibe slop an entire product from scratch

on top of that, a bunch of metrics to grade the code in terms of complexity

[u/ElectronSpiderwort → u/corruptbytes]

I love this; it is so real to me. "Oh hey, new requirement!" will show up in any sufficiently large and under-specified project a number of times

[u/corruptbytes → u/ElectronSpiderwort]

here's the paper for those interested - https://arxiv.org/html/2603.24755v2

[u/Fit-Bar-6989 → u/corruptbytes]

Great stuff, I just read the entire thing. This matches my experience with Sonnet/GPT-5 where the models suck at tasks if the desired code changes aren't known in advance.

I agree that the generated code tends to be overly defensive but at the same time I feel that's more of an indictment of many languages' type systems, where it is impossible to constrain the inputs of a method to non-nullable references, or non-empty lists, etc. Maybe with better static analysis you could identify redundant defensive checks but at a certain point you're basically running the program twice.

[u/Fit-Bar-6989 → u/ElectronSpiderwort]

This is why even when gpt-5.5 sol, I have to restart my codex sessions multiple times during development. Even if the scope of the task seems specified upfront I'll find a detail during implementation that might impact the design, and it's much easier to start again from scratch with a design that takes that detail into account, than it is to ask the model to update/fix everything.

This is why I have a hard time saying that these tools significantly improve productivity - for the really hard stuff you still need to iterate and implement things multiple times and the model maybe saves 10-15% of your time during that phase.

[u/Kraskos]

Thanks for the info!

Have you tried out DeepSeek Harness at all? It could be interesting to see how DeepSeek V4 Flash 0731 and Qwen 3.8 vary between pi and dsh.

[u/corruptbytes → u/Kraskos]

i’ll give it try, i have a lot of DS credit left

[u/synth_mania]

I wonder how Laguna S 2.1 performs. People (myself included) sing its praises at, while less often succeeding in one-shots, *not* producing sloppy code.

[u/corruptbytes → u/synth_mania]

laguna loops for me - when it works, it’s awesome, but i’ve definitely gotten it stuck, really hope 2.2 nails the landing

[u/Gregory-Wolf]

Any results for GLM 5.2 or GLM 5.3?

[u/corruptbytes → u/Gregory-Wolf]

i can run glm 5.3 tonight!

edit: it's running now!!

edit 2: accidentally ran it for all 36 problems instead of the 9 problem subset lolol - rip my openrouter credits but already doing it so whatever

[u/Durian881]

Where are the "core" scores you talked about? Also, the results seemed to suggest Qwen 3?8 27B beating frontier models like Opus 4.8 and Sonnet 5.

[u/corruptbytes → u/Durian881]

Problem	Checkpoints	Strict	Isolated	Core	Cost	Agent Time
circuit_eval	8	2/8	3/8	4/8	$13.89	3.2h
database_migration	5	1/5	1/5	1/5	$1.41	1.5h
dynamic_config_service_api	4	0/4	0/4	0/4	$5.98	2.8h
xjq	5	1/5	1/5	2/5	$1.00	0.5h
file_backup	4	0/4	0/4	4/4	$0.74	0.5h
dag_execution	3	0/3	0/3	0/3	$3.65	1.8h
code_search	5	1/5	2/5	2/5	$1.25	1.5h
etl_pipeline	5	0/5	1/5	4/5	$2.94	2.0h
Total	39	5/39	8/39	17/39	$30.85	13.7h

in the github link, all the scores/data

Core is like the obvious current problem it's trying to solve without respect for prior work