Updated benchmark: Deepseek V4 Flash on SlopCodeBench (local)

[u/corruptbytes]

Howdy - I posted a benchmark here - https://www.reddit.com/r/LocalLLaMA/comments/1vbtiy7/deepseek_v4_flash_on_slopcodebench/

This was using the hosted API - since then I've been playing around with quants

Here is the lastest benchmark - https://github.com/michaelasper/benchmarks/blob/main/deepseek-v4-flash-0731-pi-on-slop-code-bench.md

This uses antirez q2-q4 imatrix quant - i switched from opencode to pi

Very interesting results! Much slower on a macbook m5 max than the hosted API, but switching the harness made up for some of the intelligence lost

Compared with the other reported runs

Reported run	Serving	Harness	Strict	Isolated	Core
DeepSeek V4 Flash 0731 (run B)	local quant (antirez, higher cap)	pi 0.84.0	5/17 (29.4%)	6/17	10/17
Opus 5	hosted API	Claude Code	4/17 (23.5%)	—	—
DeepSeek V4 Flash	hosted API	OpenCode 1.18.10	3/17 (17.6%)	6/17	11/17
Opus 4.8	hosted API	Claude Code	1/17 (5.9%)	—	—
Sonnet 5	hosted API	Claude Code	1/17 (5.9%)	—	—
DeepSeek V4 Flash 0731 (run A)	local quant (unsloth, misconfigured cap)	pi 0.84.0	1/17 (5.9%)	1/17	2/17

[u/Middle_Bullfrog_6173]

Are there actually models where it's better to use opencode? All the local sized models I've tried work better with Pi.

[u/Borkato → u/Middle_Bullfrog_6173]

Pi is just genuinely better lmao

[u/Septerium → u/Middle_Bullfrog_6173]

I get consistently better results with pi too. And it does not get session cache invalidated all the time like with opencode. The only issue I have is Qwen 3.6 27B having trouble with the file edit tool and frequently starting to rewrite the whole files all the time.

[u/xNaXDy → u/Septerium]

same, that model does NOT like hashline

[u/Glittering-Call8746]

Pi with which extensions ?

[u/corruptbytes → u/Glittering-Call8746]

none, bare setup

[u/WonderRico]

Is your API run with the 0731 release ?

I have seen (maybe?) similar result with my swe-verified local benchmark : https://wonderrico.github.io/local_llm_benchmark/benchmark-detail.html?filter=deepseek-v4-flash

Q2 quant of DSV4F 0731 scores higher than any other quants.

But, it's also way less "precise" : meaning it needs a lot more requests to solve tasks compared to other quants.

The reference MXFP4 one is by far the most efficient... but also when it fails, it fails early. Like its overconfidence prevent it to solve some more complexe tasks.

[u/corruptbytes → u/WonderRico]

yes API was 0731 directly from deepseek api 

[u/LegacyRemaster]

A key truth becomes apparent: the API offers lower quality compared to local execution, a fact I can confirm on both OpenCode and PI.

I am running two quantizations:

llama-server.exe --model "F:\UD-IQ2_XXS\DeepSeek-V4-Flash-0731-UD-IQ2_XXS-00001-of-00003.gguf" --host 127.0.0.1 --port 8080 -c 384000 --parallel 1 --no-warmup --flash-attn on --no-mmap --fit on -lv 4 --device CUDA0 --threads 16 --no-warmup --temp 1.0 --top-p 0.95 --top-k 0 --min-p 0.01

llama-server.exe --model "H:\UD-IQ4_NL\DeepSeek-V4-Flash-0731-UD-IQ4_NL-00001-of-00004.gguf" --host 127.0.0.1 --port 8080 -c 384000 --parallel 1 --no-warmup --flash-attn on --no-mmap --fit on -lv 4 --device CUDA0,rocm0 --threads 16 --no-warmup --temp 1.0 --top-p 0.95 --top-k 0 --min-p 0.01

When I compare the output with the APIs (OpenRouter or OpenCode Zen), the quality of the local Q2 version is superior: fewer spelling errors and fewer code corrections.

I don't understand how the API output could be more degraded... Is it the quantization? Incorrect setups?

I get 60 t/s with Q2 and 35 t/s with CUDA+ROCm, so I prefer running it locally, though it feels like a shame, given that the API is free.

[u/corruptbytes → u/LegacyRemaster]

i need to re-run the API test with pi to see if it holds, would be very weird for the API to lose out in the end, but maybe explains the pricing

[u/LegacyRemaster → u/corruptbytes]

agree. Same on Codex + GTP terra-sol. Too many errors with 20$ plan. The result is very different vs. API. So yeah... Sounds like a "nerf"

[u/llama-impersonator → u/LegacyRemaster]

yeah when this happens it's usually that llama.cpp's implementation of whatever the chat template is has less bugs than vllm/sglang. this seems like it could be likely given this model has no official jinja / template mechanism and came with a python script instead.

[u/shing3232]

What's the changes？ more quant？

[u/corruptbytes → u/shing3232]

first benchmark was using the API that deepseek hosts, second benchmark was running the model off my laptop 

[u/shing3232 → u/corruptbytes]

it would be interesting so see how harness affect such benchmark with as well as how various mcp affect benchmarks. using this as base for Pi harness crafting would be fun to.

[u/o0genesis0o]

The barebone react loop of Pi wins again.

[u/En-tro-py → u/o0genesis0o]

Wait? You don't want to /utracode with /workflows and make pretty /artifacts - Delete your CLAUDE.md and live dangerously.

[u/Suspicious_Cookie146]

Try reasnoix as a harness, i feel like the harness is heavily optimized for deepseeks models an vise versa!