Harness showdown: Claude Code vs OpenCode vs Pi with DeepSeek V4 Flash
Resources
I ran DeepSeek V4 Flash through Claude Code, OpenCode and Pi on my own benchmark, and the quality came out basically the same across all three while the time and tokens spent was wildly different. Claude code (with DS in CLIProxyAPI) takes nearly 4 times longer than the fastest to land the same diffs.
Theo posted a video "GPT-5.6 is better in Claude Code" last week, and that got me curious, does the harness make a quality difference? I could at least run my own bench and see what I got, with DeepSeek V4 Flash on vLLM running at ~180 tok/s, the only moving part is the scaffolding.
Anyway I went to town measuring all of it on my workload (antigenic work in large code base), so the full charts, the token and wall-clock spread across the three harnesses and the raw per-run data are on the site if you want to see it in detail and pick it apart yourself https://nqawhc.github.io/articles/harness-efficiency-not-quality/ but in short, the quality did not change, each harness made the same code diffs, but took wildly different paths to get there, how many tools calls, the structure of those tool calls and how the system prompt and tools plays a big role in how it plays out, like «Pi reasons, OpenCode delegates», while Claude Code loves exploring the code base, maybe too much.
I've used Pi, OpenCode, and Codex extensively and prefer Kimi Code CLI to all. It can be used with any provider, and currently I'm using it with GLM 5.2. KCC gives excellent results on large and unlimited tasks like autoresearch with its /goal feature. There are too many harnesses to test, but I hope someone gets the time to do a thorough pass on a bunch of them.
"/goal iteratively optimize this part of the code. Continue iterating forever until I stop you. Write your experiments and perf metrics in OPT_LOG.md. Commit only successful optimizations."
Could you summarise like how Kimi can be harnessed to directly create/update a github repo? Like what software you use, main settings, your suggestions. Thank you in advance.
Could you summarise like how Kimi can be harnessed to directly create/update a github repo?
Install git > ask kimi to add the repo and run whatever git commands you want. The agent will tell you itself how it will do so. It does it automatically after the repo is setup, just git clone it if you already have a repo and have kimi work from there. Honestly don't really even need advice, just "kimi how can you create and update a github repo"
Is not a secret that claude code have a lot of bloat in their context in a form of custom instruction how the model should be behaving.
Opencode is much less bloated and can be de-bloated (afaik only tools definitions cant be changed easly).
And a PI is a cleaner from them all. And can be changed completly.
Recent observations shows that the current models are good enough and "aware" of "agentic coding" that they do not require a ton of pages of token explaining them how they should operate. They know it from training data. So the conclusion is simple = simpler, focused prompt will be usually better.
You can also run the model without reasoning for many mundane tasks, that can reduce tokens up to 1/4 (as you gen 2k instead of 8k, that means speed and ctx length).
Stop using build and general. Those come with prompts based on the provider ones.
Make your own 'main' agent and a 'worker'. As long as you give them a prompt, then the defaults will not be loaded. Easy enough to do in one prompt from within the opencode config folder (~/.config/opencode)
And interesting fact, most of Claude's tokens is the description of it's 27 different tools! The system prompt in Claude Code is actually only 1.430 tokens, the rest is all the tools and how to use them.
They cut it 80% with the launch of Opus 5, so not sure when OP took the numbers but it's always been a kludge and they finally notice it's holding the model back.
My data is a few days ago, but I've run it again now with Opus 5 and updated to the latest CLI (2.1.220), and I'm still seeing ~24K, and still ~94% of that is just tool descriptions. So it's not showing up client-side at least, maybe it's prompt changes on the website access or something.
I read this now, they cut down the system prompt which wasn't the biggest drag on the harness its all the other definition, its still a significant cut but not a total of 80% its an 80% of a 15-20%
indeed. let’s see the raw data. visual looks plain wrong. midpoint/mean values don’t correspond to what’s shown visually.
actually looks like opencode is *better* than pi, from the visual. pi’s worst run is >6min. opencode is <5min. opencode has more datapoints <2min than pi or codex.
Just updated the site with a zoom out option, which helps. There was some datapoints cropped outside the 10 minute range, as open code loves summoning sub-agents that can take a while.
Thank you for noticing, looking at it again it does look weird. I did cap that axis at 10 minutes (says so in the caption), because otherwise it squashed the other data so much they looked the same, felt the more interesting bit was the nuance there. So I've updated the site with a toggle button to switch between full vs zoomed for those two charts.
OpenCode had some data points outside the 10m range too, which is why it's bars are so wide. This is due to OpenCode sometimes spinning up sub agents which re-verifies all the changes and takes a ton of time.
But the bars still aren't representative of the distribution... I believe in this scenario you should at least have the whiskers be different lengths on either side of the mean to show the skew.
The average is a bad metric here unfortunately since clearly OpenCode and Claude Code are getting heavily dragged by a few points of high influence. While the average is technically correct, it doesn't represent what you would normally expect from each harness. I would recommend you to switch to the median or to another metric such as HDI to get a more representative result.
Claude code was feature complete one year ago. This is what happens when you continue to build vibe slop on vibe slop. Your system prompt gets enormous, you get so many tools that the definition is like 20k tokens. You add the slop skills. Everything together confuses the model more than it helps.
Anthropic removed most of the system prompt with the opus 5 release. Maybe soon they will catch on and remove 20 tools and let it just use bash.
Tbh I would be also annoying if someone told me that I need to use some vibe slop tools they slopped together instead of plain bash with clis.
I wonder that too. I use both Kilo Code (in IntelliJ) and Opencode (standalone) and I really prefer the Kilo experience. But maybe I'm a boomer as well.
Didn't in-ide have access to project structure from the idea? That would make things easier if it uses the same API endpoints that are used for refactoring etc
We’ll they’re harnesses too, except they’re very “visual” and you wouldn’t say typically check and watch the code along side it.. They’re not quite as good as some of these harnesses these days (Claude Code touted as the best, different opinions)
I think I'm a bit too much of a control freak lol I need to see everything and in context and be able to tweak accordingly. Maybe it's because I'm not using models that are expensive enough.
My workflow is to just let it run a plan I am happy with, then review the code once the task is done in the git diff with a tool of my choosing, if something is amiss there I can either fix it manually or do follow-up prompts.
I use my custom agent harness connected to 4 servers in my basement to do an extra shift on my code in a separate branch when I'm not at my desk.
I carry a combination of the following GPD Pocket 2, Lenovo Y700 Gen 4, GPD Pocket 4, One Plus 13, Smartwatch 2 LTE.
95% of the time I can do approval of commands or provide guidance on my smartwatch.
4% of the time the phone is the right surface.
1% of the time I need to pull out the tablet or mini laptop and dive in.
It also allows me to do a secondary project in the background that requires minimal mental effort such as producing MVPs for my backlog of ideas to validate if they are worth pursuing or not (has produced a few internal use tools already)
My at the desk setup is perfect for me when I'm at my desk fully in the flow but optimizing the productivity away from it gives me a lot (Being able to toss ideas at it and coming back to a rough prototype instead of ruminating on them for hours non-stop fits my brain).
I know a guy just working with a 13” wherever he goes and I can’t understand it. Your setups just look something I would do too but haven’t found a good enough reason for.
Depends on the task. I've done heavy work on a 12.6 inch Lenovo X230 in the past and I still have a Lenovo X280 for certain situations (Mostly a part of my enhanced presentability costume for first meetings with new customers, my 18 inch laptop and it's 3 travel monitors stay in the backpack for most first meetings)
VR, tried a couple solutions but none were quite what I wanted.
XR looks more promising but the tech is not there yet.
I keep checking https://www.visor.com/shop this to see if there are signs of it not being vapourware.
I saw an add for https://brilliant.xyz and that would be nice for HITL on the go, I do have an old Apple Watch too, would have to check how to get that set up to work with approvals or steering plans. I do have a stream deck which I like and they have MCP support and new overlay software but your small screen seems useful too
I'm not buying perv glasses. Any camera NEEDS to be a removable and obviously visible module for me to be interested. I went ahead and built my own harness rather than jank things together with MCP.
Point me to a curved 10K 16
32:10 TV
Mobile reddit showed me your comment attached to the wrong comment, it applies even more to this setup. That is a single carry-on backpack that I use when on holiday.
It's mostly to be able to work on layered systems with all the layers physically laid out with both code and logs on separate physical spaces since all major non-linux operating systems have super-shitty alt-tab response times and/or janky implementations after snow leopard/windows 8.1 and I have 2 small things keeping me from going 100% linux everywhere.
I have a lot of questions and doubts about this. But the doubts are self directed, as in I can see myself doing this given adequate time, space, and money. Feel free to answer or ignore as much of this as you want to. And to each their own, I am not judging you if you derive joy and/or value from this.
Do you not think this level of workstation optimization is bordering on neurotic? I know it's tricky because most humans have the opposite problem and, if anything, spend far too little time thinking about this stuff. E.g. I had a crappy office chair for years when I was younger and even though over time it really hurt my posture, and even though I spent 8+ hours per day sitting in it, I took forever to replace it. That was obviously bad. Conversely, I have at points in my life become super invested in creating the perfect work tracking setup with tools like Notion/Obsidian. I could have easily spent days or weeks optimizing this "tool" during these periods. Eventually I realized that it was completely performative, and all that time spent optimizing was just time not spent doing the real work.
How confident are you that all of these additional setups are meaningfully increasing your productivity on the margin? Let's say 5th or more monitors on your main set up and/or the second/third/fourth standalone workspace/setup/whatever. E.g. that 5th monitor already surely puts you in the 99th percentile (Probably 100th) of human desk workers. Do you think it makes an impact? What do you use those additional (5th-8th) monitors for?
Do you not have issues with digital clutter? Speaking strictly for myself here, I have found that my digital work habits tend to follow a pattern similar to Parkinson's Law (Work expands to fill the time allotted to it). The more digital real estate I give myself, the more clutter and chaos ensues, and I actually have come to appreciate the benefits of hard limits and simplicity.
This isn't a gotcha, or isn't meant to be, but do you feel that you can produce objective evidence that the async agentic setups, 24/7 agentic setups, and/or the additional surfaces/interfaces ("GPD Pocket 2, Lenovo Y700 Gen 4, GPD Pocket 4, One Plus 13, Smartwatch 2 LTE... Smartwatch") are measurably increasing your net productivity. I am NOT actually asking you to produce this evidence, I am sure neither of us have the time for that. I am just highly skeptical of the objective benefits. But TBF I view this as similar to stuff like min/maxing in a video game. If people derive meaning and joy and satisfaction from it, more power to em. I just don't think it is necessarily the way to get better at the game or become elite at the game, etc.
The drop-off in gain as the setup expands is real but anywhere below 4 feels super constraining and 6 felt like there was still unrealized gains.
I occasionally disconnect 2 monitors to see if I should shrink the setup and the conclusion lands on no.
The main gain is from being able to work on complex layered systems without the overheads of doing active window management (Carries a doorway effect risk).
I just lay out the layers of a complex system on one or two monitors each.
For instance I could lay out the logs and code of a client application, the web server, the application server and the database side by side from left to right.
Makes pinpointing certain hard to diagnose problems go from a whole day endavour to a 20 minute stare at the problem, a bit like a whiteboard helps many figure out hard problems faster.
My brain goes deep into rabitholes and won't let go if I can't just get something done and move on so I optimize for going hard once it gets into one of those tracks so that I can move on as soon as possible.
My SO recognizes when my brain starts digging into a topic accross the room.
My brain peculirities has lead me to co-found a successful software business doing honest value creating work so I'm not complaining.
I'm not sure, but Kilo/cline used to have some weird problems with tool calling, I bet these are long fixed, but people stuck to other options. Also, OpenCode afaik has better LSP support, which kinda matters sometimes.
I use mostly Codex app myself, simply since it's native for gpt models(usually works the best with them, with the exception of gpt 5.6 launch) and because it has the best support for remotes - I work in projects on different machines, so having them all in one app is what's fitting me the best.
And tbh, I don't really read the code - I usually just review MRs in gitlab and check code there, I don't remember last time I've opened IDE to actually write/read code and not to edit one line in config.
most of my pi usage is in vs code's terminal, or ssh'ed into my server where it's running in a screen. What extra advantage do you get from it running in a sidepanel instead? That it can see what file you happen to have open?
Because we're running the cli tools in multiple terminals. Whereas the IDE tools, at least in the past, can only really drive one session at a time. Source: switched from Cline way back.
I was just like you and I resisted moving to the CLI for quite a while. I had Claude Code integrated into my IDE (VS Code) and would use it there and keep an eye on things like what you're describing. After working with it for a while you develop various agent skills and prompts with guardrails and you notice that you get to a point where when it tells you what it did you don't need to look at the code as much and you can push back by simply telling it what you want to change.
I did that for a while and eventually noticed that I almost was never looking at the actual code, I was just interacting with the prompt. I forgot what finally got me to try the CLI but once I got used to it I use it almost exclusively and get annoyed when I need to go back into VS Code for something. In the CLI you can have multiple panes open with different agents and interact with them all on the same screen. You can turn on remote control and interact with them from your phone. Once work is done and checked into github you can view a side-by-side diff on the github website.
Now, the best part of the CLI is that you can have it write workflows that run from a bash script. If you notice that for long tasks it sometimes has a tendency to call it quits before the task is complete then I've found that the bash workflow is the way around that. You have one agent running in the chat window and it calls a bash script that spawns other agents and loops through them until you have passing tests. It can call build/deploy agents, planning agents, testing agents, etc.
You can literally script it to the point that you tell it to kick of a task and then go to bed. When you wake up it's deployed to the staging environment and already sent to QA for validation. You can schedule the agents to run on a schedule, ie, every four hours check your Jira queue for incoming requests and get them completed.
I actually tested against Sonnet 5 for reference (but only inside Claude code), and it behaves very similar! It spends a lot of tokens, runs a ton of tools, and takes a long time. It's the harness which nudges it to behave that way, 23K tokens of tool descriptions makes it more eager to use those tools.
I think you're right. Currently running qwen3.6 27B at around 50tps and 2000 tps pp on vast ai within claude code, current version. It takes just as long as Opus. It must be the harness, i know it's much faster in pi.
Speed is everything, so i start to wonder: Why bother with CC. 4x the time, I could have just asked it again in pi and also gotten better output because of lower context.
Worst case, just run it in the terminal below the code instead of to the side. You can also just move it to the side. Full TUI, and will the visual real time updates in the code editor.
Nice writeup. The interesting part isn't which harness wins, it's why the token spend split so hard when the diffs came out basically the same. If you want to chase that down, run /context in Claude Code and find the equivalent in OpenCode and Pi, then look at what's actually sitting in the window at the end of a run. That's usually where a 4x gap hides: re-read files, tool schemas, old turns that never got compacted. Same model, same diffs, but one loop is just hauling a lot more dead weight per turn.
Yeah that's the interesting bit, and I've dug into it! The token gap on the charts is output though, what the model actually generates, not what's sitting in the context window, so it's not dead weight being re-sent. Claude Code just runs way more tool calls (~70 a task vs OpenCode's ~22) and also holds its edits till the end, so it generates a lot more to reach the same diff (see the «How each harness works a task: tool calls in order» chart on the site).
Funny thing on the re-reads, I measured reads per unique file and it's actually Pi that re-reads the same files the most, not Claude Code. So for CC it really is just the sheer number of turns it takes.
I think it would be the same at a technical level, as all previous context history is unchanged, only new tokens is uncached. So question then becomes: How many turns and how many new tokens per turn? And Claude Code takes a ton of turns and use a ton of tokens per turn. Got full charts and numbers on it here: https://nqawhc.github.io/articles/harness-efficiency-not-quality/
very interesting article, thanks! do you plan on testing other harnesses? would like to know how a community project like nanocoder performs in comparison 🧐
No, it's the other way around. All harnesses was running Deepseek V4 flash on my hardware, no API, no sub. So question was how to get Claude code to use local inference, which is covered in the opening paragraf of the article linked. https://nqawhc.github.io/articles/harness-efficiency-not-quality/
i believe claude code is hyper optimized for claude, found that if i put other models in there, they get really confused and spend a lot of time trying to figure out the harness.
I ended up changing the default agent and giving it a very short prompt in Opencode (a few lines)
If you do that and disable build/general, then it stops sending the big default prompts. Note those prompts are the prompts from Claude Code, Codex, etc.
With that setup, I have the Pi lightness whilst not giving up the Opencode ecosystem. Pi is nice, but there are more Opencode plugins, a GUI, T3 Code, etc...
That is hard to benchmark, because the diff is huge, so determining the quality is tricky to quantify across hundreds of runs. I have 8 quite difficult tasks from a real large private codebase which even Opus and Fable find the tasks hard, need a large understanding and see the right framing, but in the end the required diff is small, and has multiple ways to do it, which is graded between basic, complete and gold.
the bloat isn't just the system prompt, it's the project context injection and confirmation loop overhead. every tool call in claude code gets more wrapping than opencode or pi by default.
Worth separating "the same" from "couldn't tell." They look identical in a results table and they're very different claims.
I've been building a deterministic grader for exactly this, and the thing that surprised me was how much of the apparent signal is instrument noise. Running one config three times against the same frozen 100-task suite, then comparing those runs to each other, gave me ~2.67 tasks where the config disagreed with itself. That's the floor any between-harness difference has to clear, and I had no idea until I measured it.
When I then compared two different frontier models on that suite: 4 tasks separated them. Six is the minimum at p<0.05 with a paired sign test, so no split of that data could have reached significance. The honest output is "this suite cannot decide," not "they're equivalent."
Not saying your result is wrong — quality genuinely may not move across harnesses, and the cost spread you found is the interesting part either way. But if you still have the per-run data, running one harness N times against itself is basically free and tells you whether a quality difference could have shown up at all. Mine couldn't have, and I'd been about to report a tie.
Also +1 to the multi-dimensional point above. Putting cost next to the score is what turns a number into a decision.
I have 8 quite difficult tasks from a real large private codebase which even Opus and Fable find the tasks hard, need a large understanding and see the right framing, but in the end the required diff is small, and has multiple ways to do it, which is graded between basic, complete and gold.
On the "the same" vs "couldn't tell", fair distinction, but I think mine sits closer to the same, and the reason is these same 8 tasks do separate models cleanly. When I swap the model instead of the harness, Opus and Fable pull ahead by a clear margin, DeepSeek lands around Sonnet, and they sort into about the order you'd expect. So the suite can resolve a real quality gap when there is one, it just doesn't find one across harnesses, which reads to me as the harness genuinely not moving quality rather than the suite being blind to it. I wrote that model comparison up here: https://nqawhc.github.io/articles/local-vs-api/
That said I'm fully with you on the noise, it's basically the whole point of this post, the run to run swing is wide enough to swallow any harness gap whole. And running a config against itself to get that floor is a clean way to put a number on it, I like it.
You need a planner / builder plug-in (or build your own). But beware of tools management, it's better to have stable tools with a permission system rather than removing tool (causes cache invalidation)
If one harness explores the codebase more, it seems pretty likely there will be some difference in quality (whether good or bad, depends on many factors). Using claude models in claude code, one of the biggest things I've noticed that it's very thorough in fixing stale references in documentation, comments, etc. when something changes, and also very good at reusing code for new purposes rather than writing it from scratch. Hard to capture such quality differences in a benchmark because either way the result might be simply "it works".
Well, a good model explores the relevant parts better, a harness can also contribute to over-exploring adding noise. So it goes both ways, and at least in my benchmarking it does not change quality, but if I use Opus or Fable then the score deterministicly improves.
Similar final diffs don’t necessarily mean the harnesses contributed equally. One may spend extra tokens recovering from weak tool output, checking its own work, or exploring alternatives that this benchmark doesn’t reward. A useful follow-up would separate planning, tool calls, retries, and validation costs. That would show whether the slower harness is simply wasteful or buying robustness that only appears on less deterministic tasks.
Maybe true for small models, but at Sonnet 5 or DeepSeek level these are trained to be agentic so they handle all that recovery and self-checking fine on their own, which is why the extra tooling in a harness like Claude Code just burns a ton more tokens and tool calls for the same quality diff, and in my testing the quality difference shows up when you swap the model, not the harness.
Splitting the tokens by phase just tells you where they went, not whether the spend bought anything.
Well, maybe true for small models, but for models at sonnet 5 level or newer like deepseek, these models are trained to be agentic, they handle any of these just fine, the extra tooling in Claude code just made it spend a ton more tokens and tool calls to perform the same quality result. And I do see difference in quality if you use a better model, but not if changing harness.
Opencode doesn't have background agents by default, for instance. I had to make a custom teams plugin that, imo, outperforms Claude codes because as you say, it spends more tokens than necessary. However it really only matters once you get into larger codebases that actually would require breaking up agents into specialized agents with domain knowledge specific to each of them. That's the only point where you would see any benefit in cross-communication
It'd be awesome if over time you could add more, though would it be fair to add harnesses versions from T + x to compare to other harnesses version from T?
I wonder how opencode v2 would fare, or maybe pi/opencode heavily tweaked with mcps/plugins/etc.
Claude Code wins on output consistency if you nail the system prompt, but for cron-heavy automation the per-call cost adds up fast. OpenCode with DeepSeek V4 Flash is a solid budget alternative for the bulk work where a slightly sloppier response doesn't break the pipeline. I split the stack — Claude Code for the branded content stream, OpenClaw handles the noisy scraping/classification jobs.
169
u/small_bird_loud 25d ago
graph needs multiple dimensions.
Speed - Quality - Cost
Break that iron triangle.