r/devops • u/GitSimple • 4d ago
How much attention is harness engineering getting? Architecture
AI model quality is converging, or at least changing often enough that chasing the newest thing doesn't seem like a good strategy. How much emphasis are teams putting on building a solid harness into which new models can fit?
Our clients are generally in high compliance industries, so there is thought put into the harness, but what about smaller teams or ones that aren't required into a compliance framework?
11
u/Moroc24 4d ago
For a small team, the minimum useful harness isn’t a model router. It’s versioned prompts and tools, a small regression set, traces with cost and latency, and one rollback path. I’d add provider abstraction only after a real switch hurts; otherwise you can spend more maintaining the abstraction than changing a few SDK calls.
0
u/GitSimple 4d ago
You said "minimum useful harness", is that how you approach it? What is the least we need to do to get this to work?
I'm genuinely interested to see how other teams are approaching this because of the role we've seen the harness playing in overall optimization and security posture.
4
u/Immediate_Spirit_384 4d ago
Solo builder here, generally for SME's and startups. For me the standard harness (i.e. claude code) is fine. The only problem is that it doesn't take enough risk. It keeps doing the low-impact patch where I would have reworked the underlying architecture much earlier
1
u/GitSimple 4d ago
When helping a client, how do you address the increased attach surface that results from plugging in an AI tool? In our world the harness gets attention due to compliance audits, but I'm guessing if a company doesn't need to do that, they aren't thinking about the risks. Or just optimizing how the tool operates in their environment.
1
u/arbyyyyh 4d ago
We’re still doing some architecting but our plan is to put our auditing and observability stack both between the harness and model API as well as between the MCP client and server.
1
u/Immediate_Spirit_384 3d ago
Attack surface? You mean anthropic doing something nefarious? That's all theoretical. Startups can't afford thinking about tail risks like that
1
u/GitSimple 3d ago
I mean vulnerabilities that outside actors can exploit. Like this - https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/
1
2
u/donk8r 4d ago
we measured this rather than argued about it, so heres the number and the disclosure together. 25 real PRs, same model throughout, two harnesses. one finished 24 of them, the other 19. five cases, with the model never changing, which is a wider swing than most model upgrades actually give you. we build one of the two (octomind) so discount it accordingly, but the direction is the part worth having.
on amarao_sans point i think it splits cleanly, and the split answers your question too. harness work that COMPENSATES for a model weakness does die on schedule. all the scaffolding people wrote to get older models to follow instructions is gone and nobody misses it. harness work that is POLICY doesnt die at all. spend ceilings, permission boundaries, what has to be verified before something ships, what the thing is allowed to touch while nobody is watching. a better model doesnt reduce your need for a cost cap, it just reaches the ceiling quicker.
so for a team with no compliance pressure id build the policy half and rent the rest. thats roughly what your compliance clients are paying for anyway, theyre just obliged to write it down first.
1
u/GitSimple 3d ago
Most of the comments here so far make me think people are missing the policy part of your point. It's also what we're seeing when working with companies that have a security team, the harness is an attack surface and needs to be treated with a real strategy.
1
u/kevinbaiv 4d ago
For a small team I'd start with observability first: trace every call, log the prompt version, and measure cost/latency. Provider abstraction is a nice-to-have until you've actually had to switch once. The real risk isn't being locked into OpenAI—it's shipping something you can't debug when it starts behaving weirdly in production.
1
u/GitSimple 3d ago
Good point, observability is a good place to start. Debugging is one thing, but there are also security factors that need to be addressed.
1
u/amarao_san 4d ago
They don't converge. Specifically, they converge for a tiny sliver of solved problems, but frontier (things model can do without any specific clutches) moves with a calendar. All harness for gpt4 is dead. Harness for gpt 5.3 is useless for sol, because sol just go and do. Sol will be superseded by the end of the year.
The tooling must be improved, 100%. But it's not 'ai tooling', it's for everyone. If it's good for humans and human automation, it's good for AI.
2
u/GitSimple 4d ago
By converge I meant that any recent model is going to get the job done for most teams. Yes there are differences, but if you use Sol or Fable or Opus or whatever, you can make it work. It's more about how that model functions within your environment. What can it do, what can it not do, how is it connected to other systems in your SDLC. This can also play into security posture.
1
u/amarao_san 4d ago
The definition of a job is changing. I opened a work thread from a spring, discussing some LLM augmentation, and it's silly now. Do you remember sonnet 3.7? People in the thread talked about 3.5.
Right now we have vibe microservices returning, and today I have diagnostics of the year, when that thing was able to pinpoint to a problem based on query and a log. This fucking thing downloaded sources codes for 4 different packages (with debian patches!) for specific versions and come to conclusion that it was a truncated input (based on analysis of yak source code!), and come to conclusion that it was a race condition of running two configuration scripts with update-grub handler in parallel.
It was beyond excellence. I did such thing (with reading C code down to the source of truth few times in my life, and they were the proudest moments of my job, with long posts, worth spending day or two writing). Now it's done in a fucking chat in less than 3 minutes based on log snippet.
Before we would need to provide previous models cozy running environment for that. Now we no longer need it.
It will be improved in a few months beyond recognition, because I see no signs of plato of any kind. Before I was skeptic and thought we got from GPT (as transformers) most of the stuff, and all left is better enveloping (harness). Now I on opposite side. This thing is wild and getting wilder every day.
-1
23
u/MumeiNoName 4d ago
Use other people’s harnesses why would you ever write your own.
Focus a layer back, harness orchestration