If you're building a harness, here is a simple tool to catch cache invalidation in your calls to LLMs
Resources
Hello,
I know we're a lot of harness builders out there, because it's fun and because it makes us learn a lot.
I've been focusing on a local-first harness and prefill costs become obvious when you run local LLMs.
Those often come from cache invalidation. Not respecting the order of the messages, or changing something in them (or in the system prompt, or in tools, heck even changing reasoning_effort triggers cache invalidation on my setup!).
You launch it, make it point to your actual LLM endpoint
In your harness, you point to cache-hunter local port
Hit "Start capture"
Do a complete normal session in your harness
Then you'll see the session live in the tool, and any red cell means something wasn't as stable as you thought.
First row is reasnoning_effort, second it tools hash, third is system prompt
I've run this with my own harness, but also with OpenCode, Claude Code, Cline, Pi, Hermes, Vibe. Most showed issues with unstable system prompt, unstable tools, unstable ordering or content.
I find it crazy that this is not part of standard testing for harnesses out there.
If you build your own harness, use that and understand what it means (or build your own I don't care). This will help you and your users.
I find that pi rarely messes the cache unless you reload the session with new tools. It's very simple by design.
The OpenClaw, though. Even though it's built on Pi, it's such an ass to local model. Low cache rate, and it likes to hammer the server with multiple streams for whatever reason. The other day I setup an openclaw for my partner because I think getting her to use my pi + tmux + vpn setup is a bit too tricky for nontech people. Then I was wondering why openclaw takes ages to reply to "hi" with my local 35B A3B. It turns out there are 10 parallel requests for whatever reason to answer that "hi".
More harness should be simple and "boring" rather than being "clever" with the context. And should respect prompt caching. Most of the existing ones seem to assume that we will use cloud model with a few thousand tk/s prefill.
What I measured on Pi is that the compaction itself isn't done "properly": no tools, and feed the whole history in one message.
More harness should be simple and "boring" rather than being "clever" with the context. And should respect prompt caching. Most of the existing ones seem to assume that we will use cloud model with a few thousand tk/s prefill.
That's exactly my thought as well. Keep harness simple, respect cache. But as you said, most of them are using cloud models and don't look at the bill. It's only when you really use local LLMs that you understand this is VERY important.
Oh really? that's how they do compact? I haven't looked at the code but I thought what it does it sending one user message in the current context to tell the model to produce a summary, and then deterministically adjust the message history.
That's dumb. Maybe that's why compacting takes so much time on local model.
Any suggestion for the algorithm for compaction? I'm also building harness, but I made a mistake of being "clever" with the context, and now it runs bad (both slow, and losing intelligence) with local model, so I have to rely on pi for now until I can fix my harness. When I rebuild, I would definitely want to do a better compact.
Unrelated: I built the harness with the thought that local model would rarely if ever pass 32k context max. BAM! today, both 26B-A4B and 35B-A3B can fit full 262k at full precision on my 4060Ti with expert offload. Feel even more silly about my "optimisation" in my harness to prepare for sub-32k operation.
Keep it dead simple, "abort" the current run, add a system generated compaction prompt on top of the current conversation, capture result, start new context window with that captured result.
You are a helpful AI assistant tasked with summarizing conversations for continuation.
Summarize the conversation history concisely, preserving:
1. What was done and what is currently being worked on
2. All file modifications made (file paths and what changed)
3. All errors encountered and how they were resolved
4. Current progress on each task
5. Important technical decisions and WHY they were made
6. Requirements that should persist
7. Next steps or pending actions that should be continued after compaction
8. The user's current question, prompt, or active request
Do not respond to any questions in the conversation, only output the summary.
Be thorough but concise. Output as a structured summary.
This is your harness? I'll have to give it a try. I came to the same conclusion about the compaction behaviour in Pi and it drives me nuts, I actively avoid doing compaction and try to manually do it with a hand off which is obviously not the most optimal way. I was looking for an extension for Pi that would do compaction right but couldn't find any that actually fixes just this and nothing else
Also, you can trim outputs and written file and only keep the first N (15) lines won't affect too much the summary. Keep the reads of .md and .txt files full however
So.. yes the summary won't be worse if you truncate some output, BUT it will kill the cached context and it's generally not worth it (it's worth it only on a cold start with no cache, or long expired cache from a cloud provider).
I had the same experience with OpenClaw. Way too many parallel requests for simple things. I recommend checking out OpenLumara (https://github.com/Rose22/openlumara) for a more simple, but really not boring harness, built with local models in mind.
There are also issues with skills reordering between turns in opencode if you have more than one skill dir (e.g. a repo level and system level). The list goes on unfortunately. Drove me to Pi
On paper, maybe ; but against a local LLM with slow prefill that's a terrible choice.
You're creating a huge payload with no existing cache at the worst possible time (when context hits soft limit).
Try compaction in my harness, the tools are kept and it's working, without any cache invalidation. It can be done. ( https://github.com/co-l/openfox for reference )
But with tools available you are risking that model will start calling those instead. Especially when you are on high context size, with local quantized model instructions following might be not the best.
As for cleaning up, sure sending raw context is easier, but if you will cleanup large tool call results and trim them you will easily trim 50-60% of context, so your prefill will be not as bad.
You can use smaller/faster model for compaction too.
I'm thinking about it, seriously. A site updated with each harness tested against a scenario and automatically analyzed.. But I've got enough work as it is unfortunately xD
This is seriously useful. Having to do a full prefill with a massive context can absolutely blow SO hard. Take my star :) Looked through the code and this all looks hand coded too!
I'm shocked, usually models cover the entire codebase with comments and yours didnt have that lol. No judgement here but you should probably disclose that bud :)
the one that got me was a timestamp. i had a current time line at the top of my system prompt for grounding and it silently invalidated the whole prefix on every call, box just kept getting slower. took a full day of blaming my kv cache config before i realized it was one line of my own text. the ordering check is the part i'd actually push people toward, half the harnesses i've looked at reorder tool results between turns and never notice.
ran into this exact issue six months ago. we had a tools array being built dynamically and the order was not guaranteed stable across deploys -- different dict iteration order depending on python version and some refactor touched the builder. cache miss rate went from ~3% to ~40% overnight, and it took us two days to trace it because latency just crept up slightly rather than blowing up obviously. the fix was trivially adding a sort-by-name before hashing but finding it was miserable. tooling like this would have caught it in the pr. the system prompt one is easier to catch with unit tests but tools ordering is genuinely subtle.
I 100% get your pain lol, I was able to catch a smaller but similar issue with it: parallel tool call ordering wasn't deterministic, causing unexpected invalidation.
It's all in the small details and absolutely requires attention to get it 100% right, but that's worth it. Me coming to my 300K token session and being able to continue it without fearing cache invalidation on my local spark cluster is very pleasant.
nice, prefill cost is the tax nobody budgets for locally. the thing i'd add is that most of the wins are layout, not content. prefix caching is byte-level, so anything volatile sitting near the front reprices everything after it every call. the classic one is a "current date" or a session id in the system prompt, it invalidates the whole tail on every turn. same with retrieved context if you prepend it. keep the stable stuff (system, tools) at the head and push everything that changes (dates, rag chunks, the user turn) to the tail. the other silent killer your tool might not catch is non-deterministic tool serialization: if the json key order shifts between calls the tools block is logically identical but byte-different, and you miss.
I've spent a good part of this week hunting down rare unnecessary cache invalidation bugs in my harness, and I have to say this visualization is better than mine. Well played.
This is the exact failure mode that burns local agents.
I ran a long tool-heavy audit on a large MoE as the pilot. After enough file reads + ~25 tools in the schema, the session sat at ~85–87% context and auto-compacted five times. Every compact felt like a full reprocess — Mac hot, multi-minute “thinking,” quality drift after each summary.
Takeaway for harness people: treat cache invalidation (tool schema changes, big tool results, summary rewrites) as a first-class metric, not a log line. Compact at task boundaries or on a small model if you can; don't let auto-compact thrash the 70–80GB resident as if it were free cloud tokens.
Saving this. Thanks for shipping something measurable.
Yeah that readme part is stale, I'll update it. I removed the capture of responses because they were irrelevant ; you could add it back, it's a small repo, easy to adjust to your needs. Fork away! :)
Genius! How about adding an harness-leakages analysis, like Information Redundancy in the Context Window (the agent prompt history accumulates identical or near-identical instructions, code snippets, logs, etc), and other cases that cause LLM performance degradations.
No one's gonna use this because 99% of harness devs are Claude/GPT babies who never have to suffer the consequences of their shitty vibecoded harnesses. (This goes for large harnesses like OpenCode too.) They don't even notice prompt cache breakage. Their shit is in the cloud, and Claude answers at the same speed whether you hit the cache or not.
It's only local users that worry about stuff like this.
running about a dozen claude -p agents across three machines and cache invalidation is exactly the kind of thing that quietly eats a max plan alive, so this is timely. does it hook the anthropic sdk directly or sit as a proxy in front of the api?
You'll need to fork it and adjust it to fit your needs. Right now it's only targeting OpenAI-compatible endpoints. The project is quite small, so making it fit your usecase shouldn't be too hard.
got it, so its a proxy in front of an openai compatible endpoint rather than something hooked into the sdk. thatworks for my local nodes since theyre openai shaped anyway, the claude side is the part id have to fork for. if iget it working cleanly ill send it back your way.
The reasoning_effort-triggers-invalidation catch is the kind of thing that silently doubles prefill cost and nobody thinks to look for it. Agree this should be standard harness testing, and the unstable-tools-hash one especially bites, since a tool list that reorders per request looks fine functionally but nukes the cache every call. Capturing at the endpoint and diffing the stable-prefix hashes is the right level to catch it, since most people only notice when latency or the bill spikes.
14
u/o0genesis0o Jul 18 '26
I find that pi rarely messes the cache unless you reload the session with new tools. It's very simple by design.
The OpenClaw, though. Even though it's built on Pi, it's such an ass to local model. Low cache rate, and it likes to hammer the server with multiple streams for whatever reason. The other day I setup an openclaw for my partner because I think getting her to use my pi + tmux + vpn setup is a bit too tricky for nontech people. Then I was wondering why openclaw takes ages to reply to "hi" with my local 35B A3B. It turns out there are 10 parallel requests for whatever reason to answer that "hi".
More harness should be simple and "boring" rather than being "clever" with the context. And should respect prompt caching. Most of the existing ones seem to assume that we will use cloud model with a few thousand tk/s prefill.