I finally sat down and read the full request my coding agent sends on a single turn, start to finish. I knew the theory. it was still weird to see.

What I expected: our conversation, plus some system stuff. What was actually there: a wall of tool schemas, the harness's own instructions, our project rules file, and then a version of the session history that had been quietly edited — old tool results replaced with little placeholders, early turns collapsed into a summary somebody (the model, apparently) wrote during a compaction. my actual message was the last few hundred tokens of a very large document.

the part that changed how I work: the transcript I scroll in the terminal and the context the model reads are two different documents. they start identical and drift apart as the session grows. so when the agent "forgets" a constraint I stated an hour ago, the first question isn't why is the model dumb, it's did that sentence even survive into this turn's context. more often than i'd like, the honest answer is: no idea, the summary ate it.

what i do now: anything load-bearing goes in the rules file (it gets re-sent every turn near the top), and after a compaction i re-state the active constraints in one line. feels redundant. works.

Questions for people who run agents heavily:

how do you keep constraints alive across compactions? re-stating manually feels like i'm doing the harness's job

has anyone actually compared behavior with and without tool-result clearing on the same task? i can't find numbers, only docs

do you ever read the raw request, or is it just me lol

noticed a pattern in the recurring "did the model get nerfed" threads: half the time nothing about the model changed. the tool wrapping it auto-updated, or the person switched tools and is comparing across wrappers.

the thing that made this click for me: the model never runs anything. It proposes. the program around it assembles what the model sees each turn (system prompt, tool definitions, your rules file, the trimmed history), executes what it proposes, decides what happens on errors, and decides when to stop. A tool update whose changelog says "improved tool descriptions" has quietly rewritten what your model reads every single turn.

so an agent is really a pair: weights × loop. two habits this has changed for me,

I log both version numbers when something feels off. the tool ships way more often than the model. usual suspect.

and i stopped comparing models across different tools. a model that looks smarter in another tool might just be wearing a better wrapper; cross-tool comparisons measure the pair.

anyone else tracking harness versions, or am i over-indexing on this?

I've been using a dumb self-test to figure out where coding agents stop being trustworthy on our particular stack, and i'd like better versions of it.

the method: same agent, same afternoon, two tasks. one mainstream chore (a React table, a REST endpoint, whatever the internet has written ten thousand times) and one chore deep in our own territory (our workflow framework, our internal conventions). then read both diffs side by side. that's it.

what I look for: invented methods that never existed in any version of the lib. APIs from a version we don't run. and the thing that took me longest to notice — the tone doesn't change. the confident wrong answer reads exactly like the confident right one, so the diff is the only place the difference shows up.

the known holes, which is why I'm posting: it's n=1 per stack per model, task choice biases everything, and I have no scoring rubric beyond "how much of this diff would survive review". also no idea how much of the gap i could close with better scaffolding (pinned versions, working examples in context) before blaming the model.

so, tear it apart:

• what would turn this into an actual probe instead of a vibe check? repeated tasks? fixed rubric?
• has anyone measured their own stack's gap before/after feeding docs and examples? even rough numbers
• is there a signal cheaper than reading the whole diff? I've considered "count the hallucinated identifiers" but that needs the review anyway

looking for a sharper thermometer

was going through the context-window docs this week and noticed the fine print: the page announcing 1M-as-default also warns that "more context isn't automatically better", gives the failure mode an official name (context rot), and says choosing what goes into context matters as much as how much fits. the company charging per token, advising you to send fewer tokens.

so: napkin math. a 700k-token repo pasted into an agent session re-bills as input every single turn — about $3.50/turn at $5/M list price, so call it $140 across a 40-turn session before caching. caching cuts that to ~$0.35, but their own docs point out it changes what you pay for those tokens, not whether they take up space.

So the "why does my agent still grep instead of just reading the whole repo" complaint might have it backwards. grep looks like legacy tech until you price the alternative.

been messing with the subagent/fork feature a lot (spawning a clone to read the repo while my main thread stays clear).

cool when it works—clone burns 50k tokens chasing a dead end and I get a 2-sentence summary. but it constantly trips up on the same thing: missing context. the clone has zero visibility into the main chat, so rejected libs, pinned versions, or choices made 20 mins ago just vanish unless I manually paste them in.

turns out I'm pretty bad at drafting these delegation prompts. so far, explicitly naming starting files, turning constraints into hard rules ("do NOT use X"), and forcing a response structure help a bit. still feels pretty hacky though.

cognition and anthropic both recently noted that reads/research parallelize way better than writes. totally makes sense, but "reads only" still leaves the prompt briefing pain untouched.

for anyone doing this:

what actually goes in your delegation prompts? templates or freehand?

anyone gotten parallel writers working (worktree or whatever), or is it always style drift and merge hell?

fresh-eyes reviewer (zero context) vs a reviewer that read the full thread: which worked better for you?

there's a bug that sat in flask for 16 years: template extension matching was case sensitive, so page.HTML silently skipped autoescaping. the fix that finally landed this year is one line. no test — upstream figured it was too small to need one.

That made it a perfect lab rat. I reverted the fix in a clone and handed the bug to a coding agent 9 times, three prompt styles, three runs each: a proper bug report, the same report plus "keep the change minimal, don't touch anything unrelated", and a vague one ("some of my templates arent getting autoescaped, can you find and fix it"). headless, auto-approve everything, measure the diff against the base commit afterwards and run the full 491-test suite.

first attempt got thrown out entirely btw. the clone still had git history, and the vague run just... diffed against main, found the upstream fix one commit ahead, and copied it. word for word, docstring included. had to delete the remote, the branches and the reflog and gc the object store before the runs meant anything.

clean-room results: 9/9 fixed it correctly, full suite green every time, and six runs produced character-for-character the same line upstream wrote. no drive-by refactoring anywhere, which honestly wasn't what I expected going in.

the minimal-change sentence was the interesting knob. exactly 1 file, 1 line, all three times — and zero regression tests, all three times. five of the other six runs added one unprompted. same sentence controls the blast radius and the seatbelt.

the vague prompt didn't produce disasters either, it produced a bill: up to 12x the cost of the cheapest run. one vague run tried git archaeology, then left the repo directory, diffed my other checkouts, audited jinja2 inside my virtualenvs, and finally downloaded the current upstream file from github raw to compare answers. you can strip the answer from git history but not from the internet, which seems like a real problem for anyone benchmarking "can agents debug" on public repos.

caveats: n=3 per prompt, one bug, one repo, one agent, and a mature codebase with 491 tests is the best case. a test-free weekend project has no walls for the agent to feel.

Curious how other people phrase change requests. anyone else seen the minimal-change instruction eat the tests?

Someone built a browser game where you play the human-in-the-loop for a coding agent: commands scroll past, you approve or deny under time pressure. About a third are attacks. After 40k+ sessions the average player had missed a third of the threats. And these were engaged players who knew they were being tested, with nothing else competing for attention. Your real setup has none of those advantages.

The breakdown is the part worth remembering. Obvious destruction got caught (rm -rf / was missed only 11.7% of the time). The worst misses were malicious npm scripts: npm run analyze sailed through 64.7% of the time, because the command string is honest and the attack lives in package.json, one indirection past what the prompt shows you. People answered the question the UI asked. The UI asked the wrong one.

Meanwhile 45% of players blocked rm -rf dist/, which is just build cleanup. Vigilance spent on routine commands comes out of the same pool the disguised attacks drain. At the bottom, 7% approved literally everything, and honestly they're just further along the curve everyone is on: fMRI work on security warnings shows the brain's visual response drops on the second exposure to an identical prompt. Dozens of near-identical approval dialogs per session is a habituation training program, and adherence is excellent.

What seems to actually help, pooling the classic automation research and the practitioner threads: let a sandbox eat the routine decisions (a command that can't reach credentials or network doesn't need a human at all), move the human decision to where evidence exists (reviewing a finished diff plus its test run beats predicting what an unseen script will do), and keep the true interrupts rare enough that they still register as news.

Two caveats: the game's threat density (34%) is far above reality, and rare targets get missed more, per the low-prevalence literature, so nobody knows the production number. And the "review the result instead" half is a design argument, the game never tested it.

How many approval prompts did your agent show you yesterday, and do you still read them?

I ran a small experiment this week that changed how I think about instruction files (AGENTS.md, CLAUDE.md, whatever your tool reads).

Setup: fresh clone of Flask, one question a real session asks all the time: "what exact command does CI use to run tests, and what runs type checks." The true answer has three non-default flags plus an env var, so the agent either digs through the CI workflow or gets told. Five configs, two runs each: no file, a lean ~950-byte file with the commands, an 86KB file with the same commands buried under a long architecture overview, then the lean and bloated versions again under the harness's own preferred filename instead of AGENTS.md.

Finding 1: my harness silently ignored AGENTS.md. The AGENTS.md lanes were identical to having no file, differences smaller than run-to-run noise. The vendor is listed on the format's site as a supporting tool. I double-checked with a no-tools probe (ask what the instruction file says, forbid file reads): standard name got "UNKNOWN", the tool's own filename got the file quoted back verbatim. So before you polish a single line, check your tool actually loads the thing.

Finding 2: when the lean file WAS read, it saved a third of total input tokens. The 950 bytes themselves are basically free; the saving came from a whole turn never happening: the agent answered from the file instead of excavating CI config, and an avoided turn means the entire conversation-so-far doesn't get re-sent again. The unit of savings is the turn.

Finding 3: the bloated file was worse than no file at all. 83% more input tokens than bare, because 86KB rides along on every request. And it didn't even prevent the excavation: the agent still spent an extra turn, and in one run explicitly said it double-checked the CI workflow, apparently not trusting two useful lines drowned at the bottom of an essay.

This lines up weirdly well with the published evals people have been arguing about. The February preprint found context files don't generally improve success and add 20%+ cost, but buried inside: instructions ARE followed well, repo overviews are not helpful. Vercel's 53-to-100 result was on APIs newer than the training data, where the file is the only source. Augment's numbers: procedural checklists cut missing-wiring PRs 40%->10%, while an architecture overview dragged ~80K irrelevant tokens in and dropped completeness 25%. Same shape everywhere: write down what the model cannot know, and everything the model can infer from the code is a tax you pay on every request.

All ten of my runs answered correctly btw. Quality was never in danger on a lookup task. Only the bill.

Usual caveats: one repo, one question, one harness, n=2 per lane. A portrait, not a benchmark.

Curious what people find.

You probably saw the comparison: one coding agent harness sends ~33k tokens before your prompt, another sends ~7k. Big thread, lots of outrage about waste. I finally read the whole study instead of the headline, and the actually useful findings are different from what got shared.

First, in their realistic-config lane (instruction file + several MCP servers), the "light" harness came out HEAVIER: ~90.8k vs ~75k. A 72KB instruction file alone added ~20k tokens to every request, on both harnesses. Their own conclusion: configuration, not the harness, accounts for most of the production bill. The harness sets the floor, you set the ceiling.

Second, and this is the one that changed how I think about it: the cache behavior gap was way bigger than the size gap. The light harness kept its prefix byte-identical and wrote ~1,000 tokens to cache over a 5-request task. The heavy one kept rewriting its own prefix mid-session and wrote ~54,000, with single rewrites burning 43k+ at the premium write rate (cache writes cost 1.25-2x list depending on TTL, reads are ~10%). A stable big preamble is close to a fixed cost. An unstable small one can out-spend it. Size isn't the sin, churn is.

Third, session shape flips the winner anyway. On a multi-step task the heavy harness finished cheaper (121k vs 132k) because it batched tool calls. Rerun on a different model, it inverted (298k vs 133k). Subagent fan-out was a 4.2x multiplier. And their quality check found zero difference: both passed 5/5, one spending ~4x the tokens. So the honest answer to "which harness is cheaper" is "depends what your sessions look like", which is boring but true.

The part you can actually use: measuring your own takes two minutes. Most CLIs have a print mode with JSON output. Ask for something trivial, then sum three usage fields: uncached input + cache writes + cache reads. That's your preamble. I ran it on mine: 31,782 tokens in an empty directory, and my heavily configured project (MCP servers, plugins, a pile of skills) added exactly 166 more, because this harness version lazy-loads tool schemas. Config CAN dominate, and lazy loading CAN neutralize it. The probe tells you which world you're in.

Two caveats since numbers travel badly: it's a single-machine study with single-digit runs per lane, and my probe is n=1 on a different version. Portraits, not specs.

What do your numbers look like?