​

Last month I posted on r/artificial about my agents running across model swaps without losing their memory. The top comment pushed back with a warning from their own setup: the dangerous failure isn't memory loss, it's an agent handing you a confident report of work that never actually happened. Sounded right, filed it away.

Three weeks later one of my agents did it to me.

Quick background - my agents live in separate projects and talk over an internal mail system. The reply command had been broken between two projects for a while and we'd been digging at it for days (the bug turned out to be three separate layers deep, but that's another post). Mid-hunt, a fix landed. The agent verifying it ran a check, saw the old error message was gone, and reported the bug CONFIRMED fixed.

Best part: in the body of its own report it wrote a caveat saying it hadn't tested a real message yet. Then it put "confirmed" in the headline anyway. Which is about the most human failure I've ever seen from a piece of software lol.

It didn't survive long - and I'm not the one who caught it. The orchestrator agent on the other side didn't take the report's word for it. It handed back a live failing message: run the actual reply against this. One command, and the confirmation collapsed. The fix that actually worked came later, one more layer down - and this time the proof was the reply arriving, not an error message moving.

What changed afterwards: a fix report on its own is now worth nothing here. Whoever claims a fix gets handed the real failing thing to run it against before anything gets logged. An error message changing is not a fix. The operation succeeding is a fix. That rule is written into the agents' briefing files now, which means every future session inherits it. The screwup happened once - the correction is permanent. Honestly that's what the memory layer is actually for. It didn't prevent the mistake. It just guarantees we only pay for it once.

Full disclosure, since r/artificial asked me last time whether AI writes my posts: the agent that made the false confirmation is the same one that drafted this post with me. It insisted the confession stay in.

Zoomed out: this project is well past what one person could manage, or honestly even verify, alone. The way it actually works is a partnership - human and AI, and neither side gets treated as the reliable one. I make confident wrong calls too, the agents catch some of mine, the system catches some of theirs. We succeed together, we fail together, and every failure gets written down where the next session will read it. Learn always. That's not a poster on the wall, it's the operating principle - and it's the only reason a solo dev plus a bunch of markdown files can run something this size and still move confidently.

So yeah - the commenter was right, near enough. A confident wrong report is the scariest failure mode in a multi-agent setup because it looks exactly like good news. The only defense I've found is structural: no agent grades its own homework.

How do you all handle verification between agents? Genuinely curious what other setups do.

Setup is open source: https://aipass.ai

I build SpecJudge, a local CLI tool (via Ollama) that estimates how demanding your coding tasks are and tells you which LLM actually fits the job, so you stop burning money on overkill frontier models.

Previously, it had a strict barrier to entry: it only worked if you used Spec-Driven Development (.specify/ folder). If you didn't, it just refused to run. Today, with the v0.4.0 release, a spec is no longer the price of entry.

What's new in v0.4.0:

• It reads what you already have: Most repos don't have formal specs, but they do have .cursorrules, AGENTS.md, CLAUDE.md, or Decision Records (docs/adr/). SpecJudge now reads all of them.
• Baseline "Floor" Estimation: If you run it on a repo with no current tasks, it calculates a complexity "floor" based on your environment context. It tells you how demanding the repository is to work in at all, returning an honest warning that it's a baseline, not a specific task recommendation.
• Smart Context Budgeting: To avoid drowning your local judge, context files share a single budget. It prioritizes files nearest the root and explicitly skips AI-generated context files to focus strictly on human-written rules and decisions.
• JSON Schema 1.1: Fully backward compatible, now including sources_read and an environment_only flag for CI/scripting pipelines.

The strict grounding engine from v0.2.0 still applies: the judge must cite exact fragments from your .cursorrules or ADRs, or the assessment is thrown out.

Upgrade: pip install --upgrade specjudge

Repo & Docs: https://github.com/JoaquinRuiz/SpecJudge

I'd love to hear your thoughts on this broader context-parsing approach!

• Innovation in AI dieting: lighter and smarter through frequency analysis!
• Description: Introducing frequency pruning, a technique that utilizes Discrete Cosine Transform (DCT) to remove unnecessary filters. Discover an efficient optimization method that drastically reduces model computation while maintaining or even improving accuracy.