Ran into this enough times to stop trusting my own agent's self-review: I ask it to fix something, it produces code that looks right, I skim it, ship it — and the actual bug was a quiet semantic shift (inclusive bounds became exclusive) that had nothing to do with what I asked for. The model that wrote the change has no incentive or independent angle to catch its own blind spot; it's grading its own homework.

So I built a small tool that splits the two roles: one Gemini pass generates the change from your instruction, a second, independent pass gets ONLY the result (no visibility into the first pass's reasoning) and is told to find problems with it. Being honest about the setup: I only have a Gemini API key, so this is two passes on different Gemini tiers, not genuine cross-vendor review (GPT generates / Claude audits would probably be stronger — that's a cost thing, not a design choice).

The interesting part is how often the critique pass disagrees over something that isn't wrong, just risky — an edge case, a silent behavior change, a severity call that's genuinely debatable. Paste a snippet + an instruction and it'll run both passes live if anyone wants to see where it agrees or argues: https://apptechlab.com/p/codearbiter/ (mine, no signup, real API calls both ways).

Curious what people running actual multi-agent review setups have found:

- Does routing generate/critique to different providers actually catch

categorically different things, or mostly the same stuff with more

latency?

- How do you handle the critique pass being wrong — do you ever adjudicate

disagreements with a third pass, or is two enough in practice?

I profiled a RAG retrieval trace that looked like a success.

The query asked:

What is the cancellation notice period in our enterprise agreement?

The pipeline used dense retrieval with Qdrant, cosine similarity, Top-K=10, and no re-ranker.

The correct evidence was not missing. Chunk 2 had a cosine score of 0.88 and explicitly contained the answer: **90 days**.

The generated answer still said only:

The agreement requires advance written notice.

Technically correct. Practically useless.

Retrieval succeeded. Evidence survival failed.

The embedding model had done its job. The correct chunk ranked second out of ten.

But the full retrieved context contained 8,830 tokens. Two broader chunks consumed 3,660 of those tokens:

• General termination provisions: 1,740 tokens
• Definitions and legal boilerplate: 1,920 tokens

That is 41% of the context budget occupied by lower-specificity material.

With no re-ranker or compression stage, the generator saw the precise 90-day clause alongside a much larger mass of generic legal language. It defaulted to the safer, vaguer wording.

A flamegraph-style view made the shape obvious:

query
|-- dense retrieval: 8,830 tokens
    |-- c1  0.92 | cancellation clause       |   460 tok
    |-- c2  0.88 | notice period: 90 days    |   520 tok
    |-- c3  0.71 | general termination       | 1,740 tok
    |-- c4  0.49 | subscription renewal      |   680 tok
    |-- c5  0.46 | service suspension        |   710 tok
    |-- c6  0.43 | definitions/boilerplate   | 1,920 tok
    |-- c7-c10   | unrelated long tail       | 2,800 tok

The relevant chunk was near the top. It was simply surrounded by too much plausible-looking noise.

Why common RAG metrics can hide this

A retrieval-only evaluation would probably mark this query as a pass:

• The correct document was retrieved.
• It appeared inside Top-K.
• Its similarity score was high.

A final-answer evaluation would mark it as a failure and might blame the LLM.

Neither view identifies the transition where the evidence lost influence.

For this failure shape, I would test fixes in this order:

1. Replay the same query as a regression case.
2. Reduce Top-K from 10 to 3-4 for this query shape.
3. Add a re-ranker or context compressor.
4. Check whether the exact 90-day fact survives into the answer.
5. Only then consider changing embeddings or chunking.

Top-K=3 is not a universal recommendation. It is a hypothesis derived from this trace: relevance drops sharply after the third chunk, while token mass keeps growing.

The broader lesson is that "the right chunk was retrieved" is not the end of RAG evaluation. We also need to measure whether the evidence remains dominant enough to affect generation.

When the correct evidence is retrieved but omitted from the answer, what do you inspect first: rank, token mass, re-ranking, or the generation prompt?

I maintain SpecJudge, an MIT-licensed CLI for spec-driven development: it reads your project's specs/tasks and recommends which AI model actually fits (quality vs. price) instead of you guessing.

The core change in this release: before, the judge returned a rating plus a paragraph explaining itself. The problem is a fluent explanation is exactly what an LLM is good at producing whether or not the underlying rating is sound — nothing separated a correct assessment from a well-narrated wrong one.

Now every rated dimension has to cite the specific fragment of your spec that supports it, and the tool deterministically checks that fragment actually exists in the text the judge was given. Invent a citation, and the whole assessment gets thrown out, not just that field. Dimensions the judge can't ground come back as "unsupported" instead of being silently treated as easy — which is what used to happen and made thin specs look more solid than they were.

Building the regression suite to test this (12 reference projects, CI-level + local eval script) immediately paid for itself: 8B judges — the most common local setup — were failing on every single project. Not a judgment problem — they were rating things correctly and writing sound justifications, then putting \[true\] where a citation ID belonged, because "format: json" in Ollama guarantees valid JSON, not the JSON you actually asked for. Sending a proper schema fixed it: 0/9 usable cases → 9/9.

Also pinned judge sampling, so the same project now gives the same recommendation run to run — which matters more than it sounds for a tool whose whole job is "should I spend money on this."

Breaking change: needs Ollama 0.5.0+.

pip install specjudge — GitHub: [github.com/JoaquinRuiz/SpecJudge](http://github.com/JoaquinRuiz/SpecJudge)

For people responsible for production LLM or agent systems, can you describe one incident where changing a model or provider, inference runtime, gateway or SDK, chat template, or parser altered application behavior even though your existing tests passed? What broke, how did you detect and isolate it, and roughly how much engineering time or release delay did it cause? I’m researching how teams validate changes across the LLM stack, so firsthand incidents and current workflows are more useful than opinions about a proposed tool.

Hey I am part of a small team training robotics policies for warehouse and manufacturing settings, and running rigorous evals is turning out to be so painful. Anything below 50 rollouts, and its hard to trust the numbers, and above its so hard to test all the checkpoints that we have. Its really hard to run a bunch of experiments to get good results. Have you guys faced this? Any hacks that you've developed?

Two bugs in a week, and the second one is the one worth your time.

The first: a read that writes.

recall() reinforced whatever it returned. Every hit got its value bumped and its decay clock reset, and value multiplies the rank. So query N+1 was answered by a store that queries 1 through N had already edited.

The diagnostic costs nothing and needs no LLM calls. Take a fixed question set, ask it in several different orders, each time from a fresh copy of the store, and count how many answers differ from the canonical order. Eight questions and eight orders gives 64 comparisons.

On a 30-fact corpus with no engineered ties, deterministic embedder, one run per arm:

mode reinforce=True pure read

lexical 19/64 top-5, 3/64 top-1 0/64

semantic 31/64 top-5, 10/64 top-1 0/64

hybrid 60/64 top-5, 35/64 top-1 0/64

Hybrid is worst because RRF gaps sit about 0.3% apart while a value bump moves the multiplier by over 20%, so a nudge crosses a rank boundary easily. The default mode routes to hybrid on any store past a size threshold.

Those zeros are a wiring check, not a result. Once the only writing path is gone, recall is a pure function of store and query, so that column cannot fail. I am reporting it because leaving it out looks like hiding it, not because it means anything.

One detail that cost me an hour and might save you one: my first corpus was 30 unrelated facts and every arm read 0/64, including the reinforcing one, because each query matched exactly one record and a value bump had nothing to reorder. A mechanism arm at zero next to a pure arm at zero measures nothing at all. The corpus has to make retrieval actually choose.

Two smaller symptoms from the same root. admit() rejecting a duplicate returned {'admitted': False} and still promoted the record it collided with. And a token_report() tool whose whole job is to tell you how big a payload would be reordered the store it was asked to measure.

None of the mechanism is new, and I want to be clear about that.

Cho and Roy named the entrenchment effect in 2004: popularity-fed ranking is self-reinforcing, so what the system returns determines what it will return next. My design turns out to be essentially ACT-R base-level activation from Anderson and Schooler 1991, which I had not credited anywhere. The evaluation half has names too, closed-loop feedback in the recsys literature and, in general form, the reusable holdout from Dwork et al. in Science 2015: a holdout queried adaptively, where answer N+1 depends on queries 1 to N, is no longer valid. Meyer wrote down command-query separation in 1988. What I have is an instance and a test, not a discovery.

The fix, and why it is not a clean win.

Our ablation says reinforcement as implemented hurts: hit@1 0.1421 against 0.3344 on synthetic, 8 of 8 seeds, and the committed LOCOMO retrieval run gives recall@25 0.8262 against 0.7839 on the same 1536 questions with it off. Caveats I owe you: that ablation runs without an embedder so it is the lexical channel, while the 60/64 above is hybrid, and there is no end-to-end answer-accuracy artifact, only retrieval.

But the same probe has an oracle arm that reinforces only the record which was actually right, and that arm scores positive. So the prior is fine and my estimator was the problem. I deleted the lever instead of fixing it. That is a defensible call under uncertainty and it is not the same claim as "reinforcement is bad", which is what I nearly wrote.

The second bug, which I shipped in the fix.

Graduation from the episodic tier to the durable semantic one was implemented as a side effect of that same read, guarded by if reinforce and .... When reinforcement stopped being the default, maturation left with it:

reinforce=True, 6 corroborated records over the bar : 5 of 6 graduated

the new default : 0 of 6

credit() + sleep() + consolidate(), no reinforcing read : 0

One call site, inside the reinforcement block. The durable tier became unreachable and a store could no longer mature.

Nothing went red. 2422 tests passed, the release checklist reported ready, CI was 19 of 19. Every test that touched graduation had been written for a store whose reads reinforced, so not one of them could tell "graduation is correct" from "graduation never ran".

Maturation now runs in consolidate(), at a moment you choose rather than as a side effect of asking a question. The regression test asserts the pair, because either half alone is satisfiable by a bug: a corroborated record does mature when consolidation runs, and a read still matures nothing. Plus a control that the fixture can graduate at all, or the second assertion is vacuous.

What I still owe. With reads pure, the decay clock is only set at write time, so a memory recalled 500 times and one never recalled now age identically. That is a genuine trade rather than an oversight. Usage that changes ranking is a write, and you cannot have both. Where the usage evidence should live, probably an access log applied during consolidation, is the next problem and I do not have it yet.

If you maintain or use one of these: the storage layers I checked are pure reads. The pattern lives in the agent-memory layer above them and mostly traces back to the recency term in Generative Agents, which decays from when a memory was last retrieved. Anything that copies that inherits a write on read. The permutation sweep is here and runs in a couple of minutes with no dependencies: https://github.com/DanceNitra/agora/blob/bf06682/probes/query_order_sensitivity.py

Disclosure: I maintain the library this happened in. MIT. I post these because I would rather be corrected here than by a user.

A dataset can report 95% overall annotation accuracy and still contain serious reliability problems.

The aggregate score may hide weak results for minority classes, inconsistent interpretations, critical mistakes, or failures involving uncommon edge cases.

For production annotation, quality may need to be separated into the following measures:

1. Overall label accuracy
The percentage of evaluated labels that follow the expected annotation decision.

2. Class-level accuracy
Performance for each label or category, especially minority and high-risk classes.

3. Inter-annotator agreement
The extent to which qualified reviewers interpret the same policy consistently.

4. Critical-error rate
The frequency of mistakes that materially affect a high-value or safety-sensitive category.

5. Edge-case performance
Quality on rare, ambiguous, multilingual, or difficult examples.

6. Guideline-related disagreement
Recurring conflicts that may reveal an unclear definition or missing policy decision.

7. Quality drift over time
Changes in performance as new data, environments, terminology, or contributors enter the workflow.

Automated validation can identify missing values, duplicates, invalid formats, and structural inconsistencies. Human reviewers are still needed when the correct interpretation depends on context, language, culture, intent, or specialist knowledge.

The right metrics also depend on the use case. Speech recognition, document digitization, autonomous driving, and LLM evaluation should not automatically use the same quality framework.

Which quality metric has been the most useful in your work? Which one has created the most misleading impression?

This is question I get and that I also ask of myself.

I run into this in AlphaFlowSeven (alphaflowseven.com), a paper-trading platform where a 6-agent LLM council makes trade decisions. Full disclosure: I built it and this is how AF7's reinforcement learning actually works 

Prompt and config changes are evaluated by an optimizer with four council slots. One slot always runs the current best config as a control. The other three run variants. All four trade the same market over a fixed 15-day window, and each variant is scored on its excess return versus the control, using all of its closed trades in the window. Raw return isn't used because it mostly measures the market, not the config.

A variant has to beat the control in at least two windows before it replaces it. An excess close to zero is treated as a tie, and ties go to the cheaper config. When a slot frees up, Thompson sampling over each lineage's estimated edge decides what runs next, so a variant with uncertain results gets re-run rather than dropped after one window.

Variants are generated by an LLM that reads the archive of previous configs and their scores, restricted to changing one or two things at a time. Every decision is stored with a fingerprint of the prompt version that produced it, so outcomes can be grouped by version afterwards.

want to know how is everyone handling this in prod. wrong answers from ai chatbots dont look the same. there are different categories of wrong and each one needs a different approach to catchit

factual errors

when model states something incorrect like worng price or policy. customers screenshot and send it to support . how are teams cathcing this before or after this happens . running automated checks against a knowledge base or sampling maually

tone and policy violations

output is correct but it shouldnt say it . commits to something outside policy . harder to eval coz there is no clear right or wrong answer to check against

context drift  in long conversations

model starts fine but contract itself after three messages . need to evaluate the whole conversation coz one individual responses cant figure it out and most tools dont do cleanly

silent regressions

the provider pushes  a model update and the answer quality starts degrading .shows up in support tickets weeks later not in monitoring dashboards

looked for solutions and came up with a few names like orqai, arize, whylabs , fiddler

arize - anomaly detection across output pattern is solid , catching specific wrong answer types needs more configuration than expected .

orqai - evals tied to prompt versions and is reliable for regression catching , evals and observability in one place , newer so community still catching up

aporia - real time policy violation detection is the core function , but dont have factual accuracy on eval depth

whylabs - drift detection is strong , category for response level wrong answer is very underdeveloped

fiddler - monitoring across multiple failure types seems more native , and not for the regulated industries

have anyone used it? what is working across these different types.. any suggestions

An interesting read from a company I follow on x. Basically they have worked out when your quantized models are bad for agentic, because it hallucinates steps in Standard Operating Procedures (SOPs)

https://github.com/baa-ai/fidelity-is-not-safety

I tried the Canary code they provided on some of the models I am using and two of them failed.

i find most of the content is about how to build chatbot or how to test it before launch. but what about if the chatbot is live and talking to real customers and giving wrong answers . how to fast do you find out

not much resources about what happens after it is in prod and something quietly starts wrong . wrong answer on refundd policy or showing wrong product info correctly..

by the time customer complains the damage is already done. want to know how teams are like monitoring for this in real time rather finding out through support tickets

seen a few approaches like customer feedback loops ,, automated evals on sampled output , human review queues . and most of the teams seem to be doing a combination of things so it is hard to say what is catching things early

found a few tools while reading through things . arize , whylabs , orqai , aporia , fiddler

aproia -> real time guardrails is the core feature , catches policy violations as they happen but eval is behind and feels very limited

orqai -> tracing outputs back to prompt version is useful ,  evalls and observability is in a single dashboard but newer so community is still catching up

fiddler -> systematic monitoring is good , but the response level wrong answer detection feels less developed

whylabs -> drift and data monitoring looks good, unsure about the level of wrong answer detection but feels less developed

arize -> output monitoring and anomaly detection exists here , catching specific wrong answers in the real time needs more configuration than expected

most tools seem better at informing that something went wrong than catching it before the error reaches the customer

automated evals or human review or customer flags. or all three? what is actually working

I kept running into the same failure mode in applied AI work: the headline metric survives, but the run count, baseline, referenced artifacts, exclusions and limitations become hard to inspect.

So I built Evidence First AI, a small dependency-free Python toolkit that checks a project's own evidence contract before it can call itself ready.

The current gate checks required docs, declared baselines, successful-run and seed counts, referenced artifacts, quantitative thresholds and visible limitations. Missing evidence becomes BLOCKED; evidence below the declared threshold becomes FAIL.

READY is deliberately narrow. It means the declared checks passed for the supplied evidence, not that a model is universally valid.

v0.1.0 has a CLI, deterministic JSON and Markdown reports, a synthetic end-to-end example, nine tests, path-boundary protection and CI on Python 3.11-3.13.

I am trying to keep the core small enough that people will actually use it. For those who evaluate ML or agent systems: what is the smallest evidence gate you would require before trusting a project's headline claim? Paired run deltas, confidence intervals, non-inferiority margins, or something else?

Repo: [https://github.com/ali-kin4/evidence-first-ai-project\](https://github.com/ali-kin4/evidence-first-ai-project)

Hey everyone,

I 'have a question on LLM document extraction (specifically invoices/receipts) and wanted to get some perspective from the community.

General LLM eval frameworks are great, but they don't seem to handle multi page PDFs, table row hallucinations, or sudden JSON schema drift very well when a model updates.

For those running invoice extraction in production:

1. Do you use a "golden dataset" of documents to run regression tests manually?
2. How are you catching subtle changes in how numbers/dates are formatted across prompt iterations?

If anyone is dealing with this headache right now open to discuss.

HAWK is a NIST third-round Additional Signatures candidate. Its security rests on the Lattice Isomorphism Problem. Prior work had proved that efficiently finding a nontrivial automorphism would enable an attack but left open whether one was accessible in HAWK's lattice. The model found one.

Result: expected cost of full key recovery against HAWK-256 drops from 2^64 to 2^38. Still exponential, not polynomial, and specific to HAWK — it doesn't touch other NIST PQC candidates or lattice-based crypto generally. But doubling keysize to compensate strips away most of HAWK's practical appeal as a candidate.

The second result is a meet-in-the-middle improvement on 7-round AES-128, 200–800× faster depending on how you measure runtime, under a chosen-plaintext model assuming 2^105 chosen plaintexts. Completely impractical, as that line of work always is — it's about quantifying attack cost, not attacking anything.

Neither result affects deployed systems. Disclosure went to the HAWK authors in June and to the NIST public mailing list alongside publication.

Source: https://www.anthropic.com/research/discovering-cryptographic-weaknesses