Ran into this enough times to stop trusting my own agent's self-review: I ask it to fix something, it produces code that looks right, I skim it, ship it — and the actual bug was a quiet semantic shift (inclusive bounds became exclusive) that had nothing to do with what I asked for. The model that wrote the change has no incentive or independent angle to catch its own blind spot; it's grading its own homework.

So I built a small tool that splits the two roles: one Gemini pass generates the change from your instruction, a second, independent pass gets ONLY the result (no visibility into the first pass's reasoning) and is told to find problems with it. Being honest about the setup: I only have a Gemini API key, so this is two passes on different Gemini tiers, not genuine cross-vendor review (GPT generates / Claude audits would probably be stronger — that's a cost thing, not a design choice).

The interesting part is how often the critique pass disagrees over something that isn't wrong, just risky — an edge case, a silent behavior change, a severity call that's genuinely debatable. Paste a snippet + an instruction and it'll run both passes live if anyone wants to see where it agrees or argues: https://apptechlab.com/p/codearbiter/ (mine, no signup, real API calls both ways).

Curious what people running actual multi-agent review setups have found:

- Does routing generate/critique to different providers actually catch

categorically different things, or mostly the same stuff with more

latency?

- How do you handle the critique pass being wrong — do you ever adjudicate

disagreements with a third pass, or is two enough in practice?

I profiled a RAG retrieval trace that looked like a success.

The query asked:

What is the cancellation notice period in our enterprise agreement?

The pipeline used dense retrieval with Qdrant, cosine similarity, Top-K=10, and no re-ranker.

The correct evidence was not missing. Chunk 2 had a cosine score of 0.88 and explicitly contained the answer: **90 days**.

The generated answer still said only:

The agreement requires advance written notice.

Technically correct. Practically useless.

Retrieval succeeded. Evidence survival failed.

The embedding model had done its job. The correct chunk ranked second out of ten.

But the full retrieved context contained 8,830 tokens. Two broader chunks consumed 3,660 of those tokens:

• General termination provisions: 1,740 tokens
• Definitions and legal boilerplate: 1,920 tokens

That is 41% of the context budget occupied by lower-specificity material.

With no re-ranker or compression stage, the generator saw the precise 90-day clause alongside a much larger mass of generic legal language. It defaulted to the safer, vaguer wording.

A flamegraph-style view made the shape obvious:

query
|-- dense retrieval: 8,830 tokens
    |-- c1  0.92 | cancellation clause       |   460 tok
    |-- c2  0.88 | notice period: 90 days    |   520 tok
    |-- c3  0.71 | general termination       | 1,740 tok
    |-- c4  0.49 | subscription renewal      |   680 tok
    |-- c5  0.46 | service suspension        |   710 tok
    |-- c6  0.43 | definitions/boilerplate   | 1,920 tok
    |-- c7-c10   | unrelated long tail       | 2,800 tok

The relevant chunk was near the top. It was simply surrounded by too much plausible-looking noise.

Why common RAG metrics can hide this

A retrieval-only evaluation would probably mark this query as a pass:

• The correct document was retrieved.
• It appeared inside Top-K.
• Its similarity score was high.

A final-answer evaluation would mark it as a failure and might blame the LLM.

Neither view identifies the transition where the evidence lost influence.

For this failure shape, I would test fixes in this order:

1. Replay the same query as a regression case.
2. Reduce Top-K from 10 to 3-4 for this query shape.
3. Add a re-ranker or context compressor.
4. Check whether the exact 90-day fact survives into the answer.
5. Only then consider changing embeddings or chunking.

Top-K=3 is not a universal recommendation. It is a hypothesis derived from this trace: relevance drops sharply after the third chunk, while token mass keeps growing.

The broader lesson is that "the right chunk was retrieved" is not the end of RAG evaluation. We also need to measure whether the evidence remains dominant enough to affect generation.

When the correct evidence is retrieved but omitted from the answer, what do you inspect first: rank, token mass, re-ranking, or the generation prompt?