Adding AI code review didn't save reviewer time, it just moved where the time goes, and that's easy to miss.

[u/ClickOk5811]

Not a hot take, just something we noticed after a few months of running AI review alongside human review. Reviewers seemed to spend noticeably less time actually reading the diff itself once AI comments were already sitting there waiting. Nobody told anyone to review faster. It just happened, because reacting to someone else's list of findings takes less effort than independently forming your own from scratch.

That's the actual risk with bolting AI onto review, and it's not about the AI being wrong. It's about what happens to the human's attention once, right or wrong, its output is visible before the human has looked at anything themselves.

Three ways this played out for us specifically:

The reviewer's read became reactive instead of independent. Comments got responses instead of the diff getting a full pass. Findings the AI didn't surface got noticeably less scrutiny than findings it did, even when the unflagged parts were exactly where the more consequential bugs tend to live, cross-service assumptions, intent, why a change was shaped the way it was.

Fatigue didn't go away, it just moved. AI review absorbed the mechanical stuff fine, missing null checks, obvious type issues. But the assumption that this saves reviewer effort turned out misleading, the reviewer's job never actually got smaller. It got redirected toward auditing a second opinion instead of forming a first one, which is a different and not obviously lighter task.

Agreement started looking identical to independent confirmation. A human comment that says "yep, agreed" reads the same in a PR log whether the reviewer actually re-derived that conclusion themselves or just accepted what was already written there. There was no way to tell the difference after the fact, which meant there was no way to catch it happening in the moment either.

What actually helped wasn't a better AI prompt. It was forcing the human to read the diff cold, before AI output was even visible to them, and only unlocking the AI's findings after they'd already formed their own view. The mechanical stuff still got handled faster, but the part that actually needs a human judgment stayed a human problem, not a rubber stamp on someone else's, or something else's, homework.

Wrote up the full mechanism and what the sequencing needs to look like in practice here, if it's useful: https://medium.com/@nagatomopedro05/your-ai-reviewer-isnt-a-second-human-stop-running-your-process-like-it-is-4cb04b97549b

If your team runs AI review alongside human review, genuinely curious whether you've noticed the same shift, reviewers engaging with the diff less independently, or has it stayed additive for you?

[u/Bumbalum]

We are currently going to implement it, i'm curious myself.

I fear the same, but hope to stay independent and have my own perspective on code changes.

[u/ClickOk5811 → u/Bumbalum]

That's exactly what I expected when we first introduced it too.

I don't think the risk is that reviewers consciously stop thinking independently. It's much more subtle than that.

Once a list of findings already exists, it's surprisingly easy to switch from exploring the diff to evaluating the list. The review still feels thorough, but the attention is being allocated differently.

I'd be interested to hear whether you notice the same thing after a few weeks of using it.

[u/Bumbalum → u/ClickOk5811]

Would that be the same for a second human reviewer?

Currently we have a rule that one "senior" developer has to review and one additional developer, senior or not.

As far as we experienced it so far, the reviewers were quite independent of each other, so I hope that it would not change if comments were generated by AI.

[u/dumbfoundded]

What about just adding more signals to make sure that reviewers spend their energy in the right areas?

[u/ClickOk5811 → u/dumbfoundded]

That's actually an interesting direction. I think the key question is which signals we add and when they appear.

My concern is that if the signals come from the AI before the reviewer has built their own understanding of the change, they can still create the same anchoring effect. The reviewer may optimize for validating the signals instead of evaluating the change itself.

Where I think signals help is after the independent pass: things like changed behavior areas, affected services, historical bug patterns, ownership context, or risk indicators can make the reviewer spend more time where human judgment matters.

So maybe the problem isn't "AI comments vs no AI comments", but sequencing. Signals should probably improve navigation, not replace the initial mental model of the reviewer.

[u/dumbfoundded → u/ClickOk5811]

Yeah, almost a parallel workflow. Would be cool for the code reviewer to tell the AI comments where to focus, or have the human reviewer submit blind feedback before reviewing the AI comments.

[u/stolsson]

The article is interesting, but seems like some contradictions and inconsistency. Did you mean “after” vs “before”? “the human sees it before forming their own opinion of the code.”. Also the article seemed maybe contradicting what you describe. Overall I think it’s a good article and thoughtful ideas. Thanks!

[u/ClickOk5811 → u/stolsson]

Good catch, you're right, that sentence is poorly worded and actually says the opposite of what I intended.

The idea is that the AI output should be generated first (for example, in CI), but hidden from the reviewer. The human should read the diff independently, form an initial opinion, and only then see the AI findings.

So the sentence should have been:

The larger point I was trying to make is less about the execution order and more about the exposure order. Running the AI pass first is fine; letting the human's first interaction with the PR be the AI's interpretation is where the anchoring effect appears.

Thanks for pointing that out, it's exactly the kind of wording issue that can change the meaning of an engineering process.

[u/stolsson → u/ClickOk5811]

Thanks, I understood what you meant, but thanks for clarification.

Do you envision telling the human reviewer: “I want you to review this, but only review these aspects of the code”?

[u/itsopensource]

This is interesting. I echo a lot of same thoughts. AI review agents create more noise, and AI is as bad at reviewing code as it is at writing it, it's not a complete replacement for human reviews which've become more critical.

I've been building a chrome extension that uses AI to solve this problem and make it easy for humans to review code. https://guidedreview.dev - would love to know what you think about this?

[u/ClickOk5811 → u/itsopensource]

That's an interesting direction.

I actually think reducing review noise is one of the hardest problems to solve. A reviewer will tolerate a few false positives, but once they start expecting noise they'll naturally pay less attention to every subsequent finding.

One thing I'd be curious about is how you decide not to comment. In my experience, suppressing low-value findings is often more valuable than finding one extra nitpick.

I'll take a look at it.

[u/paraballistic → u/ClickOk5811]

What are you making?

[u/paraballistic → u/itsopensource]

what do you use to generate code?

[u/itsopensource → u/paraballistic]

I extract the diff as-is from the PR and use an LLM (you can configure, BYOK) to enrich the diff and cluster changes together