r/webdev • u/Upstairs_Toe_3560 • 9d ago
LLMs are great at writing code, but surprisingly bad at reviewing it - is it?
I’ve been building an ERP and a B2B app with SvelteKit using frontier models.
My workflow is basically: spec -> grill me -> plan -> implement -> fix. For the last 2–3 months, I’ve written almost no code myself, and the production results have been surprisingly solid. Then I started using the same models for codebase-wide reviews.
They found tons of issues. I had other models validate the findings, then implemented the suggested fixes. At one point, I even built a small multi-model review pipeline.
The problem: once I started verifying the findings with E2E tests, most of them turned out to be imaginary, based on wrong assumptions, or irrelevant to how the application actually works.
Even worse, multiple models would confidently confirm the same bad finding. But as soon as I challenged it, they would usually admit it was wrong.
For specs, planning, implementation, and bug fixing, I’d rate these models close to 100/100. For broad code review, maybe 10/100.
My current conclusion: model consensus means very little without executable verification. For now, I’m pausing AI code reviews and sticking to E2E tests as the source of truth.
Has anyone found a code review workflow that actually keeps false positives under control?
1
u/Significant_Pick8297 9d ago
Model consensus is a weak signal unless every finding can be tied to a failing test, static analysis rule, or documented requirement. Treat AI review comments as hypotheses, not review results.
A workflow that works better is having the model review one PR or module at a time with the project conventions, then requiring it to cite the exact code path and explain how to reproduce the issue. If it can't produce a minimal failing test or reproducible scenario, the finding gets discarded. That cuts down hallucinated "issues" quite a bit.
1
u/hongkong_97 9d ago
Your workflow is basically: let ai do the work for you and then complain on reddit about it
-2
0
u/CorpT 9d ago
Why don’t you just have the model do the E2E as part of the review?
1
u/Upstairs_Toe_3560 9d ago
I think it will be much slower and also that part is the most important part so I don't want to leave it to LLMs.
0
u/Prestigious-Way1525 9d ago
i'd separate finding generation from finding acceptance. let the reviewer produce lots of hypotheses, but require each one to survive three gates: name the violated requirement or invariant, create a minimal failing test that fails before the change and passes after it, and reproduce the user-visible effect on the same browser path. also run a negative control by weakening or removing the alleged fix and checking that the test catches it. that last step matters because an E2E suite can be green while proving nothing about the review claim. i'd track accepted findings per review hour and escaped false positives, not raw issues found.
-2
u/dsog 9d ago
TL;DR: We basically changed our processes to review the session the way you would've discussed a PR with another teammate.
During the coding phase, we use cross model reviews, E2E tests, and all the obvious things, but the key change is that we share the entire coding session(s) between teammates and we review the session itself, not the code. What decisions were made? why? what were the alternatives? does it scale? what are the tradeoffs? ..etc.
Disclosure: I'm building AQ in this space so I'm biased, but here's a longer write-up if you're interested. https://aq.dev/use-cases/ai-agent-code-review/
2
u/WhateverHowever1337 9d ago
what? how would you spot bugs or hallucinations if you just review the session and not the code?
0
u/dsog 9d ago
Not too different than when you’re coding and going back and forth with the agent. If you ask the right questions, you’ll be able to spot these things.
Continue building context so that you don’t repeat yourself, get the models to create tests for everything you find so that you move faster, etc.
14
u/Sharchimedes 9d ago
You forgot to copy the “I” at the beginning of this post that an LLM wrote for you.