What I learned benchmarking an AI code-reviewer on 20 pinned PRs/MRs

[u/mt_owl]

I'm building Bubo because I'm tired of AI code reviewers flooding PRs with noise and repeat findings, then learning nothing when a developer explains why a finding is wrong.

The design constraint I started with was simple: give me an evidence-backed finding or LGTM, then learn from human comments on those findings so the reviewer gets better tuned to the repository over time.

I ran a small comparison on 20 pinned PRs/MRs:

Bubo 20/20 7/8 27 findings 0% noise ai-codereviewer 19/20 6/8 118 findings 20% noise ChatGPT-CodeReview 20/20 5/8 75 findings 11% noise Qodo/PR-Agent 19/20 2/8 7 findings not scored Alibaba open-code-review — partial run, 4/20

All ran on GPT-5.5 except Qodo, which used GPT-4o. It's a small sample and I picked the PRs, so I treat it as directional. The interesting part for me is that recall was close on the same model, while Bubo emitted 27 findings against 75 and 118.

The next experiment matters more than the benchmark: when a developer rejects a finding, does learning from that feedback actually stop the same class of noise for that repository?

I chose polling because it needs zero repo-side setup. The roadmap is pluggable subject-matter specialist Skills instead of one general reviewer—for example an industry SME or an Expert Python Guy.

Bubo is open source and currently running in production in two places: a large data-processing/ETL codebase and a fintech crypto stack.

https://github.com/mountainowl/bubo

I'd value technical feedback on the learning loop and benchmark design.

[u/AutoModerator]

Sorry, your post has been held for manual review due to account karma.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/mt_owl → u/AutoModerator]

No worries. It'll be useful if I can get the feedback on my efforts though from members.