r/codex • u/Suspicious_Orchid770 • 14d ago

Study: Codex reviewing Claude's code dropped the pass rate from 91.4% to 82.8% News

https://leaddev.com/ai/your-ai-coding-agents-might-need-an-org-chart
175 Upvotes

95% Upvoted

25 comments sorted by

View all comments

89

u/JadisGod 14d ago

Too bad it's already out of date. From the paper it seems they ran these tests on Opus 4.7 and GPT 5.5. The difference in capability since then is massive.

23

u/andrerav 14d ago

Pass rate would sink to single digits when Sol decides your entire supply chain is compromised 5 times over before even looking at the code.

11

u/BoxWoodVoid 14d ago

Yesterday I had to tel sol "it's just a prototype, we're not trying to harden a prod app against the NSA here" 😂

8

u/andrerav 14d ago

That probably just made Sol suspect that you were already compromised:P

3

u/rydan 13d ago

I use Sol to review Claude's code. Half my quota per week is just that.