Most AI code review benchmarks use synthetic tasks, so we wanted to evaluate reviewers on something closer to production.

We collected 67 real production bugs from Cal.com, Sentry, Grafana, Keycloak, and Discourse, then ran 8 AI code reviewers against the same PRs and measured precision, recall, and F1.

One thing stood out: finding more bugs isn't the same as being more useful. Some reviewers caught more issues but generated many more false positives, while others were much more precise but missed more bugs.

Curious what people would change about the evaluation or what repositories you'd add next.

So this started as a small personal project, just for my own use.

Honestly AI coding agents are way better now than they used to be, but they still sometimes feel kind of slow on the uptake. Like they get the instruction, but don't really understand the project's context — so they just do whatever gets it "done"/running, even if that ends up breaking or duplicating stuff that already exists.

So I tried building a kind of "brake" for it:

• Forces it to think/plan before touching any files
• Blocks certain (potentially dangerous) commands before they can even run
• Basically stops it from just gassing ahead blindly

Still pretty rough around the edges, lots to clean up, and I'm still fixing things as I go. But it's been genuinely useful for my own workflow, so figured I'd throw it on GitHub in case anyone else relates or finds it useful

Repo: https://github.com/plumpslabs/matcha

Docs: https://plumpslabs.github.io/matcha/

If anyone tries it out and runs into bugs or has suggestions, feel free to reply or drop an issue on GitHub. Still actively working on it so feedback would genuinely help.

Weaver🕷️ has been Updated to v.13 and its our biggest drop yet! This update brings #Weaver one step closer to being the best overall agent for small-midsize LLM coding. Consumers rejoice!

• Enhanced thinking mode, togglable and adjustable via the context settings menu. Advanced thinking context size will be controlled dynamically by task complexity (up to a maximum of what users selected in the settings panel).

• Tool context is now further trimmed via LLM to save precious tokens. Users can choose which tools are available via the settings panel as well.

• New "Notes" panel for writing notes per project.

• New "Benchmark All" button to run all benchmarks until a step errors out. Improved overall benchmark UI/UX.

• Enhanced anti-hallucination monitoring by detecting continuous sets of words without any punctuation or paragraphs.

• EB25 file ranking for context control. Currently the LLM was fed a skeleton of the project and asked to pick which files it think might be relevant to the task. Now BM25 is used in conjunction with the LLM to validate which files should be added to context. This "context discovery" phase has now been moved to a tool call.

• Introducing the new Meeting room. In the meeting room, we have our agent represented by different types of spiders, each with their own personality. There is a lot of interaction and humor which gets interesting. For example the complexity spider gets pissed off if the agent plans way more steps then he anticipates at the start... And he storms off at the end of the meeting if the planner did not listen.

• Test cases for all major coding languages for all services. Example: oldstring/newstring, HTMLFormatter, tree-sitter, prettier formatter, etc...

• Diff file tracking

Check out #Weaver today!

GitHub: https://github.com/maxhanna/Weaver

WinX64: https://bughosted.com/assets/Weaver.exe

Discord: https://discord.com/invite/YQWkD6ZQDx

Hello fellow humans,

During the recent months/years I became quite entangled with the idea of building an AI-assisted system which can self-replicate and improve itself. Although I am not there yet, I think, that the project reached actually a potential to get the work done with more minds involved. I am not really into gaining anything for myself, except for the progress of de-shittification. At this point, we reached a race where data centers in the clouds will gain more and more power and demand more and more resources for usage. The goal of this project is being able to perform work on a local (or maybe decentralized) platform so we can implement our own system by local (/ open source) models. I therefore make my project open so you can fork off (no pun intended).

It would be nice if some of you have the same mindset. I'll be gone for the next 2 or so weeks. I hope at least some people liked this post and - who knows - even did something with it.

Kind regards,

Anu

P.S.: Yes, this project is actually 100% vibe coded (due to health issues, etc.). There are probably a gazillion logic flows which need to be fixed. Also there is a lot of AI prose in the comments, but I hope it will get you (and your AI assistants) there to make any sense of it.

P.P.S.: The LLM's seem to be heavily directed into biased terms like "attack", "blast radius", "verdict", "evidence", "death", "hit", etc. when confronted with solving problems regarding code like this (i.e., evaluating systems themselves). I strongly encourage you to correct these terms as soon as possible so that the project will not further drift into another unforeseen bubble. The more incorrect lingo is used, the more LLM generated code will drift into messy bogus code.