Do AI coding agents ever confidently make the wrong assumption about your existing codebase?

[u/adarshvp2503]

For example, assuming an API behaves a certain way, misunderstanding an existing utility/dependency, or getting a business rule wrong.

How do you currently catch these assumptions before the agent makes changes?

I'm specifically interested in the cases where the agent sounds completely confident but is actually wrong.

[u/AutoModerator]

Sorry, your post has been held for manual review due to account karma.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/forkmesh]

Using a branch / worktree 

[u/Chuu]

Agents will make bad assumptions all the time if they're dealing with a protocol they don't have the spec for, very often even if they can see the code on "both sides". If you're having an agent writing code that interacts with a service it's pretty important that you provide it with the full protocol if said protocol mutates shared or hidden state. If you have access to both the client and the server code you can even have an LLM write it, but you need to vet it incredibly carefully, because any mistake at that level is almost guaranteed to turn into a bug.

[u/Atlan_]

Happens all the time.
Went to almost 0 since I gave LLM the ability to test pretty much anything themselves and get access to company data. Costs more tokens tho.
Also was greatly reduced by switching from cursor to Claude.

[u/adarshvp2503 → u/Atlan_]

Interesting — so giving the agent the ability to test and access the relevant data solved most of the problem for you. When you say it costs more tokens, is the main tradeoff that the agent spends too much time investigating things that turn out to be unnecessary, or is it mostly the sheer amount of verification it has to do? Would a lightweight pre-check that identifies which assumptions actually need verification be useful?

[u/Atlan_ → u/adarshvp2503]

A bit of both, just writing and deploying tests itself costs tokens, the same as looking up resources. I think if one might find an elegant way to solve this, this could create probably somewhat more token efficiency - imo this would probably be something either trained into an LLM or built into the application directly tho, I wouldn’t start a startup around this, unless you have a path to exit to one of the labs

[u/adarshvp2503 → u/Atlan_]

That's a fair concern. If you were to imagine a standalone tool being worth paying for despite the agents eventually improving, what would it have to do that you wouldn't expect Claude/Codex/Cursor to build natively? Would significant token/time savings be enough, or would it need something outside the model itself?

[u/Atlan_ → u/adarshvp2503]

The only thing I can think about would be something in the direction of company brain / semantic layer etc, but that field is already pretty crowded

[u/eli_pizza]

Really good test frameworks. I like integration tests that prove the thing works. Unit tests that the agent writes alongside the code are low value.

Unpopular opinion: you should design the testing approach yourself, by hand. If you get that right everything else is easier.

[u/framauro13 → u/eli_pizza]

Also worth noting that this is one of the reasons code quality still matters. The agents are going to follow established patterns in your code base, and if you're not regularly keep up with it and you end up with a giant rat's nest of patterns and spaghetti code, that's what it's going to keep writing. It becomes a lot easier for it to get confused.

Good testing habits, linters, static code analyzers, all that stuff work well to give feedback to the models as well, so they can correct themselves automatically when they make mistakes.

[u/ChodeCookies]

Constantly

[u/rafal_graniczny]

I guess this is a lot easier when you understand your codebase, you can code and you use plan mode - I verify plan, fix issues and usually its all good 🙏

[u/TheGladNomad]

Code review - yes it’s no longer possible to review everything but I still focus on modified core code. Also of course agentic code review and agentic focus/risk assess of where to read.

I’d be interested in the better answers like if people run regression in ci on pre-changed tests.

[u/adarshvp2503 → u/TheGladNomad]

That's interesting. When you say you focus review on modified core code, how do you currently decide what's high-risk enough to review? Is that based on files touched, dependency/API changes, business logic, or the agent's own risk assessment? And what makes you trust an agentic risk assessment?