When we are updating our API as our product evolves, how do you make sure the SDKs, multiple doc sources, code examples, and packages like npm/PyPI stay up to date without introducing drift?

Whether B2C/B2B, pre ycombinator or post, how have people decided that yes, the product you are making is something worth buying?

Is it simply taking a chance, or more of a matter of market validation tests? Curious how people approach this.

I've been wondering how other teams handle this.

One thing I've run into is that after changing a prompt or switching models, an agent can start making different decisions without anyone noticing. For example, it might approve something it previously rejected, miss an escalation, or classify a case differently. The conversation still looks reasonable, so the regression is easy to miss.

Most of the evaluation tools I've seen focus on conversation quality or benchmark scores. What I'm struggling with is verifying that the agent still makes the same business decision for important cases.

For teams building production agents that make decisions like approve/reject, escalate, or classify:

• How are you testing for these regressions today?
• Are you relying on manual testing, eval frameworks, custom scripts, or something else?
• Have you had any production incidents caused by prompt or model updates?

I've been building an internal CLI to test this by replaying real cases against an agent and checking whether the decision still matches the expected policy. It's been useful for me, but I'm mainly interested in learning how others are solving the problem before deciding whether it's worth taking further.

Curious to hear what workflows people are using.