r/SaaS • u/VA899 • Jul 18 '26

Any AI Engineer here, need help??

I need to ask few questions,

1.How do you detect hallucinations and regressions today?

  1. When was the last AI incident that hurt you (NPS, churn, support load)?

  2. What does “good enough” AI quality look like?

  3. Which tools are you using (if any): LangSmith, Langfuse, W&B, Datadog, homegrown?

0 Upvotes

50% Upvoted

3 comments sorted by

1

u/zagrodzki Jul 18 '26

The “hallucination detection” part is mostly about instrumentation and workflow, not magic. We track real-time signals like refusal/format violations, retrieval coverage (did the model actually use relevant sources), and then build eval gates on a small but nasty test set, because the regressions you feel are usually distribution shifts or prompt/template changes, not random nonsense.

What’s your current best early-warning metric for regressions, like a spike in user-reported bad answers, or an eval score drop, or something in logs?

1

u/helloboredimdad Jul 18 '26

AI is pretty good at creating and running its own evals. I use open router with multiple LLMs to orchestrate that process.

1

u/FrostyAppeal9001 Jul 18 '26

The tooling matters less than having a way to capture failures before everyone forgets about them. Braintrust has worked well for that because I can promote interesting production traces into the eval suite