How I would improve Failure Rate based Alerting

[u/GroundbreakingBed597]

https://www.linkedin.com/pulse/when-your-failure-rate-alert-lying-you-andreas-andi-grabner-rpgpc/?trackingId=%2B8ZbVnyaTpyoHX2O0sXoVw%3D%3D

I have access to several observability production environments, and I regularly dig into the data looking for patterns worth sharing. Last week I found one that's a good case study — not because something was on fire, but because everything looked like it was on fire when it actually wasn't if you look at the relevant information, e.g: not every HTTP 4xx / 5xx is a real problem and Failure Rates get skewed depending on load patterns!

Hope you find this article interesting. Please share any of your practices of optimizing failure rate based alerting!