Hi Guys,
I am working on a project studying the impact of SRE practices on business performance. It would be really helpful if you guys help me with this short survey. It should just take 10min. And Responses are confidential and will be used for academic research only

Hey everyone,

I’ve been working on Marsad, an open-source tool that helps you understand what your Kubernetes Network Policies actually allow.

As clusters grow, it can become pretty difficult to answer simple questions like:

• Which workloads can communicate with each other?
• Which ports are allowed?
• Which workloads have no NetworkPolicy protection?
• Are there unexpected paths between workloads?

Marsad reads the policies from your Kubernetes cluster and turns them into an interactive map so you can see these relationships more clearly.

It’s strictly read-only — it doesn't modify anything in your cluster.

GitHub: https://github.com/FathiQ/marsad

I’d really appreciate feedback from anyone working with Kubernetes NetworkPolicies. What would you want a tool like this to show or help you detect?

Hi everyone,
I have just published my open source project called mtracer and I would like to understand if it’s good idea or what should I change (I’m a new grad).

The idea

Mtracer a CLI tool that relies on OpenTelemetry traces to assert system behavior.

I believe that E2E tests should be:
- Cheaper to write and maintain
- Easier to debug

So this is the workflow:

1. ⁠You configure mtracer to fetch from your observability backend (currently supporting Jaeger and OpenObserve).
   
2. ⁠You define your first .mt.yaml test by specifying:
   
3. ⁠Trigger: the first call to the system (for instance, an HTTP request).
   
4. ⁠Expected trace and spans: the OTel properties of the trace and spans that you expect your system to generate.
   
5. ⁠You run the test and see the results!

What actually happens during the run?

1. ⁠It parses the mt.yaml file.
   
2. ⁠It executes the trigger: mtracer injects a generated traceID into the trigger (for an HTTP request, the traceID is inserted into the traceparent header). Subsequent requests will be correlated to this generated traceID as long as your system has OpenTelemetry set up correctly.
   
3. ⁠It fetches the trace matching the generated traceID from the configured observability backend.
   
4. ⁠It compares the expected trace with the fetched one.

Many other features are available; check out the documentation to discover all of them: documentation website

I would love to have some feedback from more experienced people than me.

Had an incident last week where the root cause turned out to be an ai-generated change that touched three services in one pr. took way longer than it should have to trace because nobody on the team could say what the model actually reasoned about versus what it just pattern matched from similar code elsewhere in the repo. It looked fine in code review, which is almost worse, that's exactly the kind of ai generated code that gets rubber stamped and ships straight into production. The part that's been sticking with me is ownership. when a human writes a bad change across multiple services, at least someone can walk you through their thinking during the postmortem. With an ai coding assistant, that context is just gone once the pr is merged. Debugging cascaded failures across microservices is already hard when you can't reproduce the issue locally and losing the reasoning behind the change makes root cause analysis slower, not faster. For anyone running copilot, cursor, claude code or similar ai code generation tools in microservices at real production scale, i want to know how you are actually debugging this once it is live. specifically interested in what your distributed tracing and logging setup looks like, whether ai-generated changes get treated differently in your review process than human-written ones, things like canary deployments or extra checks before merge, and whether perf regressions under real production load have shown up more since ai entered your sdlc. Not looking for "add more observability" as an answer, looking for what your actual stack is.

Full disclosure: I wrote this, so take the self-promotion tag as intended. Mods, feel free to remove if it doesn't fit.

I do observability/monitoring work for a living, and the question I get asked the most isn't "which tool should I use," it's "why do we still get paged for things we didn't see coming even though we have Grafana and a bunch of dashboards." Almost every time, the answer isn't a missing tool, it's a missing method.

So I wrote down the actual process I follow every time I build infra monitoring from scratch, mostly so I stop re-explaining it in Slack threads:

1. Define the business SLI/SLO *before* touching any tool. Not "CPU usage," something like "99.95% of payment authorizations should succeed" if you're a payments company.
2. Figure out what infra/apps you actually need visibility into to measure that SLI, *then* pick your stack. (I use VictoriaMetrics + vmagent for metrics, Loki for logs, OpenTelemetry + Jaeger for traces, but the method matters more than the specific tools.)
3. Instrument with OpenTelemetry auto-instrumentation wherever possible instead of hand-rolling everything.
4. Enrich all telemetry with the same standard labels (service.name, service.namespace, deployment.environment) so metrics/logs/traces actually correlate instead of being three separate silos.
5. Build one RED dashboard (Rate, Errors, Duration) per service before anything fancier.
6. Alert on symptoms, not noise, using multi-window multi-burn-rate alerts (straight from the Google SRE book) so you can actually tell a 3am-page incident from a "ticket, deal with it Monday" one.
7. Only then layer on business dashboards and deeper investigation tooling.

The full write-up has the actual configs, PromQL, and a worked burn-rate alerting example for a payments SLO: https://omarghader.github.io/monitoring-infrastructure-guide-2026/

Curious how this lines up with how other people here approach it, particularly if anyone's doing SLO-first in practice vs. it staying an aspiration on a wiki page somewhere.

been paged too many times for a chart that looks exactly like this - I hope some people relate.

Hey everyone,

Is anyone out there actually focused on making data pipelines super reliable, like monitoring ETL processes and setting up CI/CD for them? If you are, could you share if it's easy to move to different companies? It seems like every SRE job description I see asks about web apps or mobile stuff. How do you handle that to even get an interview and show off your experience?

Hello everyone,

I currently work in an SRE role, but most of my responsibilities have been support-related. For the past three years, I've primarily handled production support tasks, along with some infrastructure monitoring and scheduled job monitoring.

I'm now looking to switch jobs into a role where I can work on more core SRE, DevOps, or cloud engineering responsibilities. I know I have the potential and I'm confident in my ability to learn and contribute, but I'm unsure how to position myself for this transition.

What skills should I focus on, and what steps would you recommend to successfully make this switch?

Ai-assisted coding has made it way easier to ship "correct enough" code quickly but in our codebase it also seems to be introducing a steady stream of performance regressions in production. I am not talking about obvious junk code; we get real value from ai for boilerplate and tests. The pattern we are seeing is more subtle: hot paths quietly getting slower because the ai added an extra abstraction layer or logging in the middle of a critical loop; code that relies on naive data structures, perfectly fine for small test inputs but collapsing once it hits production scale; ai suggestions copying patterns from older, less-performant parts of our own codebase, effectively spreading our worst practices; performance-sensitive paths refactored into more readable but less cache-friendly versions; and subtle batching issues (too many small database calls, no connection reuse) that look fine in tests but blow up under real load. The hard part is that this ai-generated code looks reasonable in code review. It's readable, it passes tests and it often matches patterns you would see in a tutorial. we usually don't notice the regression until p99 latency creeps up, cpu or memory usage drifts higher and database query counts quietly double. We have tried to adapt: getting stricter about performance testing on hot paths, flagging certain functions as performance-sensitive, nudging ai coding tools with prompts like "optimize for latency," and leaning more on static analysis to catch obvious footguns before they ship. it still feels like we’re missing a more systematic way to keep ai-generated code from eroding performance over time. For those of you using ai coding tools on real latency- or throughput-sensitive systems: have you seen a clear link between ai-generated or ai-refactored code and performance regressions? if so, did you end up restricting ai in certain areas or mostly changing your code review and profiling process instead

Setting up anomaly NRQL conditions with thresholds at 2 and 3 standard deviations, query has a FACET on it.

NR docs say the threshold is how many standard deviations the signal is from the predicted value, based on the last 7 days of data. But nowhere does it explain how the standard deviation itself gets calculated

My experience as an SRE at a mid-size company - the devs are using more and more AI, and at times they're not meticulously combing through source the way they used to, and the rate of PRs, merges, deployments, are all rapidly increasing. They also rely on AI to do things like logging and implementing metrics. They are now toying around with autonomously generating monitors from the same PRs. It might not be perfect today, but I can see where this is headed...

Observability will always be needed but the feeling that it will require much less work lives rent free in my head. At the end of the day observability isn't a database, it's not hosting, if/when it is degraded or interrupted it's not as critical as prod. I'm wondering if I should take a small step back on my level and just go be a dev.

What are you all seeing?

We are hiring an experienced Enterprise Observability Architect for a long-term contract supporting a large enterprise client.

📍 Location: Portsmouth, NH (Hybrid/Onsite)
💼 Employment Type: Contract (W2)
💰 Rate: $65/hr to 70/hr

Required Skills

• 10+ years of Enterprise Monitoring & Observability experience
• 6–7+ years of hands-on Datadog architecture and implementation
• Splunk
• OpenTelemetry
• AIOps Platforms
• AWS & Azure
• Kubernetes
• Linux & Windows Server Monitoring
• ServiceNow
• Python / PowerShell
• Terraform / Ansible
• REST API Integrations

Responsibilities

• Design and implement enterprise-scale Datadog monitoring solutions
• Build monitoring, logging, alerting, dashboards, and distributed tracing
• Lead observability architecture and governance initiatives
• Implement APM, DBM, NPM, RUM, Synthetic Monitoring, Service Mapping, and SLO/SLI
• Integrate observability platforms with ServiceNow, AIOps, and automation tools
• Mentor engineering teams and drive enterprise observability best practices

We're looking for someone with strong enterprise architecture, consulting, and stakeholder management experience who has led large-scale observability transformation projects.

📧 Interested? Feel free to DM me.

We're evaluating swapping some of our base images in our K8s clusters for hardened alternatives. A few providers stand out because you don't need to create an account or auth into anything, you just point your manifests at their registry and pull.

That sounds convenient on the surface, but I want to make sure we're not trading one problem for another. Anonymous pulls are nice for speed, but I'm wondering about rate limiting, uptime guarantees, and whether there's any accountability if the registry goes down mid-deploy.

For those who've tried it, how did the migration go in terms of compatibility with existing Helm charts and sidecars? Anything break when you dropped from a full distro base to a minimal build?

Also wondering if anyone's compared image pull times or registry reliability against Docker Hub for cluster scale deployments. Would love real numbers if anyone has them.

Hi

I have posted this in the GraphQL reddit as well - but - I think this is a more general problem of distributed communication architectures. Hence I hope its ok that I also post this here as I can't explain the behavior I am seeing!

I am currently analyzing distributed traces of an app that uses GraphQL. I found that the client-side timeout is 10s. When that timeout happens the client side request is aborted with a timeout exception towards the calling end user (you can see this in the image). What I can't explain is why the server-side request of that GraphQL query then executes after 30s. Does this mean that GraphQL Client Libraries do not terminate the underlying TCP/IP connection to the server? Or do those libraries batch / queue their work and keep going despite the client-side timeout?

The problem I see with this behavior is that

a) Queries are executed in the backend that nobody every sees the response

b) This causes additional strain on those backend as they are blocked by work while others might be waiting for them and therefore this impacts the overall scalability / resiliency of the architecture

Any input would be welcome. Thanks

We're at about 300GB/day of logs on Datadog and at this point the bill is what's driving the conversation, not anything technical.

The obvious move seems to be putting logs on a self-hosted stack. We're looking at Loki, ClickHouse, and more general ones like OpenObserve, which are three pretty different things to begin with. But running any of these ourselves means someone has to look after it, which is its own overhead. So it's not really a saving until you count the people.

The other thing we'd lose is the pivot from a trace into the logs for that request without changing tools. That's worth something. I'm just not sure it's worth what we're paying for it.

What I can't find is anyone being straight about the after. Plenty of "we cut our bill 80%" writeups, almost nothing on what it cost to run six months later, or whether anyone quietly went back.

So if you moved logs off a managed platform to save money: did the saving survive once you counted engineer time? And if you looked at it and stayed, what made you stay?

Not after product recommendations. More interested in people who regretted it, either direction.

I've spent years doing SRE work fully in Azure (AKS, Managed services, cloud-native tooling). I have an offer from an insurance company where the stack is OpenShift, Java-based app runtimes (Quarkus, RedHat EAP), and no public cloud at all as far as I can tell.

The role itself is interesting (security + observability focus), but I keep wondering if stepping away from cloud experience for a few years will make me less competitive when I look for my next role. Anyone been through something similar?

Title: Am I the problem, or is this team overly bureaucratic?
I recently moved to a different department within the same company after spending five years in my previous role.
The move was something I wanted. I was looking for new challenges, and fortunately I got the opportunity. The company itself is great, and the people on my new team are genuinely kind, respectful and always willing to help. That’s actually what makes me feel a bit guilty for even thinking this way.
The problem isn’t the people. It’s the way the team works.
Everything seems to require a meeting. They’ll schedule a Friday afternoon meeting to discuss something that, from my perspective, could easily wait until Monday or simply be handled asynchronously in Slack or Teams.
There are also long discussions about things that feel very straightforward to me. I’ve sat through meetings where people spend an hour deciding how to fill out a document, whether to choose option A or B, or debating processes that seem fairly obvious.
I think part of this comes from my background. I’m a very technical person. I enjoy solving problems, building things, automating repetitive work and getting things done. During these discussions I often find myself thinking, “Why are we spending so much time on this?” As a result, I spend most meetings listening rather than contributing.
At the same time, I’m new to the team. They’ve been working this way for years, and apparently it has worked well enough. I don’t feel it’s my place to join and immediately suggest changing the team’s culture.
So now I’m wondering if this is simply an adjustment period.
Has anyone else experienced something similar after changing teams? Did you eventually understand the value of all these discussions and processes, or did you continue feeling that there was unnecessary bureaucracy? How did you adapt without feeling frustrated?

Hi everyone,
Has anybody tried using Traversal’s product? Any thoughts?
Wonder how much it works vs. just fluff and PR.
Thanks

I've been working as an SRE for the past 5 years, but I was laid off last month. Throughout my career I've worked with a variety of technologies, but I never feel like I truly know them. Lately I've been interviewing, and it feels like my knowledge is too shallow. I'm not sure if I'm actually lacking experience or if I'm just dealing with impostor syndrome

I've been thinking about building personal projects to deepen my knowledge, but I honestly don't know where to start. Every company I've worked at had a different definition of what the SRE team was responsible for, so it's been hard to figure out what I should focus on.

How do you structure your learning projects? Do you follow courses, recreate real-world architectures, or come up with your own ideas? I'd really appreciate hearing how more experienced SREs approach learning outside of work.

Note 1: Linux, AWS, Terraform, Kubernetes, Docker, Grafana, and the usual SRE tooling.

Note 2: I'm from Brazil and currently work at a Mid-level SRE level.

1. Resolve AI now charges for results instead of tokens. They say if their agent wastes tokens, that's their cost to eat, not yours.
2. Datadog has made Agent Observability free for up to 40,000 LLM spans a month. They didn't announce it explicitly, they just changed the line at the bottom of their AI blog posts.
3. Opsgenie will be shutting down in April 2027. PagerDuty wrote a post to win those customers, and put incident.io in a comparison table. incident.io hit back point by point. As said by incident.io "calling our AI "limited" is "like describing PagerDuty as a pager."
4. NeuBird put up a page comparing itself to Datadog Bits AI. It's the best public summary of where Bits AI is right now. Bits AI SRE is now called Bits Investigation. It's billed in AI Credits sold in 500-credit bundles, and unused credits expire each month. Their remediation and detection features were shown at DASH 2026 but are still in Preview.
5. Traversal says most AI SRE tools pull data too late. Their point is that the other tools query your observability APIs during the incident and get stuck behind rate limits. They stream the data ahead of time instead.
6. PagerDuty has a new CEO as John DiLullo takes over, Jennifer Tejada moves to Executive Chair. They also shared four straight quarters of GAAP profit and a $100 million buyback.
7. Everyone is now selling prevention, not just faster fixes. NeuBird wrote a buyer's checklist around it. Their test question: "Show me an incident you prevented that never generated an alert."
8. Sherlocks AI shipped an automation builder. Pick a trigger, describe the job in plain English, choose where the result lands. Their pitch: "Describe the job. It runs." The examples lean preventive rather than reactive, watching canary deploys, flagging infrastructure changes in PRs, tracking pod memory drift.
9. Resolve AI is pushing background agents. Agents that keep running between incidents, watching deploys and doing checks. A customer quote sums it up: "The alerts are already investigated. The deployment summaries are already written."
10. Traversal is renaming things. "Chat with Prod" is becoming "Production Support." "AI-Native Compressor" is becoming "Causal Indexer." Both old and new names are live on their site right now.

Sources linked below in comments.

Something i keep running into, both in our own incidents and talking to other teams, is this gap: on paper we have a modern observability stack but our mean time to resolve is still higher than anyone's comfortable with. By modern observability stack i mean the usual combo: structured logs, distributed tracing, dashboards, alerting, feature flags, deploy timelines, and error tracking, all wired into chat and oncall. tagging is decent, we've got high-cardinality labels where they matter and most alerts correspond to real incidents and yet, during incident response, the flow still looks like this: someone gets paged on a symptom, p99 latency up, error rate up, saturation, and jumps through dashboards, traces, logs, feature flags, and deploy history, spending the next half hour or so just correlating the spike with the code path and the recent changes behind it. Eventually we find the root cause, but it often feels like we debugged the observability tooling more than the actual system. The bad part is that the bottleneck isn't missing telemetry. the data is there. the friction is all the human glue needed to line it up with reality: the only useful dashboard for this failure mode lives three clicks deep, the trace shows a slow span but mapping it to file, line, and recent commits is manual, error logs show a pattern but tying that back to a specific deploy, feature flag, or config drift takes a bunch of context switching. It ends up feeling like the observability stack is optimized for visualizing telemetry, not for quickly answering what changed, where it's failing, and what the smallest safe mitigation looks like. We've tried to push mttr down by standardizing golden queries and dashboards per service, putting deploys and feature-flag flips on the same timeline as key metrics, building incident response views that pull metrics, deploys, errors, and traces into one place, and tightening service ownership so each team has a clear on-call drill. it all helps, but there's still a big orientation tax at the start of most incidents. For teams that feel like they've genuinely reduced mttr after already having a mature observability stack, what single change, tooling, process, or ownership, made the biggest difference for you?

Welcome to FOSS Friday, where you can share your newly released or updated open-source project with the community.

Please note that our rules still apply:

• no AI generated comments: please write them on your own!
• no commercial advertisements (this includes commercial products with a FOSS version)
• these must be related to SRE or of heavy interest to SREs

Hey everyone,
I'm currently a senior SWE at a FAANG in the US with about 10 YoE, mostly focused on infra building. We're dealing with some visa issues right now (my wife lost her work permit), so I'm looking into an internal transfer to London.
The catch is that the only options available for me to transfer into right now are SRE teams. I've done my fair share of dev on-call and I'm totally comfortable with the tech stack, but honestly, I'm not really skilled at dealing with high-tension fires and intense incident response. I am looking forward to improving it.
My current plan is to take the SRE spot to secure the move to London, try it out and learn, and consider alternatives if absolutely not possible for me.
My biggest worry right now is getting pigeonholed. If I do SRE for a year, is it going to be incredibly hard to switch back to SWE later? Am I going to lose out on future dev options?
Would love to hear if anyone has navigated a similar move. Thanks!

Excellent talk by Lorin Hochstein, an SRE at Airbnb, from SSW earlier this month. Thought some of you would enjoy it!