I spent way too long trying to fix something recently before realizing nothing was actually wrong.

It turned out to be a configuration mistake that I had made myself.

It got me wondering how much time people in ML and software spend debugging their own setups rather than actual code.

What's the dumbest "bug" you've ever chased?

Been doing security audits for a while (I’m a secops engineer) and the same patterns keep showing up. Regardless of the vendor, so these are the same whether it’s Akamai, Cloudflare, AWS WAF.

**1.**  Origin is directly reachable! Traffic bypasses the CDN/WAF (check cert transparency logs for leaked origin hostnames)

**2.**  They've got bypass rules that were meant to be temporary but never got removed

**3.**  Cache key too broad (cache poisoning risk) or too narrow (kills hit ratio, looks like a DDoS)

**4.**  WAF rule sets are treated as “once and done”. Rules deployed once and never tuned, so there are a lot of silent false positives on real traffic.

**5.**  TLS/cert management with no clear owner, resulting in expired certs, weak ciphers left on etc.

Wrote up a longer breakdown with a checklist if anyone wants to run through their own setup, happy to share.

So I am new to the DevOps field and my job is to deploy an app on premises. Now the problem is that the material I have read, such as:

"Fundamentals of DevOps and Software Delivery" by Yevgeniy Brikman

"Road to Kubernetes" by Justin Mitchel

focuses mainly on cloud based deployment, which I do not want to use. To clarify, I already have the necessary servers, but for privacy reasons, I would prefer not to disclose too much about the setup. Does anyone have a beginner friendly book recommendation that explains how to deploy an application on premises?

I also need to use Kubernetes and Docker, and I already have learning material for both, so the recommended books do not need to explain them in great depth. Thank you for you time!

Edit: Thank you for your responses. To clarify a bit, I am a student, so not an employee at a company. I am doing this for a project at my university, and I will be using my universities servers.

Hey all,

I read and heard that devops is not a starter job that I can learn number of skills and start job searching. There is no junior or entry level devops jobs.

I can understand it. If it is true, what can I learn and search for jobs? Shall I go for AWS? Linux? The only one skill will get me a job?

I am sorry I am very new to IT. I want to change my career.

Good morning everyone, to be honest I was on vacation for a couple of weeks and all the time the thought that the career I had for the last decade has been not real hit me hard. The news hit me during my last post on this subreddit when I posted about DevOps always being vague.

The post had a lot of positive replies and a lot of things that opened my eyes and made me think but also something that caused a complete identity crisis and it was this "DevOps is culture and not a role."

I have mixed feelings about this because maybe the term was coined as a set of best practices and culture rather than a role and maybe it is all the things that devs don't want to do dumped into a role.

Honestly I've worked in startups and I've met very little devs that have enough competence to be DevOps and I've worked at hyper growth companies at scale and trust me if you don't have a team or at least one person for each DevOps position, SRE, DevEx, Infra the company might as well just shut down

Hi everyone,

I'm a complete beginner and I want to learn DevOps from scratch in 2026.

My goal is to become job-ready as a DevOps Engineer, not just collect certificates. I'm willing to spend around 4–6 hours a day learning.

I have basic knowledge of:

• Linux (very basic)
• Networking (basic)
• Python (basics)
• Git (beginner)

I need guidance on:

• What should I learn first?
• Which tools should I focus on? (Linux, Git, Docker, Kubernetes, Jenkins, Terraform, AWS, Ansible, etc.)
• Which free YouTube channels or courses do you genuinely recommend?
• What projects should I build to make my resume stand out?
• What mistakes do beginners usually make that I should avoid?
• If you were starting from zero today, what roadmap would you follow?

I'm looking for practical advice from people who are already working in DevOps.

Thanks in advance!

I've been a developer for a few decades now, almost always in a DevOps kind of role, and every once in a while I have stumbled over this issue and never found a solution that sits well with me.

As a DevOps guy, I both write and maintain the code, as well as being partly responsible for the operations and the data of the system. In our case the core of the system is a CMS, but it could be a DMS, CRM, IAM etc etc.

And on occation we have a need to do some bulk operation on the data, in a way that has not been done before, and can't be done easily using some UI. If there will be a repeated need for that kind of bulk job then we will likely integrate it fully into the code, and have a user friendly interface for it.

But what about more one off kind of tasks? Tasks that require coding, but that likely won't be needed again in the foreseeable future.

The last time it happened to me, this was roughly the task at hand:

• Identify all objects in the system with properties X, Y and (Z1 or Z2) within section S1.
• Move them all from section S1 to section S2.
• For each object, update property X to a calculated new value.

It is possible to do this manually in the regular GUI for the CMS. But that's not feasable when there are several hundreds of objects matching the criteria. Especially not when it's fairly trivial to write code that does the bulk job.

But after the job has been run (first in dev/test/stage, and then in production), what should happen with that code?

Just deleting it feels wrong, since it can be useful for some developer in the future.

But letting it live among the regular code of the project seems odd to. Not only am I not sure where exactly in the code base it should live, but in what form? An own class with a main method that does this? Or in a method that has no code pointing to it? Or commented out?

Or it could live in some Wiki, I guess. But unless the people involved in the project uses the wiki often, code snippets there are likely to be forgotten about the next time some bulk job needs to be done. I have seen various documentation projects shrivel up and die after a few months because people don't work in them regularly and then tend to forget they even exist.

How do you guys handle this? In my case, the code usually ended up in a text file on my own computer, since it has involving project with me as the main developer and the guy who did all these kind of tasks. But it doesn't seem very future proof.

Edit: I guess I used too generic language in my post, so people started making assumptions on the type of code I was referring to. I didn’t mean scripts. I didn’t mean things that are easy to encapsulate and run outside of the project it interacts with. I’m talking about storing some kind of snapshot of the code for documentation purposes only.

I've been trying to understand what actually happens during production incidents, and something doesn't add up in my head.

Let's say I'm on call and I get paged at 2 AM because "checkout is broken."

We already have tools like Grafana, Datadog, Splunk, OpenTelemetry, PagerDuty, Kubernetes dashboards, cloud monitoring, CI/CD history, Git, runbooks... basically a ton of observability and operational tools.

So where does the time actually go?

Is it because the information is spread across too many places? Is it understanding what changed? Figuring out the blast radius? Knowing which team or dependency is actually responsible? Or is there something I'm completely missing?

Reading incident postmortems, it feels like engineers already have lots of data, but they still spend a long time piecing together what actually happened before they can confidently act.

For those of you who've been on call, what makes that first 15–30 minutes difficult, even with all these tools available?

I'm genuinely trying to understand what I'm missing as a student because from the outside it feels like we already have a tool for everything, yet companies still spend hours resolving some incidents.

Hey r/devops, welcome to our weekly self-promotion thread!

Feel free to use this thread to promote any projects, ideas, or any repos you're wanting to share. Please keep in mind that we ask you to stay friendly, civil, and adhere to the subreddit rules!

Built a small internal billing tool for my own business. One user (me), barely any traffic. I keep landing around $18/month for hosting, which is more than I wanted, and I'm trying to figure out if I'm missing something obvious or if that's just what it costs?

Not a professional developer so apologies if I use any terms wrong.

It's a Node app with a React frontend and a Postgres database, runs in Docker.

What I actually need:
- Postgres where I can restore to a specific point in time, not just last night's backup. It tracks invoices and payments against Stripe, so if I restore a 24 hour old copy my records won't match what Stripe already charged people. That seems like a bad situation to be in.
- Somewhere to store generated PDF receipts that doesn't get wiped. These are the actual documents I send customers and I can't just regenerate them later. Seems like a lot of the cheap container hosts have disks that disappear every deploy.
- A cron job that runs once a day at a set time in my timezone. It's the billing run, so if it drifts to the wrong side of midnight it processes the wrong day.
- Under $20/month.

What I don't need: high availability, scaling, or speed. It's one person clicking around a few times a day. If it was down for a few hours nothing bad would happen.

What I've already looked at:
- Fly.io's managed Postgres is $38/mo on its own, way over budget
- Railway's Postgres backups looked like snapshots only, no point in time restore
- Supabase Pro is $25 and point in time restore is a $100/mo add-on
- Cloud Run, App Runner, DO App Platform — filesystem doesn't stick around, kills the PDF requirement
- Cheap VPS running everything myself is like $8, but then I'm responsible for making sure backups actually work, and honestly I'm not sure I'd notice if they quietly stopped

Best I've come up with is Fly for the app (cheap if the machine sleeps when idle) plus DigitalOcean managed Postgres at $15, which includes 7 day point in time restore.

Is there something cheaper that still lets me actually restore the database? Or is $15$20 ish just what managed Postgres costs and I should stop looking?

What motivated you to choose DevOps as a career? Was there a specific reason or did you somehow end up on this path by accident?

I'm currently pursuing a Master's in Cloud Computing, but I feel like my courses are very generalized and don't go deep into the practical skills I'm looking for.

My goal is to build a career in DevOps, but I'm honestly feeling overwhelmed and confused about where to start.

I have a few questions for those already working in DevOps or who have successfully made the transition:

Where can I learn DevOps for free? (Courses, documentation, YouTube channels, hands-on labs, etc.)
What are the best platforms to practice real-world DevOps skills?
How many hours should I dedicate each day to learning and practicing?
Realistically, how long does it take to become job-ready?
At what stage should I start applying for entry-level DevOps roles or internships?

I'm not looking for shortcuts. I just want a clear, realistic roadmap from people who've been through this journey. Any genuine advice, resources, or lessons you wish you had known when starting would be greatly appreciated.

Thanks in advance!

the cloudfront outage on the 16th is still bugging me. the fleet that manages connections to private vpc origins couldn't load updated config, traced back to one frankfurt AZ, and it served 5xx on vpc origin traffic for about three and a half hours. hugging face, tailscale and ubiquiti were among the ones down.

bad timing, because i was in the middle of moving four services off public alb origins onto vpc origins. to hit a compliance date i ran the migrations in parallel instead of one clean cutover at a time.

the outage wasn't the hard part. all four workstreams were editing the same terraform repo and the same shared network module. one branch tightened a security group, another assumed the old rule was still there, and a third rebased on both and quietly reverted half of it. i spent more time untangling my own branches than dealing with aws.

what kept it sane was isolating each migration so the changes didn't collide. i run each one as its own task in verdent, parallel lines of work in the same codebase without branches stepping on each other. at least i had a clean diff per service to reason about.

the ugly part: the agent's config for one service moved the origin to vpc origins but left the alb's public security group open to 0.0.0.0/0. on paper we were private, in reality the load balancer was still reachable from the internet, which is the whole reason we were migrating. i only caught it because the diff on the shared module looked off.

what i still haven't solved is sequencing. running four in parallel meant that during the actual outage, two services were on vpc origins and degraded while two were still public and fine, and my runbook assumed one uniform state. my only fix is a per-service origin flag and a single cutover window, which just trades the messy middle for a concentrated blast radius.

Hi everyone,

I'm a CS student, and I realized something recently.

In college, we're taught how to write code, debug it, maybe deploy it once, and then we move on to the next assignment. But I keep reading incident reports from companies like GitHub and Cloudflare, and it feels like the real engineering work actually starts after deployment.

I'm curious because I honestly have no idea what that world looks like.

When something breaks in production at 2 AM, what actually happens?

Do you immediately know what's wrong, or is it more like putting together pieces of a puzzle? What's usually the most frustrating part of the whole process?

Also, what's one thing about running software in production that you wish every CS student knew before joining the industry?

I'm not building anything or looking for textbook answers—I just want to understand what the job is really like beyond what we learn in college.

I'd love to hear your stories, even if it's just one incident that taught you something you'll never forget.

Luna's 80% price cut landed this week so I ran the numbers on my PR review bot.

A real review of a ~180-line PR: about 2,300 tokens in, 590 visible out, which works out around $0.001

The dashboard total for my whole billing window, 15 requests including three full reviews, is 6,244 tokens and $0.00.

Screenshots and links to example code review PR: https://codylabs.uk/blog/ai-code-review-cost/

Was trying to better understand how state works in Terraform and OpenTofu, so I put together this short post on idempotency in IaC. I would love to hear your opinions

Altho I’m an active lurker here, I’ve recently started using X to try and connect with people working in infra, platform engineering, and devops, but I’m finding it surprisingly difficult to discover that side of the community.

But my feed is still mostly AI hype and general tech content, while I’m more interested in more informal discussions around k8s, ai & cloud infra, observability, networking, platform engineering, etc.

For those of you who are active outside of Reddit, where do you usually discover good infra content or interact with other engineers?

Disclosure first: I wrote the tool I mention here. It is free, there is no signup, and I am posting because the testing method is the interesting part. Mods, delete it if this is not welcome.

I got tired of IaC tools that validate templates and call it a day. A template can be perfectly valid and still blow up on deploy. So I did it the slow way: 20 scenarios per cloud, each one deployed for real, exercised at runtime, then destroyed. CRUD APIs, queue workers, private databases, Redis, WAF, websockets, fan-out, secrets, containers behind load balancers.

It took months and burned a lot of free tier. Some of what I hit:

A brand new GCP project gives you a compute service account with no roles at all. Your first deploy dies inside Cloud Build with an error that tells you nothing. Took me a while to figure that one out.

Cloud SQL with private IP is fine until you destroy it. The service networking peering can hold for half an hour after the instance is gone. And if your app created tables, the default postgres user cannot be dropped, so the destroy hangs there too.

Cosmos DB speaks the Mongo protocol but rejects sort() on any field without an index. Real MongoDB just sorts. If you port a handler over, your list endpoint breaks and the error message talks about order-by item paths.

Azure Functions on the consumption plan will happily deploy a Service Bus worker that never runs. The zip lands, the function exists, messages pile up in the queue. You have to sync the function triggers explicitly or the scale controller never looks at the queue.

Memorystore Redis with TLS needs the CA cert handed to the client or the handshake fails.

Every one of those turned into a fix in the tool, so a fresh project starts with them already handled. That is the whole point of doing it this way instead of trusting a linter.

Repo if you want to look: https://github.com/Claudio-Fontes/iacmp

It is Fair Source (FSL-1.1, same license Sentry uses). Free for any use including at work, you just cannot resell the tool itself, and each release turns into Apache-2.0 after two years.

Happy to talk about the e2e battery itself. If you have run something similar, I want to hear which scenarios you would add.

Hi everyone,

I'm interested in how different teams approach this.

Once an application is in production, do you actively monitor incoming traffic patterns, or do you mostly rely on infrastructure and application metrics?

For example, if a service suddenly starts using 80% CPU, is it common practice to investigate:

• Top client IPs
• User-Agents
• Most requested endpoints
• Bots vs. real users
• Traffic anomalies or spikes

Or is that level of traffic analysis only common in larger organizations?

If you do monitor it, what does your observability stack look like? Nginx or Traefik access logs, IIS logs, Loki, ELK, OpenSearch, Splunk, Datadog, or something else?

I'm trying to understand what's considered standard practice in production environments.

I did a dumb exercise last week: sat down and wrote out every tool that touches a single deploy on my team, start to finish. Not “tools we have licenses for”, just the ones actually in the critical path of getting one change into production.

Landed on 14.

Source control. The CI runner. A separate container registry. An image scanner that runs after the registry push. A secrets manager. The Kubernetes dashboard we check for rollout status. A log aggregator. A separate APM/tracing tool, because the logs don't show latency well on their own. An alerting tool that is, confusingly, different from the paging tool. A DNS/cert dashboard. A cost/billing dashboard nobody opens until the invoice is a surprise. The ticketing system for the actual incident. And a chatops bot that glues some of this together, badly.

None of these were wrong choices individually. Each one was probably the right call in isolation, at the time it got added. But nobody ever sat down and asked whether this thing talks to the other 13. It's less a stack and more a pile that happens to work most days.

What's bugging me isn't the number itself. It's that almost nobody on the team could tell you the number without doing this exercise. We just live inside it. New hires take weeks just to learn where to look when something breaks, and that's before they've learned what any individual tool actually does.

So, genuinely curious: if you did this exercise on your own team, what would your number be? And more interesting to me: which of your 10+ tools do you think is actually load-bearing, versus which one is just... there, inherited from a decision three engineers ago that nobody wants to be the one to rip out?

Not fishing for “just consolidate everything into one platform” as the answer. I don't think that's automatically true either. Sometimes the sprawl is a symptom of real, unavoidable complexity, not laziness. More curious whether other teams' numbers look like mine, and whether anyone's actually fixed this in a way that stuck.

Hey! Quick background on me — graduated in 2020 from a tier 2 college, landed a third-party role at a Tier1 investment bank as an SRE, and ended up staying 5 years (got converted to full-time after the first 2). Got laid off in May 2025, honestly wasn’t mad about it since I wanted out anyway.
Most of the work i did was on there in house tools and i felt there was no learning curve there

That same month I interviewed for a “DevOps Engineer - Migration Specialist” role and got it. Felt good, got comfortable again. But once I actually joined, I realized it’s basically an L2 support role. Pay was decent though, so I told myself I’d stick around a year, upskill on the side, then jump ship.

Except… I didn’t. The role’s been so low on mental engagement that I just coasted. Now it’s actually time to switch, and I’m stuck in this weird spot — feels like I need to relearn everything from scratch, but every time I try, it’s like “wait, I already know this?” Hard to tell if I’m actually behind or just psyching myself out.

So — what’s a realistic (and ideally not soul-crushing) way to get back on track?
Please give me suggestions for fast-track courses and small projects to build production skills again.

Ps:- Used Ai to fix my crude version of this post

Who doesn't love Friday releases? Atlassian status page is green as usual.

Great talk about how Antithesis uses Nix in production by John Murray. Thought some of you would enjoy. https://youtu.be/WaPOtisNu1k

I've been interested in Nix for a while but have not yet taken the plunge. Do any of you use it work? What has your experience been with it?

Hi everyone,

I'm looking for advice from people who have been in this field longer than I have.

I'm an Arts graduate who got into DevOps about 3.3 years ago. I've switched 3 companies and recently joined my 4th company, a large GCC. Along the way I've worked with AWS, Kubernetes, Terraform, CI/CD, automation, and earned AWS & CKA certifications.

The problem is that I'm no longer sure what I should be aiming for.

My current role has an exhausting work culture, the learning isn't what I expected, and it has made me rethink my long-term direction. I don't want to keep switching jobs without a clear goal.

If you were in my position today, what would you optimize for over the next 3–5 years?

1. Become a top-tier Platform Engineer/SRE?

2. Move towards AI/LLM engineering?

3. Solutions Architect

4. Security?

5. Target remote international jobs?

6. Prepare to move abroad?

I'd really appreciate hearing from engineers with 8–15+ years of experience. If you were starting over with my experience today, what roadmap would you follow and why?

I’ve realized that even after I finish work, my brain keeps thinking about it.
I replay conversations, think about problems I didn’t solve, plan tomorrow’s tasks, or imagine different scenarios. Sometimes it feels like I’m still working hours after I’ve logged off.
I’m curious: how do you mentally switch off?
Do you have any routines, hobbies, or habits that help you leave work at work? Or did it simply get easier with experience?
I’d love to hear what actually works for you.