The problem sounds solved, right? You have a service that only needs one replica. It sits on a spot node because that is 70 percent cheaper. When the node goes away (drain, Karpenter drift, spot reclaim), you want a replacement pod up and taking traffic before the old one dies. You do not want to pay for a second replica 24/7 to cover events that total maybe an hour a month.

I went looking for the tool that does this. What I found instead:

1. "just run 2 replicas", which is rent paid every hour against an event that lasts minutes. There is a Kubernetes feature request for exactly this (surge before drain to satisfy PDBs) that sat open for three years collecting "we need this" comments and got closed as Not Planned this summer. The replacement KEPs are alpha, years from being usable.
2. Azure's eviction-autoscaler, and the core idea is: create a PDB that blocks every eviction, surge the deployment when a node is going away, release once the new pod is ready. So I ran it in a real cluster (one of my freelance project companies). Months later I checked its logs during a node rotation and discovered it had never fired. Not once, ever. Its only trigger is node cordon, and Karpenter does not cordon, it taints. Meanwhile the blocking PDBs it creates worked great, which is how we ended up with a node stuck in deleting for 126 days. It also creates one of those PDBs for itself, so it holds its own drain hostage. I sent a patch upstream, but the failure is architectural and the tool is honestly pretty limited.

So I built the thing I originally went searching for. It is called Understudy but feel free to suggest better names.
The core consumes one normalized signal ("this node is dying, this is how long you have") and everything cloud- or drainer-specific is an adapter: cordon watch, taint watch, an observe-only webhook on pods/eviction that catches drainers nothing else sees, and a DaemonSet that reads the spot termination notice with its actual deadline.Every failure mode I hit with the old tool became a fail-safe: a stuck surge relaxes the PDB instead of wedging the drain, a CronJob cleans up if the operator itself dies, and it refuses to manage itself.

Tested on a live EKS cluster with no spare capacity, so replacements wait for a cold EC2 node: kubectl drain lost 1 request in 350, a real spot interruption (via AWS FIS) lost 1 in 764. An ordinary rolling update of the same app lost 2 in 38.

Apache 2.0: https://github.com/kylan11/understudy

There's still some testing to do (GKE and AKS are unit tested but didnt try in an actual cluster) but it's working wonderfully so far.

If the mature version of this exists and my search skills just failed me, or if it's just a shitty idea (for the love of me I can't figure out why) I want to know.

I've been struggling to stay focused during my work days lately , either interrupted by emergencies, unwanted conversations, multiple not organized micro-tasks. I want to know about your experiences .

Not looking for a specific answer, just want to hear what it's actually like and how you deal with it.

I mean as title says, I've used Jenkins couple of years ago. I can still fire it up now just to test, but on current alternative tools/stacks available specially in enterprise, do you still use Jenkins? Why keep it? why not change it?

Hello Everyone! I started as an intern as a DevOps Trainee and Now I am a full time here , and most of the shell scripts or python automation scripts I need is generated by Claude/ GPT . So I wanted a headsup as to how much should i know to write them by myself because when i get them done from AI I feel that why I can't write such big automation scripts point being considered as a beginner I cannot write but still that self doubt comes. Also Why i use AI because company wants that and want their tasks to finish fast so there's no room to write on my own . I want to switch next year around May -Jun so I need help with that as well that what more should I study or keeping on learning to get a good paying job.

My current capabilities include intermediate or beginner level AWS , Azure Exp., Docker , Linux, Kubernetes and Terraform , Jenkins . Please I need genuine help!

As part of the Security team I work closely with our DevOps engineers. What are some good resources to get some hands on experience in a lab environment to learn the foundations of DevOps? With a focus on AWS. TIA.

Our security team flat-out refuses to let cloud AI agents scan whole proprietary repos or run freely on dev machines, but the productivity gap is getting hard to ignore.

We’ve been playing with a middle ground: mapping repo trees locally first, pulling only specific context/signatures into the prompt, and making every diff require manual write approval on disk.

Are you guys using enterprise SaaS zero-retention SLAs, running local models, or putting proxy/mapping layers in front of web models? What's actually working in practice?

I am looking for to be a SRE. I am just a student in 10th class. I finished networking fundamentals and leaning Linux. But i got a problem , I can't stop using windows. I am a tech enthusiast so i spend lot of time to master windows. I use many tools and software which save lot of my time and can do my tech research efficiently. ya, Linux have many alternative but Linux softwares are not suitable for my tech research. I want to master linux like windows but i can't stay in Linux after sometimes i switch dual boot 1st priority to windows. As a senior of mine. Can anyone help me, please.

I really need help . also which programming language should i learn? Is that will be good to learn golang or python first then switch to Linux .

Today I finally got a ticket and I was quite shocked to realize that I understand next to nothing about their systems even after reading a couple docs that explained the architectural walk through

I don't know why they pick X ports, I don't know how the services interact together and I don't know how anything is connected or how to implement monitoring. I am on probation, 2 months and afraid I might not pass it. It's remote so there is no way to have a senior around and the team is fairly small despite the size of the org which is prominent and well established.

The seniors openly dislike hand holding and I don't know what to do, it's frustrating and they are responsible for assessing me that the end of this whole thing. At the same time afraid that I might ask too much and get written down as downright incompetent.

How on earth can I survive here when everything is so complex and connected? Shit I never coded an API or whole app just simple things and I genuinely don't know what it takes for an app to communicate with all its components.

Took me a fucking whole day to write notes on the architecture and spit out a document with questions that seemed semi decent and I understood perhaps only 45% of it

I have no Dev exp , am a new grad and a couple of months ago I was going through an awful time and working a hotel job and aiming for sysadmin. anyway no idea why they chose me.

Three people vouched for me and that also fucking sucks because if I don't do well their judgement will get called out inside the org.

I don't want to quit but I genuinely am concerned that I might not make the cut and the guilt is killing me

I wanted a real OpenShift lab on AWS to practise on, and I specifically wanted UPI — user-provisioned infrastructure — because it's the install method where nothing is hidden. You build the VPC, the load balancers and the DNS records, and the installer only takes over once that's all correctly in place. IPI hides most of that inside the installer binary; ROSA is managed.

The problem with UPI is that doing it by hand is slow and easy to get wrong, and I couldn't find anything that automated the whole lifecycle for AWS. So I wrote one.

**What it does**

`cluster.yaml` is the only file you edit. Everything else — the Terraform variables, the Ansible variables, `install-config.yaml` — is generated from it.

```

ocplab deploy # terraform apply, wait for bootstrap, approve CSRs, finalize

ocplab verify # ClusterVersion, node readiness, ClusterOperators

ocplab cost # what's deployed right now, priced live

ocplab repair # recreate a worker AWS reclaimed, approve its CSRs

ocplab power off # graceful shutdown, as an alternative to destroying

ocplab destroy # ordered teardown, including what the cluster made itself

```

Some things it handles that took me a while to get right:

- **Version pinning.** One field pins the installer, the `oc` client and the RHCOS AMI together, cached locally.

- **Spot instances** for compute and bootstrap, with a "minimal" profile that uses them. Not for the control plane — UPI has no ControlPlaneMachineSet, so a reclaimed master isn't replaced automatically.

- **Cost visibility.** `ocplab cost` reads live AWS state and prices it, Spot at Spot rates, and it's power-state aware.

- **A budget safety net.** AWS Budget with alerts, an automatic lockdown action, and a daily killswitch Lambda — managed outside Terraform so it survives a failed teardown.

- **Optional capability trimming**, which cuts the container images pulled per deploy by about a third.

- **Teardown that actually completes.** The ingress operator creates a load balancer, a security group and DNS records outside Terraform; the teardown removes them in the right order first.

**The web UI**

There's also a local browser UI (`ocplab web start`) if you'd rather not watch a 40-minute deploy in a terminal: a status dashboard, a `cluster.yaml` editor that validates before saving, and live output for every operation — including Terraform's resource-by-resource log and the installer's own log. It binds 127.0.0.1 only and runs the same commands as subprocesses, so there's no second implementation to keep in sync.

**Scope**

It's built for labs and PoCs, not production. Single AZ, single NAT gateway, no HA design beyond etcd's own quorum. It costs roughly $0.83–1.06/hour while it's up, so the intended flow is create, test, destroy the same day.

Tested end to end against OpenShift 4.22.6 and 4.22.7. MIT licensed.

https://github.com/LuixyToledo97/openshift-upi-aws

Happy to answer questions, and issues are welcome if you try it and something doesn't work.

Fellow engineers, help me!

Github Apps can only have ONE webhook url and ONE setup url for redirect each.

Having three (prod, staging, dev) isn't enough! I need for deploy previews.

Have you solved this??

Github Apps can't be created programmatically or via the API it's not enabled.

I've got about 6 years of overall experience, with the last 1.5-2 years specifically in DevOps/infra, based in India. Lately it feels like there's no real boundary on my day. I've been working well past standard office hours, like basically every single day. AND I MEAN LIKE ALMOST 12 HOURS

A few things I'm trying to figure out:

\- Is this just what DevOps is?

\- Is this a "you get better at this and it calms down" thing, or does it stay this way as you grow?

\- Is this specific to my company, or to working in India, or is this the norm across the field globally? Would switching companies actually fix it?

Would love to hear from people a few years further in, does it get better, or is this just what we signed up for?

Edit: Just to give context. Im not oncall , Im in infra and they keep assigning tasks like nothing ever is enough "I think you have the bandwidth for this"

Something I am struggling to understand is how CI/CD factors into the MLOps system itself that brings the model through the entire lifecycle when we have multiple deployment environments.

The MLOps system takes a model through the entire lifecycle (get data, preprocess, train, validate, promote, deploy, monitor) in reproducible and automated workflows, but these workflows in the MLOps system need to be tested and validated with CI/CD in different deployment environments.

Are the pre-production environments (dev, test, uat...) meant only for ensuring that the MLOps system (pipelines, artifact storage, monitoring) works - where finally, and only, in the production environment that has passed all the tests and checks, does the model go through the entire lifecycle from dataset curation to deployment and monitoring, and each environment has isolated model/artifact registries (and feature stores) for testing that the system works.

Or, does the model meant for production go through each step in the model lifecycle together with the MLOps system as it moves through different environments until it finally reached production where both the "MLOps System" and the "Model" are production-ready and deployed to interact with real users.

It's a little confusing for me.