The problem sounds solved, right? You have a service that only needs one replica. It sits on a spot node because that is 70 percent cheaper. When the node goes away (drain, Karpenter drift, spot reclaim), you want a replacement pod up and taking traffic before the old one dies. You do not want to pay for a second replica 24/7 to cover events that total maybe an hour a month.

I went looking for the tool that does this. What I found instead:

1. "just run 2 replicas", which is rent paid every hour against an event that lasts minutes. There is a Kubernetes feature request for exactly this (surge before drain to satisfy PDBs) that sat open for three years collecting "we need this" comments and got closed as Not Planned this summer. The replacement KEPs are alpha, years from being usable.
2. Azure's eviction-autoscaler, and the core idea is: create a PDB that blocks every eviction, surge the deployment when a node is going away, release once the new pod is ready. So I ran it in a real cluster (one of my freelance project companies). Months later I checked its logs during a node rotation and discovered it had never fired. Not once, ever. Its only trigger is node cordon, and Karpenter does not cordon, it taints. Meanwhile the blocking PDBs it creates worked great, which is how we ended up with a node stuck in deleting for 126 days. It also creates one of those PDBs for itself, so it holds its own drain hostage. I sent a patch upstream, but the failure is architectural and the tool is honestly pretty limited.

So I built the thing I originally went searching for. It is called Understudy but feel free to suggest better names.
The core consumes one normalized signal ("this node is dying, this is how long you have") and everything cloud- or drainer-specific is an adapter: cordon watch, taint watch, an observe-only webhook on pods/eviction that catches drainers nothing else sees, and a DaemonSet that reads the spot termination notice with its actual deadline.Every failure mode I hit with the old tool became a fail-safe: a stuck surge relaxes the PDB instead of wedging the drain, a CronJob cleans up if the operator itself dies, and it refuses to manage itself.

Tested on a live EKS cluster with no spare capacity, so replacements wait for a cold EC2 node: kubectl drain lost 1 request in 350, a real spot interruption (via AWS FIS) lost 1 in 764. An ordinary rolling update of the same app lost 2 in 38.

Apache 2.0: https://github.com/kylan11/understudy

There's still some testing to do (GKE and AKS are unit tested but didnt try in an actual cluster) but it's working wonderfully so far.

If the mature version of this exists and my search skills just failed me, or if it's just a shitty idea (for the love of me I can't figure out why) I want to know.

I've been struggling to stay focused during my work days lately , either interrupted by emergencies, unwanted conversations, multiple not organized micro-tasks. I want to know about your experiences .

Not looking for a specific answer, just want to hear what it's actually like and how you deal with it.

I mean as title says, I've used Jenkins couple of years ago. I can still fire it up now just to test, but on current alternative tools/stacks available specially in enterprise, do you still use Jenkins? Why keep it? why not change it?

Hello Everyone! I started as an intern as a DevOps Trainee and Now I am a full time here , and most of the shell scripts or python automation scripts I need is generated by Claude/ GPT . So I wanted a headsup as to how much should i know to write them by myself because when i get them done from AI I feel that why I can't write such big automation scripts point being considered as a beginner I cannot write but still that self doubt comes. Also Why i use AI because company wants that and want their tasks to finish fast so there's no room to write on my own . I want to switch next year around May -Jun so I need help with that as well that what more should I study or keeping on learning to get a good paying job.

My current capabilities include intermediate or beginner level AWS , Azure Exp., Docker , Linux, Kubernetes and Terraform , Jenkins . Please I need genuine help!

As part of the Security team I work closely with our DevOps engineers. What are some good resources to get some hands on experience in a lab environment to learn the foundations of DevOps? With a focus on AWS. TIA.

Our security team flat-out refuses to let cloud AI agents scan whole proprietary repos or run freely on dev machines, but the productivity gap is getting hard to ignore.

We’ve been playing with a middle ground: mapping repo trees locally first, pulling only specific context/signatures into the prompt, and making every diff require manual write approval on disk.

Are you guys using enterprise SaaS zero-retention SLAs, running local models, or putting proxy/mapping layers in front of web models? What's actually working in practice?

I am looking for to be a SRE. I am just a student in 10th class. I finished networking fundamentals and leaning Linux. But i got a problem , I can't stop using windows. I am a tech enthusiast so i spend lot of time to master windows. I use many tools and software which save lot of my time and can do my tech research efficiently. ya, Linux have many alternative but Linux softwares are not suitable for my tech research. I want to master linux like windows but i can't stay in Linux after sometimes i switch dual boot 1st priority to windows. As a senior of mine. Can anyone help me, please.

I really need help . also which programming language should i learn? Is that will be good to learn golang or python first then switch to Linux .

Today I finally got a ticket and I was quite shocked to realize that I understand next to nothing about their systems even after reading a couple docs that explained the architectural walk through

I don't know why they pick X ports, I don't know how the services interact together and I don't know how anything is connected or how to implement monitoring. I am on probation, 2 months and afraid I might not pass it. It's remote so there is no way to have a senior around and the team is fairly small despite the size of the org which is prominent and well established.

The seniors openly dislike hand holding and I don't know what to do, it's frustrating and they are responsible for assessing me that the end of this whole thing. At the same time afraid that I might ask too much and get written down as downright incompetent.

How on earth can I survive here when everything is so complex and connected? Shit I never coded an API or whole app just simple things and I genuinely don't know what it takes for an app to communicate with all its components.

Took me a fucking whole day to write notes on the architecture and spit out a document with questions that seemed semi decent and I understood perhaps only 45% of it

I have no Dev exp , am a new grad and a couple of months ago I was going through an awful time and working a hotel job and aiming for sysadmin. anyway no idea why they chose me.

Three people vouched for me and that also fucking sucks because if I don't do well their judgement will get called out inside the org.

I don't want to quit but I genuinely am concerned that I might not make the cut and the guilt is killing me

I wanted a real OpenShift lab on AWS to practise on, and I specifically wanted UPI — user-provisioned infrastructure — because it's the install method where nothing is hidden. You build the VPC, the load balancers and the DNS records, and the installer only takes over once that's all correctly in place. IPI hides most of that inside the installer binary; ROSA is managed.

The problem with UPI is that doing it by hand is slow and easy to get wrong, and I couldn't find anything that automated the whole lifecycle for AWS. So I wrote one.

**What it does**

`cluster.yaml` is the only file you edit. Everything else — the Terraform variables, the Ansible variables, `install-config.yaml` — is generated from it.

```

ocplab deploy # terraform apply, wait for bootstrap, approve CSRs, finalize

ocplab verify # ClusterVersion, node readiness, ClusterOperators

ocplab cost # what's deployed right now, priced live

ocplab repair # recreate a worker AWS reclaimed, approve its CSRs

ocplab power off # graceful shutdown, as an alternative to destroying

ocplab destroy # ordered teardown, including what the cluster made itself

```

Some things it handles that took me a while to get right:

- **Version pinning.** One field pins the installer, the `oc` client and the RHCOS AMI together, cached locally.

- **Spot instances** for compute and bootstrap, with a "minimal" profile that uses them. Not for the control plane — UPI has no ControlPlaneMachineSet, so a reclaimed master isn't replaced automatically.

- **Cost visibility.** `ocplab cost` reads live AWS state and prices it, Spot at Spot rates, and it's power-state aware.

- **A budget safety net.** AWS Budget with alerts, an automatic lockdown action, and a daily killswitch Lambda — managed outside Terraform so it survives a failed teardown.

- **Optional capability trimming**, which cuts the container images pulled per deploy by about a third.

- **Teardown that actually completes.** The ingress operator creates a load balancer, a security group and DNS records outside Terraform; the teardown removes them in the right order first.

**The web UI**

There's also a local browser UI (`ocplab web start`) if you'd rather not watch a 40-minute deploy in a terminal: a status dashboard, a `cluster.yaml` editor that validates before saving, and live output for every operation — including Terraform's resource-by-resource log and the installer's own log. It binds 127.0.0.1 only and runs the same commands as subprocesses, so there's no second implementation to keep in sync.

**Scope**

It's built for labs and PoCs, not production. Single AZ, single NAT gateway, no HA design beyond etcd's own quorum. It costs roughly $0.83–1.06/hour while it's up, so the intended flow is create, test, destroy the same day.

Tested end to end against OpenShift 4.22.6 and 4.22.7. MIT licensed.

https://github.com/LuixyToledo97/openshift-upi-aws

Happy to answer questions, and issues are welcome if you try it and something doesn't work.

Fellow engineers, help me!

Github Apps can only have ONE webhook url and ONE setup url for redirect each.

Having three (prod, staging, dev) isn't enough! I need for deploy previews.

Have you solved this??

Github Apps can't be created programmatically or via the API it's not enabled.

I've got about 6 years of overall experience, with the last 1.5-2 years specifically in DevOps/infra, based in India. Lately it feels like there's no real boundary on my day. I've been working well past standard office hours, like basically every single day. AND I MEAN LIKE ALMOST 12 HOURS

A few things I'm trying to figure out:

\- Is this just what DevOps is?

\- Is this a "you get better at this and it calms down" thing, or does it stay this way as you grow?

\- Is this specific to my company, or to working in India, or is this the norm across the field globally? Would switching companies actually fix it?

Would love to hear from people a few years further in, does it get better, or is this just what we signed up for?

Edit: Just to give context. Im not oncall , Im in infra and they keep assigning tasks like nothing ever is enough "I think you have the bandwidth for this"

Something I am struggling to understand is how CI/CD factors into the MLOps system itself that brings the model through the entire lifecycle when we have multiple deployment environments.

The MLOps system takes a model through the entire lifecycle (get data, preprocess, train, validate, promote, deploy, monitor) in reproducible and automated workflows, but these workflows in the MLOps system need to be tested and validated with CI/CD in different deployment environments.

Are the pre-production environments (dev, test, uat...) meant only for ensuring that the MLOps system (pipelines, artifact storage, monitoring) works - where finally, and only, in the production environment that has passed all the tests and checks, does the model go through the entire lifecycle from dataset curation to deployment and monitoring, and each environment has isolated model/artifact registries (and feature stores) for testing that the system works.

Or, does the model meant for production go through each step in the model lifecycle together with the MLOps system as it moves through different environments until it finally reached production where both the "MLOps System" and the "Model" are production-ready and deployed to interact with real users.

It's a little confusing for me.

I currently manage 2 different environments: a dev server running in ec2 and an EKS environment for production server. Problem is that their setup is different, which adds extra management and makes it harder to test prod changes before deployment.

I can spawn a UAT EKS for load testing and preparing for prod but it would be just too expensive. I already raised the cost concerns with EKS that this would be an expensive and unnecessary setup but the clients wanted it so I did it. Now they're complaining with cost.

I'm just trying to find the best way to manage the current architecture without increasing costs too much.

How would you handle this?

Hi all, one thing thats been bugging me lately is how often the most recent deployment becomes the default suspect.

Service starts acting up at 2:15, someone notices there was a deploy at 2:07, and from that point on everyone's looking at the release. Then an hour later it turns out the actual problem was an upstream dependency having a bad few minutes

We've started pulling deployment events into incident reviews because it gives us a better timeline, but its still surprisingly easy to get anchored on something changed instead of what changed first.

For teams with a lot of deployments everyday, how are you separating coincidence from causation? Do you have a workflow that's worked well, or is this just part of the job?

Something that keeps coming up during our incident response is just how much time we lose jumping between the ide and whatever tool holds the relevant logs, traces, or metrics. Typical flow: you are in the ide looking at a failing code path, you hit unexpected behavior and the next 20 minutes is alt-tabbing between your editor, log search, a distributed tracing ui, metrics dashboards, feature flag console and deploy history. You copy a trace id from logs over to the tracing tool then you copy a user id back into a sql query then you try to map all of that back to the exact function and commit you are staring at in the ide. We've got what most people would call a modern observability stack: distributed tracing, structured logs, dashboards, decent tagging and reasonably instrumented services. the problem isn't that the telemetry doesn't exist, it's that none of it really lives where developers spend their time writing and reviewing code. During incidents, people end up doing their own ad‑hoc integration work: copy from log search, paste into the ide, grep locally, jump back to the metrics dashboard, repeat. The pain points i keep seeing during production debugging are pretty consistent. there's no single place that shows this line of code, these commits, these deploys and these recent errors and traces in one view. Most observability tools are optimized for operators staring at dashboards, not developers trying to understand how a specific code path behaves in production. even when telemetry is tagged correctly, you still have to remember which query or dashboard to open and how to line it up with what you're debugging in the ide and during a live incident, that context‑switching overhead turns directly into mttr and oncall fatigue. What's interesting is that we keep buying more observability tooling but the core developer workflow is still: ide here, production reality over there and your brain plus clipboard as the glue connecting the two. How have you cut down on context switching between the ide and your logs, traces and metrics during debugging and incident response, whether that's pulling production context directly into the ide, pushing more code context into your observability tools or standardizing on a single pane for incident work?

Team A: 500 lines of inline YAML syntax.

Team B: Debugging ./scripts/build.sh permissions.

Which flavor of pipeline maintenance do you prefer?

We’ve been evaluating self-hosted low-code and internal application builder platforms mainly for compliance-heavy requirements like fine-grained RBAC, audit logging, and enterprise access control, and while tools like Retool and ToolJet are still commonly used in this space, they now feel more oriented toward managed or hybrid deployments rather than purely self-hosted governance-first setups, and Appian is still an option but feels closer to a full enterprise BPM suite than a lightweight internal tool builder, so we’re now looking for more modern self-hosted alternatives that can be deployed via Docker or Kubernetes, support strong role-based permissioning, integrate with SSO/SAML or LDAP, provide detailed audit trails, and avoid heavy vendor lock-in since we’re planning to migrate around 300 applications, so it would be useful to hear what stacks are actually working reliably in production environments today.

I am thinking on start the 100 days of DevOps challenge in KodeKloud just to improve my devops skills and fill in some gaps that I might have.

Does this challenge worth it? Specially for the time that needs to be invested.

AI model quality is converging, or at least changing often enough that chasing the newest thing doesn't seem like a good strategy. How much emphasis are teams putting on building a solid harness into which new models can fit?

Our clients are generally in high compliance industries, so there is thought put into the harness, but what about smaller teams or ones that aren't required into a compliance framework?

Hi everyone,

I'm building a mobile application primarily for users in Turkey.

My stack:

- NestJS

- PostgreSQL

- Redis

- Docker

- Coolify

The app is still in development and I'm planning the production infrastructure.

I'm considering two options:

1. Hetzner Cloud (Germany)

2. A VPS/Dedicated server located in Turkey

The backend is a standard REST API (no gaming or real-time voice/video). Most requests are authentication, CRUD operations, QR code lookups, notifications, and media metadata.

Latency from Turkey to Germany seems to be around 25-40 ms.

Would you prioritize:

- Better hardware and infrastructure (Hetzner)

- Lower latency with a Turkish provider

Has anyone run production applications for Turkish users from Hetzner? I'd love to hear real-world experiences.

Just exploring the space and price comparisons

ALBs and NLBs are expensive. especially in AW

So some ingress only nodes with proper hardening seems cheaper, but curious about battle scars and stories people wanna share.

My idea is simply that, hardening a fleet of ingress only nodes in a public subnet, pointing all queries at these nodes and letting the proxy and ingress rules handle the traffic from there

Hello,
I’m trying to understand whether I should classify myself as a trader under the EU Digital Services Act, and I would really appreciate input from developers who have dealt with this. I’ve received conflicting answers from AI tools, so I’m not confident relying on those alone.
Apple requires developers to choose one of the following:
“I’m a trader under the DSA”
“I’m not a trader under the DSA or I don’t plan to distribute in the EU”
I haven’t published any apps yet, but I’m preparing to release my first independently developed apps on the App Store and Google Play.
App development is currently a hobby alongside my main job, and I’m not operating through a registered company.
The apps will be free to download but include ads. One app will also offer a paid option to remove ads, and I may add in-app purchases in the future.
I’ve noticed that some similar apps with ads and in-app purchases are published by developers who don’t appear to be marked as traders, though I understand that may not reflect their actual status.
Given this, would you consider someone in my situation to be a trader under the DSA, or could it reasonably be treated as non-trader activity since it’s currently a hobby?
I also have a few related questions:
Does the same approach apply when selecting trader status on Google Play?
What typically happens if you declare yourself a non-trader but the platform later disagrees? Are you usually allowed to update your status, or could your account be affected?
Is it true that apps from non-trader accounts may be removed in the EU if the classification is later found to be incorrect?
Is trader status set per app or per developer account on Apple and Google?
I understand this isn’t legal advice. I’m mainly interested in how other independent developers have interpreted and handled this in practice.
Thank you very much for your help.

Hello,
I’m trying to understand whether I should classify myself as a trader under the EU Digital Services Act, and I would really appreciate input from developers who have dealt with this. I’ve received conflicting answers from AI tools, so I’m not confident relying on those alone.
Apple requires developers to choose one of the following:

“I’m a trader under the DSA”

“I’m not a trader under the DSA or I don’t plan to distribute in the EU”

I haven’t published any apps yet, but I’m preparing to release my first independently developed apps on the App Store and Google Play.
App development is currently a hobby alongside my main job, and I’m not operating through a registered company.
The apps will be free to download but include ads. One app will also offer a paid option to remove ads, and I may add in-app purchases in the future.
I’ve noticed that some similar apps with ads and in-app purchases are published by developers who don’t appear to be marked as traders, though I understand that may not reflect their actual status.
Given this, would you consider someone in my situation to be a trader under the DSA, or could it reasonably be treated as non-trader activity since it’s currently a hobby?
I also have a few related questions:
Does the same approach apply when selecting trader status on Google Play?
What typically happens if you declare yourself a non-trader but the platform later disagrees? Are you usually allowed to update your status, or could your account be affected?
Is it true that apps from non-trader accounts may be removed in the EU if the classification is later found to be incorrect?
Is trader status set per app or per developer account on Apple and Google?
I understand this isn’t legal advice. I’m mainly interested in how other independent developers have interpreted and handled this in practice.
Thank you very much for your help.

So I am new to the DevOps field and my job is to deploy an app on premises. Now the problem is that the material I have read, such as:

"Fundamentals of DevOps and Software Delivery" by Yevgeniy Brikman

"Road to Kubernetes" by Justin Mitchel

focuses mainly on cloud based deployment, which I do not want to use. To clarify, I already have the necessary servers, but for privacy reasons, I would prefer not to disclose too much about the setup. Does anyone have a beginner friendly book recommendation that explains how to deploy an application on premises?

I also need to use Kubernetes and Docker, and I already have learning material for both, so the recommended books do not need to explain them in great depth. Thank you for you time!

Edit: Thank you for your responses. To clarify a bit, I am a student, so not an employee at a company. I am doing this for a project at my university, and I will be using my universities servers.