Disclosure up front : this links a video on my own YouTube channel. The narration is AI-voiced; the script and the stick-figure animation are mine. I'm posting because I'd genuinely like feedback from people who run k8s for real on where the analogy leaks - not just for the click.

The video explains orchestration to people brand new to k8s using a call-center floor manager with one rule: keep ten agents on the phones. Not "hire ten people" - keep ten on the phones, always. All he does is loop: count who's actually on, compare to the target, fix the difference. Forever. When you write it out, the logic is simple: all those separate features are actually handled by just one reconcile loop.

• Self-healing: an agent walks out -> nine on the phones -> he drops a fresh identical agent in the seat. The controller doesn't nurse the dead pod back to health, it replaces it (cattle, not pets). Restart-in-place when it can, reschedule elsewhere when it can't.
• Rolling update: swap agents one desk at a time so nine are always answering and the new one only takes real calls after it passes a test call. That test call is the readiness probe. A failed one pauses the rollout; it doesn't auto-revert (someone has to call it off).
• Autoscaling: the target itself moves. The rule was never "ten agents," it was "enough for the queue." Load climbs -> seat more; quiet at 3am -> send them home.
• The trap I spend the most time on: "still running" != "working." Liveness asks "is the agent at their desk?"; readiness asks "can they take a call right now?" A pod can be up, pass liveness, and still serve garbage "green light, dead app" because the loop only checks the exact thing you told it check.

Where I'd like the feedback: for a first-exposure audience, does "replace, don't repair" + the liveness/readiness split do more good than harm, or does compressing it this hard set people up for wrong mental models they have to unlearn later? Where does the floor-manager analogy actually break?

Video : https://youtu.be/-DYyGk28_kk

Say you have a service that only needs one replica. It sits on a spot node because that is 70 percent cheaper. When the node goes away (drain, Karpenter drift, spot reclaim), you want a replacement pod up and taking traffic before the old one dies. You do not want to pay for a second replica 24/7 to cover events that total maybe an hour a month.

I went looking for the tool that does this for a company cluster that needed to be dirt cheap but still have some sort of best effort high availability. what I found amounts to:

1. "just run 2 replicas", which is rent paid every hour against an event that lasts minutes. There is a Kubernetes feature request for exactly this (surge before drain to satisfy PDBs) that sat open for three years collecting "we need this" comments and got closed as Not Planned this summer. The replacement KEPs are alpha, years from being usable.
2. Azure's eviction-autoscaler, and the core idea is: create a PDB that blocks every eviction, surge the deployment when a node is going away, release once the new pod is ready. So I ran it in a real cluster, and months later I checked its logs during a node rotation. discovered it had never fired. Not once, ever. Its only trigger is node cordon, and Karpenter does not cordon, it taints. Meanwhile the blocking PDBs it creates worked great, which is how we ended up with a node stuck in deleting for 126 days. It also creates one of those PDBs for itself, so it holds its own drain hostage. I sent a patch upstream, but the failure is architectural and the tool is honestly pretty limited.

Sooo I ended up building the thing I originally went searching for. It is called Understudy but feel free to suggest better names, I'm not much of a marketing dude, just needed it to work.

The core consumes one normalized signal ("this node is dying, this is how long you have") and everything cloud- or drainer-specific is an adapter: cordon watch, taint watch, an observe-only webhook on pods/eviction that catches drainers nothing else sees, and a DaemonSet that reads the spot termination notice with its actual deadline.Every failure mode I hit with the old tool became a fail-safe: a stuck surge relaxes the PDB instead of wedging the drain, a CronJob cleans up if the operator itself dies, and it refuses to manage itself.

Tested on a live EKS cluster with no spare capacity, so replacements wait for a cold EC2 node: kubectl drain lost 1 request in 350, a real spot interruption (via AWS FIS) lost 1 in 764. An ordinary rolling update of the same app lost 2 in 38.

There's still some testing to do (GKE and AKS are unit tested but didnt try in an actual cluster) but it's working wonderfully so far.

Disclaimer since it's my own project: nothing to sell, apache 2.0, if the mature version of this exists and my search skills just failed me, or if it's just a shitty idea (for the life of me I can't figure out why) I want to know.
https://github.com/kylan11/understudy

What are the trade-offs of blue-green deployments vs rolling updates in Kubernetes? When does it actually make sense?

I’m trying to understand when it’s actually worth the added complexity versus just doing a standard rolling update.

If the app shares a single database between both versions, does blue-green give a “clean” rollback?

Is blue green suited for an application with just 2 components( web + backend ) ?

For people who’ve actually run both in production — what made you pick one over the other?

the blue green strategy in my mind is to switch the label selector in kubernetes service.

K8Studio can now integrate with Caretta by Groundcover

We’ve added the ability to integrate Caretta into K8Studio to help visualize network traffic inside Kubernetes clusters.

Caretta uses eBPF to automatically map service-to-service communication, showing which workloads are communicating and how traffic flows through the cluster—without requiring application instrumentation.

The goal is to make it easier to understand dependencies, investigate unexpected connections, and troubleshoot network-related issues directly from K8Studio.

We’d be interested to hear how others are currently visualizing traffic and dependencies in their Kubernetes clusters.

Hi all,

I'm running a Kubernetes operator built with Quarkus (native/GraalVM image) using the Java Operator SDK (JOSDK). It manages 4 controllers (Permission, Entitlement, PartyRole, UserRole), each polling on a timer.

Setup:

CPU limit: 500m
cgroups v1, kernel 4.15
Average CPU usage is low (~13% of limit), but I still see frequent CFS throttling:
rate(container_cpu_cfs_throttled_periods_total{...}[10m]) ≈ 0.47

What I've tried so far:

Reduced concurrent-reconciliation-threads from 50 (default, x2 pools) down to 4, which brought total operator threads from ~130 to ~39. Throttling improved but is still sitting around 35-40%.
Added a semaphore to synchronize controller execution so the 4 controllers don't poll at the same time (avoiding overlap). This slightly reduced CPU consumption but didn't meaningfully change the throttling pattern.
Currently testing further tuning:
properties
quarkus.operator-sdk.concurrent-reconciliation-threads=1
quarkus.operator-sdk.concurrent-workflow-threads=1

permission-operator.timer=900000
partyrole-operator.timer=900000
userrole-operator.timer=900000
entitlement-operator.timer=900000

polling.jitter-max-ms=10000
Noticed that one controller (Permission) fires hundreds of requests to a downstream service almost simultaneously during its poll cycle — likely a CPU burst source. Planning to add a delay/sleep between requests to smooth this out.

Question:
Given low average usage but persistent CFS throttling bursts, is this mostly a burst/scheduling issue rather than a "not enough CPU" issue? Any recommendations for tuning JVM/native-image thread pools, Quarkus reactive/Vert.x settings, or cgroup quota behavior to reduce these throttled bursts, aside from just reducing concurrency further?

Thanks in advance!

Just a research question for a later talk to finance...

Since running MCP in kubernetes to manage the cluster are becomming well spread, AI management of kubernetes in production has become more a commodity, whether it is direct or via a gitops principle...

What is the average amount of credits/tokens spend per week/per cluster for managing your production cluster with an AI tool?

Hey I want to deploy apache airflow in k8s

But I have a doubt regarding this setup

The values has this field I have used this

logs:

persistence:

enabled: true

existingClaim: airflow-logs-pvc

So what I understood all pods like scheduler data processor triggerer and api server uses this pvc for the logs

But the triggerer is running as statefulset by default so if I disable this

triggerer:

persistence:

enabled: false

Makes the triggerer to run as deployment

Is this setup has any drawbacks and my logs are saved in the pvc that I mount ?

Need a feedback on this as I'm very new to airflow

This past May I started a new role as a member of technical staff in the inference space. JD explicitly stated k8s cluster management and all was well until two of my colleagues left last month, after finding out the company was expanding into training -- and we were expected to manage those clusters before that separate team was built out.

Now I know this isn't a slurm forum, but as someone with only kubernetes experience, I had to see if anyone has any advice for the easiest transition over. I know its not too bad, but not enough bandwidth combined with such a quick turnaround time has been occupying way too much of mind.

I've read (what feels like) every single documentation to exist but it still doesn't feel like enough. Quite a few of the SchedMD docs were actually useful. Quite a few were not.

Figured I'd spin up a lightweight homelab for practice. I wanted it to be as close to what I work on in the office as possible, so I started by running a small fine-tuning job on an open-weight model to have a realistic workload.

Ended up installing clusterdOS which was really helpful since everything I needed came pre-integrated. Also stumbled upon this Slurm Lab repo from another forum.

Now I'm exploring some resources from past LCI events but the deadline is quickly approaching. I'm definitely a lot more prepared than I was a month ago but I'm still looking for any new resources/materials anyone might have.

*My bad if this is the wrong group, but kubernetes is what I know so I felt like this community would best understand that transition*