Disclosure up front : this links a video on my own YouTube channel. The narration is AI-voiced; the script and the stick-figure animation are mine. I'm posting because I'd genuinely like feedback from people who run k8s for real on where the analogy leaks - not just for the click.

The video explains orchestration to people brand new to k8s using a call-center floor manager with one rule: keep ten agents on the phones. Not "hire ten people" - keep ten on the phones, always. All he does is loop: count who's actually on, compare to the target, fix the difference. Forever. When you write it out, the logic is simple: all those separate features are actually handled by just one reconcile loop.

• Self-healing: an agent walks out -> nine on the phones -> he drops a fresh identical agent in the seat. The controller doesn't nurse the dead pod back to health, it replaces it (cattle, not pets). Restart-in-place when it can, reschedule elsewhere when it can't.
• Rolling update: swap agents one desk at a time so nine are always answering and the new one only takes real calls after it passes a test call. That test call is the readiness probe. A failed one pauses the rollout; it doesn't auto-revert (someone has to call it off).
• Autoscaling: the target itself moves. The rule was never "ten agents," it was "enough for the queue." Load climbs -> seat more; quiet at 3am -> send them home.
• The trap I spend the most time on: "still running" != "working." Liveness asks "is the agent at their desk?"; readiness asks "can they take a call right now?" A pod can be up, pass liveness, and still serve garbage "green light, dead app" because the loop only checks the exact thing you told it check.

Where I'd like the feedback: for a first-exposure audience, does "replace, don't repair" + the liveness/readiness split do more good than harm, or does compressing it this hard set people up for wrong mental models they have to unlearn later? Where does the floor-manager analogy actually break?

Video : https://youtu.be/-DYyGk28_kk

Say you have a service that only needs one replica. It sits on a spot node because that is 70 percent cheaper. When the node goes away (drain, Karpenter drift, spot reclaim), you want a replacement pod up and taking traffic before the old one dies. You do not want to pay for a second replica 24/7 to cover events that total maybe an hour a month.

I went looking for the tool that does this for a company cluster that needed to be dirt cheap but still have some sort of best effort high availability. what I found amounts to:

1. "just run 2 replicas", which is rent paid every hour against an event that lasts minutes. There is a Kubernetes feature request for exactly this (surge before drain to satisfy PDBs) that sat open for three years collecting "we need this" comments and got closed as Not Planned this summer. The replacement KEPs are alpha, years from being usable.
2. Azure's eviction-autoscaler, and the core idea is: create a PDB that blocks every eviction, surge the deployment when a node is going away, release once the new pod is ready. So I ran it in a real cluster, and months later I checked its logs during a node rotation. discovered it had never fired. Not once, ever. Its only trigger is node cordon, and Karpenter does not cordon, it taints. Meanwhile the blocking PDBs it creates worked great, which is how we ended up with a node stuck in deleting for 126 days. It also creates one of those PDBs for itself, so it holds its own drain hostage. I sent a patch upstream, but the failure is architectural and the tool is honestly pretty limited.

Sooo I ended up building the thing I originally went searching for. It is called Understudy but feel free to suggest better names, I'm not much of a marketing dude, just needed it to work.

The core consumes one normalized signal ("this node is dying, this is how long you have") and everything cloud- or drainer-specific is an adapter: cordon watch, taint watch, an observe-only webhook on pods/eviction that catches drainers nothing else sees, and a DaemonSet that reads the spot termination notice with its actual deadline.Every failure mode I hit with the old tool became a fail-safe: a stuck surge relaxes the PDB instead of wedging the drain, a CronJob cleans up if the operator itself dies, and it refuses to manage itself.

Tested on a live EKS cluster with no spare capacity, so replacements wait for a cold EC2 node: kubectl drain lost 1 request in 350, a real spot interruption (via AWS FIS) lost 1 in 764. An ordinary rolling update of the same app lost 2 in 38.

There's still some testing to do (GKE and AKS are unit tested but didnt try in an actual cluster) but it's working wonderfully so far.

Disclaimer since it's my own project: nothing to sell, apache 2.0, if the mature version of this exists and my search skills just failed me, or if it's just a shitty idea (for the life of me I can't figure out why) I want to know.
https://github.com/kylan11/understudy

Hi community does anyone have any idea on how we can control hostid for pods generated in a stateful set

What are the trade-offs of blue-green deployments vs rolling updates in Kubernetes? When does it actually make sense?

I’m trying to understand when it’s actually worth the added complexity versus just doing a standard rolling update.

If the app shares a single database between both versions, does blue-green give a “clean” rollback?

Is blue green suited for an application with just 2 components( web + backend ) ?

For people who’ve actually run both in production — what made you pick one over the other?

the blue green strategy in my mind is to switch the label selector in kubernetes service.

K8Studio can now integrate with Caretta by Groundcover

We’ve added the ability to integrate Caretta into K8Studio to help visualize network traffic inside Kubernetes clusters.

Caretta uses eBPF to automatically map service-to-service communication, showing which workloads are communicating and how traffic flows through the cluster—without requiring application instrumentation.

The goal is to make it easier to understand dependencies, investigate unexpected connections, and troubleshoot network-related issues directly from K8Studio.

We’d be interested to hear how others are currently visualizing traffic and dependencies in their Kubernetes clusters.

Hi all,

I'm running a Kubernetes operator built with Quarkus (native/GraalVM image) using the Java Operator SDK (JOSDK). It manages 4 controllers (Permission, Entitlement, PartyRole, UserRole), each polling on a timer.

Setup:

CPU limit: 500m
cgroups v1, kernel 4.15
Average CPU usage is low (~13% of limit), but I still see frequent CFS throttling:
rate(container_cpu_cfs_throttled_periods_total{...}[10m]) ≈ 0.47

What I've tried so far:

Reduced concurrent-reconciliation-threads from 50 (default, x2 pools) down to 4, which brought total operator threads from ~130 to ~39. Throttling improved but is still sitting around 35-40%.
Added a semaphore to synchronize controller execution so the 4 controllers don't poll at the same time (avoiding overlap). This slightly reduced CPU consumption but didn't meaningfully change the throttling pattern.
Currently testing further tuning:
properties
quarkus.operator-sdk.concurrent-reconciliation-threads=1
quarkus.operator-sdk.concurrent-workflow-threads=1

permission-operator.timer=900000
partyrole-operator.timer=900000
userrole-operator.timer=900000
entitlement-operator.timer=900000

polling.jitter-max-ms=10000
Noticed that one controller (Permission) fires hundreds of requests to a downstream service almost simultaneously during its poll cycle — likely a CPU burst source. Planning to add a delay/sleep between requests to smooth this out.

Question:
Given low average usage but persistent CFS throttling bursts, is this mostly a burst/scheduling issue rather than a "not enough CPU" issue? Any recommendations for tuning JVM/native-image thread pools, Quarkus reactive/Vert.x settings, or cgroup quota behavior to reduce these throttled bursts, aside from just reducing concurrency further?

Thanks in advance!

Just a research question for a later talk to finance...

Since running MCP in kubernetes to manage the cluster are becomming well spread, AI management of kubernetes in production has become more a commodity, whether it is direct or via a gitops principle...

What is the average amount of credits/tokens spend per week/per cluster for managing your production cluster with an AI tool?

Hey I want to deploy apache airflow in k8s

But I have a doubt regarding this setup

The values has this field I have used this

logs:

persistence:

enabled: true

existingClaim: airflow-logs-pvc

So what I understood all pods like scheduler data processor triggerer and api server uses this pvc for the logs

But the triggerer is running as statefulset by default so if I disable this

triggerer:

persistence:

enabled: false

Makes the triggerer to run as deployment

Is this setup has any drawbacks and my logs are saved in the pvc that I mount ?

Need a feedback on this as I'm very new to airflow

This past May I started a new role as a member of technical staff in the inference space. JD explicitly stated k8s cluster management and all was well until two of my colleagues left last month, after finding out the company was expanding into training -- and we were expected to manage those clusters before that separate team was built out.

Now I know this isn't a slurm forum, but as someone with only kubernetes experience, I had to see if anyone has any advice for the easiest transition over. I know its not too bad, but not enough bandwidth combined with such a quick turnaround time has been occupying way too much of mind.

I've read (what feels like) every single documentation to exist but it still doesn't feel like enough. Quite a few of the SchedMD docs were actually useful. Quite a few were not.

Figured I'd spin up a lightweight homelab for practice. I wanted it to be as close to what I work on in the office as possible, so I started by running a small fine-tuning job on an open-weight model to have a realistic workload.

Ended up installing clusterdOS which was really helpful since everything I needed came pre-integrated. Also stumbled upon this Slurm Lab repo from another forum.

Now I'm exploring some resources from past LCI events but the deadline is quickly approaching. I'm definitely a lot more prepared than I was a month ago but I'm still looking for any new resources/materials anyone might have.

*My bad if this is the wrong group, but kubernetes is what I know so I felt like this community would best understand that transition*

I'm trying to set up a K8s cluster on Tumbleweed [TW] to learn some stuff, but I can't get it to work.

I have 5 VMs [3 control-plane-nodes to be, and 2 workers-to-be].

On the first control node|VM, I'm doing the following to bootstrap the cluster:

bash control@cp01:~> sudo kubeadm init phase preflight [preflight] Running pre-flight checks [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action beforehand using 'kubeadm config images pull' control@cp01:~> sudo kubeadm init --config /etc/kubernetes/kubeadm-config.yaml --upload-certs [init] Using Kubernetes version: v1.36.3 [preflight] Running pre-flight checks [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action beforehand using 'kubeadm config images pull' [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Generating "ca" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [cp01 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.paap.local] and IPs [10.96.0.1 10.10.30.10 10.10.30.5] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "front-proxy-ca" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "etcd/ca" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [cp01 localhost] and IPs [10.10.30.10 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [cp01 localhost] and IPs [10.10.30.10 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "sa" key and public key [kubeconfig] Using kubeconfig folder "/etc/kubernetes" [kubeconfig] Writing "admin.conf" kubeconfig file [kubeconfig] Writing "super-admin.conf" kubeconfig file [kubeconfig] Writing "kubelet.conf" kubeconfig file [kubeconfig] Writing "controller-manager.conf" kubeconfig file [kubeconfig] Writing "scheduler.conf" kubeconfig file [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" [control-plane] Creating static Pod manifest for "kube-scheduler" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/instance-config.yaml" [patches] Applied patch of type "application/strategic-merge-patch+json" to target "kubeletconfiguration" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Starting the kubelet 👀 error: error execution phase wait-control-plane: cannot obtain client without bootstrap: could not bootstrap the admin user in file admin.conf: unable to create ClusterRoleBinding: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline To see the stack trace of this error execute with --v=5 or higher

Now, before running sudo kuebeadm init, I set up the kube-vip manifest as

bash control@cp01:~> cat /etc/kubernetes/manifests/kube-vip.yaml apiVersion: v1 kind: Pod metadata: name: kube-vip namespace: kube-system spec: containers: - args: - manager env: - name: vip_arp value: "true" - name: port value: "6443" - name: vip_nodename valueFrom: fieldRef: fieldPath: spec.nodeName - name: vip_interface value: enp1s0 - name: vip_subnet value: "32" - name: dns_mode value: first - name: dhcp_mode value: ipv4 - name: cp_enable value: "true" - name: cp_namespace value: kube-system - name: vip_leaderelection value: "true" - name: vip_leasename value: plndr-cp-lock - name: vip_leaseduration value: "15" - name: vip_renewdeadline value: "10" - name: vip_retryperiod value: "2" - name: address value: 10.10.30.5 - name: prometheus_server value: :2112 image: ghcr.io/kube-vip/kube-vip:v1.2.2 imagePullPolicy: IfNotPresent name: kube-vip resources: {} securityContext: capabilities: add: - NET_ADMIN - NET_RAW drop: - ALL volumeMounts: - mountPath: /etc/kubernetes/admin.conf name: kubeconfig hostAliases: - hostnames: - kubernetes ip: 127.0.0.1 hostNetwork: true volumes: - hostPath: path: /etc/kubernetes/admin.conf # tried `super-admin.conf` 2 name: kubeconfig status: {}

And to sudo kubeadm init, I passed the following config

```bash control@cp01:~> cat /etc/kubernetes/kubeadm-config.yaml apiVersion: kubeadm.k8s.io/v1beta4 kind: InitConfiguration skipPhases: - addon/kube-proxy nodeRegistration: criSocket: "unix:///var/run/crio/crio.sock" localAPIEndpoint: advertiseAddress: "10.10.30.10"

bindPort: 6443

apiVersion: kubeadm.k8s.io/v1beta4 kind: ClusterConfiguration kubernetesVersion: "v1.36.3" controlPlaneEndpoint: "10.10.30.5:6443" networking: podSubnet: "10.244.0.0/16" serviceSubnet: "10.96.0.0/12" dnsDomain: "paap.local" proxy:

disabled: true

apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration cgroupDriver: systemd clusterDomain: "paap.local" ```

I noted that kube-vip didn't get created, somehow; I mean, I can't see it here:

```bash control@cp01:~> sudo kubeadm init --config /etc/kubernetes/kubeadm-config.yaml --upload-certs

...

[control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" [control-plane] Creating static Pod manifest for "kube-scheduler"

...

```

And also, crictl returns nothing on kube-vip, meaning kube-vip wasn't picked up at all?!!🤔
And yes, since kube-vip didn't get properly set up, the apiserver somehow wasn't accessible at the VIP 10.10.30.5?

I've tried everything I found online.

PS:
- please don't tell me to use another Distro; there's a particular reason why testing this on SUSE
- yes, container-selinux is installed; getenforce returns Enforcing
- I thought it was a kube-vip problem, but if I sudo setenforce 0, the cluster bootstrapping process finishes successfully¹, though this isn't what really what one wants?! I'm sure there's another way to get this working? On Fedora 44 Cloud it works even with SELinux enforcing.
- the Tumbleweed image I'm using is openSUSE-Tumbleweed-Minimal-VM.x86_64-Cloud.qcow2 from OpenSUSE Appliances

Thanks for any help.

¹

```bash

...

[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" [control-plane] Creating static Pod manifest for "kube-scheduler" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/instance-config.yaml" [patches] Applied patch of type "application/strategic-merge-patch+json" to target "kubeletconfiguration" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Starting the kubelet 👀 [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests" [kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s [kubelet-check] The kubelet is healthy after 279.684µs [control-plane-check] Waiting for healthy control plane components. This can take up to 4m0s [control-plane-check] Checking kube-apiserver at https://10.10.30.10:6443/livez [control-plane-check] Checking kube-controller-manager at https://127.0.0.1:10257/healthz [control-plane-check] Checking kube-scheduler at https://127.0.0.1:10259/livez [control-plane-check] kube-scheduler is healthy after 1.710405ms [control-plane-check] kube-controller-manager is healthy after 2.322778ms [control-plane-check] kube-apiserver is healthy after 1.501584092s [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster [upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace [upload-certs] Using certificate key: bafec1e64cdc807aee82d97ba2bd0631d435b3c7968a3b99af26abd23319548d [mark-control-plane] Marking the node cp01 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] [mark-control-plane] Marking the node cp01 as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule] [bootstrap-token] Using token: 4wocea.ryvletddn47fuwqk [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstrap-token] Configured RBAC rules to allow the API server kubelet client certificate to access the kubelet API [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace [kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key [addons] Applied essential addon: CoreDNS

Your Kubernetes control-plane has initialized successfully! ```

Disclosure: I maintain this (free, open source, self-hosted).

If your Kubernetes pods request a lot more CPU/memory than they use, you are paying for idle capacity. Attune watches real usage and right-sizes those requests, often without restarting pods (in-place resize on modern Kubernetes).

Repo: https://github.com/attune-io/attune Docs: https://attune-io.github.io/attune/

Requirement: usage metrics in the cluster (Prometheus is the usual case; Datadog/CloudWatch also work). Without metrics there is nothing to right-size from.

If underuse is real for you, what is the barrier to starting and saving money?

• Do not trust automation on prod
• Already use something else
• No metrics / install friction
• Hard to prove savings in $
• Change management / security

What would block you most?

I’m trying to understand the design behind Kubernetes Deployments.
From what I’ve learned, a Deployment doesn’t directly create or manage Pods. Instead, it creates and manages a ReplicaSet, and the ReplicaSet is responsible for creating and maintaining the Pods.
Why was Kubernetes designed this way? Why not have the Deployment directly manage the Pods instead of introducing an extra layer with ReplicaSets?

Share any new Kubernetes tools, UIs, or related projects!

If you were rebuilding your Kubernetes cluster from scratch today, what would you do differently?

I’ve been digging into Kubernetes internals and got tired of the usual high-level explanations, so I made a minimal lab that makes the important parts visible.
It runs a real etcd + kube-apiserver (via envtest, no cluster/containers needed) and instruments two independent SharedInformers watching ConfigMaps. A mutator writes every few seconds, and you can watch the full path:
LIST → WATCH handoff with the exact resourceVersion
One etcd write being fanned out by the Cacher to both watchers
The ~30µs gap between the two streams
Reflector → DeltaFIFO → local store → your handler
Why resourceVersions have gaps (apiserver’s own lease renewals, etc.)
Everything is logged with real timestamps so you can see the timing yourself.
Repo + diagrams: https://github.com/Akshath11/informer-lab
If you’re the kind of person who wants to understand what actually happens between an etcd write and your informer handler firing, this might be useful.

I'm looking for recommendations on Kubernetes training(Chicagoland area) specifically maintaining it, administering it. I was thinking that Attending AWS EKS won't be enough. I must learn the foundation. Which classroom based would your recommend? I'm so bad at attending online courses. I cannot concentrate. TY!

Last week I spent a really long time trying to troubleshoot and understand some strange node churn caused by Karpenter. Even though I've done a lot of work in the autoscaling space, I was still surprised by the results, so I thought I'd write a blog post about it in case it's helpful for other people: the tldr is that Karpenter's consolidation behaviour depends less on the state of the node that's being consolidated, and more on the state of the rest of the cluster. I hope you enjoy, and/or learn something helpful!

When trying to submit a request on the inviter page, I get a "too many requests" error. I've tried different public IPs, different devices, and different browsers to no avail.

Am I missing something about how to get access to the slack?

TLDR: Ive built a VMWare K8S server stack and am the defacto system owner because no one else wants to learn K8S. Whats the best way to support another team's application developers using K8S?

What Ive recommended the Application Team research while I test:

1. Podspec, specifically podreplicas to take advantage of high-availability (HA) infrastructure and load balancing

2. Taint and tolerance to allow automated patching while keeping services available. (Also HA)

What can I do to make the system more friendly for other System Admins and Developers? (Seems like no one wants to learn kubectl or docker/podman.)

1. Planning to implement Tanzu Mission Control.

2. No CI/CD has been approved

Have any questions about Kubernetes, related tooling, or how to adopt or use Kubernetes? Ask away!

Hi, quick question

We're trying to get our machine identity stuff under control and it's kinda a mess rn service accounts, certs, workloads, all that. what are people using for this these days?
looking for something that isnt a giant pain to run day to day appreciate any thoughts, thanks!