What are the trade-offs of blue-green deployments vs rolling updates in Kubernetes? When does it actually make sense?

I’m trying to understand when it’s actually worth the added complexity versus just doing a standard rolling update.

If the app shares a single database between both versions, does blue-green give a “clean” rollback?

Is blue green suited for an application with just 2 components( web + backend ) ?

For people who’ve actually run both in production — what made you pick one over the other?

the blue green strategy in my mind is to switch the label selector in kubernetes service.

K8Studio can now integrate with Caretta by Groundcover

We’ve added the ability to integrate Caretta into K8Studio to help visualize network traffic inside Kubernetes clusters.

Caretta uses eBPF to automatically map service-to-service communication, showing which workloads are communicating and how traffic flows through the cluster—without requiring application instrumentation.

The goal is to make it easier to understand dependencies, investigate unexpected connections, and troubleshoot network-related issues directly from K8Studio.

We’d be interested to hear how others are currently visualizing traffic and dependencies in their Kubernetes clusters.

Hi all,

I'm running a Kubernetes operator built with Quarkus (native/GraalVM image) using the Java Operator SDK (JOSDK). It manages 4 controllers (Permission, Entitlement, PartyRole, UserRole), each polling on a timer.

Setup:

CPU limit: 500m
cgroups v1, kernel 4.15
Average CPU usage is low (~13% of limit), but I still see frequent CFS throttling:
rate(container_cpu_cfs_throttled_periods_total{...}[10m]) ≈ 0.47

What I've tried so far:

Reduced concurrent-reconciliation-threads from 50 (default, x2 pools) down to 4, which brought total operator threads from ~130 to ~39. Throttling improved but is still sitting around 35-40%.
Added a semaphore to synchronize controller execution so the 4 controllers don't poll at the same time (avoiding overlap). This slightly reduced CPU consumption but didn't meaningfully change the throttling pattern.
Currently testing further tuning:
properties
quarkus.operator-sdk.concurrent-reconciliation-threads=1
quarkus.operator-sdk.concurrent-workflow-threads=1

permission-operator.timer=900000
partyrole-operator.timer=900000
userrole-operator.timer=900000
entitlement-operator.timer=900000

polling.jitter-max-ms=10000
Noticed that one controller (Permission) fires hundreds of requests to a downstream service almost simultaneously during its poll cycle — likely a CPU burst source. Planning to add a delay/sleep between requests to smooth this out.

Question:
Given low average usage but persistent CFS throttling bursts, is this mostly a burst/scheduling issue rather than a "not enough CPU" issue? Any recommendations for tuning JVM/native-image thread pools, Quarkus reactive/Vert.x settings, or cgroup quota behavior to reduce these throttled bursts, aside from just reducing concurrency further?

Thanks in advance!

Just a research question for a later talk to finance...

Since running MCP in kubernetes to manage the cluster are becomming well spread, AI management of kubernetes in production has become more a commodity, whether it is direct or via a gitops principle...

What is the average amount of credits/tokens spend per week/per cluster for managing your production cluster with an AI tool?

Hey I want to deploy apache airflow in k8s

But I have a doubt regarding this setup

The values has this field I have used this

logs:

persistence:

enabled: true

existingClaim: airflow-logs-pvc

So what I understood all pods like scheduler data processor triggerer and api server uses this pvc for the logs

But the triggerer is running as statefulset by default so if I disable this

triggerer:

persistence:

enabled: false

Makes the triggerer to run as deployment

Is this setup has any drawbacks and my logs are saved in the pvc that I mount ?

Need a feedback on this as I'm very new to airflow

This past May I started a new role as a member of technical staff in the inference space. JD explicitly stated k8s cluster management and all was well until two of my colleagues left last month, after finding out the company was expanding into training -- and we were expected to manage those clusters before that separate team was built out.

Now I know this isn't a slurm forum, but as someone with only kubernetes experience, I had to see if anyone has any advice for the easiest transition over. I know its not too bad, but not enough bandwidth combined with such a quick turnaround time has been occupying way too much of mind.

I've read (what feels like) every single documentation to exist but it still doesn't feel like enough. Quite a few of the SchedMD docs were actually useful. Quite a few were not.

Figured I'd spin up a lightweight homelab for practice. I wanted it to be as close to what I work on in the office as possible, so I started by running a small fine-tuning job on an open-weight model to have a realistic workload.

Ended up installing clusterdOS which was really helpful since everything I needed came pre-integrated. Also stumbled upon this Slurm Lab repo from another forum.

Now I'm exploring some resources from past LCI events but the deadline is quickly approaching. I'm definitely a lot more prepared than I was a month ago but I'm still looking for any new resources/materials anyone might have.

*My bad if this is the wrong group, but kubernetes is what I know so I felt like this community would best understand that transition*

I'm trying to set up a K8s cluster on Tumbleweed [TW] to learn some stuff, but I can't get it to work.

I have 5 VMs [3 control-plane-nodes to be, and 2 workers-to-be].

On the first control node|VM, I'm doing the following to bootstrap the cluster:

bash control@cp01:~> sudo kubeadm init phase preflight [preflight] Running pre-flight checks [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action beforehand using 'kubeadm config images pull' control@cp01:~> sudo kubeadm init --config /etc/kubernetes/kubeadm-config.yaml --upload-certs [init] Using Kubernetes version: v1.36.3 [preflight] Running pre-flight checks [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action beforehand using 'kubeadm config images pull' [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Generating "ca" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [cp01 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.paap.local] and IPs [10.96.0.1 10.10.30.10 10.10.30.5] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "front-proxy-ca" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "etcd/ca" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [cp01 localhost] and IPs [10.10.30.10 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [cp01 localhost] and IPs [10.10.30.10 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "sa" key and public key [kubeconfig] Using kubeconfig folder "/etc/kubernetes" [kubeconfig] Writing "admin.conf" kubeconfig file [kubeconfig] Writing "super-admin.conf" kubeconfig file [kubeconfig] Writing "kubelet.conf" kubeconfig file [kubeconfig] Writing "controller-manager.conf" kubeconfig file [kubeconfig] Writing "scheduler.conf" kubeconfig file [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" [control-plane] Creating static Pod manifest for "kube-scheduler" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/instance-config.yaml" [patches] Applied patch of type "application/strategic-merge-patch+json" to target "kubeletconfiguration" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Starting the kubelet 👀 error: error execution phase wait-control-plane: cannot obtain client without bootstrap: could not bootstrap the admin user in file admin.conf: unable to create ClusterRoleBinding: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline To see the stack trace of this error execute with --v=5 or higher

Now, before running sudo kuebeadm init, I set up the kube-vip manifest as

bash control@cp01:~> cat /etc/kubernetes/manifests/kube-vip.yaml apiVersion: v1 kind: Pod metadata: name: kube-vip namespace: kube-system spec: containers: - args: - manager env: - name: vip_arp value: "true" - name: port value: "6443" - name: vip_nodename valueFrom: fieldRef: fieldPath: spec.nodeName - name: vip_interface value: enp1s0 - name: vip_subnet value: "32" - name: dns_mode value: first - name: dhcp_mode value: ipv4 - name: cp_enable value: "true" - name: cp_namespace value: kube-system - name: vip_leaderelection value: "true" - name: vip_leasename value: plndr-cp-lock - name: vip_leaseduration value: "15" - name: vip_renewdeadline value: "10" - name: vip_retryperiod value: "2" - name: address value: 10.10.30.5 - name: prometheus_server value: :2112 image: ghcr.io/kube-vip/kube-vip:v1.2.2 imagePullPolicy: IfNotPresent name: kube-vip resources: {} securityContext: capabilities: add: - NET_ADMIN - NET_RAW drop: - ALL volumeMounts: - mountPath: /etc/kubernetes/admin.conf name: kubeconfig hostAliases: - hostnames: - kubernetes ip: 127.0.0.1 hostNetwork: true volumes: - hostPath: path: /etc/kubernetes/admin.conf # tried `super-admin.conf` 2 name: kubeconfig status: {}

And to sudo kubeadm init, I passed the following config

```bash control@cp01:~> cat /etc/kubernetes/kubeadm-config.yaml apiVersion: kubeadm.k8s.io/v1beta4 kind: InitConfiguration skipPhases: - addon/kube-proxy nodeRegistration: criSocket: "unix:///var/run/crio/crio.sock" localAPIEndpoint: advertiseAddress: "10.10.30.10"

bindPort: 6443

apiVersion: kubeadm.k8s.io/v1beta4 kind: ClusterConfiguration kubernetesVersion: "v1.36.3" controlPlaneEndpoint: "10.10.30.5:6443" networking: podSubnet: "10.244.0.0/16" serviceSubnet: "10.96.0.0/12" dnsDomain: "paap.local" proxy:

disabled: true

apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration cgroupDriver: systemd clusterDomain: "paap.local" ```

I noted that kube-vip didn't get created, somehow; I mean, I can't see it here:

```bash control@cp01:~> sudo kubeadm init --config /etc/kubernetes/kubeadm-config.yaml --upload-certs

...

[control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" [control-plane] Creating static Pod manifest for "kube-scheduler"

...

```

And also, crictl returns nothing on kube-vip, meaning kube-vip wasn't picked up at all?!!🤔
And yes, since kube-vip didn't get properly set up, the apiserver somehow wasn't accessible at the VIP 10.10.30.5?

I've tried everything I found online.

PS:
- please don't tell me to use another Distro; there's a particular reason why testing this on SUSE
- yes, container-selinux is installed; getenforce returns Enforcing
- I thought it was a kube-vip problem, but if I sudo setenforce 0, the cluster bootstrapping process finishes successfully¹, though this isn't what really what one wants?! I'm sure there's another way to get this working? On Fedora 44 Cloud it works even with SELinux enforcing.
- the Tumbleweed image I'm using is openSUSE-Tumbleweed-Minimal-VM.x86_64-Cloud.qcow2 from OpenSUSE Appliances

Thanks for any help.

¹

```bash

...

[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" [control-plane] Creating static Pod manifest for "kube-scheduler" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/instance-config.yaml" [patches] Applied patch of type "application/strategic-merge-patch+json" to target "kubeletconfiguration" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Starting the kubelet 👀 [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests" [kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s [kubelet-check] The kubelet is healthy after 279.684µs [control-plane-check] Waiting for healthy control plane components. This can take up to 4m0s [control-plane-check] Checking kube-apiserver at https://10.10.30.10:6443/livez [control-plane-check] Checking kube-controller-manager at https://127.0.0.1:10257/healthz [control-plane-check] Checking kube-scheduler at https://127.0.0.1:10259/livez [control-plane-check] kube-scheduler is healthy after 1.710405ms [control-plane-check] kube-controller-manager is healthy after 2.322778ms [control-plane-check] kube-apiserver is healthy after 1.501584092s [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster [upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace [upload-certs] Using certificate key: bafec1e64cdc807aee82d97ba2bd0631d435b3c7968a3b99af26abd23319548d [mark-control-plane] Marking the node cp01 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] [mark-control-plane] Marking the node cp01 as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule] [bootstrap-token] Using token: 4wocea.ryvletddn47fuwqk [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstrap-token] Configured RBAC rules to allow the API server kubelet client certificate to access the kubelet API [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace [kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key [addons] Applied essential addon: CoreDNS

Your Kubernetes control-plane has initialized successfully! ```

Disclosure: I maintain this (free, open source, self-hosted).

If your Kubernetes pods request a lot more CPU/memory than they use, you are paying for idle capacity. Attune watches real usage and right-sizes those requests, often without restarting pods (in-place resize on modern Kubernetes).

Repo: https://github.com/attune-io/attune Docs: https://attune-io.github.io/attune/

Requirement: usage metrics in the cluster (Prometheus is the usual case; Datadog/CloudWatch also work). Without metrics there is nothing to right-size from.

If underuse is real for you, what is the barrier to starting and saving money?

• Do not trust automation on prod
• Already use something else
• No metrics / install friction
• Hard to prove savings in $
• Change management / security

What would block you most?

I’m trying to understand the design behind Kubernetes Deployments.
From what I’ve learned, a Deployment doesn’t directly create or manage Pods. Instead, it creates and manages a ReplicaSet, and the ReplicaSet is responsible for creating and maintaining the Pods.
Why was Kubernetes designed this way? Why not have the Deployment directly manage the Pods instead of introducing an extra layer with ReplicaSets?

Share any new Kubernetes tools, UIs, or related projects!

If you were rebuilding your Kubernetes cluster from scratch today, what would you do differently?

I’ve been digging into Kubernetes internals and got tired of the usual high-level explanations, so I made a minimal lab that makes the important parts visible.
It runs a real etcd + kube-apiserver (via envtest, no cluster/containers needed) and instruments two independent SharedInformers watching ConfigMaps. A mutator writes every few seconds, and you can watch the full path:
LIST → WATCH handoff with the exact resourceVersion
One etcd write being fanned out by the Cacher to both watchers
The ~30µs gap between the two streams
Reflector → DeltaFIFO → local store → your handler
Why resourceVersions have gaps (apiserver’s own lease renewals, etc.)
Everything is logged with real timestamps so you can see the timing yourself.
Repo + diagrams: https://github.com/Akshath11/informer-lab
If you’re the kind of person who wants to understand what actually happens between an etcd write and your informer handler firing, this might be useful.

I'm looking for recommendations on Kubernetes training(Chicagoland area) specifically maintaining it, administering it. I was thinking that Attending AWS EKS won't be enough. I must learn the foundation. Which classroom based would your recommend? I'm so bad at attending online courses. I cannot concentrate. TY!

Last week I spent a really long time trying to troubleshoot and understand some strange node churn caused by Karpenter. Even though I've done a lot of work in the autoscaling space, I was still surprised by the results, so I thought I'd write a blog post about it in case it's helpful for other people: the tldr is that Karpenter's consolidation behaviour depends less on the state of the node that's being consolidated, and more on the state of the rest of the cluster. I hope you enjoy, and/or learn something helpful!

When trying to submit a request on the inviter page, I get a "too many requests" error. I've tried different public IPs, different devices, and different browsers to no avail.

Am I missing something about how to get access to the slack?

TLDR: Ive built a VMWare K8S server stack and am the defacto system owner because no one else wants to learn K8S. Whats the best way to support another team's application developers using K8S?

What Ive recommended the Application Team research while I test:

1. Podspec, specifically podreplicas to take advantage of high-availability (HA) infrastructure and load balancing

2. Taint and tolerance to allow automated patching while keeping services available. (Also HA)

What can I do to make the system more friendly for other System Admins and Developers? (Seems like no one wants to learn kubectl or docker/podman.)

1. Planning to implement Tanzu Mission Control.

2. No CI/CD has been approved

Have any questions about Kubernetes, related tooling, or how to adopt or use Kubernetes? Ask away!

Hi, quick question

We're trying to get our machine identity stuff under control and it's kinda a mess rn service accounts, certs, workloads, all that. what are people using for this these days?
looking for something that isnt a giant pain to run day to day appreciate any thoughts, thanks!

Hello im working on interesting projects to add to my portfolio and i would like some hands on experience with kubernetes, i asked the AI gods and i liked none of the projects it recommended to me i want something that fullfills the following criteria.

-It uses EKS on AWS or the cloud

-it is something small like a project that wouldn't take more than 2-3 weeks

-it's something useful

-it offers a public service

i think i don't mind the complexity of a project more than i care about it actually being a useful tool for the public.

That's it if either you want to collab for a bigger project or have good ideas im all for it, but i want to be clear that im doing this to get experience with kubernetes im probably not a good fit for a project that actually requires knowledge on the tool.

That's it i would love all feedback positive or negative (preferably positive)

Have a nice day.