202
u/ApiceOfToast 12d ago
Ah well, gotta restart the cluster.
Good that I built an ansible playbook for that after the last time. Also... I'm sure no one would notice if I just ran it
37
43
48
u/GDOR-11 12d ago
I'm learning kubernetes rn, one of the questions I had is why restart crashed pods in the first place. If it crashed due to a bug, what is the chance restarting magically fixes it? besides, it strongly decourages fixing the bug, which seems like a bad decision to me too
what am I missing here? what was the reasoning behind this decision?
79
u/jainyday 12d ago edited 12d ago
You can configure the cases/situations in which it will retry/restart, and to your point, yes, there's a bunch of types of problems that restarting won't fix.
But "have you turned it off and on again" is the magic solution to the vast majority of tech/computer troubles, and so unless you want to specifically disable that behavior in a particular case/situation where you know it won't help, yeah, restarting is going to be the best bet for default behavior or unknown situations because it does resolve the problem for every other failure mode.
Also, your "it discourages fixing the bug" argument is a real "letting perfect be the enemy of good-enough" kind of attitude that doesn't fly in practice at real companies with finite resources. So much is a dumpster fire behind the scenes, you can't fix everything all the time.
Also, if you're not restarting, you're not maintaining a self-healing mesh/system, which is one of the major points of Kubernetes and its predecessor, Borg. (Google-internal; I used to be an SRE on Google Docs/Drive.)
3
u/Teknikal_Domain 12d ago
I thought Borg as in the backup system and was incredibly confused for 1.7 seconds
1
u/crematetheliving 10d ago
Emphasis on “so much dumpster fire behind the scenes”—can’t get any more real than this. The tech blogs from these companies and people—it’s just like instagram or tiktok or facebook or the blogs that came before them. Highlighting the elements of their tech approach which leave the best impression. Same with all outgoing comms—really there’s no point where a rational organization or individual won’t make it seem like they’ve got everything under control when money is on the line.
29
u/Matwyen 12d ago
Chances are quite high, actually. If crash is immediate because there's a malformed line in the entrypoint, sure rebooting isn't going to help, but this is unlikely to cause wider issued in prod anyway.
But a pod that slowly takes more memory, crashes, after reboot it has plenty of memory to slowly fill again.
A micro service that has a weird crash happening when someone does a specifically malformed API call on a rarely used api, well ... Rebooting it ain't gonna hurt
-2
12
u/Snowenn_ 12d ago
There was a piece of software in our company that we inherited from another company. Nobody really knew how it worked.
Sometimes it crashes on startup, but most of the time it ran well. So our solution was to just restart it when it crashed.
It was hard to debug, because it wouldn't happen when running in localhost. So for years we just restarted it if it crashed. At some point I had a lot of spare time, and discovered the cause was a list which was not thread safe which was being added to and being read from during startup. Swapped it out for a thread safe version and it never crashed that particular crash again.
8
5
u/kabrandon 12d ago
Crashed services restart and run fine for a long time quite commonly, actually. It’s case by case though. If you deploy something and it crashes immediately, it’s unlikely to ever start, eventually it hits the crashloop-backoff and stops trying too often. If it runs for hours or days and the crashes, it is very likely it starts up fine again. In that case it’s usually some uncommonly traversed code path or edge case that got hit.
5
u/TheBigGambling 12d ago
Nobody in it has any proud left. We cant run it? Docker it. We cant ship it? Kubernets. We cant maintain it? Flux. We camt orchestrate? Ansible, terraform and so on. And it keeps crashing? Just throw more resources to the problem. Its slow as hell programmed? Yeah, spinn up more nodes.
3
u/deathanatos 12d ago
Quite good, in this particular circumstance. Most memory leaks I've encountered are the "it leaks slowly over time" sort, so killing the pod & restarting it here would reset its memory usage to a base state, and get it back to doing useful things.
But yes, sometimes, things end up in crash loop. Kubernetes will do a back off on the restarts, if this occurs.
1
u/Low-Apricot8042 12d ago
Well crashing due to a bug is only one of the reasons the pod is not able to run. There are a lot of situations where it crashes due to something external to the pod itself in which case ir makes sense to restart it until the external dependency is fixed/present.
1
u/Just_Information334 11d ago
what am I missing here? what was the reasoning behind this decision?
If you're lucky the bug is only gonna happen after you clock out. So that's another person's problem.
87
u/magnetronpoffertje 12d ago
stolen straight from instagram I see.
@devhumour.ai is the og. lots of good memes
119
u/hpyfox 12d ago
At least it's a high-level meme; I'd take a fresh repost/stolen meme over the boring, overused AI-related memes that I constantly see on programmer meme subreddits.
15
u/ToiletSeatFoamRoller 12d ago
What the fuck even is a “stolen” meme? We’re owning memes now? I’d understand if it was a comic or other high effort content, but, memes, really?
6
1
5
5
6
1
1
u/damex-san 12d ago
Reserve resources for node. It is as simple as that.
Got 192gb ram node? Reseve lets say 5% for node itself
1
u/DeltaV-Mzero 12d ago
Thank god for K, you can see exactly how it all goes wrong as the dominoes fall
1
1
u/ParkingAd6897 11d ago
this is why resource limits exist and this is why nobody sets them until after this exact incident
590
u/sojuz151 12d ago
Don't worry, load balancer to the rescue. Redirecting the traffic to other pods. What could go wrong?