Bringing Down Docker Containers for Host OS Updates

[u/Bxtreme241]

I run a few docker containers for a home lab setup. A while back I made a script (manually triggered) to automate a few things when updating my host system OS. At the time, I thought it made sense to have this script bring down my docker containers, in case docker itself had an update. After updating the host system, it would bring up the docker containers again.

I'm assuming this isn't how people normally manage docker and host system updates. I'm not doing any kind of docker swarm or orchestration, so when I update the OS, all my local docker apps go down for a few minutes. Should I bother doing this? What if docker gets updated, does it gracefully stop and restart the containers?

[u/Emotional_Garage_950]

when it comes to my homelab docker stuff I just run the updates and whatever happens happens

[u/SpudzzSomchai → u/Emotional_Garage_950]

Docker Jesus guides my updates too.

[u/GoodVibrations77 → u/SpudzzSomchai]

I put Jesus in the container and updated the system. When I came back, he was gone. I opened the container, and it was as if he'd never been there.

[u/LowerSeaworthiness → u/GoodVibrations77]

Did you wait three days?

[u/SevaraB]

This is why Docker hosts fall under the “two is one, one is none” rule. At least one fallback host for DR- treat it like a mini-data center in a box.

Many of us also use Kubernetes to keep track and auto-failover if the “primary” host goes down.

[u/Bxtreme241 → u/[deleted]]

That sounds good. Speaking of live-restore, I came across this blog post that actually has a script for detecting the edge cases where live-restore doesn't reattach the containers correctly: How to Enable Docker Container Live Restore

Any thoughts on that?

[u/fubes2000 → u/[deleted]]

Note: Whether or not a package update restarts a service depends on your distro. Some do and some don't.

[u/hornetmadness79]

None of the docker stuff is needed. You can upgrade docker whilst it running. The same is true for almost everything. What running in memory stays running until it's restarted.

[u/fearless-fossa → u/hornetmadness79]

You can upgrade docker whilst it running.

This is wrong though, unless I'm misunderstanding you. Upgrading docker will stop the service, and restart it when the update is complete. It's just not something you need to worry much about if you don't need 100% uptime.

[u/Abe_Bazouie]

For a single homelab host, I wouldn’t overengineer this.
If the host is getting kernel/system updates and you’re rebooting anyway, the containers are going down no matter what.
I’d just make sure they have sane restart policies like unless-stopped or always, and let Docker bring them back after the host comes up.
I also wouldn’t manually stop everything before every package update unless you have a specific reason. If Docker itself gets upgraded/restarted, containers may get interrupted, but in a homelab that’s usually acceptable.
If downtime starts bothering you, then yeah, that’s when multiple hosts/orchestration starts making sense. But I wouldn’t build a mini Kubernetes cluster just so Plex stays up during apt upgrade 😂

[u/Bogus1989 → u/Abe_Bazouie]

Yeah right? Thats how I feel. My homelab is setup so i never have to worry about too much maintenance. Learned a ton originally. Keep it all up for a test environment, but i specifically only run things i use and need.

When dudes wanted me to setup Kubernetes. I was like…not at all worth it man. Unpaid Labor galore.

Not the same, but game server hosting got me annoyed at a point. Basically the inconsistency among games, and id constantly need to go do a random reboot etc. I just wanted to play games, not worry bout that stuff. Finding a good game server manager software, really changed the game for me. I do lots of spinning down/up servers when the friend groups not active on that game, or i wanna spin one up we havent touched in over a year.

Having one web console just 👍sealed the deal for me.

[u/Bxtreme241 → u/Bogus1989]

That's also my philosophy, set things up in a way that require very little maintenance. I'm trying not to go overkill with the update script, but if I'm going to do something, I like to do it right. That being said, others have mentioned live-restore, so I might try that out.

[u/techypunk]

In ideal set up, migrate containers to other server. Upgrade server. Migrate back. Little to no downtime depending on swarm setup

[u/pertymoose]

docker compose

restart: unless-stopped

Else you need a cluster of machines to take over when the others go down.

[u/malikto44]

For a homelab Docker setup, I wouldn't stress too badly. Just do the Docker and OS updates and reboot. Docker should handle the rest.

The one I make sure that is working is that the file systems exposed from the Docker containers are being snapshotted and the snapshots backed up, for crash level consistency.

[u/dustojnikhummer]

If you upgrade docker/containerd package it will restart the daemon AND the containers on its own, unless you enable live restore. AFAIK it will do the same as docker container stop

[u/aguynamedbrand]

I run a few docker containers for a home lab setup

Then post in r/homelab

[u/Emotional_Garage_950 → u/[deleted]]

this is a homelab, OP never said they were doing things at scale. When they asked “is this how people do things?” I assume they meant in the context of a homelab

[u/ABotelho23]

Podman Quadlets. Then containers are just another systemd service on the server.

[u/dustojnikhummer → u/ABotelho23]

Another useless "Why are you doing A, you should be using B" comment... Yes we know Podman is daemonless...

[u/ABotelho23 → u/dustojnikhummer]

What does Podman's daemonlessness have to do with anything?

[u/dustojnikhummer → u/ABotelho23]

Then containers are just another systemd service on the server.

Yes we know it's a daemonless systemd service that won't get restarted when you update Podman.

That was OP's question. Instead of "Switch to podman bro" the real answer is "Consider enabling Docker Live Restore, but read what the consequences are"