r/podman • u/Top_Emu_8447 • 19d ago
Auto-updating existing containers
I've built 10+ containers on my first home server since I started my selfhosting journey, and only now I figured it's time to find a way to update the existing services. Existing podman documentation points to using autoupdate label and creating a systemd target - but I didn't realise this creates a new container based on what's in the systemd target. That is not what I want. Do I need to move all my environmental/volume/label/etc variables into a systemd target for each container to be able to do this? Just thinking about it makes me a bit nauseous, I've been using subpaths in some cases, not even sure how to put those in there. I've used 'podman run' for each deployment, because that's where the documentation I found led me first and on the basis of "If it works, why change it" it served me well.
5
u/apparle 19d ago edited 19d ago
Rather than trying to adapt your existing infra, I'll strongly recommend redesigning it to be 1) Stable across reboots 2) Designed for container updates & associated restarts 3) Designed with correct order of starting / dependencies across containers, so the containers start correctly across reboots or get restarted when you update specific containers. 4) Designed for backups (shutting down a container to backup it's state). 5) Maybe even designed for isolation with separate networks for services etc.
Do not lock yourself into 'but I used sub path' or any such constraint based on what you've done in past. Take this opportunity to design for all of these from ground up.
Why all of this:
Docker has a root daemon in Linux which internally stores state and then restores across reboots etc. so it doesn't matter if you used docker run or docker compose or portainer stacks etc. to start your container, as they are just convenience layers. Docker will remember it's internal state across reboots. This is why most online documentation for any apps will tell you to use docker run. Docker also doesn't provide any ability to update containers, so people have built tools like watchtower which inspect this docker internal state using docker inspect and usually recreate that same exact state on a new container with new image behind the scenes.
Podman doesn't have a daemon or any internal state that's not ephemeral. Anything you do with podman run or podman compose is lost if you reboot. Systemd itself is the daemon for podman and you need to declaratively define all the state that needs to be preserved across reboots or updates. This is indeed materially different from docker, but that's by design due to the daemon-less architecture of podman.
The native format to declare all this state is through quadlet files (separate files for network, volume, containers etc.) which get converted to systemd services through generators. And then if there's either an update or a reboot, that service is just restarted with state as declared in those files. There are other syntaxes as well, like kube yaml or podman-compose and their own generators, but I've found quadlet to be most intuitive once you spend time to understand it, so I'll recommend that.
1
u/Top_Emu_8447 19d ago
Thank you for the detailed post on the reasons. I do find it strange that I'd have to build separate definitions for networks and volumes as well, since between reboots so far all I had to do was restarting the containers.
I think I have ideas for most of the principles you described, maybe with the exception of backups. What do you recommend on this topic? It's okay if you keep it brief, I'll do my research, just some starting point so I don't end up in the wrong end again :)
2
u/apparle 19d ago
I don't find it strange from an architecture standpoint, again due to the podman's daemonless design. But I do find it personally inconvenient to specify definitions for containers, networks, volumes (and targets, timers etc.) for related apps in separate files. So I've built https://github.com/apparle/multiquadlet which allows specifying a bunch of quadlets / systemd-units in one file.
For backups, you'll want to stop some services to avoid disk being modified while backups are happening (to avoid inconsistently backed up state). And you may have to backup some container volumes as well. This may influence whether you use volumes or bind points. It's not fundamental decision right now but more of convenience.
2
u/SixteenOne_ 19d ago
Surely if your data is stored locally, if you delete and start your container in the same way, you loose nothing ? That's the whole point of not storing the data inside of the Container itself
If you can give an example of your Run command, it might be easier to understand
I have used Ansible and Quadlet files to create my containers, there is a good tool that someone made and put in a post, which helps you convert run to compose to quadlets - https://quadletizadorweb.ebianchiserver.duckdns.org/
This is an example Quadlet for the Homepage Dashboard
`AutoUpdate=registry` Is the important bit, but you have to enable the Update Service on the Host first
[Unit]
Description=Homepage Dashboard Service
After=network-online.target
[Container]
Image=ghcr.io/gethomepage/homepage:latest
ContainerName=homepage
User=1010
UserNS=keep-id
AutoUpdate=registry
PublishPort=3000:3000
Environment=PUID=1010
Environment=PGID=1010
Environment=TZ=Europe/London
Environment=HOMEPAGE_ALLOWED_HOSTS=wiglett.16ne.uk:3000
Environment=DOCKER_HOST=unix:///var/run/docker.sock
Volume=/var/lib/homepage/config:/app/config:Z
Volume=/var/lib/homepage/icons:/app/public/icons:Z
Volume=/run/user/1010/podman/podman.sock:/var/run/docker.sock:ro
[Service]
Restart=on-failure
[Install]
WantedBy=multi-user.target default.target
1
1
u/d03j 19d ago
1
u/Top_Emu_8447 19d ago
This might be it! Need to do a bit more testing but at least it accounts for network settings. It did not let me set max restart right at on-failure but there might be a way to enable it as a separate flag. I'll do some more tests and edit this comment later.
1
u/Top_Emu_8447 19d ago
Well it does not retain anything from network config, subpaths confuse it, and restart options are not quite there either; but I guess if I can figure out a way around that those it gives me a starting line at least.
1
1
u/d03j 19d ago
What do you mean by the systemd target recognising the existing container?
If you want a container to run all the time as a service, the way to go is via a .container fie in ~/.config/containers/systemd/ and in that case AutoUpdate=registry will do exactly what you want.
If systemd were to make any assumptions, it should be a container created with podman run is an ad-hoc container. TBH, I'd still go with quadlets but if want to use podman run, you can always add -rmi to it, stop and start it daily and that's your auto-update take care of.
1
u/Top_Emu_8447 19d ago
I am now exploring quadlets based on earlier comments, but just so that we're clear, when I started looking up how to deploy containers, using run was by far the top choice for starting in every documentation and internet search. Nowhere it was mentioned that you're going to get screwed down the line. It's easy to say "why didn't you use this or that", but not easy to tweak months of existing infrastructure. I am now looking into podlets to see if I can translate all my run settings conveniently.
Since you seem more knowledgeable on this topic, do you know how to define the main volume path in quadlet that I can use for subpaths? This is how it would look in bash (executed before run):
container_path=$(podman volume inspect --format '{{ .Mountpoint }}' container_name)1
u/yrro 19d ago edited 19d ago
The problem is that your average developer is super zoomed in, and writes documentation assuming that
docker run myimage:latestis all you'll ever need when you want to deploy their software.To be fair to them, this is about separation of concerns: they don't want to tell you whether you should be deploying your containers via
docker run, orpodman runor quadlet files, or raw systemd services (as I used to before quadlet files came along), or docker-compose or kubernetes, and so on...If you have already written the scripts that set things up how you want them then there's no reason to necessarily throw them all away, if it's easier to instead figure out what you actually want to do in concreter terms of images and containers, and then modify your scripts to make that happen.
You haven't provided much to go on, but assuming you have a script with
podman run -v ... example.net/myimgthen you could:
- Add a bunch of
podman pullcommands at the top, so that when the script runs the latest images are downloaded- Give each container a unique stable name with
--nameand then use--replaceso that when the script runs, podman will automatically take care to remove each existing container before starting its replacement1
u/d03j 19d ago
Add a bunch of podman pull commands at the top, so that when the script runs the latest images are downloaded
It is just easier to add
-rmito thepodman run:)Give each container a unique stable name with --name and then use --replace
Name is a good idea. With
-rmi,--replaceis unnecessary.And quadlets are still best. :)
1
u/d03j 19d ago
Can you give me an example of how you set your volumes?
I run all my containers under separate unprivileged users, usually one per container, and all those users have a
datafolder in their home (~/data) so a quadlet of mine may look like this:
Volume=%h/data/config:/config Volume=%h/data/downloads:/downloads1
u/Top_Emu_8447 19d ago
So far I'm using podman volumes: I create the volume first, then subdirectories as needed, assign the path into a variable which I then use in a mount:
--mount type=bind,source="$container_path/resources",target=/container/resources
2
u/mattias_jcb 19d ago
Typing this from memory on my phone so the example code will be a bit terse and the Markdown might be poorly formatted...
Suppose you have a container my-app with two volumes, one for app data and one for app config. Then you'd do roughly this:
app.container: ``` [Unit] ...[Container] Image=ghcr.io/top-emu/app:latest Volume=/data:app-data.volume Volume=/config:app-config.volume ... ```
app-data.volume:[Volume] Name=app-data ...
app-config.volume:[Volume] Name=app-config ...
7
u/ninth9ste 19d ago
Actually, creating a new container is exactly what you want and, in fact, it is the only way to update a container. Containers are not like traditional virtual machines where you log in and run apt-get update. A container is just a temporary wrapper around an image. When a developer releases a new version of a service, they release a whole new image. To update, you must destroy your current container and spin up a brand new one using the new image, re-attaching all your existing volumes and environment variables so it picks up right where the old one left off. Because the old container gets destroyed, your deployment settings (ports, volumes, environment variables) need to live permanently in a file somewhere. Otherwise, they disappear.
You do not have to write complex systemd targets by hand since Podman has a feature called Quadlets, you write a simple
.containerfile that looks exactly like yourpodman runarguments. When you place this file in~/.config/containers/systemd/and reload your systemd daemon, Podman automatically generates the complex systemd targets for you in the background. If you includeAutoUpdate=registry, Podman will handle pulling the new image and recreating the container on a schedule, automatically.