Rolling back bootc systems

We have a new built-in option available to image mode systems that typically takes more preparation with package mode operations: the rollback.

Since bootc manages state on disk, we have the previous working system available to us. Normally, we’d need to have set up snapshots or refer to a backup but bootc automatically provides us the rollback built in.

Let’s check for an available rollback option to get us back to the previous image.

Looking at the status output, we can see the section marked rollback now has image information. There are at most 3 images available at any one time: staged for next deployment, the current booted deployment, and a rollback of the last active deployment. If for some reason you needed to go back further, you can deploy directly from the registry instead.

Rollbacks are as simple as running one command. Let’s get this image back to the previous state then we can dig into what happened.

You will also find the value of rollbackQueued in the full YAML output has been updated as well. This can be useful to check before restarting a system. By adding the --format flag to the bootc status command we can get more details about the various images and operating state.

You can also check the boot order in the spec block to see what has been sent to the bootloader.

Feel free to explore the full output of bootc status --format yaml on your own.

After the reboot, the rollback image will become the booted image, and will also become the new default in the boot order. Before doing so, please make sure you are logged in to the virtual machine and not the hypervisor, the prompt should look like [core@localhost ~]$.

On the lab host, we’re going to make some changes to the Containerfile to account for how directories are managed by bootc.

In an image mode system, bootc manages available images on disk for updates and rollbacks. You just created an update, applied it, then returned to a previous version all through bootc. Behind the scenes, bootc handles the following directories differently, which is what allows for the seamless update and rollback experience.

The configuration files we added to /etc showed up after the update as a result of the merge treatment of bootc. If we’d made local changes that conflicted, we’d see the local changes rather than the new configs.

Our initial web page went in /var which means after it was unpacked from the original image, bootc treated it as local machine state. So the change in the Containerfile wasn’t applied to the running host. Since we want to control everything about our webserver from the image, we’ll need to make some changes to where we put content and how we serve it in Apache.

Let’s break down that new RUN directive.

The httpd package drops content in /var/www by default, and on bootc systems /var is machine local. For this example, to manage web content via the image, we need to move it to somewhere under bootc control. We move the default package contents to a new location in /usr then update the Apache configuration to serve pages from this new directory.

Since the heredoc support for containerfiles essentially reads it as a mini-bash script, using set -euxo pipefail will show what’s run and cause any errors in these commands to bubble up and stop the build. This way we don’t need to wait to catch an error on the host to start troubleshooting.

We’ve also changed the echo line to create the index.html in the new docroot location.

Rebuild the image with our new configuration and index page. Since this is a rebuild, podman will reuse the existing layers if there are no changes. This makes updates faster and take less space. Notice the push to the registry also only pushes those layers that contain changes.

And make sure to push it to the registry:

Now you can ssh into the virtual machine

Previously, we checked for an update, downloaded and staged it locally to be activated, then manually rebooted the system to have the update take effect. This is a very good procedure for a manual update or in places where we need to schedule any outages ahead of time, say during a maintenance window. We can do this all at once by adding a flag to the update command. This gives us a way to automate the process, like with a systemd timer. Image mode hosts ship with this timer by default.

Instead of waiting for this timer to trigger, we can immediately apply the new update and reboot.

Since we rolled back to an image that did not include our drop-in file for sudo, we will be prompted for our password. The drop-in file doesn’t have any local changes, so updates (and presence) are based on the image deployed, like our new web page. This is another thing to keep top of mind when moving between images, especially during a rollback.

Remember that the update will detail what layers are new, removed, or added, but this time will immediately reboot.

We can check for our new web page from the lab host (`[lab-user@bastion ~]$ `):

Now the output should be "Hello Red Hat Summit 2025!!"