In-Place Upgrades (leapp)
Skill Level: Advanced
1. Overview
| This exercise requires the SECONDARY terminal and conflicts with any other exercise (ex: ebpf) that uses the secondary terminal. Do not proceed if the secondary terminal is already in use with another exercise. |
In this unit we introduce LEAPP, an in-place OS upgrade tool used to:
-
analyze a host,
-
report on potential issues,
-
provide remediation assistance,
-
and finally execute the upgrade procedure.
Because the upgrade is a lengthy process, this exercise uses the secondary terminal. The intent is to dedicate the secondary terminal to the leapp upgrade, thereby leaving the primary terminal available for other exercises while the upgrade executes.
The first several sections leading up to the actual upgrade are interactive and should take around 7 - 8 minutes to complete. The sections which perform the actual upgrade and finalize it are almost completely non-interactive and each are expected to take 6 - 9 minutes to complete. Once you start them, you will have time to look at other exercises in the PRIMARY terminal while waiting for them to complete.
2. Getting Started
For these exercises, you will be using the host leapp as user root. Before you begin, confirm the node was provisioned with RHEL 9.6
Launch a 'tmux' (terminal multiplexer) in the second showroom terminal:
workshop-tmux.sh TERM2Using the second terminal for all steps below, proceed to ssh from host bastion to leapp.
ssh leappUse sudo to elevate your privileges.
[[ "$UID" == 0 ]] || sudo -iVerify that you are on the right host for these exercises.
workshop-leapp-checkhost.shYou are now ready to proceed with these exercises.
3. Current RHEL Version Verification
Verify through a few commands that this is currently running RHEL 9. These will show the current version of RHEL, the version of the running kernel and the rpm repos available to the system.
cat /etc/redhat-releaseRed Hat Enterprise Linux release 9.6 (Plow)
uname -r5.14.0-570.13.1.el9_6.x86_64
(note: your output may differ)
dnf repolistUpdating Subscription Management repositories. repo id repo name ansible-automation-platform-2.4-for-rhel-9-x86_64-rpms Red Hat Ansible Automation Platform 2.4 for RHEL 9 x86_64 (RPMs) ansible-automation-platform-2.5-for-rhel-9-x86_64-rpms Red Hat Ansible Automation Platform 2.5 for RHEL 9 x86_64 (RPMs) rhel-9-for-x86_64-appstream-rpms Red Hat Enterprise Linux 9 for x86_64 - AppStream (RPMs) rhel-9-for-x86_64-baseos-rpms Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs) rhel-9-for-x86_64-supplementary-rpms Red Hat Enterprise Linux 9 for x86_64 - Supplementary (RPMs)
You are now ready to proceed with the exercises.
| When performing an in-place upgrade in your own lab or production environments, you will need to disable any non-Red Hat repositories prior to starting the in-place upgrade process. Application upgrade(s) will need to take place after the operating system upgrade has completed. A command similar to the following could be used: |
dnf config-manager --set-disabled <my-custom-application-repo-name>
4. Installation
4.1. Tools
Install the LEAPP tools and any dependencies (this may already be done)
dnf install -y leapp-upgrade4.2. Cleanup
Delete any results from any possible previous attempts to run leapp
rm -rf /var/log/leapp /root/tmp_leapp_py34.4. Reboot
To ensure that all updates have taken effect and that we are running on the latest kernel, check the hostname is correct and reboot.
[[ $(hostname -s) == "leapp" ]] && rebootUsing the second terminal reconnect to the leapp node once it has rebooted (should be 15 - 20 seconds).
ssh leappUse sudo to elevate your privileges.
[[ "$UID" == 0 ]] || sudo -i5. Pre-Upgrade
5.1. Scan
Now run the preupgrade option to generate a preliminary report to see if there are any issues found that could prevent the upgrade from proceeding smoothly.
leapp preupgrade5.2. Report
The output from the previous command should have listed a few items that are inhibiting the upgrade of the host.
============================================================
REPORT OVERVIEW
============================================================
Upgrade has been inhibited due to the following problems:
1. Not enough space on /boot
HIGH and MEDIUM severity reports:
1. GRUB2 core will be automatically updated during the upgrade
(... output omitted ...)
Before continuing, review the full report below for details about discovered problems and possible remediation instructions:
A report has been generated at /var/log/leapp/leapp-report.txt
A report has been generated at /var/log/leapp/leapp-report.json
============================================================
END OF REPORT OVERVIEW
============================================================
Answerfile has been generated at /var/log/leapp/answerfile
Notice that the output refers you to the pre-upgrade report for details and remediations. If your system has the cockpit-leapp package installed, you can switch to using a web-broswer to step through each item and inspect the remediation options. A sample image is provided at the end of this exercise as a reference.
Look at the first several lines of the report mentioned above, /var/log/leapp/leapp-report.txt
head /var/log/leapp/leapp-report.txtThe first two lines indicate a Risk Factor, in this case high/inhibitor meaning that this issue will prevent the upgrade from proceeding entirely.
Risk Factor: high (inhibitor)
Title: Not enough space on /boot
Summary: /boot needs additional 24.09765625 MiB to be able to accommodate the upgrade initramfs and new kernel.
Related links:
- Why does kernel cannot be upgraded due to insufficient space in /boot ?: https://access.redhat.com/solutions/298263
This tells us that there is not enough space in the /boot directory to proceed. Let’s take a look:
ls -l /boottotal 859436 -rw-------. 1 root root 9431032 Apr 4 10:47 System.map-5.14.0-570.12.1.el9_6.x86_64 -rw-------. 1 root root 9431942 May 24 16:01 System.map-5.14.0-570.19.1.el9_6.x86_64 -rw-r--r--. 1 root root 614400000 Jun 10 14:49 bigfile -rw-r--r--. 1 root root 229215 Apr 4 10:47 config-5.14.0-570.12.1.el9_6.x86_64 -rw-r--r--. 1 root root 229370 May 24 16:01 config-5.14.0-570.19.1.el9_6.x86_64 drwx------. 3 root root 16384 Dec 31 1969 efi drwx------. 4 root root 65 Jun 10 13:32 grub2 (output truncated...)
Sure enough, there is a very big file, coincidentally called "bigfile" that is over 600MB. It was placed there to show what an "Inhibitor" would look like as part of the preupgrade process, so we know that it is safe to remove. The Leapp documentation (link at the end of the exercise) goes into more examples of things that could be inhibitors like device drivers that may not be supported. It also outlines environments that are not supported for an in-place upgrade such as Network based multipath and network storage that use Ethernet or Infiniband. This includes booting from SAN using FC. Note that SAN using FC is supported, just not booting from SAN.
6. Remediate
With that in mind, let’s fix the blocker/inhibitor and make sure there’s enough space in /boot
rm -f /boot/bigfiledf -h /bootFilesystem Size Used Avail Use% Mounted on /dev/vda3 960M 318M 643M 34% /boot
Much better. Now re-run the preupgrade. This time there should be no inhibitors, and it will now take a couple minutes longer:
leapp preupgradeThe output should now come back clean without any inhibitors that would prevent a successful upgrade. (about 5 minutes)
============================================================
REPORT OVERVIEW
============================================================
HIGH and MEDIUM severity reports:
1. GRUB2 core will be automatically updated during the upgrade
2. Berkeley DB (libdb) has been detected on your system
Reports summary:
Errors: 0
Inhibitors: 0
HIGH severity reports: 1
MEDIUM severity reports: 1
LOW severity reports: 3
INFO severity reports: 3
Before continuing, review the full report below for details about discovered problems and possible remediation instructions:
A report has been generated at /var/log/leapp/leapp-report.txt
A report has been generated at /var/log/leapp/leapp-report.json
============================================================
END OF REPORT OVERVIEW
============================================================
Answerfile has been generated at /var/log/leapp/answerfile
Feel free to inspect the /var/log/leapp/leapp-report.txt file. Everything should be ready to run the upgrade.
7. Upgrade
Now execute the upgrade. This will install several rpms, make some repo and other configuration changes, and will take several minutes (7 to 10 in our vm testing).
time leapp upgradeAfter several minutes, you should see an almost identical report output indicating that phase one of the upgrade has completed
Running transaction test
Transaction test succeeded.
Complete!
====> * add_upgrade_boot_entry
Add new boot entry for Leapp provided initramfs.
A reboot is required to continue. Please reboot your system.
Debug output written to /var/log/leapp/leapp-upgrade.log
============================================================
REPORT OVERVIEW
============================================================
HIGH and MEDIUM severity reports:
(output truncated...)
Answerfile has been generated at /var/log/leapp/answerfile
Reboot the system to continue with the upgrade. This might take a while depending on the system configuration.
Make sure you have console access to view the actual upgrade process.
real 7m31.299s
user 4m35.191s
sys 2m32.107s
8. Finalize the Upgrade
To finish the upgrade process, a reboot is now required. Without console access you won’t be able to see the final output in which the system boots into a special leapp initramfs and completes the in-place upgrade in single user mode. (6 - 9 minutes)
[[ $(hostname -s) == "leapp" ]] && rebootConnection to leapp closed by remote host. Connection to leapp closed.
While the upgrade is completing, here is a command you can run on the bastion to loop until the leapp host is back online
time until $( nc -z leapp 22 ) ; do echo -n "." ; sleep 3 ; done| Do not proceed until the leapp upgrade process is complete |
ssh leappUse sudo to elevate your privileges.
[[ "$UID" == 0 ]] || sudo -i9. Final RHEL Version Verification
Finally, re-run the commands from earlier to verify that the leapp node has actually been upgraded to RHEL 10.
cat /etc/redhat-releaseRed Hat Enterprise Linux release 10.0 (Coughlan)
uname -r6.12.0-55.16.1.el10_0.x86_64
dnf repolistUpdating Subscription Management repositories. This system has release set to 10.0 and it receives updates only for this release. repo id repo name rhel-10-for-x86_64-appstream-rpms Red Hat Enterprise Linux 10 for x86_64 - AppStream (RPMs) rhel-10-for-x86_64-baseos-rpms Red Hat Enterprise Linux 10 for x86_64 - BaseOS (RPMs) rhel-10-for-x86_64-supplementary-rpms Red Hat Enterprise Linux 10 for x86_64 - Supplementary (RPMs)
| The Leapp process has explicitly set the release version to "10.0". This means that updates will only be available and applied for the RHEL 10.0 version. This can be modified to a more general version "10" in order to receive ongoing updates as more point releases of RHEL 10 come out using either of the following: |
subscription-manager release --set 10 subscription-manager release --unset
10. A Word About Web Console Integration
This exercise has illustrated a very simple example of upgrading a RHEL 9 system in place to a RHEL 10 system, with a single issue that was easy to remediate and no applications running on top of the OS. In the real world, there are likely to be more issues that need to be addressed and in some cases there are known issues that will prevent an in-place upgrade (see official Red Hat documentation link below). There is a Web Console plugin that makes it easier to visualize and, in many cases, remediate upgrade inhibitors that arise. The rpm is called "cockpit-leapp" and, once installed, will enable visualization like this
11. Conclusion
To upgrade in-place or rebuild is a decision that needs to be made from one environment to the next, one group to the next, even from one system to the next. What makes sense for one application might not make sense for another. As with any OS upgrade, test in the lab and do backups!
Time to finish this unit and return the shell to its home position.
workshop-finish-exercise.sh