Many roads lead to: RHEL 8 – Part 1

The idea behind this article comes from a discussion I had regarding the question, what approach shall I take if I want to go to the newest Version of Red Hat Enterprise Linux (RHEL) but am not able to redeploy everything from scratch.

Well, one thing to consider is doing an in-place upgrade to a new version of RHEL.

In IT this question comes up often especially with traditional Applications which got deployed once, configured once and configuration was not done by automation. So the effort to redo everything is quite substantial. And if you want to move to a new Version without redoing all of your work performing an in-place upgrade is worth considering. The major benefit is that the configuration that has been already done gets preserved. 

In the next sections we will review the process of doing such an upgrade from a RHEL 7 machine to a new Version. I will give a short summary of the general functionality as well as the required steps to do and also things to consider. 

After the general review of the whole process I would like to give you a quick overview of the possible ways to run the upgrade. As always you have choice in terms of performing the steps depending on your needs and requirements. In general you have three ways to perform the upcoming tasks. This part will focus on doing it on the CLI. The next part will focus on the other two possibilities.

General workflow and procedure

Let’s start with the basics here. Leapp is a modernization framework for the OS and Applications. It is also the base for the in-place upgrade process in RHEL. The tooling allows us to move from one major RHEL Version to another. From a process perspective you split it into a pre upgrade phase and an upgrade phase with additional post tasks that optionally need to be done. 

The analysis that gets run in the pre-upgrade phase generates a report that entails all necessary details for you to decide if it is possible to perform the upgrade or not. Additionally all road blocks are being pointed out in the report as well as possible solutions to get the roadblocks out of the way. 

When viewing the analysis you will soon see, if the roadblocks being shown are simple configurational issues or if there are more complicated blockers that need to be dealt with.

But how do you deal with inhibitors that prevent you from performing the upgrade? 

There is an interesting solution for that. You can write an Actor.

Actors are steps that get executed during the upgrade process. They can simply analyze specific data and produce information or they consume the information produced by another actor and act upon the information provided to them. Based on the concept behind Leapp you get the chance to extend the upgrade workflow with customized actors that cover specific Applications or configurations in your system. This should ease and automate the final upgrade procedure even more.

Thankfully there is a great How to Guide on writing your own actors that can help you get started.

After having all inhibitors resolved you can simply run the upgrade phase. During the upgrade phase, package dependencies get resolved, new packages get downloaded and stored in cache until the next reboot. Last but not least a new initramfs image gets created to which you boot after the upgrade to check the status and do necessary post-upgrade tasks to finalize the process. Of course reports will be generated as well for you to analyze if anything did not go according to plan.

I actually forgot to mention one key thing. You should always run a backup before doing an in-place upgrade. Just to be safe. So to put all the steps together you should:

1. Make a Backup
2. Install Leapp
3. Run the pre-upgrade analysis
4. Review the analysis 
5. Remediate base on the analysis
6. Upgrade

After getting to know the path to RHEL 8 let’s dive into some more hands-on steps.

Upgrading on the CLI

I am running the following on my personal Demo machine that is running Fedora 34 with the virtualization group installed. Additionally the Cockpit Web Console has been installed on the Server/Workstation. Both will be relevant in the other parts of this article.

So I downloaded a qcow Image of a RHEL 7 machine and deployed it as a VM, to have a clean machine. After registering the System to my Red Hat Account and running multiple customizations I started preparing the machine for the inevitable RHEL 8 Upgrade. 

First of all you need to install the required packages and enable the required rpms:

# subscription-manager repos --enable rhel-7-server-rpms

# subscription-manager repos --enable rhel-7-server-extras-rpms
Code language: PHP (php)

After that we can install the leapp packages: 

# yum install leapp leapp-repository
Loaded plugins: enabled_repos_upload, package_upload, product-id, search-disabled-repos, subscription-manager
Resolving Dependencies
--> Running transaction check
---> Package leapp.noarch 0:0.12.1-1.el7_9 will be installed
---> Package leapp-repository.noarch 0:0.14.0-4.el7_9 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=============================================================================================================
 Package                   Arch            Version                  Repository                          Size
=============================================================================================================
Installing:
 leapp                     noarch          0.12.1-1.el7_9           rhel-7-server-extras-rpms           35 k
 leapp-repository          noarch          0.14.0-4.el7_9           rhel-7-server-extras-rpms          603 k

Transaction Summary
=============================================================================================================
Install  2 Packages

Total download size: 637 k
Installed size: 1.8 M
Is this ok [y/d/N]: 
Code language: PHP (php)

Now basically we can already run the pre-upgrade analysis. But before doing that, you should check back with the documentation and perform some prerequisites that might be necessary for your environment. I will not focus on that right now, but you can see all the options in the following link.

I will go ahead and run the pre-upgrade report.

# leapp preupgradeCode language: PHP (php)

I will not post the entire output as it is just too long. But I can already see that my upgrade will be inhibited. So I will have to check the generated Reports to find out what I can remediate for the process to run successfully. 

============================================================
                     UPGRADE INHIBITED                      
============================================================

Upgrade has been inhibited due to the following problems:
    1. Inhibitor: Detected loaded kernel drivers which have been removed in RHEL 8. Upgrade cannot proceed.
    2. Inhibitor: Missing required answers in the answer file
Consult the pre-upgrade report for details and possible remediation.

============================================================
                     UPGRADE INHIBITED                      
============================================================


Debug output written to /var/log/leapp/leapp-preupgrade.log

============================================================
                           REPORT                           
============================================================

A report has been generated at /var/log/leapp/leapp-report.json
A report has been generated at /var/log/leapp/leapp-report.txt

============================================================
                       END OF REPORT                        
============================================================
Code language: JavaScript (javascript)

You can see from the end of the output that three reports have been generated and their location.  

The purpose of the JSON file is to be machine readable, whereas the TXT file is human readable. The text file reflects all the risk factors for our upgrade process with a categorization in: low, medium or high

Risk factors that are a roadblock for us are marked as inhibitors. In this case two inhibitors got detected, one of them are kernel drivers that have been removed in RHEL 8 and need to be deactivated before running the workflow again. The second one is the answer file that has not yet been populated with answers for the upgrade to run. Thankfully the report provides a recommendation on how to remediate the inhibitors. 

Risk Factor: high (inhibitor)
Title: Detected loaded kernel drivers which have been removed in RHEL 8. Upgrade cannot proceed.
Summary: Support for the following RHEL 7 device drivers has been removed in RHEL 8: 
     - floppy
     - pata_acpi
Please see https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/considerations_in_adopting_rhel_8/index#removed-device-drivers_hardware-enablement for details.
Remediation: [hint] Please disable detected kernel drivers in order to proceed with the upgrade process using the rmmod or modprobe -r.
Key: b6fd580136aaf67fa42d68fb75b27f6e13f47c2d
----------------------------------------
Risk Factor: high (inhibitor)
Title: Missing required answers in the answer file
Summary: One or more sections in answerfile are missing user choices: remove_pam_pkcs11_module_check.confirm
For more information consult https://leapp.readthedocs.io/en/latest/dialogs.html
Remediation: [hint] Please register user choices with leapp answer cli command or by manually editing the answerfile.
[command] leapp answer --section remove_pam_pkcs11_module_check.confirm=True
Key: d35f6c6b1b1fa6924ef442e3670d90fa92f0d54b
----------------------------------------
Code language: JavaScript (javascript)

After running the required tasks I will just be bold and run the upgrade. If you have applications that need to be covered it might be worth it to take a look at writing custom actors to handle any inhibitors that might come up. Let’s do it:

# leapp upgradeCode language: PHP (php)

And let’s go for a coffee. After the coffee I can already see on the report that it is green *thumbs up*. Just copying the tail of the output:

Complete!
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
==> Processing phase `InterimPreparation`
====> * efi_interim_fix
        Adjust EFI boot entry for first reboot
====> * initram_disk_generator
        Creates the upgrade initram disk
====> * add_upgrade_boot_entry
        Add new boot entry for Leapp provided initramfs.
A reboot is required to continue. Please reboot your system.


Debug output written to /var/log/leapp/leapp-upgrade.log

============================================================
                           REPORT                           
============================================================

A report has been generated at /var/log/leapp/leapp-report.json
A report has been generated at /var/log/leapp/leapp-report.txt

============================================================
                       END OF REPORT                        
============================================================

Answerfile has been generated at /var/log/leapp/answerfile
Code language: PHP (php)

In the end you will be asked to reboot and do some post upgrade tasks. So let’s do that. As a side note, the reboot will take longer as usual as the new packages are being installed after the upgrade.

Finally you can log in after the upgrade to finalise the procedure. Let us check the Red Hat Release and the Kernel version:

# cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.4 (Ootpa)
# uname -r
4.18.0-305.7.1.el8_4.x86_64
Code language: PHP (php)

As you can see from the output we managed to boot into our desired Linux version. Finally we need to perform a couple of post upgrade tasks. Additionally you definitely should check on your applications just to be sure that everything is in check.

By the way the console of my virtual machine is also showing up as RHEL 8:

In the end you will remove the old kernel version and old packages that remained in the system. It is also necessary to set SELinux again to enforcing mode, as the upgrade process is setting it to permissive mode during upgrade.

Conclusion and more to come

After having run through the manual steps on the CLI to upgrade my system from RHEL 7 to 8 I have to say it was running pretty smoothly. Of course this procedure was done with a fairly standard system and might not be comparable to a complex RHEL Server with third party applications installed and lots of customisations. Such a system will take more work in the pre-upgrade phase to make sure that everything will be in order after the upgrade. I also did not cover the process on how to backup the system, as there are multiple possibilities for doing that. It is important to say that a backup should always be taken before performing an upgrade. Sometimes you also have to consider that a clean install of a new operating system might be faster than performing an upgrade and having to customise the workflow of it. Especially if you are practicing configuration management and storing every change being performed in a repository. But if you can not automate the provisioning process, the configuration of a new machine and have to undergo a lengthy approval process to get things like DNS, IPs or firewall activations, it is worth considering to perform an in-place upgrade.

This blog post was focussing on one way to RHEL 8, but there are two other options in using Leapp besides the CLI. You can run it using Satellite or the Cockpit Web Console. Whereas the Web Console is mainly visualizing the process, Satellite can be used perfectly to group your Servers and perform the pre-upgrade analysis in parallel and thus is more efficient. 

Part two of this blog post will focus on how you can make use of Leapp with Satellite and Cockpit. This is also why the title of this article is based on the popular saying: “Many roads lead to Rome”