Planning a CentOS to Rocky Linux Migration

On June 30, 2024, the ten-year run of CentOS 7 came to an end, but more significantly, the date marked all versions of CentOS Linux reaching end of life. Now, more than six months later, many organizations are still planning their migrations to other community-supported Enterprise Linux distributions. They are exploring CentOS alternatives in the hopes avoiding the squeeze from Red Hat and having to pay for Extended Life Cycle Support, on top of RHEL licenses for each host that needs to be kept secure (which is usually all of them). 

One popular migration path is CentOS to Rocky Linux. In other blogs, we’ve looked at why many have chosen Rocky Linux, but here, we'll focus on the primary considerations for organizations that haven’t made the move yet, highlight potential risks, and explain how to evaluate the readiness of a given enterprise architecture to move from one major Enterprise Linux version (6, 7, and 8) to the latest version, Enterprise Linux 9. We'll also provide a step-by-step walkthrough of a CentOS 7 to Rocky Linux 9 migration. 

Step 1: Evaluate Whether Your Applications Are Ready for Rocky Linux Migration

How were you made aware that your team has out-of-compliance, End-of-Life CentOS hosts in their architecture in the first place? Perhaps your IT department sent you a email containing a list of these hosts, as the result of a scan. Or maybe you examined a list of AMIs in your Amazon Web Services (AWS) EC2 infrastructure, and came up with a list of affected hosts. Whatever way it happened, you have the list of hosts that need an upgrade, which means you have essentially completed the first step for migration: creating an inventory of hosts that need to make the move to Rocky Linux. 

Ideally, you have a list of applications that are running on each of these hosts, and know the purpose of each. For example, “This one is a MariaDB server,” or “This runs Oracle 12c.” If you don’t, now is a good time to start building out a spreadsheet listing the CentOS hosts that need to be upgraded that includes the workload each is responsible for. Then create another inventory, for each host, of what software is installed. Find the owners of each machine, so that you can further examine what software is installed, perhaps in unconventional ways, that you might be missing. 

Without knowing what software is on which machines, you won’t know the side effects of completing a major Enterprise Linux version upgrade. The main side effect of upgrading is new versions of the kernel, new versions of software in the yum / dnf repositories, and new versions of glibc and libstdc++. And this side effect can have some major unintended consequences. 

Step 2: Decide Between Migrating in Place vs. Building a New System

The major consequences listed above are particularly important when deciding whether to do an in-place upgrade, or building a new system. You can either migrate a system to Rocky Linux in-place, or build a new system and migrate applications and data over. Each option has pros and cons, but let’s examine the three side effects of upgrading: new kernel, new versions of dnf-sourced applications, and new API and ABI versions. 

To be clear, it’s not just the ABI contracts of glibc and libstdc++ that can stymie your upgrade plans. All bets are off regarding API and ABI compatibility between all libraries and packages between major versions of EL. Another risk of in-place upgrades is, in the end, the system may have a combination of packages from different EL versions; a few libraries from EL7 and an app from EL8 running on an EL9 box. Hybrid-version systems are incredibly difficult to troubleshoot, or even rebuild if not restoring from a full backup. 

When you examine your inventory of hosts that need to be migrated off of CentOS, you need to determine if they’re bare-metal. Physical hosts are much more likely to have custom kernel modules or drivers built against the current kernel against source, source that’s often proprietary. Perhaps the host is connected to a tape library, has a PCI-E card like a graphics card, or has another peripheral that’s connected to an industrial application from a third-party manufacturer. For this reason, a hardware inventory is incredibly important: what peripherals or non-standard hardware are installed in the host? 

Most open source drivers and kernel modules are going to seamlessly work from one major EL release to another, but add-ons that aren’t shipped with the server are more likely to need a driver to be recompiled against the new kernel in Rocky Linux. Make sure you can both obtain the source code for the driver and that it compiles against the new kernel. Otherwise you might successfully upgrade to Rocky Linux, but be stuck with an application that can’t reach a critical peripheral.  

If you are on physical hardware as described above, migrating in place has the advantage that you don't need more systems. This may be the easiest (and least expensive) route. You do have to make sure that connections are stable, and the machine will be available the whole time, because if the upgrade script is interrupted, the system may be left in an unrecoverable state. This would probably require a rescue disk and some manual work to get to a point where it is usable again. If you can't take a system out of production to rebuild, then running the migration may be the best option, as the migration can be planned in a standard maintenance window. 

However, if you are on a virtualized infrastructure, or have spare hardware, it would be safer to build a new system as you want it, then migrate the data and applications over and swap out the old system with the new. But even then, a software inventory is especially important due to the upgraded dnf-sourced software and upgraded glibc/libstdc++ libraries. 

Step 3: Use a Software Inventory to Mitigate Risk

If your organization produces software with C++, it’s possible that you’re producing applications “dynamically linked” against system libraries. If you’re targeting CentOS 7, and you upgrade to Rocky Linux 9, the libraries you linked against are going to be upgraded, and the application you wrote might suddenly crash at runtime, even if it starts successfully after the upgrade. 

This is because certain standard library methods might remain the same, allowing the application to start, but others might have been deprecated or changed, causing the application to crash when they’re no longer available. Because these safety checks are completed at compile-time, a runtime error might occur. 

Even if your organization doesn’t maintain their own C++ applications, you might be installing applications from a third-party vendor that link against CentOS 7 libraries. If this vendor uses an external yum or dnf-based repository, there’s a good chance that the upgrade to Rocky might fail due to unresolved dependency issues. If the application is installed by downloading a .tar.gz, .sh, or .run file, and binaries are installed onto the host from that installer, there’s a strong possibility that the application might suffer similar crashes or incompatibilities from unexpected versions of C++ libraries, Python bindings, Lua versions, and the list goes on. 

All of the above illustrates why it's so important to have a software inventory of some kind. It could be as simple as a spreadsheet or a Software Bill of Materials (SBOM). Once you have that inventory, you can plan ahead and contact the vendors of third-party software, making sure they have an EL 8 or EL9 version of their software that can be upgraded to once your host becomes a Rocky Linux host. 

As for the dnf-sourced packages, and considering the previously mentioned issues with version numbers changing for supporting libraries, moving from CentOS 7 to Rocky Linux 9 can include some major upgrades. For example, the upgrade from MariaDB 5.5, which was released in April 2012, to MariaDB 10.5.27, which shipped in May 2023. As you can imagine, there needs to be an end-to-end plan for this upgrade, and all of the hard-to-predict "ripples" it may cause. 

What happens when you run into applications that can’t run on the new version of Rocky 8 or Rocky 9? One option would be containerizing old workloads in order to increase security, reducing the attack surface by running in an isolated container running on an up-to-date Rocky 9 host. 

Migration Services and Technical Support for Rocky Linux

Get assistance from Enterprise Architects who are directly involved with the Rocky Linux project. 

Talk to a Rocky Linux Expert 

Step 4: Determine Your CentOS to Rocky Linux Migration Path

There are a few considerations teams will need to make when migrating from CentOS to Rocky Linux. Depending on the CentOS version(s) being used, the upgrade path may require migrating to intermediary versions before arriving at the intended version. (E.g., CentOS 6 to CentOS 7, CentOS 7 to CentOS 8, CentOS 8 to Rocky Linux 8, then Rocky Linux 8 to Rocky Linux 9). 

Approaching your upgrade with this step-wise approach is also useful for mitigating the risks described above. Catching incompatibilities early on in the upgrade process will be critical for informing leadership of how long this process is actually going to take. 

Migrating CentOS 6 to Rocky Linux 

Unfortunately, there is no direct migration path from CentOS 6 to Rocky Linux. Rocky Linux starts at 8, so hosts will have to be on CentOS 8 to migrate. As described above, there are too many changes between CentOS 6 and 7, much less 8, to migrate. Once you've migrated to CentOS 7 you can migrate from CentOS 7 to CentOS 8 then migrate to Rocky Linux 9, or migrate from CentOS 8 to Rocky Linux 8 then upgrade to Rocky Linux 9. 

For CentOS 6 users there are two ways to do this. 

1. Upgrade from CentOS 6, to 7, to 8, then migrate to Rocky Linux 8, then upgrade to Rocky Linux 9. This process can take a considerable amount of time, and can run into some additional hiccups due to the package changes between major versions.
    
2. Build a new machine and migrate your data over. The best case scenario with this approach is that all third party software needed in the new machine has a new version, and all your data can be upgraded safely. The worst case is that you'll have software with no new version and will need to find alternatives or find a way to run the software anyway. This depends on the libraries being used in your CentOS 6 install. Luckily, containerization makes it easy to run older versions of software on newer systems, or even on completely different distributions entirely. 

Migrating CentOS 7 to Rocky Linux 

The migration path for CentOS 7 to Rocky Linux is similar to CentOS 6. However, migrating from CentOS 7 to Rocky Linux is a bit easier because CentOS 7 already uses systemd for service management, while CentOS 6 still uses legacy SysV init scripts. 

There are a few other changes to keep an eye out for when moving from CentOS 7 to Rocky Linux, but, aside from the service management difference, the considerations are nearly identical to CentOS 6 migration. 

Video: CentOS 7 Migration Recommendations

Migrating CentOS 8 to Rocky Linux 

Migrating from CentOS 8 to Rocky Linux 8 is relatively painless, and avoids all of the risks described above. Since it is nearly identical, there are only a few changes, which makes this the least risky and easiest to validate step in the process. The repos are swapped out for Rocky Linux repos, and a few packages are replaced (mostly branding packages, for example, the package that provides all of the logos for CentOS). 

Real-World Example: Upgrading CentOS 7 to Rocky Linux 9

In this section, we will walk through a CentOS 7 to Rocky Linux 9 migration, showing all the steps involved and potential trouble spots. 

1. Install the Latest Version of the ELevate Repository from the AlmaLinux Project

2. Install the LEAPP Package

Specifically, we are going to install the leapp-data-rocky package, which will help us move to Rocky Linux, as opposed to AlmaLinux:

3. Run the Pre-Upgrade Checks 

This stage is typically where sticky situations will show up. Even on a simple system like the one we're using for this example, the pre-upgrade is very likely to have critical errors. 

It is worth noting that this step does not change the system, so no side effects should be expected from running this command; notice that it does not require root permissions. 

The leapp-report.txt that is generated from the pre-upgrade command contains some canned resolutions to the errors it generated. Let’s try some of those answers!

There are plenty of other messages that were generated and put in the report, but only the ones we used above were necessary for the upgrade to move forward. If you have third-party repositories, there’s a possibility that upgraded versions of system dependencies might be present. You’ll have to manually remove or downgrade those packages to resolve these version inconsistencies. 

In order for the upgrade to proceed, you can’t have any errors that will inhibit the upgrade. This will appear in the “Reports Summary” under “Inhibitors.” You may address some of the warnings that appear by examining the report and testing the suggestions they provide to see if you can resolve the warnings. Keep in mind only the inhibitors are required to proceed, though. 

4. Run the Upgrade Process, Reboot, and Test

$ leapp upgrade 

Once this command completes, it’s time to reboot into the new kernel. You will need to manually intervene and select the new boot option in GRUB: ELevate-Upgrade-Initramfs 

Now that you’ve booted into the new environment, you should see that you’re running on Rocky 8. Run through your standard QA tests to make sure the services the host is providing all work as expected. Continuing the upgrade to Rocky 9 isn’t likely to fix services that are broken at this stage, so conduct a thorough check before continuing the upgrade. 

5. Upgrade to Rocky 9

The steps for continuing the upgrade to Rocky 9 are similar to the steps we took to upgrade to Rocky 8, starting with the ELevate repo:

$ yum install -y http://repo.almalinux.org/elevate/elevate-release-latest-el$(rpm --eval %rhel).noarch.rpm 

Next, we can remove package exclusions that were added from the previous LEAPP upgrade:

$ yum config-manager --save --setopt exclude='' 

You might run into a scenario like we did where we had to remove LEAPP with its dependencies, because because leapp-upgrade-el7toel8 was still installed, and it failed because it didn't match the version. Then you can install the following: 

$ yum install -y leapp-upgrade leapp-data-rocky 

And then run the LEAPP preupgrade again:
$ leapp preupgrade 

The output of the report will be similar to the previous one. After examining the report, we had to run these steps: 

Note: If root login is allowed, the report will fail. We resolved this by overriding our sshd_config:

Here was our last report before we ran the upgrade again: 

Let’s give it a shot! 

$ leapp upgrade 

Again, even on a simple system, we can get blocking errors: 

Looking at /var/log/leapp/leapp-report.txt, there are a number of warnings, including a conflict between rocky-logos 86.3-1.el8 and rocky-logos-90.15-2.el9:

file /usr/share/redhat-logos from install of rocky-logos-90.15-2.el9.x86_64 conflicts with file from package rocky-logos-86.3-1.el8.x86_64 

To resolve this, we removed rocky-logos (which also removed rocky-backgrounds), and re-ran the LEAPP upgrade. Our next step was reboot, just as we did during the CentOS 7 to Rocky 8 upgrade, and select the grub entry ELevate-Upgrade-Initramfs again, and watch it go! 

Upon rebooting, it was time to remove the excludes again: 

$ yum config-manager --save --setopt exclude='' 

Then we could remove orphaned packages, which cleans up the system and makes it more secure: 

$ rpm -qa | grep -E 'el8[.-]' | xargs rpm -e 

The same goes for the LEAPP packages, since they’re not needed anymore:

$ dnf remove $(rpm -qa|grep leapp) 

Done! Now it's time to run our E2E and integration tests. After thorough testing of this host, it will be ready to re-enter production as an upgraded system.

Final Thoughts

Organizations with a long list of hosts in their host inventory, or hosts with especially complex software inventories, may need assistance sorting out all of the complexities associated with a CentOS to Rocky Linux migration. Or they might need more time than they initially allotted for the project. Partnering with OpenLogic for CentOS long-term support or migration services can ease the burden considerably. Our Professional Services team can help you plan your migration or provide hands-on-keyboard support throughout the process. And after you have successfully migrated, our Enterprise Architects can offer Rocky Linux support up to 24-7.

Additional Resources

• White Paper - Decision Maker's Guide to Enterprise Linux
• Video - What's Next for Rocky Linux?
• Blog - Rocky Linux vs. RHEL
• Blog - AlmaLinux vs. Rocky Linux: Exploring the Key Differences
• Blog - How to Plan a Successful Linux Migration
• Blog - What's New in Rocky Linux 9.1