Disconnected Red Hat Satellite Upgrade – Upgrading RHEL7 Satellite 6.10 to RHEL8 Satellite 6.13 : Part 2

Posted on 16th Oct 202316th Oct 2023 by Scott Ryan

Following on from the previous post, you should now have a Satellite Server that is running Red Hat Enterprise Linux 7.9 with the latest patches and Satellite 6.11.5.4.

The next step of this guide is to upgrade the Satellite Server operating system from to Red Hat Enterprise Linux 8.8 – Using the LEAPP upgrade process.

Preparing for the LEAPP upgrade

This process involves following these guidelines, but for a disconnected environment.

Download required LEAPP packages on to a connected host

The leapp packages must be downloaded on a connected helper host and then copied over to the disconnected Satellite server for local installation:

[connected-host] # yum --downloadonly --downloaddir=/var/tmp/leapp-files install leapp-upgrade
[connected-host] # tar -cvf leapp-packages.tar /var/tmp/leapp-files/*

Copy the tar file to the satellite server and install the packages

[satellite-server] # mkdir /var/tmp/leapp-files
[satellite-server] # tar -xf leap-packages.tar -C /var/tmp/leapp-files
[satellite-server] # foreman-maintain packages unlock
[satellite-server] # yum install -y /var/tmp/leapp-files/*.rpm
[satellite-server] # foreman-maintain packages lock

Prepare yum repos for LEAPP Upgrade

From the previous post, the rhel8.8 DVD iso was mounted in /media/rhel-8 and the Satellite repositories were configure using the mounted Satellite DVD isos. However, the leapp utility cannot read file:/// based repositories, requiring http based repos only. In order to facilitate this, we will use the Satellite’s public HTML directory to provide html repositories.

Create a symlink from /media to /var/www/html/pub

[satellite-server] # ln -s /media /var/www/html/pub/media

Update the Satellite yum repositories to point the local http server and relevant path:

[satellite-server] # cat <<EOF> /etc/yum.repos.d/satellite-6.11-rhel8.repo
[satellite-6.11-rhel8]
baseurl=http://rhel7-satellite-demo/pub/media/satellite-6.11-rhel8/Satellite
name=Satellite-6.11-rhel8
enabled=1
gpgcheck=0

[satellite-maintenance-6.11-rhel8]
baseurl=http://rhel7-satellite-demo/pub/media/satellite-6.11-rhel8/Maintenance/
name=Satellite-Maintenance-6.11-rhel8
enabled=1
gpgcheck=0
EOF

Disable the rhel7 and Satellite-6.11-rhel7 repositories:

[satellite-server] # sed -i 's/enabled=1/enabled=0/g' /etc/yum.repos.d/rhel7.repo
[satellite-server] # sed -i 's/enabled=1/enabled=0/g' /etc/yum.repos.d/satellite-6.11-rhel7.repo

Pre-requisite tasks for LEAPP upgrade

Two kernel modules must be removed prior to the upgrade process : floppy & pata_acpi

[satellite-server] # rmmod floppy pata_acpi

Ensure that there are no NFS mount points specified in /etc/fstab:

[satellite-server] # sed -i '/\bnfs(4)?\b/s/^/#/' /etc/fstab

Run LEAPP preupgrade check

Run the leapp preupgrade command to perform the pre-upgrade phase:

[satellite-server] # leapp preupgrade --no-rhsm --iso /<path>/rhel-8-dvd.iso --enablerepo=satellite-6.11-rhel8 --enablerepo=satellite-maintenance-6.11-rhel8

Once the pre-upgrade checks have been completed, a report and an answerfile is generated in /var/log/leapp/answerfile. The file can either be edited to provide the answer to the removal of pam_pkcs11 module (uncomment the line and confirm answer as True) or run the following command:

[satellite-server] # leapp answer --section remove_pam_pkcs11_module_check.confirm=True

Re-run the leapp preupgrade command again and confirm that there are no unresolved issues or inhibitors.

Backup / Snapshot

At this stage it is highly recommended to backup or take a snapshot of the Satellite server before performing the upgrade.

Upgrade to RHEL8 using LEAPP utility

To upgrade the Satellite server to RHEL8 using the leapp utility, run the following command:

[satellite-server] # leapp upgrade --no-rhsm --iso /<path>/rhel-8-dvd.iso --enablerepo=satellite-6.11-rhel8 --enablerepo=satellite-maintenance-6.11-rhel8

The upgrade process will start. It will determine the packages it requires and at some point will reboot and continue the upgrade. Once it has completed the last stages of the upgrade process it will reboot a few times and the Satellite server will now be running RHEL8. This process can take up to an hour. Once that is completed, there are some post-upgrade tasks which are required.

Post Upgrade Tasks

Once the upgrade has been completed, complete the following post-ugprade tasks:

Check Satellite Service Status

You may run into an error where Satellite services are not started correctly and/or httpd does not start because of an error. If so run the following command to check the service status:

# satellite-maintain service list

If you find that any services are disabled, then use the following command to enable them:

# satellite-maintain service enable --only dynflow-sidekiq@,foreman-proxy,httpd,postgresql,puppetserver,redis,tomcat

The httpd service may not start due to the duplicate loading of the mpm module, if that is the case disabled the module in the file

# vi /etc/httpd/conf.modules.d/00-mpm.conf

Setup the RHEL8 Repo from the install media

cat <<EOF> /etc/yum.repos.d/rhel-8.repo
[BaseOS]
name=Red Hat Enterprise Linux 8 for x86_64 BaseOS
baseurl=file:///media/rhel-8/BaseOS
gpgcheck=0
enabled=1
 
[AppStream]
name=Red Hat Enterprise Linux 8 for x86_64 AppStream
baseurl=file:///media/rhel-8/AppStream
gpgcheck=0
enabled=1
EOF

Remove

yum config-manager --save --setopt exclude=''
cd /lib/modules && ls -d *.el7*
[ -x /usr/sbin/weak-modules ] && /usr/sbin/weak-modules --remove-kernel 3.10.0-1160.25.1.el7.x86_64
/bin/kernel-install remove 3.10.0-1160.25.1.el7.x86_64 /lib/modules/3.10.0-1160.25.1.el7.x86_64/vmlinuz
rpm -qa | grep -e '\.el[67]' | grep -vE '^(gpg-pubkey|libmodulemd|katello-ca-consumer)' | yum remove
yum remove leapp-deps-el8 leapp-repository-deps-el8
rm -rf /lib/modules/*el7*
rm -rf /var/log/leapp /root/tmp_leapp_py3 /var/lib/leapp
rm -rf /boot/vmlinuz-*rescue* /boot/initramfs-*rescue*
dnf reinstall -y kernel-core-$(uname -r)
grubby --info=ALL | grep "\.el7" || echo "Old kernels are not present in the bootloader."

If there are still el7 kernels, find them and remove them:

rpm -q kernel|grep el7
dnf remove kernel-<version>.el7.x86_64
grubby --remove-kernel=/boot/vmlinuz-<version>.el7.x86_64

ls /boot/vmlinuz-*rescue* /boot/initramfs-*rescue*
lsinitrd /boot/initramfs-*rescue*.img | grep -qm1 "$(uname -r)/kernel/" && echo "OK" || echo "FAIL"
grubby --info $(ls /boot/vmlinuz-*rescue*)

This completes the upgrade from Red Hat Enterprise Linux 7 to Red Hat Enterprise 8.

Update to latest RHEL8 Packages

The Satellite server has now been upgraded to RHEL8, but is not completely up to date with the latest security patches and bug fixes. The next step is to synchronise the BaseOS and AppStream repositories, using a helper host and then transfer the content onto the Satellite server in order to perform the update.

Use connected helper host to sync RHEL8 OS & Satellite repos

Use the reposync tool on the connected host to copy the rhel8 BaseOS and AppStream repositories. What I prefer to do is create a separate logical volume for the repository data : /repos

This allows me to manage the content separately from the rest of the OS on the connected host.

[connected-host] # reposync --gpgcheck --newpackage --download-metadata --repoid=rhel-8-for-x86_64-baseos-rpms -p /repos/
[connected-host] # reposync --gpgcheck --newpackage --download-metadata --repoid=rhel-8-for-x86_64-appstream-rpms -p /repos/

Once completed tar the rhel-8-for-x86_64-baseos-rpms and rhel-8-for-x86_64-appstream-rpms directories in the /repo directory and copy them across to the disconnected Satellite server.

Create the local repos on the Satellite server

On the Satellite server, create an additional logical volume /repos and upack the tar files into that directory.

Create the yum repo with the following command:

[satellite-server] # cat <<EOF> /etc/yum.repos.d/rhel8-local.repo
[rhel-8-for-x86_64-baseos-rpms]
name=Red Hat Enterprise Linux 8 for x86_64 BaseOS
baseurl=file:///repos/rhel-8-for-x86_64-baseos-rpms
gpgcheck=1
enabled=1

[rhel-8-for-x86_64-appstream-rpms]
name=Red Hat Enterprise Linux 8 for x86_64 AppStream
baseurl=file:///repos/rhel-8-for-x86_64-appstream-rpms
gpgcheck=1
enabled=1
EOF

The Satellite server can now update

Red Hat, rhel, Satellite, Tech Stuff

Disconnected Red Hat Satellite Upgrade – Upgrading RHEL7 Satellite 6.10 to RHEL8 Satellite 6.13 : Part 1

Posted on 30th Aug 202331st Aug 2023 by Scott Ryan

Do you find your Red Hat Satellite infrastructure environment in desperate need of an upgrade and possibly out of support? If so, this guide is here to help you get back up to a supported version of the OS and the Satellite software.

This guide will show you how to upgrade your Satellite server from Red Hat Enterprise Linux 7 & Satellite 6.10, up to a supported configuration – Red Hat Enterprise Linux 8 & Satellite 6.13.

Most of us may have grown accustomed to seamless updates, real-time data synchronisation, and immediate access to online resources. However, there are many critical infrastructures, secure environments, and sensitive data centres that operate in ‘air-gapped’ or disconnected states for security and compliance reasons. In such settings, maintaining software and ensuring that systems are up-to-date with the latest patches becomes a unique challenge.

This guide aims to demystify the process, offering step-by-step instructions and best practices to ensure your RedHat Satellite remains current, even when it’s isolated from the outside world. Whether you’re an IT administrator working in defence, finance, or any other high-security sector, this guide will help you to navigate the nuances of updating Red Hat Satellite without a direct internet connection.

Step 1 – Preparation

Backup / Snapshot:

Before doing anything, take a backup of the Satellite server. If it is a virtual machine, it can be useful to take a snapshot before doing any changes, in order to revert back quickly if something goes wrong.

Understanding Dependencies / Pre-Requisites:

Ensure that the Satellite server is patched to the latest version of RHEL7 & download all relevant DVD isos of OS and Software required : rhel7, rhel8, Satellite 6.11 (rhel7 & rhel8), Satellite 6.12 (rhel8) & Satellite 6.13 (rhel8).

Assessing the current environment :

It is important to understand what ‘disconnected environment’ really means. For this example, the Satellite server has no access to the Red Hat CDN or any upstream / sync’d repositories. Content has to be synchronised and copied onto the server directly or placed on a host that the Satellite server can access via http. For this guide, the Satellite server has no access to any repositories at all, so we will be working on the assumption that everything has to be copied locally onto the server.

Ensure there is enough disk space on the host especially the / and /var partitions. The upgrade from rhel7 to rhel8 requires a change to the postgresql data directory, so ensure that there is enough space (20GB) in the /var/lib/pgsql partition

Additionally, allow for an additional 10GB for hosting the latest rhel7 and rhel7-extras repositories. In this example, this will be in /var/repos.

Step 2 – Update the RHEL7 OS to latest patches

Before upgrading, it is imperative that the rhel7 server is updated to the latest patches. The LEAPP meta data checks which versions of the packages are required and will upgrade them if needed. However in this example of a disconnected environment, the server has no access to the Red Hat content delivery network, and neither does it subscribe to an upstream satellite server, so the latest rhel-7.9 content has to be imported manually to do the latest updates.

To do this, a helper host which has connectivity to the Red Hat CDN is required, with the reposync and yum-utils packages installed. This host is used to create a local mirror of the rhel7 repository and then transfer the content to the disconnected Satellite server.

On the connected host – Ensure it has a valid subscription and is registered, using subscription manager. Enable the rhel-7-server-rpms and rhel-7-server-extras-rpms repositories.

# subscription-manager repos enable rhel-7-server-rpms rhel-7-server-extras-rpms

Create the local mirror repositories:

# mkdir -p /var/repos
# reposync --gpgcheck -l --repoid=rhel-7-server-rpms --download_path=/var/repos --downloadcomps --download-metadata
# reposync --gpgcheck -l --repoid=rhel-7-server-extras-rpms --download_path=/var/repos --downloadcomps --download-metadata
# createrepo -v /var/repos/rhel-7-server-rpms -g comps.xml
# createrepo -v /var/repos/rhel-7-server-extras-rpms -g comps.xml
# tar -czvf rhel-7-repos.tar.gz /var/repos/*

Copy the tar files to the disconnected satellite server and untar in the /media directory (create the /media directory if required – ensuring there is enough space on the partition)

On the disconnected Server : Create the repository file in /etc/yum.repos.d

# cat <<EOF > /etc/yum.repos.d/rhel7.repo
[rhel-7-server-rpms]
baseurl=file:///media/rhel-7-server-rpms
name=rhel-7-server-rpms
enabled=1
gpgcheck=0

[rhel-7-server-extras-rpms]
baseurl=file:///media/rhel-7-server-extras-rpms
name=rhel-7-server-extras-rpms
enabled=1
gpgcheck=0
EOF

Update the host to the latest rhel7 packages:

# yum update -y

Step 3 – Download & Mount ISOs

Download the Red Hat 8.8 and Satellite 6.11, 6.12 & 6.13 dvd isos from the Red Hat portal. As of writing this, Satellite 6.11.5.4 is the latest version for RHEL7.

Download 6.11.5.4 iso for both RHEL7 & RHEL8 and copy them onto the disconnected Satellite server, along with the RHEL8.8 DVD iso.

Create the directories for all the media and mount the images to the respective directories:

# mkdir -p /media/{rhel-8,satellite-6.11-rhel8,satellite-6.11-rhel7}
# mount -o loop /var/isos/rhel-8.8-x86_64-dvd.iso /media/rhel-8
# mount -o loop /var/isos/Satellite-6.11.5.4-rhel-8-x86_64.dvd.iso /media/satellite-6.11-rhel8
# mount -o loop /var/isos/Satellite-6.11.5.4-rhel-7-x86_64.dvd.iso /media/satellite-6.11-rhel7

Mount all of the isos into the respective directories, once this is done – the next thing to do is to create the repositories for the server to use.

Setup the RHEL7, Satellite 6.11 Repositories

Create the yum repo files and ensure the base urls point to file:///media/<repo> :

# cat <<EOF > /etc/yum.repos.d/satellite-6.11-rhel7.repo
[satellite-6.11-rhel-7]
baseurl=file:///media/satellite-6.11-rhel7/Satellite
name=Satellite-6.11-rhel8
enabled=1
gpgcheck=0

[satellite-maintence-6.11-rhel-7]
baseurl=file:///media/satellite-6.11-rhel7/Maintenance/
name=Satellite-Maintenance-6.11-rhel8
enabled=1
gpgcheck=0

[ansible-rhel-7]
baseurl=file:///media/satellite-6.11-rhel7/ansible/
name=Ansible-rhel7
enabled=1
gpgcheck=0

[rhel-7-server-rhscl-rpms]
baseurl=file:///media/satellite-6.11-rhel7/RHSCL/
name=rhel-7-software-collections
enabled=1
gpgcheck=0
EOF

A yum repo list will now show all of the necessary repositories for upgrading the Satellite Server.

Upgrade Satellite Server to 6.11

We are now in a position to update the Satellite server to 6.11. Follow these instructions to upgrade the Satellite Server to 6.11. Steps 12 to 16 have been completed above. Every environment is different, so follow the instructions necessary for your environment. This guide is only showing you how to get your disconnected Satellite in a position where it can be upgraded.

Depending on how you got to Satellite 6.10 (i.e. Upgrade from 6.9 or fresh install of 6.10) you may need to create the /var/lib/pulp/media/artifact directory:

# mkdir -p /var/lib/pulp/media/artifact
# chown -hR pulp:pulp /var/lib/pulp/media/artifact

Run the command to check that there is an upgrade available:

# satellite-maintain upgrade list-versions
# satellite-maintain upgrade check --target-version 6.11 \
--whitelist="repositories-validate,repositories-setup"

You may have to add a few whitelists, for example if you have any non Red Hat packages. Once satisfied that all the checks are successful, do a backup / snapshot then commence the upgrade by running the following command:

# satellite-maintain upgrade run --target-version 6.11 \
--whitelist="repositories-validate,repositories-setup"

This upgrade process should complete in approx 45 mins, you may be asked if you want to clean up old tasks – agree to that, and by the end of it you will now be running up to date rhel7.9 & Satellite 6.11

The next step is to upgrade from Red Hat Enterprise Linux 7 to Red Hat Enterprise Linux 8.

This is described in part 2 ->

I like sharing my knowledge and experiences, in the hope that is helps others. Feel free to add comments / suggestions and if you found that this was helpful, and would like to shout me a coffee, that would be appreciated!

Virtual-Shift