openshift, Tech Stuff

Updating Disconnected OpenShift Clusters

Posted on by Scott Ryan in openshift, Tech Stuff

Intro

Disconnected OpenShift clusters are becoming increasingly popular, as more and more organisations seek to deploy and manage their own private cloud environments. In this blog post, I will show you how to mirror repository content to update a disconnected OpenShift cluster as well as some best practices for managing your environment.

Disconnected OpenShift clusters are important for a number of reasons. First and foremost, they allow organisations to maintain complete control and security over their environment, without relying on external sources for updates, patches, or security fixes. This is especially important for organisations that operate in highly regulated industries, where data privacy and security are paramount.

Updating a disconnected OpenShift cluster can be a complex process, but it’s not impossible. Here are the basic steps:

  1. Set up a local mirror repository: A local mirror repository is a copy of the OpenShift container images that you’ll use to install and manage your cluster. You can create a mirror using the oc-mirror plugin.
  2. Move the mirrored content into the registry in the disconnected / isolated environment.
  3. Configure the cluster to use the local, mirrored registry: Once you have a local mirror registry, you’ll need to apply the manifests to configure the OpenShift cluster to use it.
  4. Update the cluster!
  5. Manage the cluster: Once your disconnected cluster is updated, you’ll need to manage it. This includes applying updates, patches, and security fixes. To do this, you’ll need to periodically update your local mirror repository with the latest container images.

The oc-mirror plugin is a command-line tool that can be used to mirror images, manifests, and other artifacts, from external repositories in order to sync them to a mirrored registry for use by a disconnected cluster. This is an essential tool for organisations that need to manage OpenShift clusters in remote or offline environments.

The oc-mirror plugin works by creating a local mirror of the upstream repositories into a local registry or filesystem on removable media. This can then transferred into the disconnected environment, to update or deploy OpenShift applications and services in the disconnected cluster.

The oc-mirror plugin is a powerful tool that can be used to manage OpenShift clusters in disconnected environments. It is easy to use and can be used to mirror a wide range of artifacts from a connected cluster.

Pre-Reqs – Things you need.

  • A host which has connectivity to the upstream repositories and has the oc-mirror plugin installed
  • Pull secret from console.redhat.com to access the repositories

Download & Install oc-mirror plugin

Download the oc-mirror CLI plugin from the Downloads page of the OpenShift Cluster Manager Hybrid Cloud Console.

Extract the archive:

$ tar xvzf oc-mirror.tar.gz

If necessary, update the plugin file to be executable:

$ chmod +x oc-mirror

Copy the oc-mirror binary to /usr/local/bin

Download Pull Secret

Download the registry.redhat.io pull secret from the Red Hat OpenShift Cluster Manager.

Make a copy of your pull secret in JSON format:

cat ./pull-secret | jq . > <path>/<pull_secret_file_in_json>

Save the file either as ~/.docker/config.json or $XDG_RUNTIME_DIR/containers/auth.json on both the connected host and the host in disconnected environment.

Generate the base64-encoded user name and password or token for your mirror registry:

echo -n '<user_name>:<password>' | base64 -w0

Edit the JSON file and add a section that describes your registry to it:

  "auths": {
        "<mirror_registry>": {
        "auth": "<credentials>",
        "email": "you@example.com"
        }
  },

The JSON file should look like this:

{
  "auths": {
        "registry.example.com": {
        "auth": "BGVtbYk3ZHAtqXs=",
        "email": "you@example.com"
        },
        "cloud.openshift.com": {
        "auth": "b3BlbnNo...",
        "email": "you@example.com"
        },
        "quay.io": {
        "auth": "b3BlbnNo...",
        "email": "you@example.com"
        },
        "registry.connect.redhat.com": {
        "auth": "NTE3Njg5Nj...",
        "email": "you@example.com"
        },
        "registry.redhat.io": {
        "auth": "NTE3Njg5Nj...",
        "email": "you@example.com"
        }
  }
}

Step 1 : Mirror the Content Using oc-mirror

First thing that needs to be done is to mirror the content and the first thing to check is the update path, which can be found here

Once you have identified your upgrade path, you need to take note of all the releases you need to get to the desired release, as you will have to configure the ImageSetConfiguration to mirror all of the releases and operators, you require.

First you need to install oc mirror plugin on a host that has connectivity to the upstream content – details are here

Next, you have to configure the imageset-config.yaml to specify what you need to mirror. The below ImageSetConfiguration.yaml file is an example :
apiVersion: mirror.openshift.io/v1alpha2
archiveSize: 4                                                  
storageConfig:                                                  
  registry:
imageURL: <quay.domain.com/mirror/oc-mirror-metadata>             
skipTLS: false
mirror:
  platform:
channels:
   - name: stable-4.11
     minVersion: 4.11.0
     maxVersion: 4.11.40
     shortestPath: True
   - name: stable-4.12
     minVersion: 4.12.17
     maxVersion: 4.12.17                                        
   type: ocp
graph: true
   operators:                                               
   - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.11
    packages:
    - name: rhacs-operator
      channels:
      - name: stable-4.11
    - name: odf-operator
      channels:
      - name: stable-4.11
    - name: local-storage-operator
      channels:
      - name: stable-4.11
    - name: odf-csi-addons-operator
      channels:
      - name: stable-4.11
    - name: ocs-operator
      channels:
      - name: stable-4.11
    - name: mcg-operator
      channels:
      - name: stable-4.11
    - name: openshift-gitops-operator
      channels:
      - name: stable-4.11
    - name: oadp-operator
      channels:
      - name: stable-4.11
    - name: pipelines-operator
      channels:
      - name: stable-4.11
    - name: openshift-update-service
      channels
      - name: stable-4.11 
       - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.12
    packages:
    - name: rhacs-operator
      channels:
      - name: stable-4.12
    - name: odf-operator
      channels:
      - name: stable-4.12
    - name: local-storage-operator
      channels:
      - name: stable-4.12
    - name: odf-csi-addons-operator
      channels:
      - name: stable-4.12
    - name: ocs-operator
      channels:
      - name: stable-4.12
    - name: mcg-operator
      channels:
      - name: stable-4.12
    - name: openshift-gitops-operator
      channels:
      - name: stable-4.12
    - name: oadp-operator
      channels:
      - name: stable-4.12
    - name: pipelines-operator
      channels:
      - name: stable-4.12
    - name: openshift-update-service
      channels
      - name: stable-4.11                          
  additionalImages:
  - name: registry.redhat.io/ubi8/ubi:latest                    
  helm: {}

The above config mirrors the 4.11.0, 4.11.40 and the 4.12.17 OCP releases, as per the upgrade path, as well as the desired operators required for updating and operating the cluster. Now lets run the oc-mirror command to generate the imageset and save the contents to disk, on removable media:

oc mirror --config=./imageset-config.yaml file://<path_to_output_directory>

Depending on your network speed and your imageset configuration, this will take some time. Once it is completed, you can then transfer the content into the disconnected environment, where it will be need to be pushed into the mirror registry using the same command.

Step 2 : Copy Contents into Mirror Registry

On the helper host in the disconnected environment, attach the removable media and use the oc-mirror plugin to mirror the contents of a generated image set to the target mirror registry.

oc mirror --from=./mirror_seq1_000000.tar docker://quay.domain.com:5000

This command updates the mirror registry with the image set and generates the ImageContentSourcePolicy and CatalogSource resources.

Step 3 : Apply the generated YAML manifests

This step will change the cluster settings to use the mirror registry instead of the upstream repositories.

  1. Navigate into the oc-mirror-workspace/ directory that was generated.
  2. Navigate into the results directory, for example, results-1639608409/.
  3. Verify that YAML files are present for the ImageContentSourcePolicy and CatalogSource resources.

Apply the YAML files from the directory to the cluster, using the oc apply command:

oc apply -f ./oc-mirror-workspace/results-1639608409/

Apply the release signatures to the cluster:

oc apply -f ./oc-mirror-workspace/results-1639608409/release-signatures/

Verify that the ImageContentSourcePolicy resources were successfully applied: 

oc get imagecontentsourcepolicy --all-namespaces

Verify that the CatalogSource resources were successfully installed by running the following command:

oc get catalogsource --all-namespaces

Optional Step – Create the graph-data image

This should be done automatically when using oc-mirror, but I found that the last time I did this, there was a problem with the container and thus the pod would not run. So it may be necessary to manually create the graph-data image and push it into the mirror registry for the update service operator to function correctly.

Create a directory, download the graph data and tar the contents into the directory:

$ mkdir graph-data
$ wget https://mirror.openshift.com/pub/openshift-v4/clients/graph-data-4.12.0.tar.gz
$ tar -zxvf graph-data-4.12.0.tar.gz -C graph-data

Create the Dockerfile to build the image:

FROM registry.access.redhat.com/ubi8/ubi:8.1

RUN curl -L -o cincinnati-graph-data.tar.gz https://api.openshift.com/api/upgrades_info/graph-data

RUN mkdir -p /var/lib/cincinnati-graph-data && tar xvzf cincinnati-graph-data.tar.gz -C /var/lib/cincinnati-graph-data/ --no-overwrite-dir --no-same-owner

CMD ["/bin/bash", "-c" ,"exec cp -rp /var/lib/cincinnati-graph-data/* /var/lib/cincinnati/graph-data"]

It may be the case that you are not able to reach the internet to download the tar file. If so, download the tar file onto the removable media from the connected environment and copy across to the disconnected environment. The docker file will then look like this:

FROM registry.access.redhat.com/ubi8/ubi:8.1

COPY cincinnati-graph-data.tar.gz . 

RUN mkdir -p /var/lib/cincinnati-graph-data && tar xvzf cincinnati-graph-data.tar.gz -C /var/lib/cincinnati-graph-data/ --no-overwrite-dir --no-same-owner

CMD ["/bin/bash", "-c" ,"exec cp -rp /var/lib/cincinnati-graph-data/* /var/lib/cincinnati/graph-data"]

Build the Image:

podman build -f ./Dockerfile -t registry.example.com/openshift/graph-data:latest

Push the graph data container image to the mirror registry that is accessible to the OpenShift Update Service:

podman push registry.example.com/openshift/graph-data:latest

Conclusion

The steps above show how to use the oc-mirror command to setup a mirror registry in a completely disconnected environment. Keeping a consistent imagesetconfiguration file and filesystem location will maintain the content in mirror registry, auto-pruning if necessary.

One consideration is to ensure that the mirror registry has the same availability level as the cluster. If the registry is not available and the cluster tries to schedule new pods, it may not be able to download the images and thus, fail.

Leave a comment