Intro
Disconnected OpenShift clusters are becoming increasingly popular, as more and more organisations seek to deploy and manage their own private cloud environments. In this blog post, I will show you how to mirror repository content to update a disconnected OpenShift cluster as well as some best practices for managing your environment.
Disconnected OpenShift clusters are important for a number of reasons. First and foremost, they allow organisations to maintain complete control and security over their environment, without relying on external sources for updates, patches, or security fixes. This is especially important for organisations that operate in highly regulated industries, where data privacy and security are paramount.
Updating a disconnected OpenShift cluster can be a complex process, but it’s not impossible. Here are the basic steps:
- Set up a local mirror repository: A local mirror repository is a copy of the OpenShift container images that you’ll use to install and manage your cluster. You can create a mirror using the
oc-mirrorplugin. - Move the mirrored content into the registry in the disconnected / isolated environment.
- Configure the cluster to use the local, mirrored registry: Once you have a local mirror registry, you’ll need to apply the manifests to configure the OpenShift cluster to use it.
- Update the cluster!
- Manage the cluster: Once your disconnected cluster is updated, you’ll need to manage it. This includes applying updates, patches, and security fixes. To do this, you’ll need to periodically update your local mirror repository with the latest container images.
The oc-mirror plugin is a command-line tool that can be used to mirror images, manifests, and other artifacts, from external repositories in order to sync them to a mirrored registry for use by a disconnected cluster. This is an essential tool for organisations that need to manage OpenShift clusters in remote or offline environments.
The oc-mirror plugin works by creating a local mirror of the upstream repositories into a local registry or filesystem on removable media. This can then transferred into the disconnected environment, to update or deploy OpenShift applications and services in the disconnected cluster.
The oc-mirror plugin is a powerful tool that can be used to manage OpenShift clusters in disconnected environments. It is easy to use and can be used to mirror a wide range of artifacts from a connected cluster.
Pre-Reqs – Things you need.
- A host which has connectivity to the upstream repositories and has the oc-mirror plugin installed
- Pull secret from console.redhat.com to access the repositories
Download & Install oc-mirror plugin
Download the oc-mirror CLI plugin from the Downloads page of the OpenShift Cluster Manager Hybrid Cloud Console.
Extract the archive:
$ tar xvzf oc-mirror.tar.gz
If necessary, update the plugin file to be executable:
$ chmod +x oc-mirror
Copy the oc-mirror binary to /usr/local/bin
Download Pull Secret
Download the registry.redhat.io pull secret from the Red Hat OpenShift Cluster Manager.
Make a copy of your pull secret in JSON format:
cat ./pull-secret | jq . > <path>/<pull_secret_file_in_json>
Save the file either as ~/.docker/config.json or $XDG_RUNTIME_DIR/containers/auth.json on both the connected host and the host in disconnected environment.
Generate the base64-encoded user name and password or token for your mirror registry:
echo -n '<user_name>:<password>' | base64 -w0
Edit the JSON file and add a section that describes your registry to it:
"auths": {
"<mirror_registry>": {
"auth": "<credentials>",
"email": "you@example.com"
}
},The JSON file should look like this:
{
"auths": {
"registry.example.com": {
"auth": "BGVtbYk3ZHAtqXs=",
"email": "you@example.com"
},
"cloud.openshift.com": {
"auth": "b3BlbnNo...",
"email": "you@example.com"
},
"quay.io": {
"auth": "b3BlbnNo...",
"email": "you@example.com"
},
"registry.connect.redhat.com": {
"auth": "NTE3Njg5Nj...",
"email": "you@example.com"
},
"registry.redhat.io": {
"auth": "NTE3Njg5Nj...",
"email": "you@example.com"
}
}
}Step 1 : Mirror the Content Using oc-mirror
First thing that needs to be done is to mirror the content and the first thing to check is the update path, which can be found here
Once you have identified your upgrade path, you need to take note of all the releases you need to get to the desired release, as you will have to configure the ImageSetConfiguration to mirror all of the releases and operators, you require.
First you need to install oc mirror plugin on a host that has connectivity to the upstream content – details are here
Next, you have to configure theimageset-config.yamlto specify what you need to mirror. The below ImageSetConfiguration.yaml file is an example :
apiVersion: mirror.openshift.io/v1alpha2 archiveSize: 4 storageConfig: registry: imageURL: <quay.domain.com/mirror/oc-mirror-metadata> skipTLS: false mirror: platform: channels: - name: stable-4.11 minVersion: 4.11.0 maxVersion: 4.11.40 shortestPath: True - name: stable-4.12 minVersion: 4.12.17 maxVersion: 4.12.17 type: ocp graph: true operators: - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.11 packages: - name: rhacs-operator channels: - name: stable-4.11 - name: odf-operator channels: - name: stable-4.11 - name: local-storage-operator channels: - name: stable-4.11 - name: odf-csi-addons-operator channels: - name: stable-4.11 - name: ocs-operator channels: - name: stable-4.11 - name: mcg-operator channels: - name: stable-4.11 - name: openshift-gitops-operator channels: - name: stable-4.11 - name: oadp-operator channels: - name: stable-4.11 - name: pipelines-operator channels: - name: stable-4.11 - name: openshift-update-service channels - name: stable-4.11- catalog: registry.redhat.io/redhat/redhat-operator-index:v4.12 packages: - name: rhacs-operator channels: - name: stable-4.12 - name: odf-operator channels: - name: stable-4.12 - name: local-storage-operator channels: - name: stable-4.12 - name: odf-csi-addons-operator channels: - name: stable-4.12 - name: ocs-operator channels: - name: stable-4.12 - name: mcg-operator channels: - name: stable-4.12 - name: openshift-gitops-operator channels: - name: stable-4.12 - name: oadp-operator channels: - name: stable-4.12 - name: pipelines-operator channels: - name: stable-4.12 - name: openshift-update-service channels - name: stable-4.11 additionalImages: - name: registry.redhat.io/ubi8/ubi:latest helm: {}
The above config mirrors the 4.11.0, 4.11.40 and the 4.12.17 OCP releases, as per the upgrade path, as well as the desired operators required for updating and operating the cluster. Now lets run the oc-mirror command to generate the imageset and save the contents to disk, on removable media:
oc mirror --config=./imageset-config.yaml file://<path_to_output_directory>
Depending on your network speed and your imageset configuration, this will take some time. Once it is completed, you can then transfer the content into the disconnected environment, where it will be need to be pushed into the mirror registry using the same command.
Step 2 : Copy Contents into Mirror Registry
On the helper host in the disconnected environment, attach the removable media and use the oc-mirror plugin to mirror the contents of a generated image set to the target mirror registry.
oc mirror --from=./mirror_seq1_000000.tar docker://quay.domain.com:5000This command updates the mirror registry with the image set and generates the ImageContentSourcePolicy and CatalogSource resources.
Step 3 : Apply the generated YAML manifests
This step will change the cluster settings to use the mirror registry instead of the upstream repositories.
- Navigate into the oc-mirror-workspace/ directory that was generated.
- Navigate into the results directory, for example, results-1639608409/.
- Verify that YAML files are present for the ImageContentSourcePolicy and CatalogSource resources.
Apply the YAML files from the directory to the cluster, using the oc apply command:
oc apply -f ./oc-mirror-workspace/results-1639608409/Apply the release signatures to the cluster:
oc apply -f ./oc-mirror-workspace/results-1639608409/release-signatures/Verify that the ImageContentSourcePolicy resources were successfully applied:
oc get imagecontentsourcepolicy --all-namespacesVerify that the CatalogSource resources were successfully installed by running the following command:
oc get catalogsource --all-namespacesOptional Step – Create the graph-data image
This should be done automatically when using oc-mirror, but I found that the last time I did this, there was a problem with the container and thus the pod would not run. So it may be necessary to manually create the graph-data image and push it into the mirror registry for the update service operator to function correctly.
Create a directory, download the graph data and tar the contents into the directory:
$ mkdir graph-data
$ wget https://mirror.openshift.com/pub/openshift-v4/clients/graph-data-4.12.0.tar.gz
$ tar -zxvf graph-data-4.12.0.tar.gz -C graph-dataCreate the Dockerfile to build the image:
FROM registry.access.redhat.com/ubi8/ubi:8.1
RUN curl -L -o cincinnati-graph-data.tar.gz https://api.openshift.com/api/upgrades_info/graph-data
RUN mkdir -p /var/lib/cincinnati-graph-data && tar xvzf cincinnati-graph-data.tar.gz -C /var/lib/cincinnati-graph-data/ --no-overwrite-dir --no-same-owner
CMD ["/bin/bash", "-c" ,"exec cp -rp /var/lib/cincinnati-graph-data/* /var/lib/cincinnati/graph-data"]It may be the case that you are not able to reach the internet to download the tar file. If so, download the tar file onto the removable media from the connected environment and copy across to the disconnected environment. The docker file will then look like this:
FROM registry.access.redhat.com/ubi8/ubi:8.1
COPY cincinnati-graph-data.tar.gz .
RUN mkdir -p /var/lib/cincinnati-graph-data && tar xvzf cincinnati-graph-data.tar.gz -C /var/lib/cincinnati-graph-data/ --no-overwrite-dir --no-same-owner
CMD ["/bin/bash", "-c" ,"exec cp -rp /var/lib/cincinnati-graph-data/* /var/lib/cincinnati/graph-data"]Build the Image:
podman build -f ./Dockerfile -t registry.example.com/openshift/graph-data:latestPush the graph data container image to the mirror registry that is accessible to the OpenShift Update Service:
podman push registry.example.com/openshift/graph-data:latestConclusion
The steps above show how to use the oc-mirror command to setup a mirror registry in a completely disconnected environment. Keeping a consistent imagesetconfiguration file and filesystem location will maintain the content in mirror registry, auto-pruning if necessary.
One consideration is to ensure that the mirror registry has the same availability level as the cluster. If the registry is not available and the cluster tries to schedule new pods, it may not be able to download the images and thus, fail.
Virtual-Shift 