1 Build a verified base-image mirror with Ansible

An enterprise mirror is a trust decision. Copying an image into Nexus does not make that image authentic. Your workflow checks Red Hat’s signature before copying, carries the signature to the destination, and checks it again under the internal registry name. Each task must succeed before the next can run.

Red Hat Hardened Images is that runtime: a minimal signed image under registry.access.redhat.com/hi/…, with no package manager in the image. A Universal Base Image is a general-purpose base. This module uses the Hardened Images pull spec the cluster stores in ConfigMap lab-lightwell-repo.

The arrows separate control from data movement: Showroom starts the work over SSH, while the bastion reads the published image and writes the internal copy. Track 3 records the approved destination in application source; Track 4 builds the application image.

Lightwell Network and Lightwell Clearinghouse remediate application libraries. Those libraries run on this base. A Clearinghouse member package still needs this verified Hummingbird pull spec underneath it.

In a customer engagement, the customer subscribes to Hardened Images and records the digest in their catalog. Red Hat runs the factory and holds the signing key. Here you do the consumer half: read the pull spec, verify it, and carry that same pull spec into the mirror.

Showroom controls the workflow and reads cluster configuration. The bastion moves image data between the public catalog and Nexus. The Nexus Docker repository is hummingbird-mirror, but the image path remains hi/openjdk; the repository’s administrative name is not part of the pull spec.

This is platform automation for importing a base image. Track 5 signs your application image, which is a different artifact with a different signing identity. Lightwell libraries arrive through the Maven proxy in Track 2; they run on this runtime base.

By the end of this module, you will be able to:

Open your service in the Red Hat Demo Platform catalog. Under Details, expand the OpenShift provision messages. Find the SSH hostname, port, username, and password. Use the values for your own cluster: the port is not necessarily 22, and the service name suffix may differ from the deployment GUID.

In Showroom, confirm that the Ansible runtime and SSH tools are ready. Provisioning installs Ansible in your persistent home directory. You do not install software on your laptop.

The commands should print the Ansible version, paths to the SSH tools, and Showroom’s cluster identity. If a command is missing, ask the instructor to repair provisioning before continuing.

The first connection may ask you to confirm a host-key fingerprint. Compare it with trusted lab access information or have the instructor confirm it. Enter the provisioned password at the password prompt. ssh-copy-id installs your public key on the bastion; the private key stays in Showroom. If you protected the private key with a passphrase, start ssh-agent and add ~/.ssh/track1 before running Ansible.

The cluster has been prepared with two digests for the published Hummingbird runtime hi/openjdk. Read the digests by running the following commands.

As you can see, the two digests describe the hi/openjdk image. The first command prints the pull spec: one line with registry.access.redhat.com, hi/openjdk, and @sha256:. The digest after @ is the image index, one manifest that can list more than one architecture. ConfigMap lab-lightwell-repo in namespace lightwell-repo stores that line as hummingbird_source_pullspec. The second command prints a different sha256:, then registry.access.redhat.com. That sha256 is the amd64 manifest, the platform image this cluster runs. ConfigMap lab-showroom in namespace showroom stores it as hummingbird_amd64_manifest. The same index may also list arm64. That listing is an example, and the labs do not use it. It shows that one index can list more than one architecture. In the Exercise section you pass that pull spec to cosign verify.

Red Hat signs every Hardened Images catalog image at build time. This Showroom shell already has cosign on PATH. On a Red Hat Enterprise Linux host the same key is a file under /etc/pki/sigstore, and the product documentation mounts that directory into a cosign container. This terminal has no such directory, so the Exercise uses the URL below.

The key URL is https://security.access.redhat.com/data/63405576.txt. The playbook command includes --insecure-ignore-tlog, which skips the transparency log and checks the signature against that key. This Showroom cosign warns that skipping the log is insecure. Success is the two checks under that warning: the cosign claims were validated, and the signatures were verified against the specified public key. See Verify the integrity of Red Hat Hardened Images.

Track 5 signs your application image with in-cluster Red Hat Trusted Artifact Signer. That signature uses a short-lived Fulcio certificate. This module uses Red Hat’s published Hardened Images key. If the key URL fails to load on this cluster, stop and tell an instructor.

Read the destination before completing the playbook. destination is this cluster’s Route host. repo is the administrative Nexus repository name, not another path segment in the image name. A newly provisioned cluster has no scored image; pre_mirror=false describes provisioning, not the current registry contents after a run.

Lab preparation has already created workshop-templates/hummingbird-mirror-pipeline on Gitea after the server became ready. The seed files are maintained in the workshop content repository on GitHub; provisioning imports the mirror directory from the configured workshop revision. You clone only the prepared Gitea repository. No repository creation or GitHub access is needed for this exercise.

mirror-base.yml is the workflow you complete. prepare.yml discovers cluster inputs from Showroom, copies the lab’s Linux tools to the bastion, and writes an ImageSet and temporary registry authentication file. publish.yml checks the verified payloads and records the result. inventory.yml selects the single bastion. The WORKSHOP_REVISION file records which workshop commit supplied these starters.

Your local Git checkout should contain all four Ansible files and the workshop revision. The starter is public within the lab, so cloning requires no password. If the repository is missing or empty, ask the instructor to repair preparation. If you are returning to this exercise, use the existing clone rather than cloning over it. Edit locally; Track 1 requires no push to the operator-managed starter. Your application repositories are a later Track 3 exercise.

Edit inventory.yml. Set ansible_host, ansible_port, and ansible_user to the values from your service details. Keep ansible_ssh_private_key_file: ~/.ssh/track1. Do not put the password in this file.

Edit the two variables near the top of mirror-base.yml:

Read the task list before running it. The source cosign verify task must precede oc-mirror. cosign copy follows the image mirror and transfers the signature and available attached metadata using the storage format this Nexus deployment supports. The destination cosign verify must precede publish.yml. Both verification tasks use the same key and the same index digest.

register retains each command’s result. changed_when: false says that verification observes state. A nonzero command result still fails the task. The rescue block clears the consumable result and fails the play; the always block removes temporary registry credentials. Neither block permits a failed mirror to publish success.

The lab commands retain two explicit environment accommodations: --insecure-ignore-tlog skips transparency-log verification, and the destination commands relax registry TLS verification for the lab Route. The expected Cosign warning explains the first. These are not customer defaults; a customer workflow must establish registry CA trust and its required transparency-log policy.

The syntax check should report no errors, and the bastion should respond with pong. Ansible’s ping executes a Python module over SSH; it is not an ICMP test. The lab expects a Linux amd64 bastion with Python. The supplied playbook transfers Cosign and oc-mirror from Showroom into ~/track-1-mirror/bin on that bastion without sudo.

Before running the pipeline, you recorded the destination’s state in Record the destination before mirroring. Now inspect the mirrored image and compare it with that saved observation. The commands below write the current state to after.txt and compare it with the existing before.txt; do not repeat the earlier capture and overwrite your starting state. Use the recorded destination rather than a mutable tag, and inspect the bastion’s actual ImageSet alongside the saved mirror log.

You know you have mirrored the correct image when the destination’s image-index digest and the digest of each platform manifest match those recorded for the source image. In this output, Manifest List is the signed index digest. Each Digest belongs to the platform identified by its OS and Arch lines; do not mistake that child digest for the index. The ImageSet must name the exact published pull spec, and mirror.log must show that the corresponding image was processed successfully. On a newly provisioned cluster, the diff changes from an image-not-found response to image metadata; git diff --no-index returns 1 when there are differences, which is expected here. On a repeat run, the before and after metadata may be identical and the mirror may reuse existing content. That is successful repeatability, not a requirement to change the image digest. If oc image info fails after the run, stop and investigate before continuing.

These observations prove registry availability and image identity. The playbook’s source and destination Cosign tasks establish publisher trust by verifying both signatures with the Red Hat public key you configured. Review their saved evidence next.

The playbook invalidates the previous consumable result at the start of every run. A failed retry cannot expose an older success as the result of the new attempt. The registry may still contain previously copied bytes; this workflow controls publication of an approved pull spec, not registry admission.

The result must be verified, and the destination must name the internal registry with the original index digest. Both Cosign tasks must have succeeded in the Ansible run. Inspect their saved verification payloads:

In both payloads, critical.image.docker-manifest-digest must match the published index digest. These files record the successful Cosign results from this run; displaying the JSON does not perform another signature verification. Together with the successful playbook run, matching image metadata, and approved destination, they complete your Track 1 review.

Keep your playbook and verification evidence in Showroom. In Track 3, you will use the verified internal pull spec as the runtime base for your application. Continue to Track 2: enterprise dependencies.

Red Hat publishes Hardened Images, builds the runtime, and holds the signing key. You keep the public key and use it to check both pull specs. Your Ansible workflow now implements the mirror pipeline in the diagram: a failed source verification stops the copy, a failed copy stops the sequence, and a failed destination verification prevents publication of an approved destination.

The index digest keeps both checks on the same published bytes. A tag alone can move after verification. A copied image without a valid Red Hat signature still fails the destination check, even if the registry contains a .sig tag. Choosing the verification key is part of the trust decision: a different key can accept a signature Red Hat did not create.

This automation belongs on the platform side, wherever the customer imports approved bases. A customer can run the playbook from a controlled Ansible environment against its bastion, with registry CA trust and the required transparency-log policy. A physically disconnected destination also needs an archive-transfer procedure; this cluster uses a connected mirror. Later tracks sign your application image, apply application policy, and refuse an unsigned digest at deployment.

You can automate this mirror pipeline with tools other than Ansible. OpenShift Pipelines, built on Tekton, can run its steps as Kubernetes tasks, while Jenkins or GitLab CI/CD can coordinate the same commands through pipeline stages. Choose an execution environment with access to the source and destination registries, and preserve the same controls: verify the pinned image with Red Hat’s trusted public key, copy the image and its signatures, then verify the internal digest. Configure failures to stop copying or publication, and release the destination pull spec to downstream builds only after verification succeeds.