5.3 Add and verify the admission signing key

In this module you prove that the application digest also carries a signature you can verify with a public key, without calling Fulcio.

This final module in Track 5 prepares the application image for public-key admission in Track 6. Module 5.1 established its keyless signature through Trusted Artifact Signer. The upcoming exercise adds a second signature to the same digest using a cosign key pair held in Showroom: the private key signs the image, and the public key verifies that signature and becomes the trust material for admission. The existing keyless signature remains available.

This demonstrates a verification approach that can be used in a disconnected environment when the image, signature, and trusted public key are available there. The workshop cluster remains connected and retrieves the signature from its registry, so the exercise does not demonstrate physical network isolation or offline artifact transfer. Disconnected transfer planning identifies transfer requirements and optionally collects public evidence. It does not mirror TUF, transfer images or signatures, or verify a destination. Design and test a destination procedure separately.

Why this control exists

This workshop uses Fulcio to issue the signing certificate and TUF to distribute verification trust material. Other keyless verification workflows depend on the chosen verifier, trusted-root mechanism, and bundle format. A keyless verifier uses certificate and trust-root evidence; it does not ask Fulcio to issue a new certificate at verification time. This lab adds a public-key verification path for Track 6 admission. You add that second signature here, on the same digest. The keyless signature stays on the image.

Native ImagePolicy signedEmail cannot hold the 5.1 Kubernetes service-account URI. 6.1 GitOps + admission therefore renders CRI-O policy as policyType: PublicKey from the PEM you publish. That PEM is ConfigMap lab-cosign-pubkey in namespace tssc-admission. 5.1 Sign draws the keyless column. This module is the other signature: ~/lab-trust/cosign.key signs, and cosign.pub is what admission will trust.

Two signatures on one application digest; only the learner public key is published to the admission ConfigMap

As the figure shows, module 5.1 signed with Fulcio, Rekor, and TUF. In this module you generate a key pair on the Showroom terminal, sign the same digest with cosign sign --key and --tlog-upload=false, and publish cosign.pub. Module 6.1 reads that public key as policyType: PublicKey.

Lightwell Network and Lightwell Clearinghouse artifacts may already have their own signatures. This Check is the application image. Clearinghouse does not replace the key pair.

The private key stays on this terminal. It does not go in Git, and it does not go in a screenshot. cosign.key is the file to keep out of the commit.

In a customer engagement, choose and test the destination verifier before collecting keys, trust roots, signatures, and any bundles. Keyless verification does not universally require a live TUF mirror; the verifier and evidence format determine its requirements. You generate the pair on this terminal and sign the digest you already built. Red Hat does not operate a data diode on this cluster. An empty COSIGN_PASSWORD is for this non-interactive lab terminal. A customer plant uses a real passphrase.

Learning objectives

By the end of this module, you will be able to:

  • Generate a cosign key pair under /home/lab-user/lab-trust and sign the application digest with --key and --tlog-upload=false

  • Verify that digest with cosign verify --key

  • Publish cosign.pub to ConfigMap lab-cosign-pubkey so 6.1 can render ImagePolicy

  • Explain that a Lightwell library signature is a different layer from this application key

The digest you already signed

cosign is already on the Showroom PATH. Success is a dockerImageReference for spring-boot-lw-poc:latest in the build namespace, and the saved verified reference from 5.1. An existing key pair is expected on a return visit; reuse it.

Read the 5.1 digest
BUILD_NS="$(oc -n gitea get configmap lab-gitea -o jsonpath='{.data.student_build_namespace}')"
IMAGE="$(cat /home/lab-user/lightwell-signing-evidence/image.txt)"
CURRENT_IMAGE="$(oc -n "$BUILD_NS" get istag spring-boot-lw-poc:latest -o jsonpath='{.image.dockerImageReference}')"
[[ "$IMAGE" == *@sha256:* && "$IMAGE" == "$CURRENT_IMAGE" ]] || { echo "Refresh 5.1 verification for the current application digest"; exit 1; }
oc -n "${BUILD_NS}" get istag spring-boot-lw-poc:latest \
  -o jsonpath='{.image.dockerImageReference}{"\n"}'
ls /home/lab-user/lab-trust 2>/dev/null || echo 'lab-trust empty (expected before the Exercise)'

Exercise

Generate or reuse your signing key

Generate the key pair in ~/lab-trust if cosign.key is not already there. export HOME=/tmp/lab-home exists so cosign login does not write a Docker config into the real home directory. The --key path and the ConfigMap --from-file path stay on /home/lab-user/lab-trust, because a tilde would follow the temporary HOME. --tlog-upload=false keeps this signature off the public Rekor log. That is the disconnected procedure: the signature travels with the digest, and verify uses --insecure-ignore-tlog on the next step.

ImagePolicy signedEmail cannot store a service-account URI. The public key is what 6.1 will render. Success is cosign.pub on disk, cosign sign exiting 0, and ConfigMap lab-cosign-pubkey in namespace tssc-admission holding that PEM. Put HOME back to /home/lab-user before the Check.

Prepare the key pair
BUILD_NS="$(oc -n gitea get configmap lab-gitea -o jsonpath='{.data.student_build_namespace}')"
IMAGE="$(cat /home/lab-user/lightwell-signing-evidence/image.txt)"
CURRENT_IMAGE="$(oc -n "$BUILD_NS" get istag spring-boot-lw-poc:latest -o jsonpath='{.image.dockerImageReference}')"
[[ "$IMAGE" == *@sha256:* && "$IMAGE" == "$CURRENT_IMAGE" ]] || { echo "Refresh 5.1 verification for the current application digest"; exit 1; }
mkdir -p /home/lab-user/lab-trust
cd /home/lab-user/lab-trust
if [[ ! -f cosign.key ]]; then
  COSIGN_PASSWORD="" cosign generate-key-pair
fi
test -s cosign.key && test -s cosign.pub || { echo "Incomplete key pair; recover it before continuing"; exit 1; }

Sign the verified digest and publish the public key

Use the same immutable IMAGE selected above. The signing command must succeed before you publish the public key; stop and investigate any error.

export HOME=/tmp/lab-home
mkdir -p "${HOME}"
cosign login -u unused -p "$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
  image-registry.openshift-image-registry.svc:5000
COSIGN_PASSWORD="" cosign sign --key /home/lab-user/lab-trust/cosign.key \
  --tlog-upload=false \
  --allow-http-registry --allow-insecure-registry \
  "${IMAGE}" || { echo "Signing failed"; exit 1; }
# ImagePolicy signedEmail cannot be a SA URI. Publish the public key for 6.1.
oc -n tssc-admission create configmap lab-cosign-pubkey \
  --from-file=cosign.pub=/home/lab-user/lab-trust/cosign.pub \
  --dry-run=client -o yaml | oc apply -f -
# Tilde follows HOME. Put the real home back before the Check (or use /home/lab-user/lab-trust).
export HOME=/home/lab-user

Verify with the published public key

Export the public key from tssc-admission/lab-cosign-pubkey, compare it with /home/lab-user/lab-trust/cosign.pub, and verify the saved application digest with that exported key. --insecure-ignore-tlog matches the sign, which did not upload a transparency-log entry. Success is exit code 0 on your application digest. The 5.1 keyless signature has to remain verifiable. Stripping the Fulcio signature fails that later use.

Verify with the public key
BUILD_NS="$(oc -n gitea get configmap lab-gitea -o jsonpath='{.data.student_build_namespace}')"
IMAGE="$(cat /home/lab-user/lightwell-signing-evidence/image.txt)"
CURRENT_IMAGE="$(oc -n "$BUILD_NS" get istag spring-boot-lw-poc:latest -o jsonpath='{.image.dockerImageReference}')"
[[ "$IMAGE" == *@sha256:* && "$IMAGE" == "$CURRENT_IMAGE" ]] || { echo "Refresh 5.1 verification for the current application digest"; exit 1; }
export HOME=/home/lab-user
LAB_TRUST=/home/lab-user/lab-trust
test -f "${LAB_TRUST}/cosign.pub"
oc -n tssc-admission get configmap lab-cosign-pubkey -o jsonpath='{.data.cosign\.pub}' > /tmp/track5-admission.pub
python3 -c 'import pathlib, sys; sys.exit(pathlib.Path(sys.argv[1]).read_bytes() != pathlib.Path(sys.argv[2]).read_bytes())' "$LAB_TRUST/cosign.pub" /tmp/track5-admission.pub \
  || { echo "Admission public key differs from the local key"; exit 1; }
cosign login -u unused -p "$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
  image-registry.openshift-image-registry.svc:5000
cosign verify --key /tmp/track5-admission.pub \
  --insecure-ignore-tlog \
  --allow-http-registry --allow-insecure-registry \
  "${IMAGE}" > /tmp/track5-key-verification.json
VERIFY_STATUS=$?
cat /tmp/track5-key-verification.json
test "$VERIFY_STATUS" -eq 0 || { echo "Public-key verification failed"; exit 1; }
cp /tmp/track5-key-verification.json /home/lab-user/lightwell-signing-evidence/

Review

Confirm that the saved digest still matches the application tag, the exported admission public key matches your local cosign.pub, and verification with that exported key succeeds. Repeat the keyless verification in 5.1 to confirm that both signatures remain usable on the same digest. Keep the private key under /home/lab-user/lab-trust; neither Git nor the admission ConfigMap needs it.

Explain the boundary of this exercise: verification uses the public key and registry-hosted signature without a transparency-log lookup. The registry is still reachable, and no physical network isolation or mirrored-TUF transfer has been demonstrated. The appendix provides planning and public-evidence collection only; a destination transfer and verification procedure remains separate work.

Record what the verification exercise demonstrates

The workshop stores your short answer to “Did this exercise demonstrate a physically air-gapped environment?” in ConfigMap report-14-disconnected-verify, in namespace lw-poc-validate. This is separate from your saved signature-verification output and the admission public-key ConfigMap. The answer describes the exercise’s scope: you verified the application signature with a public key while the cluster and registry remained connected.

Record the required answer not-air-gap in the field named diode_honest. That field name belongs to the workshop’s assessment; it does not refer to a data diode you need to configure. The following command updates only your answer, then displays it for confirmation.

oc -n lw-poc-validate patch configmap report-14-disconnected-verify \
  --type=merge -p '{"data":{"diode_honest":"not-air-gap"}}'
oc -n lw-poc-validate get configmap report-14-disconnected-verify \
  -o jsonpath='{.data.diode_honest}{"\n"}'

The output should be not-air-gap. The validation Job below checks this answer and verifies the application signature using the published admission public key. Recording the answer does not replace successful signature verification. An empty value or REPLACE_ME leaves the short-answer requirement incomplete. validate-docs contains reference examples and is not where your answer belongs.

Run the module validation

The scored Check is Job validate-14-disconnected-verify in lw-poc-validate. Direct nav URLs still work (honor system). Unlimited retries. There is no Solve.

Run the Validate Job
oc -n lw-poc-validate delete job validate-14-disconnected-verify --ignore-not-found
TPL='job-14.yaml'
oc -n lw-poc-validate get configmap validate-job-templates \
  -o go-template="{{ index .data \"${TPL}\" }}" | oc create -f -
oc -n lw-poc-validate logs -f job/validate-14-disconnected-verify

Expect CHECK PASSED: before you continue. CHECK FAILED: names the object to fix. Delete the Job and create it again as often as you need.

The Validate Job reads ConfigMap tssc-admission/lab-cosign-pubkey and runs cosign verify --key on ImageStream tag spring-boot-lw-poc:latest in the build namespace. The PEM in that ConfigMap has to be the public key that signed the digest. REPLACE_ME fails the Check. Task cosign-sign-keyless is the 5.1 Sign Check. Track 6 still opens if you skip this module. TUF root.json and the Fulcio and Rekor PEMs are a different exercise. Use Disconnected transfer planning to identify the materials required by a destination verifier. This module’s artifact verifies with cosign.pub.

Carry the result into Track 6

Keep the final image.txt, both verification outputs, and the policy reports. Track 6 promotes that signed digest and reads tssc-admission/lab-cosign-pubkey. If another build moves the tag, stop and refresh signing, policy, and SBOM evidence for the replacement digest before promotion.

Official documentation

Key takeaways

  • Track 5 is the keyless signature and a key-based signature you created on the same digest.

  • cosign.pub is published to tssc-admission/lab-cosign-pubkey. 6.1 admission uses it.

  • The private key stays on the Showroom terminal.

  • The cluster stayed on the network. You verified using a public key against a connected registry, without demonstrating offline transfer or destination verification.