Module 5: WeatherNow — Agentic Release Investigation and GitOps Recovery
Start here: Log in to OpenShift
Open the Showroom Terminal tab and authenticate using your assigned workshop credentials before running any other oc commands:
oc login --server=https://api.cluster-%GUID%.example.com:6443 --username=user-guid --password='password' --insecure-skip-tls-verify
Verify that oc whoami returns user-guid. This is a disposable lab environment: the + -p + option can expose your password in shell history and process listings. Do not reuse this password on a personal or production cluster.
Module 4 ended with a healthy WeatherNow app in {user}-gitops. The fault was a bad readiness probe. Module 5 introduces a separate release failure: the new WeatherNow image tag does not exist, so the new ReplicaSet cannot pull it. Students diagnose the failed rollout, investigate with Lightspeed or a student-owned analysis-only AgenticRun (where secure tenant access is enabled), and restore the source through Gitea.
This does not repeat Module 4’s incident and does not modify either National Parks stack.
Duration: 30 minutes.
| Time | Mission |
|---|---|
4 min |
1 — Check a healthy WeatherNow rollout and GitOps source |
6 min |
2 — Commit a bad image tag and observe ImagePullBackOff |
8 min |
3 — Diagnose the rollout using Lightspeed and live evidence |
7 min |
4 — Create and inspect your own analysis-only AgenticRun if authorized |
5 min |
5 — Restore the image in Git and verify release recovery |
|
Safety and ownership
|
Mission 1: Confirm the healthy release baseline (4 minutes)
-
Use the student project and verify Module 4 was repaired:
export STUDENT="$(oc whoami)" export NS="$STUDENT-gitops" oc get deployment weathernow -n "$NS" \ -o custom-columns='NAME:.metadata.name,READY:.status.readyReplicas,IMAGE:.spec.template.spec.containers[0].image' oc get application weathernow-lab -n "$NS" -o json | \ jq '{sync:.status.sync.status, health:.status.health.status, revision:.status.sync.revision}' oc get deployment weathernow -n "$NS" -o json | \ jq '.spec.template.spec.containers[0].readinessProbe'Expected: WeatherNow 1/1 Ready, image quay.io/openshift-workshop-applications/weathernow:latest, readiness probe port http, GitOps Synced/Healthy. If Module 4 is not recovered, stop and repair it first.
-
Capture the working external Route:
export WEATHER_HOST="$(oc get route weathernow -n "$NS" -o jsonpath='{.spec.host}')" curl -sS -o /dev/null -w 'WeatherNow HTTP %{http_code}\n' "https://$WEATHER_HOST/"
Checkpoint 1: The current image and Route work.
Mission 2: Release an invalid image tag (6 minutes)
-
In the student-owned Gitea repository main, open:
content/modules/ROOT/examples/module04-weathernow/weathernow.yamlFind the Deployment/weathernow container image (not the Service or Route).
-
Replace only this value:
image: quay.io/openshift-workshop-applications/weathernow:latestWith a deliberately nonexistent tag:
image: quay.io/openshift-workshop-applications/weathernow:module05-intentional-bad-tagDo not edit the resource name, pull secrets, Service or quota. Commit:
lab(module05): simulate WeatherNow image pull failure -
Observe the new Deployment generation and its image pull errors:
oc get application weathernow-lab -n "$NS" -o json | \ jq '{sync:.status.sync.status, health:.status.health.status, revision:.status.sync.revision}' oc get deployment weathernow -n "$NS" -o json | \ jq '.spec.template.spec.containers[0].image' oc get pods -n "$NS" -l app=weathernow oc get events -n "$NS" --sort-by=.lastTimestamp | tail -20The new Pod should eventually show ErrImagePull or ImagePullBackOff. The old ready Pod may still serve the Route during RollingUpdate; record what actually happened.
Checkpoint 2: GitOps has accepted the wrong source revision and the new image cannot be pulled.
Mission 3: Evidence-backed Lightspeed diagnosis (8 minutes)
-
Open Lightspeed → Troubleshooting and ask:
Investigate the new Deployment/weathernow revision in namespace user-guid-gitops. The current Gitea revision points to a WeatherNow image tag that cannot be pulled. Use read-only tools to inspect Deployment/ReplicaSet/Pod conditions, image references, and Pod events. Compare the new image with the last known working tag and explain why an older ready Pod may still answer HTTP 200. Explain Synced versus Healthy, give the smallest source-controlled GitOps repair and cite actual evidence. Do not read Secrets or change any resource. Do not access other student namespaces. -
Independently verify:
oc get deployment weathernow -n "$NS" -o json | \ jq '{image:.spec.template.spec.containers[0].image,available:.status.availableReplicas,updated:.status.updatedReplicas}' oc get replicasets,pods -n "$NS" -l app=weathernow oc describe deployment weathernow -n "$NS" oc get events -n "$NS" --sort-by=.lastTimestamp | tail -20An absent tag can cause manifest unknown or similar registry errors; you should not invent a specific message. This is not a MongoDB fault, a scheduling issue, or a reason to modify cluster-wide registry credentials.
Checkpoint 3: You can identify the exact wrong image and propose a Git commit restoring the old tag.
Mission 4: AgenticRun in the student namespace (7 minutes)
-
Check per-student permissions:
oc auth can-i create agenticruns.agentic.openshift.io -n "$NS" oc auth can-i get agenticruns.agentic.openshift.io -n "$NS"If either check says no, that is the secure default: continue using Lightspeed, or review an authorized instructor demo. A CLI installed in your home directory does not grant RBAC.
-
If the instructor enabled secure self-service for this tenant, create an analysis-only run:
if [ "$(oc auth can-i create agenticruns.agentic.openshift.io -n "$NS")" = yes ]; then cat <<YAML | oc create -n "$NS" -f - apiVersion: agentic.openshift.io/v1alpha1 kind: AgenticRun metadata: generateName: weather-image- namespace: $NS spec: request: >- Analyze WeatherNow image pull failure in $NS only. The new Deployment/weathernow image tag is module05-intentional-bad-tag; compare it with the last known-good latest tag, Pod events and ReplicaSet rollout. Propose a Gitea correction. No mutations, no Secrets, no unrelated namespaces. analysis: agent: default YAML else echo "Student AgenticRun denied: continue with read-only troubleshooting." fi -
Inspect your own run and AnalysisResult, if permitted:
if [ "$(oc auth can-i list agenticruns.agentic.openshift.io -n "$NS")" = yes ]; then oc get agenticruns,analysisresults -n "$NS" oc agentic run list -n "$NS" fiDo not approve an execution stage or request automatic changes. An analysis may report Failed or missing permissions; record those facts.
Checkpoint 4: The analysis explains the bad image or records a real limitation; it has not edited Kubernetes or Git.
Mission 5: Repair Git, verify and close (5 minutes)
-
In student Gitea main, restore:
image: quay.io/openshift-workshop-applications/weathernow:latestCommit:
fix(module05): restore working WeatherNow image tag -
Verify automatic reconciliation and a working new revision:
oc get deployment weathernow -n "$NS" -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}' oc rollout status deployment/weathernow -n "$NS" --timeout=120s oc get application weathernow-lab -n "$NS" -o json | \ jq -r '[.status.sync.status,.status.health.status] | join(" / ")' curl -sS -o /dev/null -w 'WeatherNow HTTP %{http_code}\n' "https://$WEATHER_HOST/" oc get deployment nationalparks parksmap mongodb-nationalparks -n "$NS"
Checkpoint 5: The restored image is running, GitOps is Synced/Healthy, Route returns HTTP 200 and the two National Parks stacks remain unaffected.
Completion
| Evidence | During incident | After correction |
|---|---|---|
Gitea WeatherNow image |
Invalid module05 tag |
Latest |
New Pod |
ErrImagePull / ImagePullBackOff |
1/1 Ready |
GitOps source |
Bad image commit |
Repair commit |
Route |
May still serve old Pod |
Healthy new release |
AgenticRun |
Read-only analysis or documented fallback |
No automatic mutation |
You have completed the OpenShift 5 application lifecycle: deploy, introduce a controlled failure, investigate real evidence, review an agentic proposal if authorized and recover by changing the source of truth.
Troubleshooting
| Symptom | Check |
|---|---|
Invalid tag has no effect |
Confirm Argo CD reconciled the student Gitea commit, not a GitHub copy |
Old Pod still serves HTTP 200 |
Normal rolling update can preserve availability; inspect the new Pod and events |
Student AgenticRun creation denied |
Expected until instructor verifies sandbox isolation; do not request cluster-admin |
Recovery won’t roll out |
Check the live image tag, Deployment conditions and Application source revision; do not delete the App/PVC |
WeatherNow absent |
Revisit Module 4 setup before starting Module 5 |