Module 5: WeatherNow — Agentic Release Investigation and GitOps Recovery

Start here: Log in to OpenShift

Open the Showroom Terminal tab and authenticate using your assigned workshop credentials before running any other oc commands:

oc login --server=https://api.cluster-%GUID%.example.com:6443 --username=user-guid --password='password' --insecure-skip-tls-verify

Verify that oc whoami returns user-guid. This is a disposable lab environment: the + -p + option can expose your password in shell history and process listings. Do not reuse this password on a personal or production cluster.

Module 4 ended with a healthy WeatherNow app in {user}-gitops. The fault was a bad readiness probe. Module 5 introduces a separate release failure: the new WeatherNow image tag does not exist, so the new ReplicaSet cannot pull it. Students diagnose the failed rollout, investigate with Lightspeed or a student-owned analysis-only AgenticRun (where secure tenant access is enabled), and restore the source through Gitea.

This does not repeat Module 4’s incident and does not modify either National Parks stack.

Duration: 30 minutes.

Time Mission

4 min

1 — Check a healthy WeatherNow rollout and GitOps source

6 min

2 — Commit a bad image tag and observe ImagePullBackOff

8 min

3 — Diagnose the rollout using Lightspeed and live evidence

7 min

4 — Create and inspect your own analysis-only AgenticRun if authorized

5 min

5 — Restore the image in Git and verify release recovery

Safety and ownership
  • All changes happen in your own Gitea main branch, in the separate WeatherNow path.

  • Application/weathernow-lab and Deployment/weathernow live in {user}-gitops.

  • Do not change the National Parks manifests, Shared ClusterResourceQuota, Secrets, Namespace, ApprovalPolicy, or Argo CD controller.

  • Do not patch the live Deployment or issue oc rollout undo: GitOps would eventually restore its declared state. Repair the Git source instead.

  • AgenticRun permissions are disabled by default. Students may create runs only when both oc auth can-i checks pass following an independently verified tenant-isolation rollout. Analysis-only; never approve execution.

Mission 1: Confirm the healthy release baseline (4 minutes)

  1. Use the student project and verify Module 4 was repaired:

    export STUDENT="$(oc whoami)"
    export NS="$STUDENT-gitops"
    oc get deployment weathernow -n "$NS" \
      -o custom-columns='NAME:.metadata.name,READY:.status.readyReplicas,IMAGE:.spec.template.spec.containers[0].image'
    oc get application weathernow-lab -n "$NS" -o json | \
      jq '{sync:.status.sync.status, health:.status.health.status, revision:.status.sync.revision}'
    oc get deployment weathernow -n "$NS" -o json | \
      jq '.spec.template.spec.containers[0].readinessProbe'

    Expected: WeatherNow 1/1 Ready, image quay.io/openshift-workshop-applications/weathernow:latest, readiness probe port http, GitOps Synced/Healthy. If Module 4 is not recovered, stop and repair it first.

    S01: healthy WeatherNow image and GitOps baseline
  2. Capture the working external Route:

    export WEATHER_HOST="$(oc get route weathernow -n "$NS" -o jsonpath='{.spec.host}')"
    curl -sS -o /dev/null -w 'WeatherNow HTTP %{http_code}\n' "https://$WEATHER_HOST/"
    S02: working WeatherNow Route

Checkpoint 1: The current image and Route work.

Mission 2: Release an invalid image tag (6 minutes)

  1. In the student-owned Gitea repository main, open:

    content/modules/ROOT/examples/module04-weathernow/weathernow.yaml

    Find the Deployment/weathernow container image (not the Service or Route).

    S03: WeatherNow image field in student Gitea
  2. Replace only this value:

    image: quay.io/openshift-workshop-applications/weathernow:latest

    With a deliberately nonexistent tag:

    image: quay.io/openshift-workshop-applications/weathernow:module05-intentional-bad-tag

    Do not edit the resource name, pull secrets, Service or quota. Commit:

    lab(module05): simulate WeatherNow image pull failure
    S04: deliberately incorrect WeatherNow image tag and Gitea commit
  3. Observe the new Deployment generation and its image pull errors:

    oc get application weathernow-lab -n "$NS" -o json | \
      jq '{sync:.status.sync.status, health:.status.health.status, revision:.status.sync.revision}'
    oc get deployment weathernow -n "$NS" -o json | \
      jq '.spec.template.spec.containers[0].image'
    oc get pods -n "$NS" -l app=weathernow
    oc get events -n "$NS" --sort-by=.lastTimestamp | tail -20

    The new Pod should eventually show ErrImagePull or ImagePullBackOff. The old ready Pod may still serve the Route during RollingUpdate; record what actually happened.

    S05: new WeatherNow Pod image pull failure and events

Checkpoint 2: GitOps has accepted the wrong source revision and the new image cannot be pulled.

Mission 3: Evidence-backed Lightspeed diagnosis (8 minutes)

  1. Open Lightspeed → Troubleshooting and ask:

    Investigate the new Deployment/weathernow revision in
    namespace user-guid-gitops. The current Gitea revision points
    to a WeatherNow image tag that cannot be pulled.
    
    Use read-only tools to inspect Deployment/ReplicaSet/Pod
    conditions, image references, and Pod events. Compare the
    new image with the last known working tag and explain
    why an older ready Pod may still answer HTTP 200.
    Explain Synced versus Healthy, give the smallest
    source-controlled GitOps repair and cite actual evidence.
    Do not read Secrets or change any resource.
    Do not access other student namespaces.
    S06: Lightspeed investigates WeatherNow image pull failure
  2. Independently verify:

    oc get deployment weathernow -n "$NS" -o json | \
      jq '{image:.spec.template.spec.containers[0].image,available:.status.availableReplicas,updated:.status.updatedReplicas}'
    oc get replicasets,pods -n "$NS" -l app=weathernow
    oc describe deployment weathernow -n "$NS"
    oc get events -n "$NS" --sort-by=.lastTimestamp | tail -20

    An absent tag can cause manifest unknown or similar registry errors; you should not invent a specific message. This is not a MongoDB fault, a scheduling issue, or a reason to modify cluster-wide registry credentials.

    S07: live image tag and pull failure events

Checkpoint 3: You can identify the exact wrong image and propose a Git commit restoring the old tag.

Mission 4: AgenticRun in the student namespace (7 minutes)

  1. Check per-student permissions:

    oc auth can-i create agenticruns.agentic.openshift.io -n "$NS"
    oc auth can-i get agenticruns.agentic.openshift.io -n "$NS"

    If either check says no, that is the secure default: continue using Lightspeed, or review an authorized instructor demo. A CLI installed in your home directory does not grant RBAC.

    S08: WeatherNow AgenticRun permission gate
  2. If the instructor enabled secure self-service for this tenant, create an analysis-only run:

    if [ "$(oc auth can-i create agenticruns.agentic.openshift.io -n "$NS")" = yes ]; then
      cat <<YAML | oc create -n "$NS" -f -
    apiVersion: agentic.openshift.io/v1alpha1
    kind: AgenticRun
    metadata:
      generateName: weather-image-
      namespace: $NS
    spec:
      request: >-
        Analyze WeatherNow image pull failure in $NS only.
        The new Deployment/weathernow image tag is
        module05-intentional-bad-tag; compare it with the last
        known-good latest tag, Pod events and ReplicaSet rollout.
        Propose a Gitea correction. No mutations, no Secrets,
        no unrelated namespaces.
      analysis:
        agent: default
    YAML
    else
      echo "Student AgenticRun denied: continue with read-only troubleshooting."
    fi
    S09: permitted student-created WeatherNow analysis-only run
  3. Inspect your own run and AnalysisResult, if permitted:

    if [ "$(oc auth can-i list agenticruns.agentic.openshift.io -n "$NS")" = yes ]; then
      oc get agenticruns,analysisresults -n "$NS"
      oc agentic run list -n "$NS"
    fi

    Do not approve an execution stage or request automatic changes. An analysis may report Failed or missing permissions; record those facts.

    S10: WeatherNow run analysis evidence or instructor fallback

Checkpoint 4: The analysis explains the bad image or records a real limitation; it has not edited Kubernetes or Git.

Mission 5: Repair Git, verify and close (5 minutes)

  1. In student Gitea main, restore:

    image: quay.io/openshift-workshop-applications/weathernow:latest

    Commit:

    fix(module05): restore working WeatherNow image tag
    S11: WeatherNow image restored in student Gitea
  2. Verify automatic reconciliation and a working new revision:

    oc get deployment weathernow -n "$NS" -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
    oc rollout status deployment/weathernow -n "$NS" --timeout=120s
    oc get application weathernow-lab -n "$NS" -o json | \
      jq -r '[.status.sync.status,.status.health.status] | join(" / ")'
    curl -sS -o /dev/null -w 'WeatherNow HTTP %{http_code}\n' "https://$WEATHER_HOST/"
    oc get deployment nationalparks parksmap mongodb-nationalparks -n "$NS"
    S12: restored WeatherNow GitOps release and unaffected National Parks workloads

Checkpoint 5: The restored image is running, GitOps is Synced/Healthy, Route returns HTTP 200 and the two National Parks stacks remain unaffected.

Completion

Evidence During incident After correction

Gitea WeatherNow image

Invalid module05 tag

Latest

New Pod

ErrImagePull / ImagePullBackOff

1/1 Ready

GitOps source

Bad image commit

Repair commit

Route

May still serve old Pod

Healthy new release

AgenticRun

Read-only analysis or documented fallback

No automatic mutation

You have completed the OpenShift 5 application lifecycle: deploy, introduce a controlled failure, investigate real evidence, review an agentic proposal if authorized and recover by changing the source of truth.

Troubleshooting

Symptom Check

Invalid tag has no effect

Confirm Argo CD reconciled the student Gitea commit, not a GitHub copy

Old Pod still serves HTTP 200

Normal rolling update can preserve availability; inspect the new Pod and events

Student AgenticRun creation denied

Expected until instructor verifies sandbox isolation; do not request cluster-admin

Recovery won’t roll out

Check the live image tag, Deployment conditions and Application source revision; do not delete the App/PVC

WeatherNow absent

Revisit Module 4 setup before starting Module 5