Debugging & Troubleshooting

In the OCP Console, click on the weather-frontend pod. The Events tab at the bottom shows the error with colour-coded severity - you’ll see the image pull failure immediately.

You can also check via CLI:

Look at the Events section. You should see:

The image tag 9.9 doesn’t exist. Let’s confirm the image reference:

Update the image to the correct tag (latest):

You should see 1/1 Running.

Takeaway: ImagePullBackOff = wrong image name, missing tag, or no pull credentials. Always check oc describe pod → Events.

The pod keeps starting and crashing. In the OCP Console, click the weather-api pod, then the Logs tab. The log viewer shows the error message with real-time streaming. If the pod has restarted, use the Container dropdown to select the previous container’s logs.

You can also check via CLI:

You should see:

The app is trying to read a config file that doesn’t exist.

Create the missing config file as a ConfigMap and mount it into the container. We’ll also fix the container command to run the actual application:

You should see 1/1 Running.

Takeaway: CrashLoopBackOff = the app starts then exits. Always check oc logs and oc logs --previous. Common causes: missing config, bad command, failed health checks, out of memory.

You may have noticed weather-cache doesn’t appear in the pod list at all. In the OCP Console, navigate to Workloads → Deployments (project: ops-track-demo) and click weather-cache. The Events tab immediately shows the FailedCreate error explaining why no pod was created.

Let’s also check via CLI:

It shows 0/1 ready - the pod was never created. When a pod can’t even be created, check the ReplicaSet events:

You should see:

A LimitRange in the namespace caps containers at 2Gi memory, but the deployment requests 512Gi (a typo - should be 512Mi). Let’s confirm:

Fix the memory typo - change 512Gi to 512Mi and 1Ti to 1Gi:

You should see 1/1 Running. The pod can now be created because the resource requests fit within the LimitRange.

Takeaway: When a pod never appears at all, check the ReplicaSet events (oc describe replicaset). LimitRanges and ResourceQuotas can silently prevent pods from being created.

You should see:

The deployment references a ConfigMap that doesn’t exist:

Create the missing ConfigMap with the keys the app expects:

Restart the pod to pick up the new ConfigMap:

You should see 1/1 Running. Confirm the environment variables are injected:

You should see:

Takeaway: CreateContainerConfigError = a referenced ConfigMap or Secret doesn’t exist. Check oc describe pod for "not found" messages. Create the missing resource and restart the pod.

This pod is Running - so what’s wrong? In the OCP Console, navigate to Networking → Services (project: ops-track-demo) and click weather-proxy. The Pods tab shows 0 Pods - immediately revealing that the service can’t find any matching pods.

Check via CLI as well:

You should see ENDPOINTS is empty (<unset>). The service has no backends.

Compare the service selector with the pod labels:

Spot the difference? The service selector says weather-proxi (typo) but the pod label is weather-proxy.

Patch the service selector to match the actual pod label:

You should now see the pod’s IP in the ENDPOINTS column. Traffic will flow to the pod.

Takeaway: When a pod is Running but the service isn’t working, check if the service selector matches the pod labels. A single character typo can break routing. Use oc get endpointslice to verify.

Check the pod status:

You’ll see weather-db in CrashLoopBackOff. Check the logs:

The log shows: mkdir: cannot create directory '/var/lib/pgsql': Permission denied. The container is trying to create a data directory but doesn’t have permission.

This is a Security Context Constraint (SCC) issue. Check which SCC the pod is running under:

It shows restricted-v2. This SCC assigns a random user ID to the container for security - the container does NOT run as root. The directory /var/lib/pgsql is owned by root inside the image, so the random UID can’t write to it.

This is the most common source of pod failures on OpenShift. Containers that expect to run as root will fail under restricted-v2.

Mount an emptyDir volume at the data path. The container can always write to emptyDir volumes regardless of its UID:

The pod should now be Running. Check the logs to confirm the database initialized:

You should see Database ready on port 5432.

Takeaway: When you see Permission denied in a container log, check the SCC with oc get pod -o jsonpath='{.metadata.annotations.openshift\.io/scc}'. Under restricted-v2, containers run as a random UID and can’t write to root-owned directories. Use emptyDir volumes for writable paths, or use images designed to run as non-root.