Observability & Logging

View the Prometheus pods that are already running:

Query cluster CPU usage using the console metrics viewer:

You’ll see a time-series graph showing cluster CPU usage over time - far more useful than a point-in-time number.

For quick CLI snapshots, you can also use:

Navigate to Observe → Dashboards in the OpenShift console. Key dashboards include:

OpenShift can also scrape metrics from your own applications. User workload monitoring is not enabled by default - it has been pre-enabled on this workshop cluster. Verify it’s running:

If pods are running, any application that exposes a /metrics endpoint can be scraped by creating a ServiceMonitor or PodMonitor resource.

Any application that exposes a Prometheus /metrics endpoint can be scraped by creating a ServiceMonitor pointing at it. This is how HPA gets custom metrics for scaling decisions.

Check what alerting rules exist:

You can also view alerts in the console under Observe → Alerting.

You can define your own alerts for application namespaces. Here’s an example that fires when a pod has been restarting. We’ll create it in the openshift-monitoring namespace where the built-in alerting rules live:

Verify the rule was created:

The rule exists but nothing is triggering it. Let’s fix that - deploy a pod that crash-loops so you can see the alert go from inactive to pending to firing:

While the restarts accumulate, navigate to Observe → Alerting → Alerting Rules in the console and search for HighPodRestartRate.

Click on the rule to see its details. After about 2-3 minutes (once the pod has restarted a few times and the for: 1m duration passes), you’ll see the alert transition from Inactive → Pending → Firing:

Now switch to the Alerts tab. Filter by Alert State: Firing - you should see HighPodRestartRate with crashloop-test as the source pod:

This is the view on-call engineers live in. You just created a rule, triggered it with a real failure, and watched it fire. Clean up the crash-looping pod:

Your alert fired - but who got notified? Right now, nobody. Alertmanager has receivers configured (Default, Watchdog, Critical) but none of them actually send notifications anywhere. Alerts fire, appear in the console, and that’s it.

In production, that means an overnight incident goes unnoticed until someone checks the console in the morning. Let’s fix that by wiring up email notifications.

The logging pipeline works like this:

Each layer is managed by a separate operator, so you can upgrade or troubleshoot them independently.

Three operators are needed. Each one manages a different part of the pipeline. First, create the required namespaces and operator groups:

Install the Loki Operator from the Software Catalog - this is the same process for any operator in OperatorHub.

Loki needs object storage for log data. This cluster has OpenShift Data Foundation (ODF) which provides S3-compatible storage via NooBaa. In production, you’d point this at AWS S3, Azure Blob, or any S3-compatible store.

Create an S3 bucket for log storage:

Wait for the bucket to be provisioned:

Now extract the S3 credentials, create the CA cert ConfigMap, and deploy the LokiStack - all in one block so the variables stay in scope:

The ClusterLogForwarder tells Vector (the collector) what to collect and where to send it. The collector runs as a DaemonSet - one pod per node - so it captures logs from every container on every node.

Create the collector service account with the required permissions:

Wait for the LokiStack gateway CA bundle to be created (the ClusterLogForwarder needs it):

Create the ClusterLogForwarder - this defines the pipeline:

The UIPlugin adds the Observe → Logs view to the OpenShift console:

Wait for the collector pods to start before verifying the full stack:

You should see the logging operator, collector pods (one per node), and multiple Loki component pods all Running.

After the UIPlugin is created, you may see a notification in the console:

Click Refresh to reload the console with the new Logs view. This may log you out of the console - if so, log back in with:

Navigate to Observe → Logs in the console. You’ll see three log types:

Use the filters to search by namespace, pod name, or severity level:

Filter by severity (Critical, Error, Warning, Info, Debug):

Try it now: select Infrastructure logs, then filter by severity Warning or Error. These are the log entries that matter during an incident - pod evictions, OOM kills, certificate expiry warnings.

The histogram at the top shows log volume over time - spikes in the error count correlate with incidents. Click on a spike to zoom into that time window.

Tempo is the backend that stores and indexes traces - similar to how Loki stores logs. We’ll use TempoMonolithic, a single-binary deployment suitable for workshops and small clusters.

Create the namespace and deploy TempoMonolithic with multi-tenancy enabled:

Wait for Tempo to be ready:

The tracing pipeline needs two sets of permissions: the collector needs write access to push traces, and users need read access to query them.

The collector is the central receiving station for trace data. Apps send traces to the collector, and the collector forwards them to Tempo for storage. Think of it like Vector for logs - a pipeline component that sits between producers and storage.

This collector:

Verify the collector is running:

You should see both the Tempo pod and the collector pod Running.

Like the logging UI, the Traces view needs a UIPlugin:

You may see a console notification to refresh - click Refresh to reload.

The collector is ready but nothing is sending traces yet. Let’s send a test trace to verify the full pipeline:

This simulates what an instrumented application does - sends a span describing a single operation (GET /api/weather) with timing data.

Loading the Traces view may log you out of the console. If so, log back in with:

Navigate to Observe → Traces in the console. Select the Tempo instance (tracing-system / sample) and tenant (dev):

You should see the workshop-demo trace. Click on it to see the trace detail view - a timing waterfall showing each span:

With a single service this shows one span. In a real microservices application, you’d see multiple spans stacked in a waterfall - frontend → API → database - making it immediately obvious where latency is coming from.

The collector is now ready for production use. Developers point their apps at otel-collector.tracing-system.svc:4317 using OpenTelemetry SDKs, and traces automatically appear here.

What you learned:

Together, metrics, logs, and traces give you the three pillars of observability - each answers a different question about your cluster and workloads.

Key operational commands: