Tools of the trade

These are some of the many Community Created Tools available to speed up debugging, issue resolution and problem identification when analyzing issues with an OpenShift cluster. Red Hat does not maintain and/or provide support on these tools. Please see each projects GitHub page, if available, for assistance.

Tools we will use

During this lab, we will run through a number of exercises that show the power and effective use of the following 7 tools.

omc

The omc tool (OpenShift Must‑Gather Client) is a CLI that lets engineers inspect and query OpenShift must-gather archives using oc-like commands to view resources, logs, metrics, and diagnostics.

You can find and download omc from github at: omc releases

For more in-depth details on how to collect a must-gather and for alternative must-gather commands for other OpenShift components, you can refer to the Gathering data about your cluster section of our doc.

kubectl-dev_tool

kubectl-dev_tool is a kubectl plugin that provides developer-friendly commands for Kubernetes workflows—like port-forwarding, log streaming, interactive pod shells, and simplified deploy/rollback operations—wrapped in shorter, higher-level subcommands. It focuses on speeding local development and debugging by combining common kubectl actions into convenient, opinionated shortcuts.

For our use case we will be using the audit function of the kubectl-dev_tool to filter by user, resource, verb, and sort the data in an easy to ready format.

Installation

To install kubectl-dev_tool from github use ensure golang is installed and run the command go get or you can follow the build instructions located in the cluster debug tools repository.

Example: go get github.com/openshift/cluster-debug-tools/cmd/kubectl-dev_tool

Usage

In order to use the kubectl-dev_tool you need to collect the audit logs from the problematic cluster. You can find more information on using the gather_audit_logs sub-command with must-gather in the gather_audit_logs section of our docs.

yq and jq

jq is a lightweight command-line JSON processor that reads JSON, applies queries and filters to transform, extract, and format data, and outputs the result—useful for slicing, mapping, and aggregating JSON in scripts and pipelines.

yq is a command-line YAML processor (also supports JSON) that applies jq-like queries and transformations to read, update, convert, and write YAML/JSON—useful for scripting config files and pipelines.

Installation

jq and yq are provided by most Package Managers

Example: dnf install jq yq -y

Usage

When to use each tool is based on your preferred output. The version of yq we use will convert YAML to JSON unless you specify to keep the output YAML as seen in the example below.

$ omc get node worker-00002c77 -o json | jq '.status.conditions[]'
{
  "lastHeartbeatTime": "2025-10-13T15:23:37Z",
  "lastTransitionTime": "2025-09-11T02:08:38Z",
  "message": "kubelet has sufficient memory available",
  "reason": "KubeletHasSufficientMemory",
  "status": "False",
  "type": "MemoryPressure"
}
{
  "lastHeartbeatTime": "2025-10-13T15:23:37Z",
  "lastTransitionTime": "2025-09-11T02:08:38Z",
  "message": "kubelet has no disk pressure",
  "reason": "KubeletHasNoDiskPressure",
  "status": "False",
  "type": "DiskPressure"
}
{
  "lastHeartbeatTime": "2025-10-13T15:23:37Z",
  "lastTransitionTime": "2025-09-11T02:08:38Z",
  "message": "kubelet has sufficient PID available",
  "reason": "KubeletHasSufficientPID",
  "status": "False",
  "type": "PIDPressure"
}
{
  "lastHeartbeatTime": "2025-10-13T15:23:37Z",
  "lastTransitionTime": "2025-09-11T02:10:10Z",
  "message": "kubelet is posting ready status",
  "reason": "KubeletReady",
  "status": "True",
  "type": "Ready"
}
$ omc get node worker-00002c77 -o yaml | yq -Y '.status.conditions[]'
lastHeartbeatTime: "2025-10-13T15:23:37Z"
lastTransitionTime: "2025-09-11T02:08:38Z"
message: kubelet has sufficient memory available
reason: KubeletHasSufficientMemory
status: "False"
type: MemoryPressure
---
lastHeartbeatTime: "2025-10-13T15:23:37Z"
lastTransitionTime: "2025-09-11T02:08:38Z"
message: kubelet has no disk pressure
reason: KubeletHasNoDiskPressure
status: "False"
type: DiskPressure
---
lastHeartbeatTime: "2025-10-13T15:23:37Z"
lastTransitionTime: "2025-09-11T02:08:38Z"
message: kubelet has sufficient PID available
reason: KubeletHasSufficientPID
status: "False"
type: PIDPressure
---
lastHeartbeatTime: "2025-10-13T15:23:37Z"
lastTransitionTime: "2025-09-11T02:10:10Z"
message: kubelet is posting ready status
reason: KubeletReady
status: "True"
type: Ready

etcd-ocp-diag

This python script allows you to quick identify common etcd performance errors, review statistics, and easily review when errors have occurred so that you can quickly pinpoint if the problem is your underlying disks or point you to other areas that may be causing the issues.

You can find and download etcd-ocp-diag from github at: etcd-ocp-diag-script

ocp_insights

OpenShift LightSpeed, formerly OpenShift Insights, is a data collection tool that collects information similar to an OpenShift Must-Gather and uploads that data to Red Hat’s servers for proactive analysis. When a must-gather is collected from a cluster running the Insights Operator, the latest Lightspeed Archives are collected. Using these archives along with the ocp_insights script, we can quickly triage a cluster to determine all of the important information in seconds to allow us to get an understand of the make up and design of the cluster along with any issues currently affecting the cluster.

You can find and download ocp_insights from github at: ocp_insights