Tools of the trade
These are some of the many Community Created Tools available to speed up debugging, issue resolution and problem identification when analyzing issues with an OpenShift cluster. Red Hat does not maintain and/or provide support on all of these tools. Please see each projects GitHub page, if available, for assistance.
Tools we will use
During this lab, we will run through a number of exercises that show the power and effective use of the following 8 tools.
omc
The omc tool (OpenShift Must‑Gather Client) is a CLI that lets engineers inspect and query OpenShift must-gather archives using oc-like commands to view resources, logs, metrics, and diagnostics.
You can find and download omc from github at: omc releases
For more in-depth details on how to collect a must-gather and for alternative must-gather commands for other OpenShift components, you can refer to the Gathering data about your cluster section of our doc.
kubectl-dev_tool
kubectl-dev_tool is a kubectl plugin that provides developer-friendly commands for Kubernetes workflows—like port-forwarding, log streaming, interactive pod shells, and simplified deploy/rollback operations—wrapped in shorter, higher-level subcommands. It focuses on speeding local development and debugging by combining common kubectl actions into convenient, opinionated shortcuts.
For our use case we will be using the audit function of the kubectl-dev_tool to analyze audit log data by user, resource, verb, and sort it in an easy to ready format.
Installation
To install kubectl-dev_tool from github ensure golang is installed and run the go install command or follow the build instructions located in the cluster debug tools repository.
go install github.com/openshift/cluster-debug-tools/cmd/kubectl-dev_tool@latestUsage
In order to use the kubectl-dev_tool you need to collect audit logs from the problematic cluster. You can find more information on using the gather_audit_logs sub-command with must-gather in the gather_audit_logs section of our docs.
yq and jq
jq is a lightweight command-line JSON processor that reads JSON, applies queries and filters to transform, extract, and format data, and outputs the result. This is useful for slicing, mapping, and aggregating JSON in scripts and pipelines.
yq is a command-line YAML processor (also supports JSON) that applies jq-like queries and transformations to read, update, convert, and write YAML/JSON—useful for scripting config files and pipelines.
Installation
jq and yq are provided by most Package Managers. yq for Red Hat Enterprise Linux is located in the EPEL repository.
dnf install jq yq -yUsage
When to use each tool is based on your preferred output. The version of yq we use will convert YAML to JSON unless you specify to keep the output YAML as seen in the example below.
$ omc get node worker-00002c77 -o json | jq '.status.conditions[]'
{
"lastHeartbeatTime": "2025-10-13T15:23:37Z",
"lastTransitionTime": "2025-09-11T02:08:38Z",
"message": "kubelet has sufficient memory available",
"reason": "KubeletHasSufficientMemory",
"status": "False",
"type": "MemoryPressure"
}
{
"lastHeartbeatTime": "2025-10-13T15:23:37Z",
"lastTransitionTime": "2025-09-11T02:08:38Z",
"message": "kubelet has no disk pressure",
"reason": "KubeletHasNoDiskPressure",
"status": "False",
"type": "DiskPressure"
}$ omc get node worker-00002c77 -o yaml | yq -Y '.status.conditions[]'
lastHeartbeatTime: "2025-10-13T15:23:37Z"
lastTransitionTime: "2025-09-11T02:08:38Z"
message: kubelet has sufficient memory available
reason: KubeletHasSufficientMemory
status: "False"
type: MemoryPressure
---
lastHeartbeatTime: "2025-10-13T15:23:37Z"
lastTransitionTime: "2025-09-11T02:08:38Z"
message: kubelet has no disk pressure
reason: KubeletHasNoDiskPressure
status: "False"
type: DiskPressure
---etcd-ocp-diag
This python script allows you to quickly identify common etcd performance errors and review statistics and errors that have occurred so you can quickly pinpoint issues with your underlying disk or point you to towards other areas that may be causing the issue.
You can find and download etcd-ocp-diag from github at: etcd-ocp-diag-script
ocp_insights
OpenShift LightSpeed, formerly OpenShift Insights, is a data collection tool that collects information similar to an OpenShift Must-Gather and uploads that data to Red Hat’s servers for proactive analysis. When a must-gather is collected from a cluster running the Insights Operator, the latest Lightspeed Archives are collected. Using these archives along with the ocp_insights script, we can quickly triage a cluster to determine all of the important information in seconds to allow us to get an understand of the make up and design of the cluster along with any issues currently affecting the cluster.
You can find and download ocp_insights from github at: ocp_insights
ovnkube-trace
ovnkube-trace, is a GO based command line tool that takes a set of arguments, inspects your running OpenShift cluster and translates all of the pod IP and MAC information to perform low level ovn-trace, ovs-appctl ofproto/trace and ovn-detrace commands on your defined workloads.
|
Because this lab is not taking place on a running OpenShift cluster, we will show you examples of the ovnkube-trace command for each exercise and actually use the lower level ovn-trace that ovnkube-trace produces. |