Intro to omc

In this module we will at a subset of the features in omc that allow you to quickly review an OpenShift Must-Gather. omc and the oc client have common sub-commands like get and describe, so we will not cover them directly. Instead we will focus on some of the handy omc specific commands to get at harder to find and interpret information.

Using the omc use command

To get started you want to verify that omc is installed in your path and working
Once you have confirmed this, you want to call the omc use command to tell omc the must-gather to utilize for review
Then run the omc get clusterversion to verify that you are using the must-gather

cd ~/Module2/

omc version

omc version: v3.12.2
hash: b7eee6a
https://github.com/gmeghnag/omc

omc use must-gather.local

Must-Gather    : /home/lab-user/Module2/must-gather.local/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-1689ebb475fcb1d8021057f26ef090534fbe210716392ec68bdeba70774dc04f
Project        : default
ApiServerURL   : https://api.ci-ln-m6sdtw2-76ef8.aws-4.ci.openshift.org:6443
Platform       : AWS
ClusterID      : b26e0177-d855-48ed-aefd-79de416fcc06
ClusterVersion : 4.18.30
ClientVersion  : 4.19.4
Image          : quay-io-openshift-release-dev-ocp-v4-0-art-dev

omc get clusterversion

NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.18.30   True        False         8m      Cluster version is 4.18.30

Like with oc, you can use -o yaml to print out addition information about the object:

omc get clusterversion -o yaml

apiVersion: v1
items:
- apiVersion: config.openshift.io/v1
  kind: ClusterVersion
  metadata:
    creationTimestamp: "2025-12-15T20:48:21Z"
    generation: 2
    name: version
    resourceVersion: "35472"
    uid: 3c5ff6ba-55ae-416c-9702-ad5012cd8aff
  spec:
    clusterID: b26e0177-d855-48ed-aefd-79de416fcc06
  status:
    availableUpdates: null
    capabilities:
      enabledCapabilities:
      - Build
      - CSISnapshot
      - CloudControllerManager
      - CloudCredential
      - Console
      - DeploymentConfig
      - ImageRegistry
      - Ingress
      - Insights
      - MachineAPI
      - NodeTuning
      - OperatorLifecycleManager
      - OperatorLifecycleManagerV1
      - Storage
      - baremetal
      - marketplace
      - openshift-samples
      knownCapabilities:
      - Build
      - CSISnapshot
      - CloudControllerManager
      - CloudCredential
      - Console
      - DeploymentConfig
      - ImageRegistry
      - Ingress
      - Insights
      - MachineAPI
      - NodeTuning
      - OperatorLifecycleManager
      - OperatorLifecycleManagerV1
      - Storage
      - baremetal
      - marketplace
      - openshift-samples
    conditions:
    - lastTransitionTime: "2025-12-15T20:48:54Z"
      message: The update channel has not been configured.
      reason: NoChannel
      status: "False"
      type: RetrievedUpdates
    - lastTransitionTime: "2025-12-15T20:48:54Z"
      message: Kubernetes 1.32 and therefore OpenShift 4.19 remove several APIs which
        require admin consideration. Please see the knowledge article https://access.redhat.com/articles/7112216
        for details and instructions. This cluster is GCP or AWS but lacks a boot
        image configuration. OCP will automatically opt this cluster into boot image
        management in 4.19. Please add a configuration to disable boot image updates
        if this is not desired. See https://docs.redhat.com/en/documentation/openshift_container_platform/4.18/html/machine_configuration/mco-update-boot-images#mco-update-boot-images-disable_machine-configs-configure
        for more details.
      reason: AdminAckRequired
      status: "False"
      type: Upgradeable
    - lastTransitionTime: "2025-12-15T20:48:54Z"
      message: Capabilities match configured spec
      reason: AsExpected
      status: "False"
      type: ImplicitlyEnabledCapabilities
    - lastTransitionTime: "2025-12-15T20:48:54Z"
      message: Payload loaded version="4.18.30" image="registry.build11.ci.openshift.org/ci-ln-m6sdtw2/release@sha256:3bf71637442f3e5e5d92dfbcba6e7d496afbbe6c3af975f043c5deaf792e6edf"
        architecture="amd64"
      reason: PayloadLoaded
      status: "True"
      type: ReleaseAccepted
    - lastTransitionTime: "2025-12-15T21:23:16Z"
      message: Done applying 4.18.30
      status: "True"
      type: Available
    - lastTransitionTime: "2025-12-15T21:23:16Z"
      status: "False"
      type: Failing
    - lastTransitionTime: "2025-12-15T21:23:16Z"
      message: Cluster version is 4.18.30
      status: "False"
      type: Progressing
    desired:
      image: registry.build11.ci.openshift.org/ci-ln-m6sdtw2/release@sha256:3bf71637442f3e5e5d92dfbcba6e7d496afbbe6c3af975f043c5deaf792e6edf
      version: 4.18.30
    history:
    - completionTime: "2025-12-15T21:23:16Z"
      image: registry.build11.ci.openshift.org/ci-ln-m6sdtw2/release@sha256:3bf71637442f3e5e5d92dfbcba6e7d496afbbe6c3af975f043c5deaf792e6edf
      startedTime: "2025-12-15T20:48:54Z"
      state: Completed
      verified: false
      version: 4.18.30
    observedGeneration: 2
    versionHash: PqPMX3E67Uk=
kind: List

Using the omc certs inspect command

The certs inspect command allows you to inspect all certificates in ConfigMaps and Secrets that are in the must-gather
Additionally, the command highlights CertificateSigngingRequest which can help resolve issues related to nodes joining a cluster.

omc certs inspect

NAME                                             KIND                        AGE   CERTTYPE    SUBJECT                                                                             NOTBEFORE                       NOTAFTER
kube-root-ca.crt                                 ConfigMap                   43m   ca-bundle   CN=kube-apiserver-lb-signer,OU=openshift                                            2025-12-15 20:35:02 +0000 UTC   2035-12-13 20:35:02 +0000 UTC
kube-root-ca.crt                                 ConfigMap                   43m   ca-bundle   CN=kube-apiserver-localhost-signer,OU=openshift                                     2025-12-15 20:35:02 +0000 UTC   2035-12-13 20:35:02 +0000 UTC
...
openshift-service-ca.crt                         ConfigMap                   43m   ca-bundle   CN=openshift-service-serving-signer@1765831901                                      2025-12-15 20:51:41 +0000 UTC   2028-02-13 20:51:42 +0000 UTC
csr-258lr                                        CertificateSigningRequest   33m   ca-bundle   CN=system:ovn-node:ip-10-0-91-110.ec2.internal,O=system:ovn-nodes                   2025-12-15 20:52:46 +0000 UTC   2025-12-16 20:35:04 +0000 UTC
csr-4c9l4                                        CertificateSigningRequest   10m   ca-bundle   CN=system:node:ip-10-0-110-54.ec2.internal,O=system:nodes                           2025-12-15 21:15:43 +0000 UTC   2025-12-16 20:35:04 +0000 UTC
...

Using the omc etcd command

The etcd command allows you to view the health and the status of your etcd cluster.
It includes two options, etcd health and etcd status

omc etcd health

+--------------------------+--------+-------------+-------+
|         ENDPOINT         | HEALTH |    TOOK     | ERROR |
+--------------------------+--------+-------------+-------+
| https://10.0.110.19:2379 | true   | 6.670232ms  |       |
| https://10.0.37.185:2379 | true   | 10.701713ms |       |
| https://10.0.97.114:2379 | true   | 14.002023ms |       |
+--------------------------+--------+-------------+-------+

omc etcd status

+--------------------------+------------------+---------+----------------+----------+-----------+------------+-----------+------------+--------------------+--------+
|         ENDPOINT         |        ID        | VERSION | DB SIZE/IN USE | NOT USED | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+--------------------------+------------------+---------+----------------+----------+-----------+------------+-----------+------------+--------------------+--------+
| https://10.0.110.19:2379 | bf9b0fa63f084685 | 3.5.18  | 60 MB/55 MB    | 9%       | true      | false      |         8 |      41342 |              41342 |        |
| https://10.0.97.114:2379 | c05a125dae594a9b | 3.5.18  | 60 MB/55 MB    | 9%       | false     | false      |         8 |      41342 |              41342 |        |
| https://10.0.37.185:2379 | f7b31c824b48036d | 3.5.18  | 60 MB/55 MB    | 9%       | false     | false      |         8 |      41342 |              41342 |        |
+--------------------------+------------------+---------+----------------+----------+-----------+------------+-----------+------------+--------------------+--------+

Using the omc haproxy command

The haproxy backends command displays all of the haproxy configured backends

omc haproxy backends

The example must-gather used for this module does not contain HAProxy backends, so the above command will not return any data. The output below is an example output.

NAMESPACE	NAME			 INGRESSCONTROLLER	SERVICES	      PORT	     TERMINATION
aap		frost-prod    		 default  		frost-prod-service    http(8052)     edge/Redirect
ecomm		app-api-blue-p4lb5	 default	  	ecomm-api-blue	      https(8443)    reencrypt/Redirect
ecomm		app-api-prod-kg8l6	 default		plaid-api-prod	      https(8443)    passthrough/Redirect

Using the omc machine-config command

The machine-config diff command allows you to compare two MachineConfig objects. This is different from the built-in omc get machineconfigs which only displays a single MachineConfigs.
The diff option will open the selected machine-configs in vim-diff to quickly review of the two MachineConfigs to help identify changes.

omc get machineconfigs

NAME                                               GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE
00-master                                          a51e963bdc11e1b49404d111c94e6257a637059f   3.4.0             38m
00-worker                                          a51e963bdc11e1b49404d111c94e6257a637059f   3.4.0             38m
01-master-container-runtime                        a51e963bdc11e1b49404d111c94e6257a637059f   3.4.0             38m
01-master-kubelet                                  a51e963bdc11e1b49404d111c94e6257a637059f   3.4.0             38m
01-worker-container-runtime                        a51e963bdc11e1b49404d111c94e6257a637059f   3.4.0             38m
01-worker-kubelet                                  a51e963bdc11e1b49404d111c94e6257a637059f   3.4.0             38m
97-master-generated-kubelet                        a51e963bdc11e1b49404d111c94e6257a637059f   3.4.0             38m
97-worker-generated-kubelet                        a51e963bdc11e1b49404d111c94e6257a637059f   3.4.0             38m
98-master-generated-kubelet                        a51e963bdc11e1b49404d111c94e6257a637059f   3.4.0             38m
98-worker-generated-kubelet                        a51e963bdc11e1b49404d111c94e6257a637059f   3.4.0             38m
99-master-generated-registries                     a51e963bdc11e1b49404d111c94e6257a637059f   3.4.0             38m
99-master-ssh                                                                                 3.2.0             42m
99-worker-generated-registries                     a51e963bdc11e1b49404d111c94e6257a637059f   3.4.0             38m
99-worker-ssh                                                                                 3.2.0             42m
rendered-master-1acb4208fc6aef31cf85cb429bb918f7   a51e963bdc11e1b49404d111c94e6257a637059f   3.4.0             38m
rendered-worker-40c4ec5da61bfe22cb638e0699876a42   a51e963bdc11e1b49404d111c94e6257a637059f   3.4.0             38m

omc machine-config diff rendered-master-1acb4208fc6aef31cf85cb429bb918f7 rendered-worker-40c4ec5da61bfe22cb638e0699876a42

You can :q twice to close out and exit from the split screen vim-diff.

Using the omc node-logs command

The node-logs command lists the node-logs collected from the OpenShift control-plane nodes and are in the must-gather.
By running omc node-logs crio, for example, will output crio-logs for all of the control-plane nodes.

omc node-logs

The following node service logs are available to be displayed:
- NetworkManager
- crio
- kubelet
- machine-config-daemon-firstboot
- machine-config-daemon-host
- openvswitch
- ostree-finalize-staged
- ovs-configuration
- ovs-vswitchd
- ovsdb-server
- rpm-ostreed

is it possible to read the content by executing 'omc node-logs <SERVICE>'.

omc node-logs kubelet | head -n 10

Dec 15 20:49:28.345190 ip-10-0-110-19 systemd[1]: Starting Kubernetes Kubelet...
Dec 15 20:49:28.845114 ip-10-0-110-19 kubenswrapper[2442]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Dec 15 20:49:28.845114 ip-10-0-110-19 kubenswrapper[2442]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Dec 15 20:49:28.845114 ip-10-0-110-19 kubenswrapper[2442]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Dec 15 20:49:28.845114 ip-10-0-110-19 kubenswrapper[2442]: Flag --register-with-taints has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Dec 15 20:49:28.845114 ip-10-0-110-19 kubenswrapper[2442]: Flag --pod-infra-container-image has been deprecated, will be removed in a future release. Image garbage collector will get sandbox image information from CRI.
Dec 15 20:49:28.845114 ip-10-0-110-19 kubenswrapper[2442]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Dec 15 20:49:28.847145 ip-10-0-110-19 kubenswrapper[2442]: I1215 20:49:28.847038    2442 server.go:211] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Dec 15 20:49:28.855238 ip-10-0-110-19 kubenswrapper[2442]: W1215 20:49:28.855214    2442 feature_gate.go:330] unrecognized feature gate: ManagedBootImages
Dec 15 20:49:28.855238 ip-10-0-110-19 kubenswrapper[2442]: W1215 20:49:28.855231    2442 feature_gate.go:330] unrecognized feature gate: ConsolePluginContentSecurityPolicy

Using the omc ovn command

The ovn subnets command will output all of the ovn subnets on the cluster

omc ovn subnets

HOST/NODE                     ROLE                   NODE SUBNET     NODE TRANSIT-SWITCH-IP
ip-10-0-110-19.ec2.internal   control-plane,master   10.130.0.0/23   100.88.0.3/16
ip-10-0-110-54.ec2.internal   worker                 10.129.4.0/23   100.88.0.11/16
ip-10-0-37-185.ec2.internal   control-plane,master   10.129.0.0/23   100.88.0.4/16
ip-10-0-62-69.ec2.internal    worker                 10.129.2.0/23   100.88.0.7/16
ip-10-0-89-71.ec2.internal    worker                 10.130.4.0/23   100.88.0.12/16
ip-10-0-97-114.ec2.internal   control-plane,master   10.128.0.0/23   100.88.0.2/16

Using the omc prometheus command

The prometheus command provides several options to output Prometheus alertgroup, alertrule, and target.

omc prometheus alertgroup | head -n 10

GROUP                              FILENAME                                                                                                           AGE
CloudCredentialOperator            openshift-cloud-credential-operator-cloud-credential-operator-alerts-2ee2f7c9-a542-4f11-a391-ac8c0ee54224.yaml     11s
cluster-machine-approver.rules     openshift-cluster-machine-approver-machineapprover-rules-74b6c87d-07ed-4a75-aa52-e24905a846b4.yaml                 11s
node-tuning-operator.rules         openshift-cluster-node-tuning-operator-node-tuning-operator-465e993d-cb8e-44d5-88b0-7bc0f00cc5e1.yaml              2s
SamplesOperator                    openshift-cluster-samples-operator-samples-operator-alerts-72d9774f-1e50-4c05-b511-7630f32c9544.yaml               8s
default-storage-classes.rules      openshift-cluster-storage-operator-prometheus-f0c28350-cf99-4adb-b2b7-932aa3faf0da.yaml                            3s
kubernetes-storage                 openshift-cluster-storage-operator-prometheus-f0c28350-cf99-4adb-b2b7-932aa3faf0da.yaml                            9s
storage-operations.rules           openshift-cluster-storage-operator-prometheus-f0c28350-cf99-4adb-b2b7-932aa3faf0da.yaml                            25s
storage-selinux.rules              openshift-cluster-storage-operator-prometheus-f0c28350-cf99-4adb-b2b7-932aa3faf0da.yaml                            11s
cluster-operators                  openshift-cluster-version-cluster-version-operator-04905bfe-dfb3-4c78-8748-1da7e81b2591.yaml                       27s

omc prometheus alertrule | head -n 10

RULE                                               SEVERITY   STATE      AGE   ALERTS   ACTIVE SINCE
CloudCredentialOperatorTargetNamespaceMissing      warning    inactive   11s   0        ----
CloudCredentialOperatorProvisioningFailed          warning    inactive   11s   0        ----
CloudCredentialOperatorDeprovisioningFailed        warning    inactive   11s   0        ----
CloudCredentialOperatorInsufficientCloudCreds      warning    inactive   11s   0        ----
CloudCredentialOperatorStaleCredentials            warning    inactive   11s   0        ----
MachineApproverMaxPendingCSRsReached               warning    inactive   11s   0        ----
NTOPodsNotReady                                    warning    inactive   2s    0        ----
NTODegraded                                        warning    inactive   2s    0        ----
SamplesRetriesMissingOnImagestreamImportFailing    warning    inactive   8s    0        ----

omc prom alertrule -s firing

RULE       SEVERITY   STATE    AGE   ALERTS   ACTIVE SINCE
Watchdog   none       firing   23s   1        15 Dec 25 21:22 UTC

omc prom alertrule -s pending

RULE                                 SEVERITY   STATE     AGE   ALERTS   ACTIVE SINCE
ClusterNotUpgradeable                info       pending   27s   1        15 Dec 25 21:22 UTC
AlertmanagerReceiversNotConfigured   warning    pending   0s    1        15 Dec 25 21:23 UTC

omc prometheus target | head -n 10

TARGET                                             SCRAPE URL                                      HEALTH   LAST ERROR
openshift-apiserver-operator-677968d5cc-qt2zb      https://10.130.0.19:8443/metrics                up
apiserver-69dbd4bc64-bpn8b                         https://10.128.0.65:17698/metrics               up
apiserver-69dbd4bc64-b4vbt                         https://10.129.0.63:17698/metrics               up
apiserver-69dbd4bc64-xxlbw                         https://10.130.0.89:17698/metrics               up
apiserver-69dbd4bc64-bpn8b                         https://10.128.0.65:8443/metrics                up
apiserver-69dbd4bc64-b4vbt                         https://10.129.0.63:8443/metrics                up
apiserver-69dbd4bc64-xxlbw                         https://10.130.0.89:8443/metrics                up
authentication-operator-57cb7b4459-jvxsp           https://10.130.0.24:8443/metrics                up
oauth-openshift-7cb79d6c79-c7fwc                   https://10.128.0.56:6443/metrics                up