Install Kata, Break It, Fix It

Value Proposition

Installing the Sandboxed Containers Operator is necessary but not sufficient. The Kata runtime does not activate until a KataConfig resource tells the operator which nodes to configure. That configuration step is deceptively easy to get wrong: a single label mismatch between the kataConfigPoolSelector in KataConfig and the label applied to the Kata NodePool in Module 2 causes the operator to proceed silently — no error, no warning, zero nodes configured.

This module closes the installation loop. You apply KataConfig with careful attention to the selector, watch the DaemonSet install Kata binaries onto each node in real time, and verify the result at two layers: the cluster-level kata RuntimeClass and the containerd runtime registry on the node itself.

Then you deliberately break it. A kata pod without the correct node toleration stays Pending forever — the scheduler cannot place it on any node. You reproduce it, trace the FailedScheduling event with oc describe, and fix it by adding the missing toleration. The module closes with a discussion of production-appropriate toleration strategy.

Learning Objectives

By the end of this module you will be able to:

Configure KataConfig with a kataConfigPoolSelector that correctly matches the Kata NodePool node label from Module 2
Observe the DaemonSet-based installation as it installs Kata binaries onto each node without rebooting
Verify Kata installation at two layers: the kata RuntimeClass in the cluster and the kata handler in containerd’s runtime registry
Reproduce and trace a Kata scheduling failure caused by a missing node toleration
Fix the scheduling failure by adding the correct toleration to the pod spec

Section 1 — Apply KataConfig

The kataConfigPoolSelector

KataConfig does not install Kata binaries on every node — it installs them only on nodes that match the kataConfigPoolSelector. The selector is a standard Kubernetes label selector. If it does not exactly match the label applied to the Kata NodePool nodes in Module 2, the operator proceeds with zero nodes configured and reports no error.

Confirm the label is present on the Kata nodes before applying KataConfig:

oc get nodes -l node-role.kubernetes.io/kata= \
  -o custom-columns=NAME:.metadata.name,INSTANCE:.metadata.labels."node\.kubernetes\.io/instance-type"

Both Standard_D4s_v5 nodes should appear. If this returns nothing, the label from Module 2 was not applied correctly — stop and re-apply it before continuing.

Configure ACR Registry Credentials

The Sandboxed Containers Operator reads the OpenShift release image manifest to locate the Kata extension payload. In ARO HCP, release images are mirrored to a private Azure Container Registry (arohcpocpprod.azurecr.io). The operator’s internal HTTP client reads credentials from the filesystem — not from Kubernetes pull secrets — so those credentials must be explicitly mounted into the operator pod before KataConfig will work.

Extract the ACR credentials from the cluster pull secret and create a Docker config secret in the operator namespace:

oc get secret pull-secret -n openshift-config \
  -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | \
  jq '{"auths": {"arohcpocpprod.azurecr.io": .auths["arohcpocpprod.azurecr.io"]}}' \
  > /tmp/acr-config.json

oc create secret generic acr-docker-config \
  --from-file=config.json=/tmp/acr-config.json \
  -n openshift-sandboxed-containers-operator \
  --dry-run=client -o yaml | oc apply -f -

Capture the controller-manager container name and patch the deployment to mount the credentials:

CONTAINER=$(oc get deployment controller-manager \
  -n openshift-sandboxed-containers-operator \
  -o jsonpath='{.spec.template.spec.containers[0].name}')

oc patch deployment controller-manager \
  -n openshift-sandboxed-containers-operator \
  --type=strategic -p "{
  \"spec\": {
    \"template\": {
      \"spec\": {
        \"volumes\": [{\"name\": \"acr-docker-config\", \"secret\": {\"secretName\": \"acr-docker-config\"}}],
        \"containers\": [{
          \"name\": \"${CONTAINER}\",
          \"env\": [{\"name\": \"DOCKER_CONFIG\", \"value\": \"/tmp/docker-cfg\"}],
          \"volumeMounts\": [{\"name\": \"acr-docker-config\", \"mountPath\": \"/tmp/docker-cfg\", \"readOnly\": true}]
        }]
      }
    }
  }
}"

Wait for the operator to roll out with the new credentials:

oc rollout status deployment/controller-manager \
  -n openshift-sandboxed-containers-operator

This step is specific to ARO HCP. The cluster’s release images are mirrored to a private ACR that uses managed identity authentication at the node level. The operator’s Go HTTP client cannot use managed identity — it reads credentials from $DOCKER_CONFIG/config.json on disk. Mounting the pull secret at that path gives the operator the Bearer token it needs to inspect the release image manifest and locate the Kata extension payload.

Prepare Kata Nodes for Installation

Two steps are required before applying KataConfig in ARO HCP. In self-managed OpenShift, the MachineConfig Operator handles both automatically. In ARO HCP, the MCO is managed internally by the service and does not apply these changes to customer NodePools.

Add the `kata-oc` node label

The Sandboxed Containers Operator’s install DaemonSet uses the node selector node-role.kubernetes.io/kata-oc=. This label is normally applied by the MCO when it processes a MachineConfigPool. In ARO HCP, it must be applied manually:

oc label nodes -l node-role.kubernetes.io/kata= \
  node-role.kubernetes.io/kata-oc=

Confirm the label is present on both Kata nodes:

oc get nodes -l node-role.kubernetes.io/kata-oc= \
  -o custom-columns=NAME:.metadata.name,ROLES:.metadata.labels."node-role\.kubernetes\.io/kata-oc"

Both Kata nodes should appear. If this returns nothing, the label was not applied — check that the Kata NodePool nodes have node-role.kubernetes.io/kata= first.

Remove the `NoSchedule` taint temporarily

The Kata NodePool was created with a kata=true:NoSchedule taint to prevent regular workloads from landing on Kata nodes. The osc-rpm-install DaemonSet does not include a toleration for this taint — it expects the MCO to have already configured the nodes before the taint is in effect. In ARO HCP, the taint must be removed during installation.

oc adm taint nodes -l node-role.kubernetes.io/kata-oc= \
  kata=true:NoSchedule-

The trailing - removes the taint. This is safe — the Kata nodes only have Kata workloads, and no non-Kata workloads will be scheduled here during the brief installation window. The taint is re-added after installation completes in Section 2.

Apply KataConfig

Apply the KataConfig resource, setting the kataConfigPoolSelector to match node-role.kubernetes.io/kata:

oc apply -f - <<'EOF'
apiVersion: kataconfiguration.openshift.io/v1
kind: KataConfig
metadata:
  name: example-kataconfig
spec:
  kataConfigPoolSelector:
    matchLabels:
      node-role.kubernetes.io/kata: ""
EOF

Confirm the resource was created and note the status columns:
```
oc get kataconfig example-kataconfig
```
Expected output:
```
NAME                 INPROGRESS   COMPLETED   TOTAL   AGE
example-kataconfig                                    Xm
```
The three status columns track the DaemonSet rollout as it runs:
- INPROGRESS — nodes currently being processed by the install DaemonSet
- COMPLETED — nodes that have finished installing Kata binaries
- TOTAL — total number of nodes targeted by the kataConfigPoolSelector
  
  All three are empty immediately after creation. They populate once the DaemonSet pods begin running on each node. If TOTAL remains empty after several minutes, the kataConfigPoolSelector matched zero nodes — verify the label with oc get nodes -l node-role.kubernetes.io/kata=.

The kataConfigPoolSelector value node-role.kubernetes.io/kata: "" must be an exact match — same key, same value — as the label applied in Module 2. An empty string ("") is a valid label value in Kubernetes. If the selector uses any other value, KataConfig installs on zero nodes silently.

Section 2 — Watch the Kata Installation Rollout

When KataConfig is applied, the Sandboxed Containers Operator creates the osc-rpm-install DaemonSet, which runs one pod per matching node. Each pod installs the kata-containers RPM packages directly onto the node and exits. Nodes do not reboot.

In ARO HCP, the MachineConfigPool API is not exposed — node configuration is managed internally by the service. Progress is tracked through the KataConfig status columns and the DaemonSet pod state.

Installation typically takes 3–5 minutes per node. Use two terminals to observe it in parallel.

Terminal 1 — Watch the KataConfig Status

Watch the KataConfig status columns for progress:
```
oc get kataconfig example-kataconfig -w
```
As the rollout proceeds, the columns populate:
- INPROGRESS — number of nodes currently being processed
- COMPLETED — number of nodes with Kata binaries installed
- TOTAL — total number of nodes targeted
  
  Wait for COMPLETED to equal TOTAL before continuing to Section 3. Press Ctrl+C once complete.

Terminal 2 — Watch the Install DaemonSet

In a second terminal, watch the install pods come up on each Kata node:
```
oc get pods -n openshift-sandboxed-containers-operator \
  -l name=osc-rpm-install -w
```
The operator creates one pod per Kata node. Each pod installs the kata-containers RPM packages directly onto the node and exits when done:
- ContainerCreating → Running — pod started, RPM installation in progress
- Completed — RPM installation finished, node ready for kata workloads
  
  In ARO HCP, Kata nodes do not reboot during installation. The DaemonSet installs binaries in place using an overlay filesystem. This is different from self-managed OpenShift where the MCO triggers a full node reboot to apply a MachineConfig.

ARO HCP does not expose the machineconfigpool API. In self-managed OpenShift, the MCO applies a MachineConfig that triggers a rolling node reboot — you would see nodes cycle through Ready,SchedulingDisabled → NotReady,SchedulingDisabled → Ready. That sequence does not occur in ARO HCP. The KataConfig status columns (INPROGRESS, COMPLETED, TOTAL) are the authoritative progress indicator here.

While You Wait — Observe the Installation Internals

Installation runs in the background while you watch. Use this time to make it visible rather than waiting passively.

Watch the Operator DaemonSet

The Sandboxed Containers Operator creates a DaemonSet to push Kata binaries onto each matching node. Watch it come up:

oc get daemonset -n openshift-sandboxed-containers-operator -w

The DaemonSet rolls one pod per Kata node. Each pod handles the binary installation on its node and exits when done. This is how the operator delegates node-level work without needing direct host access.

Stream the Operator Logs

The controller-manager logs every step of the rollout:

oc logs deployment/controller-manager \
  -n openshift-sandboxed-containers-operator -f \
  | grep -iE "kata|node|install|daemon"

You will see the operator detecting the KataConfig, creating the DaemonSet, and tracking node progress. Press Ctrl+C when you are done observing.

Inspect the Node Before Kata Is Installed

Open a debug shell on a Kata node while it is still waiting to be processed. Capture the containerd runtime list — you will compare this to the same output in Section 3 after installation completes:

KATA_NODE=$(oc get nodes -l node-role.kubernetes.io/kata= \
  -o jsonpath='{.items[0].metadata.name}')

oc debug node/$KATA_NODE -- chroot /host \
  sh -c 'crictl info | grep kata || echo "kata handler: NOT REGISTERED"'

kata handler: NOT REGISTERED. The compact JSON from crictl info contains no kata entries yet. After Section 3’s verification step, the same command will show kata handler strings in the output — making the before/after change concrete.

Re-add the `NoSchedule` Taint

Once COMPLETED equals TOTAL, re-add the taint to restore workload isolation on the Kata nodes:

oc adm taint nodes -l node-role.kubernetes.io/kata-oc= \
  kata=true:NoSchedule

Confirm the taint is back:

oc get nodes -l node-role.kubernetes.io/kata-oc= \
  -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

Both nodes should show [map[effect:NoSchedule key:kata value:true]]. Kata pods with the correct toleration will still schedule here — all other workloads are blocked.

Section 3 — Verify Kata Installation

Kata installation must be verified at two layers. A RuntimeClass present in the cluster confirms the API is registered. The kata handler present in containerd’s runtime registry on the node confirms the binary is actually installed and usable.

Layer 1 — Cluster: RuntimeClass

Confirm the kata RuntimeClass now exists:
```
oc get runtimeclass kata
```
Expected output:
```
NAME   HANDLER   AGE
kata   kata      Xm
```
The HANDLER value kata tells containerd which runtime binary to call when a pod requests this class. If this resource is absent, the DaemonSet installation did not complete or the kataConfigPoolSelector matched zero nodes.

Layer 2 — Node: containerd Runtime Registry

Capture a Kata node name:

KATA_NODE=$(oc get nodes -l node-role.kubernetes.io/kata= \
  -o jsonpath='{.items[0].metadata.name}')
echo $KATA_NODE

Open a debug shell on the Kata node:
```
oc debug node/$KATA_NODE
```

Inside the debug shell, inspect the containerd runtime registry:

chroot /host crictl info | python3 -c "
import json,sys
def find(obj):
    if isinstance(obj,dict):
        for k,v in obj.items():
            if 'kata' in k and isinstance(v,dict) and 'RuntimePath' in v:
                print(f'{k}: {v.get(\"RuntimeType\",\"-\")} -> {v.get(\"RuntimePath\",\"-\")}')
            find(v)
    elif isinstance(obj,list):
        [find(i) for i in obj]
find(json.load(sys.stdin))
"

Expected output:

kata: vm -> /usr/bin/containerd-shim-kata-v2
kata-nvidia-gpu: vm -> /usr/bin/containerd-shim-kata-v2
kata-remote: vm -> /usr/bin/containerd-shim-kata-v2
kata-se: vm -> /usr/bin/containerd-shim-kata-v2
kata-snp: vm -> /usr/bin/containerd-shim-kata-v2
kata-snp-nvidia-gpu: vm -> /usr/bin/containerd-shim-kata-v2
kata-tdx: vm -> /usr/bin/containerd-shim-kata-v2
kata-tdx-nvidia-gpu: vm -> /usr/bin/containerd-shim-kata-v2

OSC installs eight kata runtime variants. All share the same shim binary (containerd-shim-kata-v2) and RuntimeType: vm — confirming every variant runs workloads inside a micro-VM, not as a standard Linux process. The variants differ in hardware security technology:

kata — standard QEMU/KVM, used in this lab
kata-remote — peer pods (workload VM runs in Azure, not on the node)
kata-snp / kata-tdx / kata-se — confidential computing variants (AMD SEV-SNP, Intel TDX, IBM Secure Execution)
kata-*-nvidia-gpu — GPU passthrough variants

If this output is empty, the DaemonSet installation did not complete on this node — check oc get kataconfig -o yaml and look for the node listed under status.kataNodes.failed.

Exit the debug shell:
```
exit
```

Section 4 — Break It: Trigger the Scheduling Failure

The Kata NodePool has a kata=true:NoSchedule taint — the same taint you added in Module 2 to prevent regular workloads from landing on these nodes. Any pod that requests the kata RuntimeClass must also declare a toleration for that taint. Without it, the scheduler cannot place the pod on any node and it remains Pending indefinitely.

This is a realistic field failure. The RuntimeClass exists, KataConfig completed, the node is Ready — everything looks correct — but the pod never starts.

Deploy a Kata Pod Without a Toleration

Create a namespace for the test:
```
oc new-project kata-test
```

Deploy a Kata pod that is missing the node toleration:

oc apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: kata-test-pod
  namespace: kata-test
spec:
  runtimeClassName: kata
  containers:
  - name: ubi
    image: registry.access.redhat.com/ubi9/ubi-minimal
    command: ["sleep", "infinity"]
EOF

Observe that the pod stays Pending:
```
oc get pod kata-test-pod -n kata-test
```

Trace the Failure

Examine the pod events to find the scheduling message:
```
oc describe pod kata-test-pod -n kata-test | grep -A 5 Events
```
The events section will contain a FailedScheduling message similar to:
```
Warning  FailedScheduling  ...  0/4 nodes are available: 2 node(s) had
untolerated taint {kata: true}: value true, effect NoSchedule, 2 node(s)
didn't match Pod's node affinity/selector.
```
The two regular worker nodes are excluded by the RuntimeClass node affinity (they have no kata handler). The two Kata nodes are excluded because the pod has no toleration for kata=true:NoSchedule. No valid node exists — the pod is stuck.

The kata RuntimeClass automatically injects a node affinity that requires node-role.kubernetes.io/kata-oc= — this is how OpenShift ensures kata pods only schedule on nodes where the kata handler is installed. Combined with the NoSchedule taint, kata pods must carry both the correct runtimeClassName and the matching toleration to land on a kata node.

Section 5 — Fix It: Add the Toleration

Add the Missing Toleration

Delete the failing pod:

oc delete pod kata-test-pod -n kata-test

Re-apply the pod with the correct toleration:

oc apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: kata-test-pod
  namespace: kata-test
spec:
  runtimeClassName: kata
  tolerations:
  - key: kata
    value: "true"
    effect: NoSchedule
  containers:
  - name: ubi
    image: registry.access.redhat.com/ubi9/ubi-minimal
    command: ["sleep", "infinity"]
EOF

Confirm the pod reaches Running state:
```
oc get pod kata-test-pod -n kata-test -w
```
Wait for STATUS to show Running, then press Ctrl+C.

Confirm the pod landed on a Kata node:

oc get pod kata-test-pod -n kata-test \
  -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName,RUNTIME:.spec.runtimeClassName

Production Consideration: Toleration Strategy

Every kata workload in your cluster must carry the kata=true:NoSchedule toleration. The recommended approach is to embed this in a pod template at the Deployment or StatefulSet level rather than setting it pod by pod. Teams adopting Kata as a runtime standard can enforce toleration presence through an admission webhook or OPA policy — ensuring no kata workload accidentally loses the toleration during a manifest update.

Summary

What You Learned

The kataConfigPoolSelector is the most common Kata misconfiguration. A label mismatch causes zero nodes to be configured with no error output. Verify with oc get nodes -l node-role.kubernetes.io/kata= before applying KataConfig.
KataConfig tracks the rollout through status columns. In ARO HCP, the machineconfigpool API is not exposed. Progress is visible in the INPROGRESS, COMPLETED, and TOTAL columns of oc get kataconfig. The DaemonSet installs Kata binaries in place — nodes do not reboot.
Kata installation requires two-layer verification. The kata RuntimeClass in the cluster proves the API is registered. The kata handler in the containerd runtime registry on the node proves the binary is installed. Both must be present for Kata pods to run.
Kata pods require a toleration for the kata=true:NoSchedule taint. Without it, the pod stays Pending indefinitely — the RuntimeClass node affinity routes it to Kata nodes, but the taint blocks scheduling. The FailedScheduling event names the untolerated taint directly.
Tolerations belong in the workload template, not as a one-off fix. Embed the toleration at the Deployment or StatefulSet level so it survives manifest updates. Enforce it cluster-wide with an admission webhook or OPA policy for teams running Kata at scale.

Key Takeaways for Customer Conversations

"KataConfig is installed but pods aren’t using Kata — what’s wrong?"

Check the kataConfigPoolSelector first. Run oc get nodes -l node-role.kubernetes.io/kata= — if it returns nothing, the label either wasn’t applied or doesn’t match the selector exactly. KataConfig silently skips nodes whose labels don’t match.

"How do I know Kata is actually installed on the node and not just registered in the API?"

Two-layer verification: oc get runtimeclass kata confirms the API object. Then oc debug node/<kata-node> and chroot /host crictl info | grep kata confirms the handler is registered in containerd’s runtime registry. Both must pass before you trust that a Kata pod will run.

"A customer’s Kata pod is stuck in Pending. What do I check?"

Run oc describe pod <name> and look at the Events section. A FailedScheduling event with "untolerated taint {kata: true}" means the pod is missing tolerations: [{key: kata, value: true, effect: NoSchedule}]. Add it to the pod spec and re-apply. If the taint message is absent, check that the runtimeClassName: kata is set — without it the pod won’t target kata nodes at all.

"Does Kata installation disrupt my cluster in ARO HCP?"

No. In ARO HCP, Kata installation does not reboot nodes. The osc-rpm-install DaemonSet installs the kata-containers packages in place on each node. Nodes remain Ready throughout. The only preparation required is temporarily removing the kata=true:NoSchedule taint during installation, which is safe since no non-Kata workloads are targeting those nodes.