Appendix - Technical Foundations: Agentic Troubleshooting
This appendix explains how the Agentic Troubleshooting environment is built, and above all how access control works: what the AI is allowed to see and change, under which identity, and who decides. Read it after (or alongside) the Agentic Troubleshooting module if you want to understand what happens behind each exercise.
Everything described here is deployed by OpenShift GitOps (Argo CD) from Helm charts in the lab repository, under automation/gitops/.
In One Sentence
In the chat, Lightspeed acts with your own permissions through the OpenShift MCP Server; in autonomous runs, each step of the agent runs under its own short-lived service account, which can read the cluster but can only change it with the exact permissions listed in the remediation option a human approved.
Two Modes, Two Identities
| User-initiated (Exercises 1–3) | Autonomous agentic runs (Exercise 4) | |
|---|---|---|
Started by |
You, in the Lightspeed chat (troubleshooting mode) |
The alerts adapter, when a critical alert fires |
Identity used against the cluster |
Your console user token |
A per-run, per-step service account created by the agentic operator |
What it can read |
Whatever your RBAC allows |
Cluster-wide read ( |
What it can change |
Whatever your RBAC allows, after you approve sensitive tools |
Only the RBAC rules of the remediation option that was approved, only during execution |
Human control |
Tool approval prompts in the chat |
|
Building Blocks
All components run in the openshift-lightspeed namespace unless stated otherwise.
OpenShift Lightspeed
-
Lightspeed operator — deploys the Lightspeed service, its console plugin and its PostgreSQL conversation cache from the
OLSConfigCR namedcluster. -
OLSConfig— declares the LLM provider (credentials in thellm-credentialssecret), enables theMCPServerfeature gate, and registers the OpenShift MCP Server as Lightspeed’s tool provider:mcpServers: - name: openshift-mcp url: http://openshift-mcp-server.openshift-mcp.svc.cluster.local:8080/mcp headers: - name: kubernetes-authorization valueFrom: type: kubernetes -
OpenShift MCP Server (
openshift-mcpnamespace) — the same server used in the MCP Server module, running in token passthrough mode. See Technical Foundations: MCP Server for how it works.
The Agentic Layer
-
Agentic operator
lightspeed-agentic-operator— watchesAgenticRunresources and drives each one through analysis, execution and verification. Each step runs in an ephemeral sandbox pod that hosts the LLM-backed agent and is deleted when the step ends. -
Alerts adapter
lightspeed-agentic-alerts-adapter— polls Alertmanager every 30 seconds and creates anAgenticRunfor each new firing alert routed to an allowed receiver. It is configured by thealerts-adapter-configConfigMap. -
Agentic console — the Agentic Runs pages in the OpenShift console, and the
oc agenticCLI plugin.
The agentic operator is configured with these custom resources:
| Resource | Role |
|---|---|
|
The OpenAI-compatible LLM endpoint and its credentials secret ( |
|
Which LLM provider and model the agent uses, and the time limits for each step (1800 seconds here) |
|
Which stages proceed automatically and which wait for a human decision |
|
One investigation: the alert or request, its current phase, and its results |
|
The human decisions for one run: approve or deny a stage, and which remediation option to execute |
|
What each step produced, written by the sandbox pod of that step |
Observability
-
Cluster Observability Operator with a
UIPluginthat enables Perses dashboards in the console (Observe > Dashboards (Perses)). -
PersesGlobalDatasourcepointing at the Thanos querier, used by the inline charts that Lightspeed generates (Exercise 3). -
User workload monitoring, enabled so that the
PrometheusRuleresources of the demo applications raise alerts such asPaymentErrorRateHigh.
Demo Workloads
Pre-deployed applications with intentional faults, in the payments, inventory, catalog-service, user-imports and service-mesh namespaces. Each fault leaves a causal trail (alert, metrics, logs, events) for the AI to follow — for example a bad reporting-service release that exhausts PostgreSQL connections, or an image tag typo (laetst) behind a PodDisruptionBudget alert.
User-Initiated Troubleshooting: Your Own Permissions
You (console) ──▶ Lightspeed chat ──▶ LLM decides which tools to call
│
│ kubernetes-authorization: <your token>
▼
OpenShift MCP Server ── same token (passthrough) ──▶ kube-apiserver / Thanos
│
▼
your RBAC decides what succeeds
-
Lightspeed does not have a privileged identity of its own for chat tool calls. With
valueFrom.type: kubernetes, it forwards the token of the user logged in to the console, and the MCP server passes it on to the Kubernetes API. A user with read-only access gets a read-only assistant. -
Tool approval adds a human check on top of RBAC. The
OLSConfigfieldspec.ols.toolsApprovalConfig.approvalTypecontrols it:-
tool_annotations(the default, used in this lab) — tools that the MCP server annotates as sensitive, such aspod_exec, require you to click Approve in the chat (Exercise 1). -
always— every tool call requires approval. -
never— no approval prompts.
-
-
The mitigation plan suggested in the chat is advice only: nothing is changed unless you, or a tool you approve, change it.
Autonomous Agentic Runs: Per-Step Service Accounts
From Alert to AgenticRun
The alerts adapter has a narrow identity, the lightspeed-agentic-alerts-adapter service account:
-
monitoring-alertmanager-viewinopenshift-monitoring— to read firing alerts. -
create,listandgetonagenticruns— to start investigations.
It cannot read or change workloads itself. By default filtering.allowedReceivers is empty, so no runs are created. In Exercise 4 you uncomment critical, which makes the adapter process alerts routed to the critical receiver. The postRunDelay of 1 hour prevents the same alert from being investigated again right after a run finishes.
The Three Stages and Their Gates
AgenticRun ──▶ Analysis ──▶ Proposed ──▶ Execution ──▶ Verification ──▶ Completed
▲ ▲ ▲
gate: Analysis gate: Execution gate: Verification
(Automatic in Ex. 4) (Manual) (Automatic)
The ApprovalPolicy named cluster defines a gate for each stage. A stage starts when the policy says Automatic, or when an AgenticRunApproval contains an approval for that stage. The lab ships with Analysis and Execution set to Manual; Exercise 4 switches Analysis to Automatic so investigations start immediately, while Execution stays Manual. maxConcurrentRuns: 5 limits how many runs can be active at the same time.
Who Runs Each Step
For every step of every run, the agentic operator creates a dedicated service account in openshift-lightspeed, named after the step and the run UID:
| Step | Service account | Permissions |
|---|---|---|
Analysis |
|
Read-only: copies of the |
Execution |
|
Read-only bindings, plus a |
Verification |
|
Read-only, plus permission to create its |
The read-only permissions are not hardcoded. The operator discovers every ClusterRoleBinding whose subject is the lightspeed-agent service account, and creates a copy of each one for the step’s service account. In this lab, the openshift-agentic-lightspeed chart creates two such bindings:
lightspeed-agent-cluster-reader ──▶ ClusterRole cluster-reader
lightspeed-agent-monitoring-view ──▶ ClusterRole cluster-monitoring-view
All these service accounts and bindings are labelled with the run UID and deleted when the step or run ends.
Where Write Permissions Come From
During analysis, the agent proposes one or more remediation options. Each option lists the exact actions it would take, and the RBAC rules it needs to take them — for example patch on deployments named reporting-service in the payments namespace — with a justification for each rule.
When you click Execute remediation on an option:
-
The console writes your decision to the run’s
AgenticRunApproval. A mutating webhook of the agentic operator records who approved and when, inspec.approver, from the authenticated request — the approver cannot be spoofed by the client. -
The operator creates a
RoleandRoleBindingin each target namespace (and aClusterRoleandClusterRoleBindingfor cluster-scoped rules) with exactly the rules of the selected option, bound to the execution service accountls-exe-<run-uid>. -
The execution sandbox pod runs with that service account, performs the actions, and reports an
ExecutionResult. -
The execution RBAC is deleted afterwards. The verification step runs read-only again.
This is why the agentic operator itself is bound to cluster-admin in this lab: Kubernetes only lets a service account grant permissions it holds itself, and the operator must be able to grant whatever an approved option requires.
| Approving an option means approving its RBAC rules. Review both the proposed actions and the permissions they require before clicking Execute remediation. |
Who Can Approve
Approving is an ordinary Kubernetes write: whoever can create or update agenticrunapprovals in openshift-lightspeed can approve or deny runs — from the console, or with oc agentic run approve / oc agentic run deny. In this lab you are a cluster administrator, so you can. In production, grant this permission to the operations leads who own remediation decisions, and keep it separate from the permission to create AgenticRuns.
Inspect It Yourself
-
View the gates of the approval policy:
oc get approvalpolicy cluster -n openshift-lightspeed -o jsonpath='{.spec}' | jq -
View the read-only bindings that every step inherits:
oc get clusterrolebinding lightspeed-agent-cluster-reader lightspeed-agent-monitoring-view \ -o custom-columns=BINDING:.metadata.name,ROLE:.roleRef.name,SUBJECT:.subjects[0].name -
View the narrow permissions of the alerts adapter:
oc auth can-i create agenticruns -n openshift-lightspeed \ --as=system:serviceaccount:openshift-lightspeed:lightspeed-agentic-alerts-adapter oc auth can-i patch deployments -n payments \ --as=system:serviceaccount:openshift-lightspeed:lightspeed-agentic-alerts-adapterThe first command returns
yes, the secondno. -
While a run is in progress, list the per-step service accounts and the bindings created for them:
oc get serviceaccounts -n openshift-lightspeed | grep -E '^ls-' oc get clusterrolebindings | grep -E '^ls-' -
While an execution is in progress, find the Role created from the approved option:
oc get roles -A | grep ls-exec -
See who approved each run:
oc get agenticrunapprovals -n openshift-lightspeed \ -o custom-columns=NAME:.metadata.name,APPROVER:.spec.approver.username,AT:.spec.approver.approvedAt
Key Points to Remember
-
Chat mode inherits your identity. Lightspeed forwards your token through the MCP server, so your RBAC applies, and sensitive tools need your approval in the chat.
-
Autonomous mode uses least privilege per step. Analysis and verification can only read. Execution receives write permissions only for the approved option, only for the duration of that step.
-
Humans gate the risky stage. The
ApprovalPolicydecides which stages need a decision, and the approver’s identity is recorded on every approval. -
Permissions are visible Kubernetes objects. Every grant is a labelled
Role,ClusterRoleor binding that you can inspect and audit with standard tools.