Module 3: Full Automation — AI-Assisted Remediation

Before diving into the hands-on steps, this section compares what changes between the quick win approach and the full automation approach, introduces the orchestration pipeline and the two applications you will remediate — one low-risk that merges automatically, one higher-risk that waits for your approval.

The back half of both paths is the same. Full automation only replaces who decides what to change and whether it may ship.

Event-Driven Ansible is deliberately thin here. It is a sensor: it recognises the .rhlw artifact event and triggers a job template, which posts to an Ansible Orchestrator webhook. Everything after that is a node on an Orchestrator canvas, recorded as it runs.

Your project contains three workflows, not one, and the split is the first thing worth understanding:

They are separate because the Orchestrator cannot pause a run and resume it on an outside event. Rather than hold a job open for the hours a human review might take, each stage is its own short execution, woken by a webhook. The Snow incident is what ties them back into one story.

All the preparation — opening the parent incident, asking TPA which applications contain the component, classifying each against its policy, raising a child incident per application — is a single Prepare Remediation node. It is one Ansible job running several plays, so the canvas shows you the decision points rather than the plumbing.

The merge — by the Orchestrator or by you — is where the full automation path rejoins the quick win path:

The Merged workflow does not get a webhook when the build finishes — Tekton does not send one. Instead it polls the GitLab commit status that Pipelines-as-Code already mirrors from the PipelineRun. That is why one Merged execution can run for several minutes: it is watching your build.

Each repository carries a .lightwell/policy.json file. It is the primary control that decides whether a human is interrupted — but it is not the only one. If any step in the merge request pipeline fails (unit tests, regression scan, licence compliance), the merge is blocked regardless of policy, and the change requires human review:

The policy travels with the code, in the repository, reviewed like any other change. There is no approval matrix in a wiki that nobody has opened since the last audit.

The commands in this module use oc (the OpenShift CLI) to interact with your cluster. If you already logged in during Module 2 and the terminal still shows your user name for oc whoami, skip this step. Otherwise log in as your tenant user by running the following command in the terminal on the right:

You should see Login successful and your user name displayed. This gives you access to your dedicated namespaces on the cluster.

Start with the easy one. Playing the Lightwell release engineer role once more, you will publish a remediated build of a dependency that benefits-mgmt-app declares directly in its pom.xml.

Run the following command in the terminal on the right:

This is the same script you used in Module 2. It publishes the remediated JAR to the lightwell-remediated repository in your artifact manager and pushes the matching OSV advisories to TPA.

Switch to the Ansible Orchestrator tab on the right (or open it in a new tab) and log in with {user} / {password}. Your project is lightwell-%GUID%, and it holds the three workflows listed earlier.

Click Lightwell Remediation — Start, then open the Runs view to see the executions this remediation produced. There are three, one per workflow, and they completed at different times — Start first, then Merged, then Deploy as each webhook arrived.

Within a few seconds of the artifact landing you should see a new execution. Watch the nodes light up in order:

Open the Start run. The canvas shows the shape of the flow, and the Run details table underneath names each node as it completes:

The candidate list comes from TPA and nowhere else. If TPA has no SBOM containing the component, the loop has nothing to iterate over — the Orchestrator will not invent an application to remediate.

Look at the order again. Opening the parent incident is the first thing Prepare Remediation does — before TPA is queried, before anything is classified, before a single line of anyone’s pom.xml is read. The platform raises the ticket and then does the work, which is the opposite of how most automation behaves.

Switch to the Snow tab on the right (or open it in a new tab) — no login required. Your incident is at the top of the list, with the most recent state.

What you are looking at is a small hierarchy, not one ticket:

The children are where the detail goes. Each application’s loop iteration comments on its own child as it completes — merge request opened, verifier passed, awaiting merge — so two applications remediated from one event do not interleave their history into a single unreadable timeline. The parent gets short milestone notes with links down into the children, and nothing else. An operator reads the parent to find out where things are, and opens a child to find out what happened.

The comments are not just prose, either. Open one and the merge request, the pipeline run and (later) the ArgoCD application and the running route are all clickable, so the ticket is a working console rather than a transcript.

Your two incidents are at the top of the list, both raised by Lightwell AO (Ansible Orchestrator). Open the parent — the one whose description starts with remediate:

Now follow the Child incidents link. The child is where the per-application detail lives — the policy that applied, the merge request, and what the agent reported:

This is the part that tends to matter most to the people who sign off on automation. The audit trail is not reconstructed from git history afterwards — it is written as the work happens, by the thing doing the work. Nothing in this flow is retrofitted to look compliant.

Leave the tab open. You will come back to it at the end, and the shape of what has accumulated there is the point.

In the loop, the Orchestrator tasks the impact-analyzer agent with creating the fix. The agent reads the application’s pom.xml, bumps the dependency to the remediated .rhlw version, pushes a branch, and opens a merge request.

Switch to the GitLab tab on the right (or open it in a new tab) and go to acme-%GUID%/benefits-mgmt-app → Merge requests. If the list is empty, the Orchestrator has already auto-merged it — click the Merged tab to find it. The title follows a fixed shape:

Yours will name the dependency you just published, with its own .rhlw number.

The coloured dot is the application’s Acme Inc. risk class, not the severity of the CVE. That distinction is easy to miss and worth pausing on: CVE-2022-42889 is a critical-severity flaw, yet this merge request is green, because benefits-mgmt-app is a low-risk application and the fix is therefore allowed to merge itself. The wire-transfer-svc merge request later in this module carries a 🔴 for the opposite reason. A reviewer scanning twenty merge requests is triaging by what the change can break, not by how frightening the CVE sounds.

The diff itself is almost boring — the same single-line pom.xml change Renovate made in Module 2. The interesting part is the description, which is the artifact a human is actually expected to read:

Scroll to Automated Validation. When the agent first opens the merge request, every row says Pending — the agent proposed the change but did not test it. If the pipeline has already started or finished by the time you look, some or all rows will show real results instead (for example, 4/4 tests passed). Either way, the point is the same: the table is filled by the pipeline, not by the agent:

The merge request is created before any testing has happened, so every row in the Automated Validation table starts as Pending. A Pipelines-as-Code pipeline, build-pull-request, runs against the open merge request, executes mvn verify together with the CycloneDX SBOM plugin, and then rewrites that one section of the description with real results.

Back on the Orchestrator canvas, the MR Verifier node confirms the merge request is healthy and every check reported, then the Auto-approve? condition reads the approval value that travelled with the application’s own policy.

For benefits-mgmt-app that value is auto_approve, so the canvas takes the green branch to Merge MR (auto-approved). The Orchestrator marks the merge request ready and merges it. No human is asked, and no human is notified. The child incident gets a comment recording the merge, and the loop moves on to the next application.

Switch back to the GitLab tab and open the merge request again. The block you watched earlier has cleared and the merge has already happened — performed by the automation account, with you never touching it:

Read those three panels top to bottom, because they are in causal order:

Check the timestamps while you are there. The merge happened roughly two minutes after the request was opened, not seconds — the Orchestrator was waiting for build-pull-request to report before it merged. It could not have jumped that queue even if it had tried.

This is the volume case, and it is the whole economic argument for the platform. An internal HR application, a compatible rebuild of a library, every automated check green: there is no decision here that a person adds value to, so no person is asked. Multiply that by forty remediations a week and the time saved is not the merge click — it is the context switch that surrounds it.

Merging to main starts the build pipeline. Switch to the OpenShift Console tab on the right (or open it in a new tab), navigate to Pipelines → Pipelines, select the PipelineRuns tab, and make sure the build-%GUID% namespace is selected. You should see the run for benefits-mgmt-app:

The task graph has one more box than the list above: run-tests and maven-build are separate tasks, and there is no sign task at all. Signing happens out of band — Tekton Chains observes the completed run and signs the image afterwards, which is why the signature shows up as the small shield badge next to the PipelineRun name rather than as a step you can watch.

That last step is what closes the loop. The pipeline does not deploy anything — it makes a git commit in the deployment repository, and ArgoCD does the rest. Switch to the ArgoCD tab on the right (or open it in a new tab). Choose Log in via OpenShift. The first time you do this, OpenShift asks you to approve the connection:

Find the acme-apps-%GUID% application and look at the two panels at the top:

Sync Status reads Synced and Last Sync reads Sync OK, but the useful part is the commit quoted under both: promote benefits-mgmt-app to 5598d3ade…, authored minutes ago. That is the pipeline’s promote-to-deployment-repo task writing the new image digest into the deployment repository, and the sync beside it is ArgoCD noticing and rolling the deployment in your {user}-app namespace.

Nothing here was triggered by a person, and the pipeline deployed nothing. It changed a file in git; ArgoCD reconciled the cluster to match. The answer to "what is running, and why" is a commit.

Now go back to the Topology view from earlier and refresh it. The deployment is still called benefits-mgmt-app-prod and the ring is still complete — but click it and compare the pod name with the one you noted earlier. It is different. The old pod is gone and a new one is serving traffic, built from the remediated dependency, and nobody connected to the cluster to make that happen.

The application keeps serving traffic throughout. That is not a staging trick: the remediated dependency is the real upstream library with the fix applied, so the application genuinely runs on it.

Publish a remediation for a dependency that wire-transfer-svc declares — the payment-critical, PCI-scoped service. Run the following command in the terminal on the right:

Everything you just watched happens again, identically: incident opened, TPA queried, application classified, agent dispatched, merge request opened, pipeline runs, validation table filled in. Follow it on the canvas if you like — it is the same shape.

Then the run stops.

On the canvas, the Auto-approve? condition does not see auto_approve, so it takes the other branch: a comment on the child incident recording that the merge request is ready and awaiting a GitLab merge. Then the execution ends.

Read that again, because it is easy to misread as a failure. The Orchestrator has not crashed, timed out, or parked a job that is burning a worker somewhere. It has done everything it is permitted to do and stopped. The merge request is open in GitLab with your name on it, and nothing further happens in this entire platform until a person acts.

Open the Lightwell Remediation — Start run for this trigger and zoom in on the right-hand side of the canvas. The False branch of Auto-approve? carries the green tick, and the path onward to Merge GitLab MR (auto_approve) is dashed, meaning it was never taken:

Compare that with the low-risk run earlier, where the same condition ticked True and ran straight into Merge GitLab MR. The green tick marks the branch that was taken — it is a record of a decision, not an error.

Check Snow first. The child incident for wire-transfer-svc records that the merge request is ready and hands the decision to you — please review and merge in GitLab when you are satisfied; CI/CD will continue from there:

Now go and read the merge request in acme-%GUID%/wire-transfer-svc. Everything you need is in one page:

Read that last point again, because it is the whole argument. Every automated check passed. The platform still refused to merge it. The policy says that on a system that moves customer money a person signs off even on a change that looks perfect, and the platform is not allowed to talk itself out of that — not even with a full table of green ticks to point at.

Note the Recommended Action block at the bottom. For benefits-mgmt-app it said the change was safe to take. Here it reads:

Work through the page in order:

"Regardless of automated results" is the policy in one phrase. Every check passed. None of that changes the answer.

That click is the only manual step in the entire module, and it is the one the whole design exists to protect. Everything up to it was preparation so that your decision is informed; everything after it is automation again.

From here it is the same spine as the low-risk application: merging fires the GitLab webhook, Event-Driven Ansible relays it, and a short Orchestrator run records the merge on the child incident. build-push-to-main builds, generates the SBOM, signs via Tekton Chains, pushes to the OCI registry and promotes the imageTag into acme-deployment; ArgoCD syncs and the new pod rolls out in {user}-app. Each stage reports back as it completes.

Switch to the TPA tab on the right (or open it in a new tab). Sign in through SSO with {user} / {password} if needed. The dashboard tells you how much it is holding for this tenant, and when the last SBOM arrived — which should be the build you just watched:

In the left-hand navigation, expand SBOMs and choose All SBOMs:

Every build each application has ever produced is listed, with its dependency and vulnerability counts. On a fresh tenant that is a short list, and both of the applications you just remediated appear twice — once as shipped, once as rebuilt.

You do not need to open anything to see the result — the list itself is the evidence. For each application the original build and the remediated one sit next to each other, so the before and after are on one screen:

v1.0-build is the application as it shipped; chore(deps)… is the build produced after remediation. Same application, one dependency different:

The two tell different stories, and the difference is worth understanding.

For benefits-mgmt-app the headline number moves: two vulnerabilities fewer, one of them critical, and one dependency fewer. The vulnerable commons-text is gone and the .rhlw build stands in its place.

For wire-transfer-svc the total does not move at all. It stays at 90, with one critical becoming one high. That is not a failed remediation — it is an honest picture of a payment system carrying 238 dependencies and a long tail of known issues that this one fix was never going to touch. The build changed; the backlog did not.

Notice what none of this required: nobody wrote a report and nobody updated a spreadsheet.

The verification is not a separate exercise bolted on at the end. The pipeline uploaded the SBOM as part of the build, which means the evidence for the next remediation is already in place. That is what makes the loop a loop.

Switch back to the Snow tab on the right (or open it in a new tab).

The parent incident is now Resolved. Read its thread from the top — it is deliberately short, a sequence of milestones with links down into the children:

Now open a child. That is where the work actually is: the merge request link, the CVE, the verifier outcome, the pipeline run, the ArgoCD application and the route the fixed code is now serving on. The parent tells an operator where things stand; the child tells them what happened.

Notice what resolved it. Not the merge, and not a green pipeline — the deploy. SNOW finalize only runs in the Deploy workflow, so the ticket stays open until the remediated code is actually running. A build that passes and never ships does not close anything.

That is also why three workflows rather than one long-running job. The Start run ends when the merge requests are ready; merge and deploy arrive later as webhooks and start their own short executions. The incident is what stitches them back into a single story — which is precisely the job a ticket is supposed to do.

You have just run the Full Automation path end to end: