Section 2 - Remediation Workflow

Objective

For this section, we will build another workflow called the Remediation Workflow. The goal for this workflow is to take the information we learned from Section 1 with Red Hat AI and now prompt Ansible Lightspeed to automatically create an Ansible Playbook to fix the issue. By combining these two workflows, we can create an AIOps workflow capable of self-healing infrastructure.

remediation_workflow

By separating these two parts into distinct workflows, we allow for manual review of the response from Red Hat AI and can take corrective action if necessary. This creates a natural stopping point in our automation workflow where we can ensure the automation is performing as expected.

⚠️ Before we get started you need to make sure you have run the first workflow.
⚠️ Reminder that the way we purposely broke httpd with the ❌ Break Apache Job Template was by inserting InvalidDirectiveHere into the configuration file and restarting the service. This crashed the httpd service as you can see by running the systemctl status httpd.service Linux command.

Verify webserver failure

  1. Connect to your bastion host either via SSH or Visual Studio Code. The connection information is provided by your instructor.

  2. SSH over to node1

[lab-user@bastion ~]$ ssh node1
Last login: Tue Apr  8 15:56:41 2025 from 192.168.0.37
[ec2-user@node1 ~]$

  1. Verify httpd is down with the systemctl status httpd.service command

[ec2-user@node1 ~]$ sudo systemctl status httpd.service
× httpd.service - The Apache HTTP Server
     Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; preset: disabled)
     Active: failed (Result: exit-code) since Tue 2025-04-08 15:56:15 UTC; 1h 11min ago
   Duration: 1min 9.506s
       Docs: man:httpd.service(8)
    Process: 4099 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exited, status=1/FAILURE)
   Main PID: 4099 (code=exited, status=1/FAILURE)
     Status: "Reading configuration..."
        CPU: 58ms

Apr 08 15:56:15 node1.example.com systemd[1]: Starting The Apache HTTP Server...
Apr 08 15:56:15 node1.example.com httpd[4099]: AH00526: Syntax error on line 35 of /etc/httpd/conf/httpd.conf:
Apr 08 15:56:15 node1.example.com httpd[4099]: Invalid command 'InvalidDirectiveHere', perhaps misspelled or defined by a module not included in the server configuration
Apr 08 15:56:15 node1.example.com systemd[1]: httpd.service: Main process exited, code=exited, status=1/FAILURE
Apr 08 15:56:15 node1.example.com systemd[1]: httpd.service: Failed with result 'exit-code'.
Apr 08 15:56:15 node1.example.com systemd[1]: Failed to start The Apache HTTP Server.
[ec2-user@node1 ~]$

As you can see in the above output, the httpd.service has failed.

Create Remediation Workflow

  1. Log in to the web UI for Ansible Automation Platform if you are not already logged in.

  2. In the left navigation menu, click on Automation Execution → Templates.

automation_execution_templates

  1. Scroll down and select the Remediation Workflow Workflow job template

create_workflow
💡 The last workflow (specifically the job template Build Ansible Lightspeed Job Template) used the ansible.controller.workflow_job_template module to automatically create this blank workflow. We could have Ansible automatically create the entire workflow but we want to illustrate each part and what it is doing. The most important part is that Ansible automatically created a survey with output from Red Hat AI. You will see this called out below when we run the workflow. Ask your instructor if you need clarification.

  1. Click on the blue View workflow visualizer button

visualizer_button

  1. Verify you have a window that looks like this:

currently_no_nodes_workflow

  1. Click on the blue Add step

  2. Fill out the following values

Parameter Value

Node type

Job Template

Job Template

🧠 Lightspeed Remediation Playbook Generator

Convergence

Any

Node alias

(You can leave this blank)
add_step_lightspeed_remediation

  1. Click on the blue Next.

blue_next_button

  1. Review and click the blue Finish button.

blue_finish_button

Your workflow will now look like this:

remediation_workflow_part_one

  1. Click on the three dots (kebab menu) next to the 🧠 Lightspeed Remediation Playbook Generator node

  2. Click on ⊕ Add step and link

workflow_add_step_and_link

  1. Fill out the following values

Parameter Value

Node type

Job Template

Job Template

🧾 Commit Fix to Gitea

Status

Run on success

Convergence

Any

Node alias

(You can leave this blank)
remediation_workflow_part_two

  1. Click on the blue Next.

blue_next_button

  1. Review and click the blue Finish button

blue_finish_button

Your workflow will now look like this:

remediation_workflow_part_two_final

  1. Click on the three dots (kebab menu) next to the 🧾 Commit Fix to Gitea node

16 Click on ⊕ Add step and link

workflow_add_step_and_link

  1. Fill out the following values

Parameter Value

Node type

Project Sync

Job Template

Lightspeed-Playbooks

Status

Run on success

Convergence

Any

Node alias

(You can leave this blank)
remediation_workflow_part_three
⚠️ Notice this node type is different. This is a built-in node type that will force a Project Sync to the repository that will hold our auto-generated Ansible Playbooks. This is effectively a git pull The previous node: 🧾 Commit Fix to Gitea synced the Ansible Playbook from Ansible Lightspeed to Gitea, now we are retrieving that playbook from Gitea to use in Ansible Automation Platform.

  1. Click on the blue Next.

blue_next_button

  1. Review and click the blue Finish button.

blue_finish_button

Your workflow will now look like this:

remediation_workflow_part_three_final

  1. Click on the three dots (kebab menu) next to the 🧠 Lightspeed-Playbooks node

  2. Click on ⊕ Add step and link

workflow_add_step_and_link

  1. Fill out the following values

Parameter Value

Node type

Job Template

Job Template

⚙️ Build HTTPD Remediation Template

Status

Run on success

Convergence

Any

Node alias

(You can leave this blank)
remediation_workflow_part_four

  1. Click on the blue Next.

blue_next_button

  1. Review and click the blue Finish button

blue_finish_button

Your workflow will now look like this:

remediation_workflow_part_four_final

You are now done creating the workflow!

  1. Click on the blue Save button at the top.

save_button

  1. Exit out of the Workflow Visualizer by clicking the x at the top right

  2. Click the 🚀 Launch template button to execute the Remediation Workflow

launch_template_button
💡 This is the really cool part! There are two text boxes here. The left one is the value we use to prompt Ansible Lightspeed to generate an Ansible Playbook. This value was manually created by the Ansible Tech Marketing team. The 2nd text box was dynamically created from the last workflow by Red Hat AI. If you are happy with the Red Hat AI prompt, feel free to copy and paste it over the first text box to test out the solution. By separating AIOps into two workflows, it gives us (the administrator) the ability to review AI responses for accuracy before we deploy the next workflow!
workflow_prompt

  1. Review the prompt AI generated (right side) and correct the left side with the red required asterick *. Feel free to copy and paste from the right side to the left side.

This is the step that allows us to course correct any output that AI has provided us. We can review what it showed us before we prompt Ansible Lightspeed to make sure the prompt is legitimate.

  1. Click on the blue Next.

blue_next_button

  1. Review and click the blue Finish button.

blue_finish_button

  1. Wait for the workflow to complete

When the workflow completes you will see a green ✅ Success

remediation_workflow_success
🎉 You have used Ansible Automation Platform in an AIOps workflow to automatically create an Ansible Job Template to fix your issue. This has created an Ansible Job Template called 🔧✅ Execute HTTPD Remediation. You could also add this to the workflow, but it might be another natural breakpoint where you schedule this job template during an approved change window. For example, some organizations might only want to apply changes to production servers on Sundays at 3AM.

Review the AI-Generated Playbook in Gitea

Before we proceed with executing the final job, let’s take a step back and review what was automatically created for us.

Red Hat AI was able to:

  • Analyze the Apache failure logs collected by Filebeat and forwarded to RHEL AI

  • Generate a root cause explanation for the issue (RCA)

  • Create a concise and relevant prompt based on the RCA

  • Use that prompt to call Ansible Lightspeed

  • Ansible Lightspeed with IBM watsonx Code Assistant, generated an Ansible Playbook to fix the issue

  • The playbook was automatically pushed to Gitea in the lightspeed-playbooks repository

This showcases a complete AI-driven automation loop — from incident detection to automated remediation content generation — all before a human steps in. Gitea here acts as the version control source of truth, where you can review the generated fix.

How to Review the Playbook

Gitea Access Details

Use the following information to log in to Gitea and review the generated playbook:

Component Value

Gitea URL

Gitea Web UI

Username

lab-user

Password

{ssh_password}

  1. Login using the credentials provided.

  2. Navigate to the lightspeed-playbooks repository.

  3. Review the most recent commits — you should see one that corresponds to the Lightspeed-generated fix.

💡 You can open the playbook file and inspect the contents. This is your opportunity to validate the AI-generated output before executing it. Just like in production systems, automated changes should be reviewed—either manually or through automated policy enforcement tools.
Pro Tip: If you’re planning to use this in a real-world setup, consider inserting an approval step here or sending the commit as a Pull Request to a staging branch. This way, your remediation workflow includes guardrails for human validation before running automated fixes.

Fix webserver issue

  1. In the left navigation menu, click on Automation Execution → Templates.

automation_execution_templates

  1. Scroll down to the 🔧✅ Execute HTTPD Remediation job template and click the 🚀 to launch the job.

  2. Type in node1 for the Limit

limit_node1
💡 A limit in Ansible Automation Platform is equivalent to --limit with the ansible-playbook command. The limit specifies the host or hosts you want to run the Ansible Job on versus the entire inventory, effectively limiting it to only running on those host(s).

  1. Click on the blue Next.

blue_next_button

5 Review and click the blue Finish button.

blue_finish_button

Check Success

  1. Connect to your bastion host either via SSH or Visual Studio Code. The connection information is provided by your instructor.

  2. SSH over to node1

[lab-user@bastion ~]$ ssh node1
Last login: Tue Apr  8 15:56:41 2025 from 192.168.0.37
[ec2-user@node1 ~]$

  1. Verify httpd is fixed with the systemctl status httpd.service command

[ec2-user@node1 ~]$ sudo systemctl status httpd.service
● httpd.service - The Apache HTTP Server
     Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; preset: disabled)
     Active: active (running) since Tue 2025-04-08 17:16:02 UTC; 1min 55s ago
       Docs: man:httpd.service(8)
   Main PID: 6024 (httpd)
     Status: "Total requests: 0; Idle/Busy workers 100/0;Requests/sec: 0; Bytes served/sec:   0 B/sec"
      Tasks: 177 (limit: 48028)
     Memory: 23.5M
        CPU: 193ms
     CGroup: /system.slice/httpd.service
             ├─6024 /usr/sbin/httpd -DFOREGROUND
             ├─6025 /usr/sbin/httpd -DFOREGROUND
             ├─6026 /usr/sbin/httpd -DFOREGROUND
             ├─6027 /usr/sbin/httpd -DFOREGROUND
             └─6028 /usr/sbin/httpd -DFOREGROUND

Apr 08 17:16:02 node1.example.com systemd[1]: Starting The Apache HTTP Server...
Apr 08 17:16:02 node1.example.com httpd[6024]: Server configured, listening on: port 80
Apr 08 17:16:02 node1.example.com systemd[1]: Started The Apache HTTP Server.
[ec2-user@node1 ~]$

If you see (running) above it means httpd has been fixed! You have successfully completed the lab.

Takeaways

You have

  • Created a Workflow template that automatically creates an Ansible Playbook, pushes the Ansible Playbook to Gitea and then automatically creates a Job Template to fix the issue.

  • Fixed the httpd service using the prompt generated by Red Hat AI

  • Completed an AIOps workflow, by utilizing two different AI endpoints

  • Used the limit feature in a Job Template

aiops-cool

Complete

You have completed module 01.