Section 1 - Log Enrichment and Prompt Generation Workflow

Objective

In this section, we will build the first workflow called the AI Insights and Lightspeed prompt generation workflow. This workflow will be triggered by Event-Driven Ansible (EDA) whenever an issue is detected in the system. Specifically, this demo is set up to respond to an Apache (httpd) service failure.

This workflow will check the status of the Apache service, notify a Mattermost channel with details, ask Red Hat AI to analyze the incident, and finally create a new job template using the AI-generated prompt for Ansible Lightspeed.

Enrichment and Prompt Generation Workflow

By organizing these steps into a workflow, we simulate how multiple tools can collaborate to enrich a ticket or alert and prepare the remediation plan—all before a human even reviews the problem.

⚠️ This workflow will be automatically triggered by EDA once the ❌ Break Apache job template is run. You must create this workflow before simulating the failure.

Introducing AI with Ansible

Red Hat AI is already deployed in this lab and powered by the Granite LLM model. You can interact with this AI using simple curl commands or trigger it through Job Templates via Ansible Automation Platform.

Red Hat AI is deployed using two Ansible Content Collections:

  • redhat.ai (Certified)

  • infra.ai (Validated)

These are available as part of your AAP subscription, giving your organization a quick and validated path to deploy generative AI models using automation.

Try It Yourself: Talk to the AI!

Before diving into the workflow, let’s see how easy it is to test this out yourself.

  1. Open the Terminal in the lab.

  2. Run the following curl command:

curl -X POST http://rhelai.example.com:8000/v1/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer {ssh_password}" \
  -d '{
    "model": "/root/.cache/instructlab/models/granite-8b-lab-v1",
    "prompt": "What is the capital of USA?",
    "max_tokens": 100,
    "temperature": 0
  }' | jq -r '.choices[0].text'

You can also try real-world prompts like:

curl -X POST http://rhelai.example.com:8000/v1/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer {ssh_password}" \
  -d '{
    "model": "/root/.cache/instructlab/models/granite-8b-lab-v1",
    "prompt": "systemd[1]: httpd.service: Failed with result exit-code. Suggest a fix for Apache.",
    "max_tokens": 150,
    "temperature": 0.6
  }' | jq -r .choices[0].text

This returns a JSON response with helpful AI suggestions. This is exactly what’s being automated in the next step via the 🤖 RHEL AI: Analyze Incident job template.

Exploring Mattermost Integration

In this lab environment, we’ve included Mattermost, a lightweight, open-source chat client that serves as a central place to view error logs and AI insights as part of your AIOps workflow.

Think of Mattermost as a stand-in for ServiceNow in this demonstration.

💡 In a real-world production environment, you can easily replace Mattermost with ServiceNow (or any other ITSM tool). This enables a more advanced automated workflow, where tickets are automatically created and enriched with logs and insights — all before a human even picks up the ticket!

Accessing Mattermost

Component Value

Mattermost URL

Mattermost

Username

ansibleadmin@ansible.com

Password

ansible123!

  1. Open mattermost using the above details

  2. Click on “View in Browser” to open the console.

  3. Login using the credentials provided.

  4. Click on the team "Automators".

Once logged in, click on the Town Square channel from the left-hand navigation.

Click on Town Square

What You’ll See in Town Square

In the Town Square channel, you will eventually see:

  • 🔍 Error logs from the Apache HTTPD service (collected by Filebeat and forwarded via Kafka)

  • 🧠 AI Insights generated from the logs using RHEL AI

This real-time feed mimics how production environments might use automated ticket enrichment — by capturing logs and insights and sending them directly to a ticketing system like ServiceNow.

You won’t see any messages just yet — this step is simply to explain where Mattermost fits in and why this channel matters. Logs and insights will start appearing once the first workflow runs.
💡 With this integration, you are essentially watching AI and automation work together — detecting the issue, diagnosing it, and preparing remediation guidance before anyone intervenes.

Build the Workflow

System URL Credentials

Ansible Automation Platform

AAP is preloaded in the lab interface. Click link if you want to open it in full tab: AAP Web UI

Username: lab-user Password: {ssh_password}

  1. Login to Ansible Automation Platform.

  2. Go to Automation Execution → Templates.

automation_execution_templates

  1. Click Create template → Create workflow job template.

create_workflow

  1. Fill in the details:

Parameter Value

Name

AI Insights and Lightspeed prompt generation

Organization

Default

  1. Click Create workflow job template.

create_workflow_job_template

  1. You’ll see the empty workflow visualizer.

currently_no_nodes_workflow

  1. Add the Apache Service Status Check node:

Parameter Value

Node type

Job Template

Job Template

⚙️ Apache Service Status Check

Convergence

Any

Node alias

(You can leave this blank)
Add Apache Status Check Step

  1. Click Next, then Finish.

blue_next_button
blue_finish_button

  1. Visual after first node:

Workflow after Apache node

  1. Add RHEL AI: Analyze Incident step:

  2. Click on the three dots (⋮ kebab menu) next to the ⚙️ Apache Service Status Check node.

  3. Click on ⊕ Add step and link to insert the next node into the workflow.

workflow_add_step_and_link
Parameter Value

Node type

Job Template

Job Template

🤖 RHEL AI: Analyze Incident

Status

Run on success

Convergence

Any

Node alias

(You can leave this blank)
Add RHEL AI Step

  1. Click Next, then Finish.

blue next button
blue finish button

  1. Workflow with two nodes:

After RHEL AI step

  1. Add Notify via Mattermost:

  2. Click on the three dots (⋮ kebab menu) next to the 🤖 RHEL AI: Analyze Incident node.

  3. Click on ⊕ Add step and link to insert the next node into the workflow.

workflow_add_step_and_link
Parameter Value

Node type

Job Template

Job Template

📣 Notify via Mattermost

Status

Run on success

Convergence

Any

Node alias

(You can leave this blank)
Add Mattermost Step

  1. Click Next, then Finish.

After Mattermost step

  1. Add Build Ansible Lightspeed Job Template:

  2. Click on the three dots (⋮ kebab menu) next to the 📣 Notify via Mattermost node.

  3. Click on ⊕ Add step and link to insert the next node into the workflow.

workflow_add_step_and_link
Parameter Value

Node type

Job Template

Job Template

⚙️ Build Ansible Lightspeed Job Template

Status

Run on success

Convergence

Any

Node alias

(You can leave this blank)
Add Lightspeed JT Creator Step

  1. Click Next, then Finish.

blue next button
blue finish button

  1. Final workflow visual:

Final Workflow

  1. Click Save to finalize.

save button

Trigger the Workflow

  1. Run the ❌ Break Apache job template. This inserts an invalid directive in Apache config and restarts the service.

run break apache

  1. Go to Automation Decisions(Event-Driven Ansible) → Rulebook Activations. Confirm EDA(Event-Driven Ansible) picked up the event.

eda trigger capture

  1. Go to Automation Controller → Jobs. Confirm workflow execution. When the workflow completes you will see a green ✅ Success.

workflow triggered jobs

  1. Go to Templates and you should be able to see a new job template called "🧠 Lightspeed Remediation Playbook Generator" generated. This step also creates a blank Workflow Job Template called "Remediation Workflow" that will be used in next challenge.

Final Checks & Observations

Before you move on to the remediation phase, take a moment to explore the Mattermost Town Square channel.

Here’s what to look for:

  • 🛑 HTTPD Error Logs: These logs were automatically collected from the webserver.

  • 🧠 AI Insights (RCA): Red Hat AI parsed the logs and generated a root cause analysis. These insights help you understand exactly why the failure occurred.

This is what real-world AIOps looks like! Imagine this in a production setting:

💡 A ticket appears in your ITSM tool with: - Attached logs - Root Cause Analysis (RCA) - A prompt ready for Lightspeed to generate the fix

And all of this before anyone has even opened the ticket.

We used Mattermost in this lab because it’s a lightweight, open-source chat platform that’s easy to run per student. But the same flow would work with ServiceNow, Jira, or any ITSM of your choice.

💡 Think of this as automated ticket enrichment. Logs and RCA can be pre-attached to incidents or alerts, helping teams respond faster and more effectively.

Mattermost logs and RCA message preview

🎯 In short: You now have an intelligent incident response pipeline that goes from failure detection to context to fix suggestion — powered by Red Hat AI and Ansible Automation Platform.

Summary

You created a workflow that:

  • Uses logs for root cause analysis via Red Hat AI

  • Notifies Mattermost

  • Prepares a Lightspeed prompt for automated playbook generation

In the next step, we’ll use that prompt to fix Apache automatically!

Complete

You have completed this module. Move forward to the next one to use the created templates.