Notifications
Module goals
-
Understanding how notification and alerting work in RHACS
-
Setup a notification alert to a Microsoft teams channel
-
Setup a notification alert to a Slack channel
How do notifications and alerts work in RHACS
Red Hat Advanced Cluster Security for Kubernetes (RHACS) offers robust notification and alerting mechanisms to enhance the security and observability of Kubernetes environments.
Notification and Alerting Mechanisms
Policy Violations
RHACS uses a set of pre-defined and custom policies to monitor the cluster for security violations. These policies can cover a wide range of security aspects including image vulnerabilities, misconfigurations, runtime behaviors, and compliance checks. When a policy is violated, RHACS generates a violation event. These events are the primary triggers for notifications and alerts.
Alert Configurations
RHACS supports integrations with various notification and alerting systems such as Slack, Microsoft Teams, email, PagerDuty, Splunk, and more. For systems not directly supported, custom webhooks can be configured to send alerts to any endpoint that can process HTTP POST requests.
Setting Up Notifications
Administrators can configure notification channels through the RHACS console. This involves providing the necessary credentials and endpoints for the chosen notification service. These notifications can be scoped to specific policies, allowing for granular control over which violations generate alerts and where those alerts are sent.
Alert Management
Policies can be assigned different severity levels (Low, Medium, High, Critical). Alerts can be configured to trigger based on the severity level of the violation. Depending on the integration, alerts can be routed to different teams or systems. For example, critical violations can be sent to on-call engineers via PagerDuty, while lower-severity alerts might be sent to a Slack channel for informational purposes.
Runtime Alerts
RHACS continuously monitors Kubernetes clusters for runtime anomalies. Alerts can be configured to trigger on suspicious activities such as unexpected network connections, process executions, or file system changes.By analyzing the runtime behavior of applications, RHACS can alert on deviations from known good states or behaviors. Configuring Alerting in RHACS
Setup a Microsoft Teams Integration
Let’s setup a basic Microsoft teams integration.
| For this lab I have setup a free personal account. If you do not have access to an account simply follow along and complete the steps that you are able to. Please head to the Microsoft site |
| The following instructions have not been verified. |
-
In the New Teams client, select Teams and navigate to the channel where you want to add an Incoming Webhook.
-
Select More options ••• on the right side of the channel name.
-
Select Manage channel.
-
Screenshot shows the Manage channel option under More options for Teams 2.1
-
Select Edit.
-
Search for Incoming Webhook and select Add.
-
Select Add.
| If you’ve already added an Incoming Webhook, the Configure option appears. Select Configure to create an Incoming Webhook. |
-
Provide a name for the webhook and upload an image if necessary.
-
Select Create.
-
Screenshot shows the name and image fields to be filled to create the webhook.
-
Copy and save the unique webhook URL present in the dialog. The URL maps to the channel and you can use it to send information to Teams.
-
Select Done. The webhook is now available in the Teams channel.
Setup a Slack Integration
Create a Slack Webhook
-
Create a Slack App
-
Log in to Slack: Go to the link: Slack API site and log in with your Slack credentials.
-
Create a New App: Click on "Add Apps"
-
Click on "From scratch" .
You will need to choose a name for the app and select the workspace where it will be used.
| You can add features such as webhooks or bot users later, but for now, focus on setting up the Incoming Webhooks. |
-
Once your app is created, go to the app’s settings and find the "Incoming Webhooks" feature.
-
Toggle Webhooks On: You will see an option to enable webhooks. Toggle it to “On” to allow incoming webhooks for your app.
-
After enabling Incoming Webhooks, scroll down to the "Webhook URLs for Your Workspace" section.
-
Add New Webhook to Workspace: Click the "Add New Webhook to Workspace" button.
A new window will open, allowing you to choose a Slack channel where messages sent by the webhook will appear. Authorize this action by clicking the "Allow" button after selecting the desired channel.
-
Copy the Webhook URL: Once the webhook is authorized, a unique Webhook URL will be generated. You will use this URL in the next step.
Intergrate the Slack webhook into RHACS
-
Access the RHACS Console
-
Go to Platform Configurations → Integrations
-
Select Notifier Integrations → Slack
-
Click the New integration button to set up a new notification system.
A new window will appear where you can choose the type of integration. Select Slack from the list.
| You will need to configure the Slack integration by providing the following information: |
Integration Name: Give your integration a name, like "Slack Webhook for Security Alerts." Webhook URL: Paste the Slack Webhook URL you generated earlier. Default Channel: This is typically auto-populated based on the webhook URL, but you can verify or specify the correct Slack channel. *Message Format: *Choose the format for your messages. This could be simple text, or you could customize it further using Block Kit if supported.
-
Hit Test and hopefully recieve a "The test was successful" message.
Next we have to configure our policies to alert based on specific triggers.
Configure Notifications
Balancing security alerting in a team is crucial to ensure that the team is responsive to genuine threats while minimizing alert fatigue. RHACS enables this by letting you configure your alert channels and attach them to the policies you car about.
Let’s test this out. You will use the runtime enforcement policy that was used in the previous module.
-
On the left-hand side of the application, click the Platform Configuration tab and select Policy Management.
-
Filter through the policies to find Ubuntu Package Manager Execution or use the search bar to select Policy.
-
Once you have found the policy Ubuntu Package Manager Execution, click on the runtime enforcement policy you created earlier
There should now be a notifier option on the right hand side of the screen.
-
Click your new webhook
-
Save the policy
Next we are going to run the same runtime apt command as we did earlier.
-
Run the following command in the terminal
POD=$(oc get pod -l app=ctf-web-to-system -o jsonpath="{.items[0].metadata.name}")
oc exec $POD -i --tty -- /bin/bash[demo-user@bastion ~]$ POD=$(oc get pod -l app=ctf-web-to-system -o jsonpath="{.items[0].metadata.name}")
oc exec $POD -i --tty -- /bin/bash
node@ctf-web-to-system-6db858448f-hz6j2:/app$| If you see node@ctf… you’ve confirmed you have a shell and access to the Java application. |
-
Run the Ubuntu package manager in this shell:
apt update-
Examine the output and expect to see that the package manager attempts to perform an update operation:
Sample output
node@ctf-web-to-system-6db858448f-stwhq:/$ apt update
Reading package lists... Done
E: List directory /var/lib/apt/lists/partial is missing. - Acquire (13: Permission denied)
node@ctf-web-to-system-6db858448f-stwhq:/$ command terminated with exit code 137-
Lastly, review your Slack or Teams channel for the alert.
Congrats!
A task to complete on your own.
You know the drill
Add the notifier to the Runtime Policy 'Netcat Execution Detected'. Enable enforcement of the policy and watch the "visa-processor" container for a violation. This container runs an nc command every minute or so.
Your mission is completed with a Notification to Slack or Teams.
