Policy Management

Module goals

Create build, deploy and runtime policies
Create policies from filters
Clone and edit policies based on the ACS defaults

RHACS Policy Management Basics

RHACS has many built-in policies to detect activity related to attacker goals: gain a foothold, maintain a presence, move laterally, and exfiltrate data. The continuous runtime monitoring observes all container activity and will automatically respond to events with appropriate enforcement and notification. However, that would be missing out on an opportunity - RHACS wants to go one step further, to take advantage of containers' ephemeral, immutable nature, to improve security in a measurable way.

We want to use runtime incidents and vulnerabilities as a learning opportunity to improve security going forward by constraining how our containers can act. We achieve this by creating policies and implementing them early in the CI/CD process.

As we move into this next section, let’s focus on identifying and enforcing a runtime policy in the cluster. For the upcoming example, we will focus on stopping the Ubuntu package manager from being run on pods in our cluster. Our example container ctf-web-to-system has this package manager in the container. Let’s ensure that it never calls for updates while present in our clusters.

RHACS Runtime Policy Basics

RHACS observes container processes and collects this information to enable you to craft policies to prevent behavior that you don’t like. This information can also create baseline policy configurations that the user can update. Runtime policies can include all build-time and deploy-time policy criteria but they can also include data about process executions during runtime.

The example below demonstrates how security may want to block a package manager from downloading any packages to the container. This runtime enforcement option is the first in the process of shifting left. After runtime enforcement, you will want to stop the package manager from being used in the container altogether.

For the Runtime stage, RHACS stops all pods that match the conditions of the policy. It does not scale the deployment down to zero.

Procedure

On the left-hand side of the application, click the Platform Configuration tab and select Policy Management.

Filter through the policies to find Ubuntu Package Manager Execution or use the search bar to select Policy.

Once you have found the policy Ubuntu Package Manager Execution, click on it to learn more.

If you click the actions button, you will see how easy it is to edit, clone, export or disable these policies. We also recommended cloning the policies and adding or removing specific filters as you need them.

Prevent execution of package manager binary

Package managers like apt (Ubuntu), apk (Alpine), or yum/dnf (RedHat) are binary software components used to manage and update installed software on a Linux® host system. They are used extensively to manage running virtual machines. However, using a package manager to install or remove software on a running container violates the immutable principle of container operation.

This policy demonstrates how RHACS detects and avoids a runtime violation, using Linux kernel instrumentation to detect the running process and OpenShift® to terminate the pod for enforcement. Using OpenShift to enforce runtime policy is preferable to enforcing rules directly within containers or in the container engine, as it avoids a disconnect between the state that OpenShift is maintaining and the state where the container is operating. Furthermore, because a runtime policy may detect only part of an attacker’s activity inside a container, removing the container avoids the attack.

Enable enforcement of policy

Click the Actions button, then click Edit policy.

Select the Policy Behavior tab by hitting next or clicking the tab.

Enable runtime enforcement by clicking the inform and enforce button.
Configure enforcement behavior by selecting Enforce at Runtime.

Go to the Review Policy tab
Review the changes
Click save

Make sure to save the policy changes! If you do not save the policy, the process will not be blocked!

Testing the configured policy

Next, we will use tmux to watch OpenShift events while running the test so you can see how RHACS enforces the policy at runtime.

Make sure that you are signed into the bastion host with OpenShift access when running the following commands.

Procedure

In the terminal, start tmux with two panes:

tmux new-session \; split-window -v \; attach

Next, run a watch on OpenShift events in the first shell pane:

oc get events -w

Press Ctrl+b THEN o to switch to the next pane. (Ctrl+b THEN o)
Exec into our Java application by getting the pod details and adding them to the following command.

POD=$(oc get pod -l app=ctf-web-to-system -o jsonpath="{.items[0].metadata.name}")
oc exec $POD -i --tty -- /bin/bash

[demo-user@bastion ~]$ POD=$(oc get pod -l app=ctf-web-to-system -o jsonpath="{.items[0].metadata.name}")
oc exec $POD -i --tty -- /bin/bash
node@ctf-web-to-system-6db858448f-hz6j2:/app$

If you see node@ctf… you’ve confirmed you have a shell and access to the Java application.

Run the Ubuntu package manager in this shell:

apt update

Examine the output and expect to see that the package manager attempts to perform an update operation:

Sample output

node@ctf-web-to-system-6db858448f-stwhq:/$ apt update
Reading package lists... Done
E: List directory /var/lib/apt/lists/partial is missing. - Acquire (13: Permission denied)
node@ctf-web-to-system-6db858448f-stwhq:/$ command terminated with exit code 137

Examine the oc get events tmux pane (The pane on the bottom), and note that it shows that RHACS detected the package manager invocation and deleted the pod:

0s          Normal    Killing                  pod/ctf-web-to-system-6db858448f-hz6j2    Stopping container ctf-web-container
0s          Normal    AddedInterface           pod/ctf-web-to-system-6db858448f-qp85v    Add eth0 [10.128.2.162/23] from ovn-kubernetes
0s          Normal    Pulling                  pod/ctf-web-to-system-6db858448f-qp85v    Pulling image "quay.io/jechoisec/ctf-web-to-system-01"
0s          Normal    Pulled                   pod/ctf-web-to-system-6db858448f-qp85v    Successfully pulled image "quay.io/jechoisec/ctf-web-to-system-01" in 262ms (263ms including waiting)
0s          Normal    Created                  pod/ctf-web-to-system-6db858448f-qp85v    Created container ctf-web-container
0s          Normal    Started                  pod/ctf-web-to-system-6db858448f-qp85v    Started container ctf-web-container

After a few seconds, you can see the pod is deleted and recreated. In your tmux shell pane, note that your shell session has terminated and that you are returned to the Bastion VM command line.

Congrats!

You have successfully stopped yourself from downloading malicious packages! However, the security investigative process continues, as you have now raised a flag that must be triaged! We will triage our violations later in this module.

Type exit in the terminal, use ctrl+c to stop the 'watch' command, and type exit one more time to get back to the default terminal.

CONTENT → Worth talking about the differences in enforcement?

Introduction to deploy-time policy enforcement

Deploy-time policy refers to enforcing configuration controls in the cluster and before deployment in the CI/CD process and the configuration of applications inside the cluster. Deploy-time policies can include all build-time policy criteria, but they can also include data from your cluster configurations, such as running in privileged mode or mounting the Docker socket.

There are two approaches to enforcing deploy-time policies in RHACS:

In clusters with listen and enforce AdmissionController options enabled, RHACS uses the admission controller to reject deployments that violate policy.
In clusters where the admission controller option is disabled, RHACS scales pod replicas to zero for deployments that violate policy.

In the next example, we are going to configure a Deploy-Time policy to block applications from deploying into the default namespace with the apt|dpkg application in the image.

Prevent the Ubuntu Package Manager in the ctf-web-to-system image from being deployed

Navigate to Platform Configuration → Policy Management
On the Policy Management page, type Policy then Ubuntu into the filter bar at the top.

This time we are going to edit a different policy. Specifically related to the Build & Deploy phases.

Click on the Ubuntu Package Manager in Image options (The three dots on the right side of the screen) and select Clone policy

Make sure to CLONE the policy. Cloning policies ensure the defaults don’t change.

Give the policy a new name. Something you will remember. The best practice would be to add a description for future policy enforcers as well. For example;

Next, update the policy to inform and enforce while clicking on the deploy stage only.

Make sure to unselect the Build lifecycle before moving forward.

Now, we want to target our specific deployment with an image label.

Click on the Policy criteria tab.
Click on the Deployment metadata dropdown on the right side of the browser.
Find the Namespace label and drag it to the default policy criteria.
Type default under the namespace criteria

Your policy should look like this,

Lastly, go to the Review Policy tab
Review the changes

There is a preview tab on the right side of the page that will show you all of the affected applications with the introduction of this policy.

Click Save

Now, let’s test it out! We’re going to deploy a simple Ubuntu application to the cluster.

cat << EOF > ubuntu-deployment.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ubuntu-deployment
  labels:
    app: ubuntu
spec:
  replicas: 3
  selector:
    matchLabels:
      app: ubuntu
  template:
    metadata:
      labels:
        app: ubuntu
    spec:
      containers:
      - name: ubuntu
        image: ubuntu:latest
        command: ["/bin/bash", "-c", "--"]
        args: ["while true; do echo hello world; sleep 10; done"]
        ports:
        - containerPort: 80
        resources:
          limits:
            memory: "128Mi"
            cpu: "500m"
          requests:
            memory: "64Mi"
            cpu: "250m"
EOF

Next, apply the deployment manifest.

oc apply -f ubuntu-deployment.yml

[lab-user@bastion ~]$ oc apply -f ubuntu-deployment.yml
Error from server (Failed currently enforced policies from StackRox): error when creating "ubuntu-deployment.yml": admission webhook "policyeval.stackrox.io" denied the request:
The attempted operation violated 1 enforced policy, described below:

Policy: Ubuntu Package Manager in Image - Deploy
- Description:
 ↳ Alert on deployments with components of the Debian/Ubuntu package management
 system in the image.
- Rationale:
 ↳ Package managers make it easier for attackers to use compromised containers,
 since they can easily add software.
- Remediation:
 ↳ Run `dpkg -r --force-all apt apt-get && dpkg -r --force-all debconf dpkg` in the
 image build for production containers.
- Violations:
 - Container 'ubuntu' includes component 'apt' (version 2.7.14build2)
 - Container 'ubuntu' includes component 'dpkg' (version 1.22.6ubuntu6)

In case of emergency, add the annotation {"admission.stackrox.io/break-glass": "ticket-1234"} to your deployment with an updated ticket number

Another option for enforcement is to use the "deployment check" CLI command.

Set variables to connect to RHACS Central.

export ROX_CENTRAL_ADDRESS={acs_route}
cd ~/
export TUTORIAL_HOME="$(pwd)/demo-apps"

Verify the ctf-web-to-system application against the policies you’ve created.

roxctl -e $ROX_CENTRAL_ADDRESS:443 deployment check --file $TUTORIAL_HOME/kubernetes-manifests/ctf-web-to-system/ctf-w2s.yml --insecure-skip-tls-verify

[demo-user@bastion ~]$ roxctl -e $ROX_CENTRAL_ADDRESS:443 deployment check --file $TUTORIAL_HOME/kubernetes-manifests/ctf-web-to-system/ctf-w2s.yml --insecure-skip-tls-verify
Policy check results for deployments: [ctf-web-to-system]
(TOTAL: 7, LOW: 4, MEDIUM: 2, HIGH: 1, CRITICAL: 0)

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|             POLICY             | SEVERITY | BREAKS DEPLOY |    DEPLOYMENT     |          DESCRIPTION           |           VIOLATION            |          REMEDIATION           |
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|   Ubuntu Package Manager in    |   LOW    |       -       | ctf-web-to-system |      Alert on deployments      |          - Container           |    Run `dpkg -r --force-all    |
|             Image              |          |               |                   |     with components of the     |  'ctf-web-container' includes  |     apt apt-get && dpkg -r     |
|                                |          |               |                   |     Debian/Ubuntu package      |    component 'apt' (version    |  --force-all debconf dpkg` in  |
|                                |          |               |                   |    management system in the    |             1.4.9)             | the image build for production |
|                                |          |               |                   |             image.             |                                |          containers.           |
|                                |          |               |                   |                                |          - Container           |                                |
|                                |          |               |                   |                                |  'ctf-web-container' includes  |                                |
|                                |          |               |                   |                                |   component 'dpkg' (version    |                                |
|                                |          |               |                   |                                |            1.18.25)            |                                |
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|   Ubuntu Package Manager in    |   LOW    |       X       | ctf-web-to-system |      Alert on deployments      |          - Container           |    Run `dpkg -r --force-all    |
|   Image - Default namespace    |          |               |                   |     with components of the     |  'ctf-web-container' includes  |     apt apt-get && dpkg -r     |
|                                |          |               |                   |     Debian/Ubuntu package      |    component 'apt' (version    |  --force-all debconf dpkg` in  |
|                                |          |               |                   |    management system in the    |             1.4.9)             | the image build for production |
|                                |          |               |                   |             image.             |                                |          containers.           |
|                                |          |               |                   |                                |          - Container           |                                |
|                                |          |               |                   |                                |  'ctf-web-container' includes  |                                |
|                                |          |               |                   |                                |   component 'dpkg' (version    |                                |
|                                |          |               |                   |                                |            1.18.25)            |                                |
|                                |          |               |                   |                                |                                |                                |
|                                |          |               |                   |                                | - Namespace has name 'default' |                                |
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
WARN:   A total of 7 policies have been violated
ERROR:  failed policies found: 1 policies violated that are failing the check
ERROR:  Policy "Ubuntu Package Manager in Image - Default namespace" within Deployment "ctf-web-to-system" - Possible remediation: "Run `dpkg -r --force-all apt apt-get && dpkg -r --force-all debconf dpkg` in the image build for production containers."
ERROR:  checking deployment failed after 3 retries: breaking policies found: failed policies found: 1 policies violated that are failing the check

You should see one of the policies you’ve created breaking the deployment process while the others are in inform-only mode.

Congrats!

You’re now enforcing against the Ubuntu package manager at runtime and deploy time. Let’s finish with enforcing at build-time!

Introduction to build-time policy enforcement

Build time policies for container images are guidelines that define how container images should be constructed. These policies aim to achieve several goals, including:

Security: Minimizing vulnerabilities and ensuring images are built with secure practices.
Efficiency: Reducing image size and build times for faster deployments.
Consistency: Maintaining a uniform structure and content across all images. Here are some key areas covered by build time policies:
Base Image: Specifying a minimal base image that only contains essential components.
Package Management: Encouraging the use of package managers for dependency installation and updates.
File Copying: Limiting what gets copied into the image to only required files and avoiding unnecessary bloat.
User Management: Defining a non-root user for the application process to run as.
Environment Variables: Storing sensitive information in environment variables outside the image.

In RHACS, build-time policies apply to image fields such as CVEs and Dockerfile instructions.

Prevent the Ubuntu package manager in the ctf-web-to-system image from being pushed to Quay

Export the following variables to make our lives easier

export QUAY_USER={quay_admin_username}
export QUAY_URL=$(oc -n quay-enterprise get route quay-quay -o jsonpath='{.spec.host}')
export ROX_CENTRAL_ADDRESS={acs_route}

Verify that the variables are correct.

echo $QUAY_USER
echo $QUAY_URL
echo $ROX_CENTRAL_ADDRESS

podman login $QUAY_URL

Use the quay admin credentials, Username: {quay_admin_username} & password: {quay_admin_password}. You can create unique user and group credentials in Quay for proper segmentation.

Download the latest Ubuntu image.

podman pull ubuntu:latest
podman tag docker.io/library/ubuntu:latest $QUAY_URL/$QUAY_USER/ctf-web-to-system:1.0

The following command is designed to mimic and build a pipeline where a container build is going through a commit/promotion step.

Check the image and push it to the local Quay instance.

roxctl --insecure-skip-tls-verify -e "$ROX_CENTRAL_ADDRESS:443" image check --image=$QUAY_URL/$QUAY_USER/ctf-web-to-system:1.0
podman push $QUAY_URL/$QUAY_USER/ubuntu:latest --remove-signatures

We are using the image check CLI option, NOT the image scan.

 },
 "components": 968,
 "cves": 82,
 "fixableCves": 80,
 "lastUpdated": "2024-05-22T21:32:48.735784107Z",
 "riskScore": 10.8,
 "topCvss": 10,
 "notes": [
 "MISSING_SIGNATURE",
 "MISSING_SIGNATURE_VERIFICATION_DATA"
 ]
}
Copying blob 03004fed5c90 skipped: already exists
...
Copying config 1cbb2b7908 done
Writing manifest to image destination

Now RHACS hasn’t broken the command since there is no enforcement of any build policies.

Let’s make a copy of the build & deploy-time policy and enforce it during the build phase.

Procedure

Navigate to Platform Configuration → Policy Management
On the Policy Management page, type Policy then Ubuntu into the filter bar at the top.
Click on the Ubuntu Package Manager in Image options (The three dots on the right side of the screen) and select Clone policy

Make sure to CLONE the policy

Give the policy a new name. Something you will remember. The best practice would be to add a description for future policy enforcers as well. For example;

Next, update the policy to inform and enforce while ensuring the Build stage checkbox is selected And select Enforce on Build at the bottom of the page.

Make sure to unselect the DEPLOY lifecycle before moving forward.

At the policy scope tab, make sure there are no exclusions or inclusions.
Lastly, go to the Review Policy tab
Review the changes.
Click Save

Now let’s test it out!

Run the following in the terminal.

roxctl --insecure-skip-tls-verify -e "$ROX_CENTRAL_ADDRESS:443" image check --image=docker.io/library/ubuntu
podman push $QUAY_URL/$QUAY_USER/ubuntu:latest --remove-signatures

--------------------------------------
|   Ubuntu Package Manager in    |   LOW    |      -       |      Alert on deployments      |   - Image includes component   |    Run `dpkg -r --force-all    |
|             Image              |          |              |     with components of the     |  'apt' (version 2.7.14build2)  |     apt apt-get && dpkg -r     |
|                                |          |              |     Debian/Ubuntu package      |                                |  --force-all debconf dpkg` in  |
|                                |          |              |    management system in the    |   - Image includes component   | the image build for production |
|                                |          |              |             image.             | 'dpkg' (version 1.22.6ubuntu6) |          containers.           |
--------------------------------------------------------------------------------------------------------------------------------------------------------+
WARN:   A total of 4 policies have been violated
ERROR:  failed policies found: 1 policies violated that are failing the check
ERROR:  Policy "Ubuntu Package Manager in Image - Build-time" - Possible remediation: "Run `dpkg -r --force-all apt apt-get && dpkg -r --force-all debconf dpkg` in the image build for production containers."
ERROR:  checking image failed after 3 retries: failed policies found: 1 policies violated that are failing the check

You should see the same violations from the previous command EXCEPT now you have a failed policy check. This woruld send an exit 0 command if this was run in any pipeline.

Understand Violations in RHACS

Violations taken together determine risk, which you covered in previous labs. In this lab, you explore how to determine the details of those violations to plan and implement their remediation.

The Violations view allows you to see these details.

Using RHACS, you can view policy violations, drill down to the actual cause of the violation, and take corrective actions.

The built-in policies identify a variety of security findings, including vulnerabilities (CVEs), violations of DevOps best practices, high-risk build and deployment practices, and suspicious runtime behaviors. You can use the default out-of-the-box security policies or your own custom policies.

Report and Resolve Violations

In this last section. We will resolve a few of the issues that we have created.

Procedure

Navigate to the Violations page.
Filter by the policy violation Ubuntu Package Manager Execution OR by the most recent policy violations. You will see a build, deploy and runtime policy violation that has been enforced one time.

Click the most recent violation and explore the list of the violation events:

If configured, each violation record is pushed to a Security Information and Event Management (SIEM) integration and is available to be retrieved via the API. The forensic data shown in the UI is recorded, including the timestamp, process user IDs, process arguments, process ancestors, and enforcement action.

After this issue is addressed, in this case by the RHACS product using the runtime enforcement action, you can remove it from the list by marking it as Resolved.

Lastly, hover over the violation in the list to see the resolution options and resolve this issue.

For more information about integration with SIEM tools, see the RHACS help documentation on external tools.

Summary

AMAZING!

In summary, we made use of the features provided by Red Hat Advanced Cluster Security for Kubernetes to display potential security violations in your cluster in a central dashboard. You crafted both deploy-time and runtime policies to help prevent malicious events from occurring in our cluster. Hopefully this lab has helped demonstrate to you the immense value provided by RHACS and OpenShift Platform Plus. Please feel free to continue and explore the RHACS lab environment.

On to CI/CD and Automation!