Policy Management

Module goals

Create build, deploy and runtime policies
Create policies from filters
Clone and edit policies based on the ACS defaults

RHACS Policy Management Basics

RHACS has many built-in policies to detect activity related to attacker goals: gain a foothold, maintain a presence, move laterally, and exfiltrate data. The continuous runtime monitoring observes all container activity and will automatically respond to events with appropriate enforcement and notification. However, that would be missing out on an opportunity - RHACS wants to go one step further, to take advantage of containers' ephemeral, immutable nature, to improve security in a measurable way.

We want to use runtime incidents and vulnerabilities as a learning opportunity to improve security going forward by constraining how our containers can act. We achieve this by creating policies and implementing them early in the CI/CD process.

As we move into this next section, let’s focus on identifying and enforcing a runtime policy in the cluster. For the upcoming example, we will focus on stopping the Ubuntu package manager from being run on pods in our cluster. Our example container ctf-web-to-system has this package manager in the container. Let’s ensure that it never calls for updates while present in our clusters.

Introduction to Runtime Policy Creation and Enforcement

RHACS observes container processes and collects this information to enable you to craft policies to prevent behavior that you don’t like. This information can also create baseline policy configurations that the user can update. Runtime policies can include build-time and deploy-time policy criteria but they can also include data about process executions during runtime. However, runtime policies that use “audit logs” event source cannot use build and deploy criteria at all as they are Kubernetes specific events.

The example below demonstrates how security may want to block a package manager from downloading any packages to the container. This runtime enforcement option is the first in the process of shifting left. After runtime enforcement, you will want to stop the package manager from being used in the container altogether.

For the Runtime stage, RHACS stops all pods that match the conditions of the policy. It does not scale the deployment down to zero.

Procedure

On the left-hand side of the application, click the Platform Configuration tab and select Policy Management.

Filter through the policies to find Ubuntu Package Manager Execution or use the search bar to select Policy.

Once you have found the policy Ubuntu Package Manager Execution, click on it to learn more.

If you click the actions button, you will see how easy it is to edit, clone, export or disable these policies. We also recommended cloning the policies and adding or removing specific filters as you need them.

Prevent execution of package manager binary

Package managers like apt (Ubuntu), apk (Alpine), or yum/dnf (RedHat) are binary software components used to manage and update installed software on a Linux® host system. They are used extensively to manage running virtual machines. However, using a package manager to install or remove software on a running container violates the immutable principle of container operation.

This policy demonstrates how RHACS detects and avoids a runtime violation, using Linux kernel instrumentation to detect the running process and OpenShift® to terminate the pod for enforcement. Using OpenShift to enforce runtime policy is preferable to enforcing rules directly within containers or in the container engine, as it avoids a disconnect between the state that OpenShift is maintaining and the state where the container is operating. Furthermore, because a runtime policy may detect only part of an attacker’s activity inside a container, removing the container avoids the attack.

Enable enforcement of policy

Click the Actions button, then click Edit policy.

Select Policy Behavior → Actions

Enable runtime enforcement by clicking the inform and enforce button.
Configure enforcement behavior by selecting Enforce at Runtime.

Go to the Review tab
Review the changes
Click save

Make sure to save the policy changes! If you do not save the policy, the process will not be blocked!

Testing the configured policy

Next, we will use tmux to watch OpenShift events while running the test so you can see how RHACS enforces the policy at runtime.

Make sure that you are signed into the bastion host with OpenShift access when running the following commands.

Procedure

In the terminal, start tmux with two panes:

tmux new-session \; split-window -v \; attach

Next, run a watch on OpenShift events in the first shell pane:

oc get events -w

Press Ctrl+b THEN o to switch to the next pane. (Hold ctrl+b then press o)
Exec into our Java application by getting the pod details and adding them to the following command.

POD=$(oc get pod -l app=ctf-web-to-system -o jsonpath="{.items[0].metadata.name}")
oc exec $POD -i --tty -- /bin/bash

[demo-user@bastion ~]$ POD=$(oc get pod -l app=ctf-web-to-system -o jsonpath="{.items[0].metadata.name}")
oc exec $POD -i --tty -- /bin/bash
node@ctf-web-to-system-6db858448f-hz6j2:/app$

If you see node@ctf… you’ve confirmed you have a shell and access to the Java application.

Run the Ubuntu package manager in this shell:

apt update

Examine the output and expect to see that the package manager attempts to perform an update operation:

Sample output

node@ctf-web-to-system-6db858448f-stwhq:/$ apt update
Reading package lists... Done
E: List directory /var/lib/apt/lists/partial is missing. - Acquire (13: Permission denied)
node@ctf-web-to-system-6db858448f-stwhq:/$ command terminated with exit code 137

Examine the oc get events tmux pane (The pane on the bottom), and note that it shows that RHACS detected the package manager invocation and deleted the pod:

0s          Normal    Killing                  pod/ctf-web-to-system-6db858448f-hz6j2    Stopping container ctf-web-container
0s          Normal    AddedInterface           pod/ctf-web-to-system-6db858448f-qp85v    Add eth0 [10.128.2.162/23] from ovn-kubernetes
0s          Normal    Pulling                  pod/ctf-web-to-system-6db858448f-qp85v    Pulling image "quay.io/jechoisec/ctf-web-to-system-01"
0s          Normal    Pulled                   pod/ctf-web-to-system-6db858448f-qp85v    Successfully pulled image "quay.io/jechoisec/ctf-web-to-system-01" in 262ms (263ms including waiting)
0s          Normal    Created                  pod/ctf-web-to-system-6db858448f-qp85v    Created container ctf-web-container
0s          Normal    Started                  pod/ctf-web-to-system-6db858448f-qp85v    Started container ctf-web-container

After a few seconds, you can see the pod is deleted and recreated. In your tmux shell pane, note that your shell session has terminated and that you are returned to the Bastion VM command line.

Congrats!

You have successfully stopped yourself from downloading malicious packages! However, the security investigative process continues, as you have now raised a flag that must be triaged! We will triage our violations later in this module.

Type exit in the terminal, use ctrl+c to stop the 'watch' command, and type exit one more time to get back to the default terminal.

Introduction to deploy-time policy enforcement

Deploy-time policy refers to enforcing configuration controls in the cluster and before deployment in the CI/CD process and the configuration of applications inside the cluster. Deploy-time policies can include all build-time policy criteria, but they can also include data from your cluster configurations, such as running in privileged mode or mounting the Docker socket.

There are two approaches to enforcing deploy-time policies in RHACS:

In clusters with listen and enforce AdmissionController options enabled, RHACS uses the admission controller to reject deployments that violate policy.
In clusters where the admission controller option is disabled, RHACS scales pod replicas to zero for deployments that violate policy.

In the next example, we are going to configure a Deploy-Time policy to block applications from deploying into the default namespace with the apt|dpkg application in the image.

Prevent the Ubuntu Package Manager in the ctf-web-to-system image from being deployed

First, delete the deployment from the cluster, we will redeploy the application after creating the policy.

oc delete -f $TUTORIAL_HOME/kubernetes-manifests/ctf-web-to-system/ctf-w2s.yml

Navigate to Platform Configuration → Policy Management
On the Policy Management page, type Policy then Ubuntu into the filter bar at the top.

This time you are going to edit a different policy for the Ubuntu package manager, specifically related to the Build & Deploy phases.

Click on the Ubuntu Package Manager in Image options (The three dots on the right side of the screen) and select Clone policy

Make sure to CLONE the policy. Cloning policies ensure the defaults don’t change.

Give the policy a new name, such as Ubuntu Package Manager in Image - Enforce Deploy. The best practice would be to add a description for future policy enforcers as well.

Next, ensure the policy by clicking on the Build stage so that only the Deploy stage is selected.

Make sure to unselect the Build lifecycle before moving forward. This will trigger an alert!

We will have to add the policy criteria back. This is because certain actions can only be done in the build, deploy, or runtime stages.

Now, we want to target our specific deployment with an image label.

Click on the Rules tab.
Click on the Image contents dropdown on the right side of the browser.
Find the Image component label and drag it to the default policy criteria.
Type apt-get under the criteria

Your policy should look like this,

In Policy behavior → Actions, click Inform and enforce + Enforce on Deploy

Lastly, go to the Review Policy tab
Review the changes

There is a preview tab on the right side of the page that will show you all of the affected applications with the introduction of this policy.

Click Save

Now, let’s test it out! We’re going to redeploy the CTF-web-to-system application from earlier.

cat $TUTORIAL_HOME/kubernetes-manifests/ctf-web-to-system/ctf-w2s.yml

apiVersion: v1
kind: Service
metadata:
  name: ctf-web-to-system-service
spec:
  selector:
    app: ctf-web-to-system
  ports:
    - protocol: TCP
      port: 80
      targetPort: 9090
  type: LoadBalancer
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ctf-web-to-system
  labels:
    app: ctf-web-to-system
    demo: roadshow
  annotation:
    app: ctf-web-to-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ctf-web-to-system
  template:
    metadata:
      labels:
        app: ctf-web-to-system
    spec:
      containers:
      - name: ctf-web-container
        image: quay-jlchp.apps.cluster-jlchp.jlchp.sandbox2037.opentlc.com/quayadmin/ctf-web-to-system:1.0
        ports:
        - containerPort: 9090

Now, apply the manifests to the cluster.

oc apply -f $TUTORIAL_HOME/kubernetes-manifests/ctf-web-to-system/ctf-w2s.yml

[lab-user@bastion ~]$ oc apply -f $TUTORIAL_HOME/kubernetes-manifests/ctf-web-to-system/ctf-w2s.yml
Error from server (Failed currently enforced policies from StackRox): error when creating "/home/lab-user/demo-apps/kubernetes-manifests/ctf-web-to-system/ctf-w2s.yml": admission webhook "policyeval.stackrox.io" denied the request:
The attempted operation violated 1 enforced policy, described below:

Policy: Ubuntu Package Manager in Image (COPY)
- Description:
    ↳ Alert on deployments with components of the Debian/Ubuntu package management
      system in the image.
- Rationale:
    ↳ Package managers make it easier for attackers to use compromised containers,
      since they can easily add software.
- Remediation:
    ↳ Run `dpkg -r --force-all apt apt-get && dpkg -r --force-all debconf dpkg` in the
      image build for production containers.
- Violations:
    - Container 'ctf-web-container' includes component 'apt' (version 1.4.9)
    - Container 'ctf-web-container' includes component 'dpkg' (version 1.18.25)

In case of emergency, add the annotation {"admission.stackrox.io/break-glass": "ticket-1234"} to your deployment with an updated ticket number

Another option for enforcement is to use the "deployment check" CLI command.

Verify the ubuntu application against the policies you’ve created.

roxctl -e $ROX_CENTRAL_ADDRESS:443 deployment check --file $TUTORIAL_HOME/kubernetes-manifests/ctf-web-to-system/ctf-w2s.yml --insecure-skip-tls-verify

[lab-user@bastion ~]$ roxctl -e $ROX_CENTRAL_ADDRESS:443 deployment check --file $TUTORIAL_HOME/kubernetes-manifests/ctf-web-to-system/ctf-w2s.yml --insecure-skip-tls-verify
Policy check results for deployments: [ctf-web-to-system]
(TOTAL: 6, LOW: 3, MEDIUM: 2, HIGH: 1, CRITICAL: 0)

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|             POLICY             | SEVERITY | BREAKS DEPLOY |    DEPLOYMENT     |          DESCRIPTION           |           VIOLATION            |          REMEDIATION           |
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|   Ubuntu Package Manager in    |   LOW    |       -       | ubuntu-deployment |      Alert on deployments      |          - Container           |    Run `dpkg -r --force-all    |
|             Image              |          |               |                   |     with components of the     |  'ctf-web-container' includes  |     apt apt-get && dpkg -r     |
|                                |          |               |                   |     Debian/Ubuntu package      |    component 'apt' (version    |  --force-all debconf dpkg` in  |
|                                |          |               |                   |    management system in the    |             1.4.9)             | the image build for production |
|                                |          |               |                   |             image.             |                                |          containers.           |
|                                |          |               |                   |                                |          - Container           |                                |
|                                |          |               |                   |                                |  'ctf-web-container' includes  |                                |
|                                |          |               |                   |                                |   component 'dpkg' (version    |                                |
|                                |          |               |                   |                                |            1.18.25)            |                                |
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|   Ubuntu Package Manager in    |   LOW    |       X       | ctf-web-to-system |      Alert on deployments      |          - Container           |    Run `dpkg -r --force-all    |
|   Image - Default namespace    |          |               |                   |     with components of the     |  'ctf-web-container' includes  |     apt apt-get && dpkg -r     |
|                                |          |               |                   |     Debian/Ubuntu package      |    component 'apt' (version    |  --force-all debconf dpkg` in  |
|                                |          |               |                   |    management system in the    |             1.4.9)             | the image build for production |
|                                |          |               |                   |             image.             |                                |          containers.           |
|                                |          |               |                   |                                |          - Container           |                                |
|                                |          |               |                   |                                |  'ctf-web-container' includes  |                                |
|                                |          |               |                   |                                |   component 'dpkg' (version    |                                |
|                                |          |               |                   |                                |            1.18.25)            |                                |
|                                |          |               |                   |                                |                                |                                |
|                                |          |               |                   |                                | - Namespace has name 'default' |                                |
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
WARN:   A total of 6 policies have been violated
ERROR:  failed policies found: 1 policies violated that are failing the check
ERROR:  Policy "Ubuntu Package Manager in Image (COPY)" within Deployment "ctf-web-to-system" - Possible remediation: "Run `dpkg -r --force-all apt apt-get && dpkg -r --force-all debconf dpkg` in the image build for production containers."
ERROR:  checking deployment failed: breaking policies found: failed policies found: 1 policies violated that are failing the check

You should see one of the policies you’ve created breaking the deployment process while the others are in inform-only mode.

Congrats!

You’re now enforcing against the Ubuntu package manager at runtime and deploy time. Let’s finish with enforcing at build-time!

Introduction to build-time policy enforcement

Build time policies for container images are guidelines that define how container images should be constructed. These policies aim to achieve several goals, including:

Security: Minimizing vulnerabilities and ensuring images are built with secure practices.
Efficiency: Reducing image size and build times for faster deployments.
Consistency: Maintaining a uniform structure and content across all images. Here are some key areas covered by build time policies:
Base Image: Specifying a minimal base image that only contains essential components.
Package Management: Encouraging the use of package managers for dependency installation and updates.
File Copying: Limiting what gets copied into the image to only required files and avoiding unnecessary bloat.
User Management: Defining a non-root user for the application process to run as.
Environment Variables: Storing sensitive information in environment variables outside the image.

In RHACS, build-time policies apply to image fields such as CVEs and Dockerfile instructions.

Prevent the Ubuntu package manager in the ctf-web-to-system image from being pushed to Quay

Export the following variables to make our lives easier

export QUAY_USER={quay_admin_username}
export QUAY_URL=$(oc -n quay-enterprise get route quay-quay -o jsonpath='{.spec.host}')
export ROX_CENTRAL_ADDRESS={acs_route}

Verify that the variables are correct.

echo $QUAY_USER
echo $QUAY_URL
echo $ROX_CENTRAL_ADDRESS

podman login $QUAY_URL

Use the quay admin credentials, Username: {quay_admin_username} & password: {quay_admin_password}. You can create unique user and group credentials in Quay for proper segmentation.

Let’s pretend as if the developers are pushing an update to the ctf-web-to-system application. First, pull and scan the related image.

The following command is designed to mimic and build a pipeline where a container build is going through a commit/promotion step. You download the image, scan for vulnerabilities, tag a newer version and upload to Quay.

podman pull $QUAY_URL/$QUAY_USER/ctf-web-to-system:1.0
roxctl --insecure-skip-tls-verify -e "$ROX_CENTRAL_ADDRESS:443" image scan --image=$QUAY_URL/$QUAY_USER/ctf-web-to-system:1.0

podman tag $QUAY_URL/$QUAY_USER/ctf-web-to-system:1.0 $QUAY_URL/$QUAY_USER/ctf-web-to-system:1.1
podman push $QUAY_URL/$QUAY_USER/ctf-web-to-system:1.1 --remove-signatures

We are using the image check CLI option, NOT the image scan. This is because we are checking for a policy violation and not grabbing a vulnerability scan output.

 },
    "notes": [
      "OS_CVES_UNAVAILABLE",
      "PARTIAL_SCAN_DATA"
    ],
    "hash": "4109274308829192377"
  },
  "components": 968,
  "cves": 92,
  "fixableCves": 89,
  "lastUpdated": "2025-01-16T23:02:11.049699566Z",
  "riskScore": 10.8,
  "topCvss": 10,
  "notes": [
    "MISSING_SIGNATURE",
    "MISSING_SIGNATURE_VERIFICATION_DATA"
  ]
}
[lab-user@bastion ~]$ podman tag $QUAY_URL/$QUAY_USER/ctf-web-to-system:1.0 $QUAY_URL/$QUAY_USER/ctf-web-to-system:1.1
podman push $QUAY_URL/$QUAY_USER/ctf-web-to-system:1.1 --remove-signatures
Copying blob 308102f44919 skipped: already exists
Copying blob b8d9a96d44df skipped: already exists
....
Copying config 1cbb2b7908 done   |
Writing manifest to image destination

Now RHACS hasn’t broken the command since there is no enforcement of any build policies.

Let’s make a copy of the build & deploy-time policy and enforce it during the build phase.

Procedure

Navigate to Platform Configuration → Policy Management
On the Policy Management page, type Policy then Ubuntu into the filter bar at the top.
Click on the Ubuntu Package Manager in Image options (The three dots on the right side of the screen) and select Clone policy

Make sure to CLONE the policy

Give the policy a new name, such as Ubuntu Package Manager in Image - Enforce Build. The best practice would be to add a description for future policy enforcers as well.

Next, update the policy to inform and enforce while ensuring the Build stage checkbox is selected And select Enforce on Build at the bottom of the page.

At the policy scope tab, make sure there are no exclusions or inclusions.
Lastly, go to the Review Policy tab
Review the changes.
Click Save

Now let’s test it out!

Run the following in the terminal.

podman pull $QUAY_URL/$QUAY_USER/ctf-web-to-system:1.1
roxctl image check --insecure-skip-tls-verify -e "$ROX_CENTRAL_ADDRESS:443"  --image=$QUAY_URL/$QUAY_USER/ctf-web-to-system:1.1

--------------------------------------
|   Ubuntu Package Manager in    |   LOW    |      X       |      Alert on deployments      |   - Image includes component   |    Run `dpkg -r --force-all    |
|     Image - Enforce Build      |          |              |     with components of the     |     'apt' (version 1.4.9)      |     apt apt-get && dpkg -r     |
|                                |          |              |     Debian/Ubuntu package      |                                |  --force-all debconf dpkg` in  |
|                                |          |              |    management system in the    |   - Image includes component   | the image build for production |
|                                |          |              |             image.             |    'dpkg' (version 1.18.25)    |          containers.           |
--------------------------------------------------------------------------------------------------------------------------------------------------------+
WARN:   A total of 4 policies have been violated
ERROR:  failed policies found: 1 policies violated that are failing the check
ERROR:  Policy "Ubuntu Package Manager in Image - Enforce Build" - Possible remediation: "Run `dpkg -r --force-all apt apt-get && dpkg -r --force-all debconf dpkg` in the image build for production containers."
ERROR:  checking image failed: failed policies found: 1 policies violated that are failing the check

You should see the same violations from the previous command EXCEPT now you have a failed policy check. This would send an exit 0 command if this was run in any pipeline.

Summary

AMAZING!

In summary, we made use of the features provided by Red Hat Advanced Cluster Security for Kubernetes to display potential security violations in your cluster in a central dashboard. You crafted both deploy-time and runtime policies to help prevent malicious events from occurring in our cluster. Hopefully this lab has helped demonstrate to you the immense value provided by RHACS and OpenShift Platform Plus. Please feel free to continue and explore the RHACS lab environment.

On to CI/CD and Automation!