Policy Management

Module goals

Create build, deploy and runtime policies
Create policies from filters
Clone and edit policies based on the ACS defaults

RHACS Policy Management Basics

RHACS has built-in policies to detect activities linked to attacker goals, like gaining access, maintaining presence, moving laterally, and exfiltrating data. Continuous runtime monitoring watches all container activities and automatically responds with enforcement and notifications. However, RHACS aims to go beyond this by leveraging the ephemeral, immutable nature of containers to improve security.

We want to turn runtime incidents and vulnerabilities into opportunities for better security by restricting container actions. This is done by creating and implementing policies early in the CI/CD process.

In the next section, we’ll focus on identifying and enforcing a runtime policy in the cluster. For this example, we’ll prevent the Alpine Linux package manager from running on pods in our cluster. We’ll use the frontend container, which includes this package manager, and ensure it cannot update while in the cluster.

Introduction to Runtime Policy Creation and Enforcement

RHACS monitors container processes and collects data to help create policies that block unwanted behavior. It also helps establish baseline policies that users can adjust. Runtime policies can include criteria from build-time and deploy-time but also account for process executions during runtime. However, runtime policies based on audit logs can’t use build and deploy criteria, as these logs are Kubernetes-specific events.

The example below shows how security may block a package manager from downloading packages. This runtime enforcement is the first step in shifting security left. After enforcing runtime policies, the next step will be to prevent the package manager from being used in the container entirely.

For the Runtime stage, RHACS stops all pods that match the conditions of the policy. It does not scale the deployment down to zero.

Procedure

On the left-hand side of the application, click the Platform Configuration tab and select Policy Management.

Filter through the policies to find Alpine Linux Package Manager Execution or use the search bar to select Policy.

Once you have found the policy Alpine Linux Package Manager (apk) in Image, click on it to learn more.

Prevent execution of package manager binary

Package managers like apt (Ubuntu), apk (Alpine), and yum/dnf (RedHat) are used to manage and update software on Linux hosts, including virtual machines. However, using a package manager to install or remove software on a running container breaks the container’s immutable nature.

This policy shows how RHACS detects and prevents such violations by using Linux kernel instrumentation to identify the running process and OpenShift to terminate the pod for enforcement. Enforcing runtime policy through OpenShift is preferred over doing it directly in the container or container engine. This ensures consistency between the state OpenShift manages and the state of the container. Additionally, since runtime policies might only detect part of an attack, removing the container entirely helps stop the attack.

Enable enforcement of policy

Procedure

Click the Actions button, then click CLONE policy. Then give it the name "Alpine Linux Package Manager Execution - Runtime"

Alpine Linux Package Manager Execution - Runtime

Select Policy Behavior → Actions

Enable runtime enforcement by clicking the inform and enforce button.
Configure enforcement behavior by selecting Enforce at Runtime.

Click Next
Review the changes then click save

Make sure to save the policy changes! If you do not save the policy, the process will not be blocked!

Testing the configured policy

Next, we will use tmux to watch OpenShift events while running the test so you can see how RHACS enforces the policy at runtime.

Make sure that you are signed into the bastion host with OpenShift access when running the following commands.

Procedure

In the terminal, start tmux with two panes:

tmux new-session \; split-window -v \; attach

Next, run a watch on OpenShift events in the first shell pane:

oc get events -n patient-portal -w

Press Ctrl+b first, and then press o to switch to the next pane. (Hold ctrl+b then press o)
Exec into our Java application by getting the pod details and adding them to the following command.

POD=$(oc get pod -l app=frontend -n patient-portal -o jsonpath="{.items[0].metadata.name}")
oc exec $POD -n patient-portal -i --tty -- /bin/bash

[demo-user@bastion ~]$ POD=$(oc get pod -l app=frontend -o jsonpath="{.items[0].metadata.name}")
oc exec $POD -i --tty -- /bin/bash
node@frontend-6db858448f-hz6j2:/app$

If you see /home/fritz you’ve confirmed you have a shell and access to the Java application.

Run the apk package manager in this shell:

apk update

Sample output

node@frontend-6db858448f-stwhq:/$ apt update
Reading package lists... Done
E: List directory /var/lib/apt/lists/partial is missing. - Acquire (13: Permission denied)
node@frontend-6db858448f-stwhq:/$ command terminated with exit code 137

Examine the output and expect to see that the package manager attempts to perform an update operation
Examine the oc get events tmux pane (The pane on the bottom), and note that it shows that RHACS detected the package manager invocation and deleted the pod:

0s          Normal    Killing                pod/frontend-78795c75fc-7nhs4             Stopping container frontend
0s          Normal    SuccessfulCreate       replicaset/frontend-78795c75fc            Created pod: frontend-78795c75fc-8q9fn
0s          Normal    Scheduled              pod/frontend-78795c75fc-8q9fn             Successfully assigned patient-portal/frontend-78795c75fc-8q9fn to ip-10-0-108-51.us-east-2.compute.internal
0s          Normal    AddedInterface         pod/frontend-78795c75fc-8q9fn             Add eth0 [10.129.3.148/23] from ovn-kubernetes
0s          Normal    Pulling                pod/frontend-78795c75fc-8q9fn             Pulling image "quay-88k5s.apps.cluster-88k5s.88k5s.sandbox139.opentlc.com/quayadmin/frontend:0.1"
0s          Normal    Pulled                 pod/frontend-78795c75fc-8q9fn             Successfully pulled image "quay-88k5s.apps.cluster-88k5s.88k5s.sandbox139.opentlc.com/quayadmin/frontend:0.1" in 61ms (61ms including waiting). Image size: 119108884 bytes.
0s          Normal    Created                pod/frontend-78795c75fc-8q9fn             Created container: frontend
0s          Normal    Started                pod/frontend-78795c75fc-8q9fn             Started container frontend

After a few seconds, you can see the pod is deleted and recreated. In your tmux shell pane, note that your shell session has terminated and that you are returned to the Bastion VM command line.

Congrats!

The security investigative process continues, as you have now raised a flag that must be triaged! We will triage our violations later in this module.

Type exit in the terminal, use ctrl+c to stop the 'watch' command, and type exit one more time to get back to the default terminal.

Introduction to deploy-time policy enforcement

Deploy-time policy enforces configuration controls both before deployment in the CI/CD process and within the cluster itself. It can include build-time policies and cluster configurations, such as privileged mode or mounting the Docker socket.

There are two ways to enforce deploy-time policies in RHACS:

With listen and enforce options enabled, RHACS rejects deployments that violate the policy using the admission controller.
When the admission controller is disabled, RHACS scales pod replicas to zero for deployments that violate the policy.

Next, we’ll set up a Deploy-Time policy to block applications from deploying into the default namespace if the image contains the apt|dpkg application.

Prevent the Alpine Linux Package Manager in the frontend image from being deployed

Procedure

First, delete the deployment from the cluster, we will redeploy the application after creating the policy.

oc delete -f $APP_HOME/skupper-demo/frontend.yml

Sample output

[lab-user@bastion ~]$ oc delete -f $APP_HOME/skupper-demo/
namespace "patient-portal" deleted
deployment.apps "database" deleted
service "database" deleted
deployment.apps "frontend" deleted
route.route.openshift.io "frontend-patient-route" deleted
service "frontend-service" deleted
deployment.apps "payment-processor" deleted
service "payment-service" deleted

Navigate to Platform Configuration → Policy Management
On the Policy Management page, type Policy then Alpine Linux Package Manager (apk) in Image into the filter bar at the top.

This time you are going to edit a different policy for the Alpine Linux Package Manager (apk), specifically related to the Build & Deploy phases.

Click on the Alpine Linux Package Manager (apk) in Image options (The three dots on the right side of the screen) and select Clone policy

Make sure to CLONE the policy. Cloning policies ensure the default policies don’t aren’t altered.

Give the policy a new name, such as Alpine Linux Package Manager in Image - Enforce Deploy. The best practice would be to add a description for future policy enforcers as well.

Alpine Linux Package Manager in Image - Enforce Deploy

Click Next
Next, deselect Build stage so that only the Deploy stage is selected.

Since certain policy criteria is specific to each stage you have to reset the policy criteria

Now, we want to target our specific deployment with an image label.

Click Next
Click on the Image contents dropdown on the right side of the browser.
Find the Image component label and drag it to the default policy criteria.
Type apk-tools under the criteria

Your policy should look like this,

In Policy behavior → Actions, click Inform and enforce + Enforce on Deploy

Click Next
Review the changes
Click Save

Now, let’s test it out! We’re going to redeploy the frontend application from earlier.

In case you forgot to run the command earlier

oc delete -f $APP_HOME/skupper-demo/frontend.yml

Now reapply the frontend service and deployment

oc apply -f $APP_HOME/skupper-demo/frontend.yml

route.route.openshift.io/frontend-patient-route created
service/frontend-service created
Error from server (Failed currently enforced policies from RHACS): error when creating "/home/lab-user/skupper-app/skupper-demo/frontend.yml": admission webhook "policyeval.stackrox.io" denied the request:
The attempted operation violated 1 enforced policy, described below:

Policy: Alpine Linux Package Manager in Image - Enforce Deploy
- Description:
    ↳ Alert on deployments with the Alpine Linux package manager (apk) present
- Rationale:
    ↳ Package managers make it easier for attackers to use compromised containers,
      since they can easily add software.
- Remediation:
    ↳ Run `apk --purge del apk-tools` in the image build for production containers.
- Violations:
    - Container 'frontend' includes component 'apk-tools' (version 2.14.6-r2)


In case of emergency, add the annotation {"admission.stackrox.io/break-glass": "ticket-1234"} to your deployment with an updated ticket number

Another option for enforcement is to use the "deployment check" CLI command.

Verify the frontend application against the policies you’ve created.

roxctl -e $ROX_CENTRAL_ADDRESS:443 deployment check --file $APP_HOME/skupper-demo/frontend.yml --insecure-skip-tls-verify

The new policy will fail the check when in enforce mode. In inform mode you will get a policy violation without the "Failed policy" error.

Congrats!

You’re now enforcing against the Alpine Linux package manager at runtime and deploy time. Let’s finish with enforcing at build-time!

Introduction to build-time policy enforcement

Build time policies for container images are guidelines that define how container images should be constructed. These policies aim to achieve several goals, including:

Security: Minimizing vulnerabilities and ensuring images are built with secure practices.
Efficiency: Reducing image size and build times for faster deployments.
Consistency: Maintaining a uniform structure and content across all images. Here are some key areas covered by build time policies:
Base Image: Specifying a minimal base image that only contains essential components.
Package Management: Encouraging the use of package managers for dependency installation and updates.
File Copying: Limiting what gets copied into the image to only required files and avoiding unnecessary bloat.
User Management: Defining a non-root user for the application process to run as.
Environment Variables: Storing sensitive information in environment variables outside the image.

In RHACS, build-time policies apply to image fields such as CVEs and Dockerfile instructions.

Prevent the Alpine Linux package manager in the frontend image from being pushed to Quay

Verify that the variables are correct to make this section smoother

echo $QUAY_USER
echo $QUAY_URL
echo $ROX_CENTRAL_ADDRESS

Make sure you are logged in to Quay

podman login $QUAY_URL -u $QUAY_USER -p sample_LSncwGJkXcayY1tUMmr8

Use the quay admin credentials, Username: sample_quayadmin & password: sample_LSncwGJkXcayY1tUMmr8. You can create unique user and group credentials in Quay for proper segmentation.

Let’s pretend as if the developers are pushing an update to the frontend application. First, pull and scan the related image.

The following command is designed to mimic and build a pipeline where a container build is going through a commit/promotion step. You download the image, scan for vulnerabilities, tag a newer version and upload to Quay.

podman pull $QUAY_URL/$QUAY_USER/frontend:0.1
roxctl --insecure-skip-tls-verify -e "$ROX_CENTRAL_ADDRESS:443" image check --image=$QUAY_URL/$QUAY_USER/frontend:0.1

--------------------------------------------------------------------------------------------------------------------------------------------------------+
|  Alpine Linux Package Manager  |   LOW    |      -       | Alert on deployments with the  |   - Image includes component   |      Run `apk --purge del      |
|         (apk) in Image         |          |              |  Alpine Linux package manager  |      'apk-tools' (version      | apk-tools` in the image build  |
|                                |          |              |         (apk) present          |           2.14.6-r3)           |   for production containers.   |
--------------------------------------------------------------------------------------------------------------------------------------------------------+
|  Alpine Linux Package Manager  |   LOW    |      -       | Alert on deployments with the  |   - Image includes component   |      Run `apk --purge del      |
|    in Image - Enforce Build    |          |              |  Alpine Linux package manager  |      'apk-tools' (version      | apk-tools` in the image build  |
|                                |          |              |         (apk) present          |           2.14.6-r3)           |   for production containers.   |
--------------------------------------------------------------------------------------------------------------------------------------------------------+
WARN:   A total of 3 policies have been violated

With the Image Check cli command you can check against the created policies

podman tag $QUAY_URL/$QUAY_USER/frontend:0.1 $QUAY_URL/$QUAY_USER/frontend:0.2
podman push $QUAY_URL/$QUAY_USER/frontend:0.2 --remove-signatures

[lab-user@bastion ~]$ podman tag $QUAY_URL/$QUAY_USER/frontend:1.0 $QUAY_URL/$QUAY_USER/frontend:1.1
podman push $QUAY_URL/$QUAY_USER/frontend:1.1 --remove-signatures
Copying blob 308102f44919 skipped: already exists
Copying blob b8d9a96d44df skipped: already exists
....
Copying config 1cbb2b7908 done   |
Writing manifest to image destination

Now RHACS hasn’t broken the command since there is no enforcement of any build policies. Letting the developer build tag and push a new image.

Let’s make a copy of the build & deploy-time policy and enforce it during the build phase.

Procedure

Navigate to Platform Configuration → Policy Management
On the Policy Management page, type Policy then Alpine Linux into the filter bar at the top.
Click on the Alpine Linux Package Manager in Image options (The three dots on the right side of the screen) and select Clone policy

Make sure to CLONE the policy

Give the policy the name Alpine Linux Package Manager in Image - Enforce Build. The best practice would be to add a description for future policy enforcers as well.

Alpine Linux Package Manager in Image - Enforce Build

Unselect the "Deploy" stage, and reset the policy criteria.
Click "Next"
Add the "image component - apk-tools" to the Rules
Click "Next"
Remove all of the exclusion scopes. You can do this by clicking the garbage tags in the scope. Exclusion scopes are limited to the build and runtime stages.
Click "Next"
Update the policy to inform and enforce while ensuring the Build stage checkbox is selected And select Enforce on Build at the bottom of the page.

Go to the Review Policy tab
Review the changes.
Click Save

Now let’s test it out!

Run the following in the terminal.

podman pull $QUAY_URL/$QUAY_USER/frontend:0.1
roxctl image check --insecure-skip-tls-verify -e "$ROX_CENTRAL_ADDRESS:443"  --image=$QUAY_URL/$QUAY_USER/frontend:0.1

WARN:   A total of 3 policies have been violated
ERROR:  failed policies found: 1 policies violated that are failing the check
ERROR:  Policy "Alpine Linux Package Manager in Image - Enforce Build" - Possible remediation: "Run `apk --purge del apk-tools` in the image build for production containers."
ERROR:  checking image failed: failed policies found: 1 policies violated that are failing the check

You should see the same violations from the previous command EXCEPT now you have a failed policy check. This would send an exit 0 command if this was run in any pipeline.

Summary

AMAZING!

Before you head to the next module. Make sure to run the following commands to redploy our example app.

POLICY_ID=$(curl -X GET \
  -H "Authorization: Bearer $ROX_API_TOKEN" \
  -H "Content-Type: application/json" \
  https://$ROX_CENTRAL_ADDRESS/v1/policies | jq -r '.policies[] | select(.name=="Alpine Linux Package Manager in Image - Enforce Deploy") | .id')

curl -X DELETE \
  -H "Authorization: Bearer $ROX_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
        "id": "$POLICY_ID"
      }' \
  https://$ROX_CENTRAL_ADDRESS/v1/policies/$POLICY_ID

oc apply -f $APP_HOME/skupper-demo/

In summary, we made use of the features provided by Red Hat Advanced Cluster Security for Kubernetes to display potential security violations in your cluster in a central dashboard. You crafted both deploy-time and runtime policies to help prevent malicious events from occurring in our cluster. Hopefully this lab has helped demonstrate to you the immense value provided by RHACS and OpenShift Platform Plus. Please feel free to continue and explore the RHACS lab environment.

On to CI/CD and Automation!