Untitled :: OpenShift Security Roadshow

Black Hat - RHACS Log4Shell Vulnerability Demonstration

In this lab, you demonstrate how to quickly stop workloads with log4shell vulnerabilities from deploying using Red Hat^® Advanced Cluster Security for Kubernetes (RHACS). You also learn how to locate workloads with log4shell.

Goals

Prevent the execution of vulnerable deployments
Report and resolve the violation

Ensure a clean environment

Start by ensuring you have a clean lab environment. Run the following command to reset any changes others may have made.

oc delete project log4shell
rm ~/deploy.yaml
ROX_CENTRAL_ADDRESS=$(oc -n stackrox get route central -o jsonpath='{.spec.host}')
POL_ID=$(curl -k -H  "Authorization: Bearer $ROX_API_TOKEN" "https://$ROX_CENTRAL_ADDRESS/v1/policies" | jq -r '.policies[] | select(.isDefault|not) | .id')
curl -k -X DELETE -H "Authorization: Bearer $ROX_API_TOKEN"  "https://$ROX_CENTRAL_ADDRESS/v1/policies/$POL_ID"

Deploy the log4shell infected container image

The next step is to deploy the log4shell laden container image to the OpenShift cluster.

Procedure

Start by creating a new project and deploying a vulnerable Log4Shell application.

oc new-project log4shell

Now, using project "log4shell" on server "https://api.cluster-fdg9s.fdg9s.sandbox674.opentlc.com:6443".

You can add applications to this project with the 'new-app' command. For example, try:

 oc new-app rails-postgresql-example

to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application:

 kubectl create deployment hello-node --image=k8s.gcr.io/serve_hostname

Next, deploy the vulnerable application by creating and applying the following deployment YAML file:

cat << EOF >deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: log4shell
  namespace: log4shell
spec:
  replicas: 1
  selector:
    matchLabels:
      deployment: log4shell
  template:
    metadata:
      labels:
        deployment: log4shell
    spec:
      containers:
      - name: log4shell
        image: quay.io/rhacs-misc/log4shell:1.0
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      restartPolicy: Always
EOF

Deploy the log4shell deployment into your new project by using the 'oc' CLI

oc create -f ./deploy.yaml

[lab-user@bastion ~]$ oc create -f ./deploy.yaml
deployment.apps/log4shell created

Ensure that the application was deployed without issues.

oc get pods -l deployment=log4shell -A

[lab-user@bastion ~]$ oc get pods -l deployment=log4shell -A
NAMESPACE   NAME                         READY   STATUS    RESTARTS   AGE
log4shell   log4shell-5d4b665947-grp8d   1/1     Running   0          63s

For the last part of this section, check to make sure that the log4shell vulnerability is in the container with a quick CLI scan using the roxctl CLI

Execute the following command in the terminal to ensure the log4shell vulnerability is present.

roxctl --insecure-skip-tls-verify -e "$ROX_CENTRAL_ADDRESS:443" image scan --image=quay.io/rhacs-misc/log4shell:1.0 --force -o table --severity=CRITICAL

misc/log4shell:1.0 --force -o table --severity=CRITICAL
Scan results for image: quay.io/rhacs-misc/log4shell:1.0
(TOTAL-COMPONENTS: 4, TOTAL-VULNERABILITIES: 3, LOW: 0, MODERATE: 0, IMPORTANT: 0, CRITICAL: 3)

----------------------------------------------------------------------------------------------------------------------------------------+
|              COMPONENT              | VERSION |      CVE       | SEVERITY |                      LINK                       | FIXED VERSION |
----------------------------------------------------------------------------------------------------------------------------------------+
| org.apache.logging.log4j:log4j-core | 2.14.1  | CVE-2021-44228 | CRITICAL | https://nvd.nist.gov/vuln/detail/CVE-2021-44228 |    2.15.0     |
+                                     +         ------------------------------------------------------------------------------------------+
|                                     |         | CVE-2021-45046 | CRITICAL | https://nvd.nist.gov/vuln/detail/CVE-2021-45046 |    2.16.0     |
----------------------------------------------------------------------------------------------------------------------------------------+
|            spring-beans             | 5.3.10  | CVE-2022-22965 | CRITICAL | https://nvd.nist.gov/vuln/detail/CVE-2022-22965 |    5.3.18     |
-------------------------------------         ------------------------------------------------------------------------------------------+
|           spring-webflux            |         | CVE-2022-22965 | CRITICAL | https://nvd.nist.gov/vuln/detail/CVE-2022-22965 |    5.3.18     |
-------------------------------------         ------------------------------------------------------------------------------------------+
|            spring-webmvc            |         | CVE-2022-22965 | CRITICAL | https://nvd.nist.gov/vuln/detail/CVE-2022-22965 |    5.3.18     |
----------------------------------------------------------------------------------------------------------------------------------------+
WARN:   A total of 3 unique vulnerabilities were found in 4 components

You can see that CVE-2021-44228 can be found in the quay.io/rhacs-misc/log4shell:1.0 container image that you just deployed.

Great job so far!

Red Hat^® Advanced Cluster Security (RHACS) web console

Red Hat Advanced Cluster Security for Kubernetes is a Kubernetes-native security platform that equips you to build, deploy, and run cloud-native applications with more security. The solution helps protect containerized Kubernetes workloads in all major clouds and hybrid platforms, including Red Hat OpenShift, Amazon Elastic Kubernetes Service (EKS), Microsoft Azure Kubernetes Service (AKS), and Google Kubernetes Engine (GKE).

Access the RHACS web console.

Procedure

Log into the RHACS console at {acs_route}
Click the "Advanced" button in your browser

Click "Proceed to {acs_route}"

Enter the RHACS credentials

RHACS Console Username:

{acs_portal_username}

RHACS Console Password:

{acs_portal_password}

Find the log4shell vulnerability in RHACS dashboard.

The next step is to use the ACS dashboard to locate the Log4shell vulnerability. The following gif will showcase how to find the vulnerability using the exact steps outlined below.

CVE-2021-44228 & CVE-2021-45046 can both be used to find the log4shell vulnerability in the dashboard.

misc log 1

Procedure

Click the Vulnerability Management Tab, then click the Workload CVE tab
Use the filter dropdown options to ensure that CVE and Name are selected.
Use CVE-2021-44228, then click the arrow or hit enter to search for the vulnerability.
Click on the 2 Deployments tab to see both the "frontend" and "log4shell" deployments affected by these CVEs.
Explore the details of the vulnerabilities present in the "log4shell" container image.

Enforce the default Log4Shell policy

Many default policies ship with RHACS. These policies notify you of violations by default but require their enforcement to enable RHACS to take action and block future deployments or scale pods to zero at runtime.

First, copy the default policy. You can do this by copying the video actions or following the Procedure listed below.

Procedure

Click on Platform Configuration → Policy Management
Click the search dropdown and select Policy
Type in Log4Shell and hit ENTER
Clone the Log4Shell: log4j Remote Code Execution vulnerability policy by clicking the three dots to the right of the policy and selecting Clone policy OR by clicking the policy, selecting actions and clicking Clone policy

Great job!

Now, let’s give the policy a new name and enforce it during the build and deploy phases.

Procedure

Give the policy an additional identifier for easier searchability. You can also leave the (COPY) label at the end.
Click Next
Click the Inform and enforce option on the Policy behavior page
Scroll down and select the Deploy and Build time enforcement behavior

This will break container builds AND not allow the deployment of the log4shell vulnerability to your clusters.

Click Next
Review the Policy criteria page. This page will outline the CVEs that we are looking to block.
Click Next
Click next on the Policy scope page. This page allows you to exclude specific clusters, namespaces and deployments. However, we do not want to do this for the log4shell vulnerability.
Review the policy and click Save

What he said ^!

Test out your policy!

Finally! It’s time try to redeploy the image that contained the log4shell vulnerability.

oc create -f ~/deploy.yaml

Examine the output and note that the deployment failed to start

[lab-user@bastion ~]$ oc create -f ~/deploy.yaml
Error from server (Failed currently enforced policies from StackRox): error when creating "/home/lab-user/deploy.yaml": admission webhook "policyeval.stackrox.io" denied the request:
The attempted operation violated 1 enforced policy, described below:

Policy: Log4Shell: log4j Remote Code Execution vulnerability - Enforcement
- Description:
 ↳ Alert on deployments with images containing the Log4Shell vulnerabilities
 (CVE-2021-44228 and CVE-2021-45046). There are flaws in the Java logging library
 Apache Log4j in versions from 2.0-beta9 to 2.15.0, excluding 2.12.2.
- Rationale:
 ↳ These vulnerabilities allows a remote attacker to execute code on the server if
 the system logs an attacker-controlled string value with the attacker's JNDI
 LDAP server lookup.
- Remediation:
 ↳ Update the log4j libary to version 2.16.0 (for Java 8 or later), 2.12.2 (for
 Java 7) or later. If not possible to upgrade, then remove the JndiLookup class
 from the classpath: zip -q -d log4j-core-*.jar
 org/apache/logging/log4j/core/lookup/JndiLookup.class
- Violations:
 - CVE-2021-44228 (CVSS 10) (severity Critical) found in component 'org.apache.logging.log4j:log4j-core' (version 2.14.1) in container 'log4shell'
 - CVE-2021-45046 (CVSS 9) (severity Critical) found in component 'org.apache.logging.log4j:log4j-core' (version 2.14.1) in container 'log4shell'


In case of emergency, add the annotation {"admission.stackrox.io/break-glass": "ticket-1234"} to your deployment with an updated ticket number

This includes the ability to bypass it in an emergency "In case of emergency, add the annotation {"admission.stackrox.io/break-glass": "ticket-1234"} to your deployment with an updated ticket number"

Review the policy violations

A complete record of the event can be found on the Violations page. Click on it and find the following enforcement notice

Procedure . Navigate to the Violation page from the left navigation bar. . Use the Filter Bar to find the Policy: Log4Shell: log4j Remote Code Execution vulnerability and select the policy name. . Explore the list of the violation events.

Summary

THUMBS UP!

You enabled Log4Shell deploy-time policy enforcement and verified that the policy prevented the log4shell container from being deployed to the OpenShift cluster.

Clean up

To prepare the lab for another user, please run the following command.

oc delete project log4shell
rm ~/deploy.yaml
ROX_CENTRAL_ADDRESS=$(oc -n stackrox get route central -o jsonpath='{.spec.host}')
POL_ID=$(curl -k -H  "Authorization: Bearer $ROX_API_TOKEN" "https://$ROX_CENTRAL_ADDRESS/v1/policies" | jq -r '.policies[] | select(.isDefault|not) | .id')
curl -k -X DELETE -H "Authorization: Bearer $ROX_API_TOKEN"  "https://$ROX_CENTRAL_ADDRESS/v1/policies/$POL_ID"

[lab-user@bastion ~]$ oc delete project log4shell
rm ~/deploy.yaml
ROX_CENTRAL_ADDRESS=$(oc -n stackrox get route central -o jsonpath='{.spec.host}')
POL_ID=$(curl -k -H  "Authorization: Bearer $ROX_API_TOKEN" "https://$ROX_CENTRAL_ADDRESS/v1/policies" | jq -r '.policies[] | select(.isDefault|not) | .id')
curl -k -X DELETE -H "Authorization: Bearer $ROX_API_TOKEN"  "https://$ROX_CENTRAL_ADDRESS/v1/policies/$POL_ID"
project.project.openshift.io "log4shell" deleted
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 31755    0 31755    0     0   258k      0 --:--:-- --:--:-- --:--:--  258k
{}

THANK YOU!

Black Hat - RHACS Log4Shell Vulnerability Demonstration

Ensure a clean environment

Deploy the log4shell infected container image

Red Hat® Advanced Cluster Security (RHACS) web console

Access the RHACS web console.

Find the log4shell vulnerability in RHACS dashboard.

Enforce the default Log4Shell policy

Test out your policy!

Review the policy violations

Summary

Clean up

Red Hat^® Advanced Cluster Security (RHACS) web console