CI/CD and Automation

Module goals

Trigger build-time violations for images and deployments on the command line and CI/CD pipelines
Enforce a container image vulnerability violation at build time
Codify an image vulnerability violation enforcement in a pipeline
Explore warnings about deployment attributes from the command line

Implementing Security Checks into your CI/CD pipeline

Implementing CI/CD security in container builds involves more than just securing your contianer images and your deployment manifests. There are several best practices to ensure that your containerized applications are secure throughout the development and deployment lifecycle.

Access Control: Use strong authentication and authorization mechanisms. Implement role-based access control (RBAC) to limit who can modify the pipeline and deploy applications.
Audit Logging: Enable and regularly review audit logs to monitor access and changes to the CI/CD pipeline.
Official and Verified Images: Always start with official images from trusted sources (e.g., Docker Hub, Red Hat, Ubuntu).
Minimal Base Images: Use minimal base images to reduce the attack surface (e.g., alpine).
Static Analysis: Integrate static code analysis tools (e.g., SonarQube, Snyk) into the pipeline to detect vulnerabilities in the code.
Image Scanning: Use tools like Clair, Anchore, or Trivy to scan container images for known vulnerabilities before deploying them.
Pin Dependencies: Specify exact versions of dependencies to avoid introducing unverified or vulnerable components.
Regular Updates: Keep dependencies and base images updated to their latest versions to benefit from security patches.
Environment Variables: Avoid hardcoding secrets in the source code. Use environment variables or secret management tools.
Secret Management Tools: Use tools like HashiCorp Vault, AWS Secrets Manager, or Kubernetes Secrets to securely manage and inject secrets.
Policy as Code: Use tools like Open Policy Agent (OPA) to define and enforce security policies as code within your CI/CD pipeline.
Compliance Checks: Automate compliance checks to ensure all deployments adhere to security standards and policies.

Now, RHACS can be implemented into any CI/CD process as a step in the build-deploy-run lifecycle. RHACS is able to acheive this through the roxctl CLI and the admission controller. RHACS can implement those image scanning, env variables and compliance checks into those pipelines.

Let’s take a look at a very simple CI/CD pipeline in OpenShift would look like.

Review a basic CI/CD pipeline

In this section, we will review a simple pipeline that has been configured in Red Hat^® OpenShift Pipelines.

Red Hat^® OpenShift Pipelines is a Kubernetes-native CI/CD (Continuous Integration/Continuous Deployment) solution that automates the process of building, testing, and deploying applications. It leverages Tekton, a powerful and flexible open-source framework, to create cloud-native pipelines that integrate seamlessly with OpenShift and other Kubernetes environments. This enables developers to define and manage their build and deployment workflows as code, ensuring consistency, scalability, and repeatability across the software development lifecycle.

Access the Red Hat^® OpenShift Container Platform (OCP) web console

Head over to the OpenShift Console. If you don’t have access follow the steps below. If you do proceed to the following section.

First, make sure you can access the Red Hat^® OCP console web console.

Procedure

Log into the OCP console at {web_console_url}
Click the rhsso option

Enter the OCP credentials

User:

{openshift_admin_user}

Password:

{openshift_admin_password}

Enter the OpenShift username {openshift_admin_user} and password: {openshift_admin_password}

You should be directed to the RHACM dashboard. RHACM is intergrated into the OpenShift dashboard.

To get to your clusters local OpenShift dashboard click the top dropdown and select local-cluster

Click on the Pipelines tab on the left side of the page and select "Pipelines"

You should see all of the default pipelines that Red Hat OpenShift and it’s operators use to update or fetch information. Most of these have not been run and are simply configuration files.

The pipeline we are looking for is called rox-pipeline and you can either find it and select it or you can change the Project to "pipeline-demo".

We haven’t run the pipeline yet so let’s start one.

Click the three dots on the right side of the screen and select start

This pipeline works by taking and image that you have created and passing it through a "roxctl image scan" and a "roxctl image check" process.

Let’s use the ubuntu:latest image that we used in the previous module.

The pipeline should fail.

Why do you think the pipeline failed?

Let’s look at the logs.

Take a look at the log snippet on the bottom right of the page.

curl: (6) Could not resolve host: ROX_CENTRAL_ENDPOINT
Getting roxctl
chmod: cannot access './roxctl': No such file or directory
/tekton/scripts/script-0-ftdft: line 5: ./roxctl: No such file or directory

This snippet is talling us that that are variable that are incorrect. With RHACS you will need API access to query central for results.

Let’s take a look at the secret file necessary for OpenShift Pipelines.

Run the following in the terminal.

export TUTORIAL_HOME="$(pwd)/demo-apps"
ls $TUTORIAL_HOME/openshift-pipelines

[lab-user@bastion pipeline]$ ls $TUTORIAL_HOME/openshift-pipelines
pipeline  secrets  tasks

As you can see, the pipelines are broken up into one pipeline, with three tasks that need a secret file. Let’s take a look at what variables are needed for the pipeline.

Run the following in the terminal.

cat $TUTORIAL_HOME/openshift-pipelines/secrets/rox-secrets.yml

[lab-user@bastion pipeline]$ cat $TUTORIAL_HOME/openshift-pipelines/secrets/rox-secrets.yml
apiVersion: v1
stringData:
  rox_central_endpoint: "$ROXCTL_CENTRAL_ENPOINT"
  # The address:port tuple for StackRox Central (example - rox.stackrox.io:443)
  rox_api_token: "$API_TOKEN"
  # StackRox API token with CI permissions
  # Refer to https://help.stackrox.com/docs/use-the-api/#generate-an-access-token
kind: Secret
metadata:
  name: roxsecrets
  namespace: pipeline-demo

Ok. So need the RHACS Central endpoint and a API token with the right permissions. Luckily we have the RHACS Central endpoint handy.

Yours is {acs_route} but you can run the following command to update the rox-secrets file with your unique route.

Run the following in the terminal.

ACS_URL=$(oc -n stackrox get route central -o jsonpath='{.spec.host}')
ACS_URL_PORT=$(echo "$ACS_URL" | sed 's/$/:443/')
sed -i "s|ROXCTL_CENTRAL_ENDPOINT|$ACS_URL_PORT|g" $TUTORIAL_HOME/openshift-pipelines/secrets/rox-secrets.yml

Verify by running the following command

cat $TUTORIAL_HOME/openshift-pipelines/secrets/rox-secrets.yml

[lab-user@bastion pipeline]$ cat $TUTORIAL_HOME/openshift-pipelines/secrets/rox-secrets.yml
apiVersion: v1
stringData:
  rox_central_endpoint: "central-stackrox.apps.cluster-mjh4d.sandbox1834.opentlc.com:443"
  # The address:port tuple for StackRox Central (example - rox.stackrox.io:443)
  rox_api_token: "API_TOKEN"
  # StackRox API token with CI permissions
  # Refer to https://help.stackrox.com/docs/use-the-api/#generate-an-access-token
kind: Secret
metadata:
  name: roxsecrets
  namespace: pipeline-demo
type: Opaque

Get an API token from RHACS

Just like the title says.

Procedure

Head back to the RHACS dashboard and on Platform Configuration → Integrations and scroll to the bottom of the page.

Click on *API Token.
Click on Generate token
Give your token a name, select Continuous Integration as the Role and set an expiration date and hit Generate

The role of CI has read only access to the RHACS Central Services. You can setup other roles but there are a few default roles like Admin, Analyst and Continuous Integration.

Now either save the API Token into the $TUTORIAL_HOME/openshift-pipelines/secrets/rox-secrets.yml file and replace API_TOKEN or export the variable and use SED with the following commands.

export ACS_API=

Then run

sed -i "s|API_TOKEN|$ACS_API|g" $TUTORIAL_HOME/openshift-pipelines/secrets/rox-secrets.yml

And verify that your file is ready by running the following and getting a similar output.

cat $TUTORIAL_HOME/openshift-pipelines/secrets/rox-secrets.yml

[lab-user@bastion pipeline]$ cat $TUTORIAL_HOME/openshift-pipelines/secrets/rox-secrets.yml
apiVersion: v1
stringData:
  rox_central_endpoint: "central-stackrox.apps.cluster-mjh4d.sandbox1834.opentlc.com:443"
  # The address:port tuple for StackRox Central (example - rox.stackrox.io:443)
  rox_api_token: "eyJhbGciOiJSUzI1NiIsImtpZCI6Imp3dGswIiwidHlwIjoiSldUIn0.eyJFeHBpcmVBdCI6IjIwMjQtMDUtMjRUMDQ6MDA6MDBaIiwiYXVkIjoiaHR0cHM6Ly9zdGFja3JveC5pby9qd3Qtc291cmNlcyNhcGktdG9rZW5zIiwiZXhwIjoxNzE2NTIzMjAwLCJpYXQiOjE3MTY1MTA4MzAsImlzcyI6Imh0dHBzOi8vc3RhY2tyb3guaW8vand0IiwianRpIjoiMTMxNmMwMGQtM2Q4Mi00NTVmLWJmNzktMzYyNzlmZTc5MWY1IiwibmFtZSI6ImNpY2QtdG9rZW4iLCJyb2xlcyI6WyJDb250aW51b3VzIEludGVncmF0aW9uIl19.Uxz2w8d_PPwS85bL4IuchaXVybBM80fSYQ2X6zAm69fNv72unXq_tne0amsrdbSlbzqE-MY9xKoLLfp0O7Y4jh0rN10g9G301M4Ure0uoXRe5X62Iso3_MBIDMY-JiZCqZ7dnHChgRj1bt7BOsLS1MOK-uGXlzFZlGDXZ_TVk0yqkyJUu-7oIq1Rd-UZJzL8FFkAUwPra33gQ_3bJJeZvbNmxFKalXoEhuLzOmrThGXqLG5Pnp5lg-6OOsFVfWbbe-HzeXAkY1qWD4GSPpaeiuqPXzmdc6N7EElpLyZzXyEUSQfqZc1SY0Rqe-8vR_qD0sk8Z8bwmbFXZ0lpyHVkLSlLBiyqeIuT8ReNPMxtYjFXnbHvrLy518TVPZYCleoz90gRuPNvkMjpp3rdXAodFNwjxW7-D8MxpMSlzMUTsoO9DiRM28ER5mU2OC5i6qeGhz0mTkwBtkmLW6Rcc54Xm6E2jPOWAi4PzK1CNGLlozfyQtCyZ0ljJ1aYOwe44Z-kEeYkd9QSjJIt2X4w4kpykimn03DEacfR-oq_2smI5J-PdcR_fUs6SZw7BdlTMx8rZcnnNDIAbOooinb0ggpDYjk2e2fbVe-ZWFtxidGApOkTDPVHiYzYqgBWCSWrqEK70spg7YTnvsuQC22DyEKRhfw7ikj5qZtO7H9L_-MfrdM"
  # StackRox API token with CI permissions
  # Refer to https://help.stackrox.com/docs/use-the-api/#generate-an-access-token
kind: Secret
metadata:
  name: roxsecrets
  namespace: pipeline-demo
type: Opaque

Lastly apply the following and head over to the OpenShift Console

oc apply -f $TUTORIAL_HOME/openshift-pipelines/secrets/rox-secrets.yml

[lab-user@bastion pipeline]$ oc apply -f $TUTORIAL_HOME/openshift-pipelines/secrets/rox-secrets.yml
secret/roxsecrets configured

Let’s try again

Procedure

Let’s make sure that you are looking at the pipeline we ran before.

Click the Actions dropdown and select Rerun. Then relax for a few seconds and grab a sip of water.

What happened? Did you expect this behavior?

Check the log snippet in bottom right of the page.

--------------------------------+
WARN:	A total of 4 policies have been violated
ERROR:	failed policies found: 1 policies violated that are failing the check
ERROR:	Policy "Ubuntu Package Manager in Image - Build-time" - Possible remediation: "Run `dpkg -r --force-all apt apt-get && dpkg -r --force-all debconf dpkg` in the image build for production containers."
ERROR:	checking image failed after 3 retries: failed policies found: 1 policies violated that are failing the check

So it looks like our build policy from the previous exercise impacted the ability to complete this pipeline run.

A task to complete on your own.

Here is your mission

Should you choose to accept it

Turn off the enforcement of the build lifecycle policy blocking this pipeline and rerun the pipeline.

Do not disable the policy. Disable the enforcement.

Your mission is completed with a successful pipeline run.

What would you do?

How would you implement roxctl into your pipelines?
Would you have enforcement off at the beginning?
Are there any organizational policies that should be enforced in all pipelines?

Summary

THUMBS UP!

In this lab, you review a basic CI/CD use case that implemented the build policy from the previous module. You then adjusted the policy so the pipeline would succeed.

Congrats! On to Compliance!