Scenario - Automatic storage management with Autopilot

In this scenario, we will configure Autopilot to grow a PVC when our virtual disk runs low on free space.

Autopilot is a rule-based engine that responds to changes from a monitoring source. Autopilot allows you to specify monitoring conditions along with actions it should take when those conditions occur.

With Autopilot, your cluster can react dynamically without your intervention to events such as:

  • Resizing PVCs when it is running out of capacity

  • Scaling Portworx storage pools to accommodate increasing usage

  • Rebalancing volumes across Portworx storage pools when they come unbalanced

In this scenario, we will use Autopilot to detect when a VM’s disk is filling up, and automatically expand that disk.

The OCP Console tab at the top of this page is already open. If prompted to log in, use username {openshift_cluster_admin_username} and password {openshift_cluster_admin_password}.

Configure our Autopilot Rule

Task 1: Create a new VM for Autopilot

Let’s create a new VM to test out Autopilot. This VM will have an additional data disk. We will mount this disk and start filling it up with data. We will also use the secret holding the public SSH key that we previously created.

cat << EOF | oc apply -f -
---
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: centos-stream9-autopilot
  namespace: vmtest
spec:
  dataVolumeTemplates:
  - metadata:
      name: centos-stream9-autopilot-ds-centos-stream9
    spec:
      sourceRef:
        kind: DataSource
        name: centos-stream9
        namespace: openshift-virtualization-os-images
      storage:
        storageClassName: px-rwx-block-kubevirt
        resources: {}
  - metadata:
      name: centos-stream9-autopilot-data-disk
    spec:
      preallocation: false
      source:
        blank: {}
      storage:
        resources:
          requests:
            storage: 5Gi
        storageClassName: px-rwx-block-kubevirt
  instancetype:
    name: u1.medium
  preference:
    name: centos.stream9
  runStrategy: Always
  template:
    spec:
      domain:
        devices: {}
        resources: {}
      terminationGracePeriodSeconds: 180
      volumes:
      - name: centos-stream9-autopilot-ds-centos-stream9
        dataVolume:
          name: centos-stream9-autopilot-ds-centos-stream9
      - name: centos-stream9-autopilot-data-disk
        dataVolume:
          name: centos-stream9-autopilot-data-disk
      - name: cloudinitdisk
        cloudInitNoCloud:
          userData: |-
            #cloud-config
            user: cloud-user
      accessCredentials:
      - sshPublicKey:
          propagationMethod:
            noCloud: {}
          source:
            secret:
              secretName: authorized-keys
EOF

Task 2: Create the Autopilot Rule

Create the Autopilot rule by running the following:

cat << EOF | oc apply -f -
---
apiVersion: autopilot.libopenstorage.org/v1alpha1
kind: AutopilotRule
metadata:
  name: volume-resize
spec:
  ##### selector filters the objects affected by this rule given labels
  selector:
    matchLabels:
      expand: "true"
  ##### namespaceSelector filters the namespaces affected by this rule
  namespaceSelector:
    matchLabels:
      expand: "true"
  ##### conditions are the symptoms to evaluate. All conditions are AND'ed
  conditions:
    # volume usage is greater than 30%
    expressions:
    - key: "100 * (px_volume_usage_bytes / px_volume_capacity_bytes)"
      operator: Gt
      values:
        - "30"
    for: 5
  ##### action to perform when condition is true
  actions:
  - name: openstorage.io.action.volume/resize
    params:
      # resize volume by scalepercentage of current size
      scalepercentage: "100"
      # volume capacity should not exceed 20GiB
      maxsize: "20Gi"
EOF

The rule will:

  • selector: Target PVCs with the Kubernetes label expand: true

  • namespaceSelector: Only look for those PVCs in namespaces that also have the label expand: true

  • conditions: Monitor whether capacity usage grows above 30%

  • actions: Automatically grow the volume by 100% of its current size (doubling it) when usage above 30% is detected

  • maxsize: Never grow the volume to more than 20Gi

Task 3: Label our namespace and Virtual Machine PVC

Our Autopilot rule only applies to PVCs that have the expand=true label, in namespaces that also have the expand=true label. In production you would usually include these labels in your namespace and PVC manifests. For this lab, we will apply them to resources that already exist.

Label the vmtest namespace:

oc label namespace vmtest expand=true --overwrite

Then label our virtual machine’s data disk PVC:

oc label pvc centos-stream9-autopilot-data-disk \
  expand=true --overwrite -n vmtest
oc get pvc centos-stream9-autopilot-data-disk -n vmtest

Take note of the size of our PVC!

Task 4: Format and mount our data disk

Because this is a new VM, let’s format our data disk and mount it to /data

# Wait for the VM to boot
until virtctl ssh cloud-user@vm/centos-stream9-autopilot -n vmtest -i ~/.ssh/id_rsa -t "-o StrictHostKeyChecking=no" -c 'lsblk'; do
    echo "waiting for VM to boot"
    sleep 10
done

# Set up the filesystem and mount the disk as /data
virtctl ssh cloud-user@vm/centos-stream9-autopilot -n vmtest \
  -i ~/.ssh/id_rsa \
  -t "-o StrictHostKeyChecking=no" \
  -c '(echo g; echo n; echo 1; echo ; echo ; echo w) | sudo fdisk /dev/vdb && sudo mkfs.ext4 /dev/vdb1 && sudo mkdir /data && sudo mount /dev/vdb1 /data'

Fill the disk and watch Autopilot respond

We will use the shred command to write data to our virtual machine’s data disk.

We could of course log in to our VM though the console, but virtctl lets us run commands in the VM through the OpenShift control plane without configuring any networking for the VM.

Task 5: Start filling the disk

Let’s execute a command to write data to the /data disk inside of our virtual machine:

virtctl ssh cloud-user@vm/centos-stream9-autopilot -n vmtest \
  -i ~/.ssh/id_rsa \
  -t "-o StrictHostKeyChecking=no" \
  -c 'sudo touch /data/file; sudo shred -n 1 -s 4G /data/file'

Task 6: Observe the Portworx Autopilot events

Run the following command to observe the state changes for Portworx Autopilot:

watch oc get events --field-selector \
  involvedObject.kind=AutopilotRule,involvedObject.name=volume-resize \
  --all-namespaces --sort-by .lastTimestamp -o custom-columns=MESSAGE:.message
It takes some time for Autopilot to detect that the disk has crossed the usage threshold in its rule. You may not see all of the states below right away. Wait until the ActiveActionsTaken event is displayed.

You will see Portworx Autopilot move through the following states as it monitors volumes and takes actions defined in Portworx Autopilot rules:

  • Initializing: Detected a volume to monitor via applied rule conditions

  • Normal: Volume is within defined conditions and no action is necessary

  • Triggered: Volume is no longer within defined conditions and action is necessary

  • ActiveActionsPending: Corrective action is necessary but not executed yet

  • ActiveActionsInProgress: Corrective action is under execution

  • ActiveActionsTaken: Corrective action is complete

Once you see ActiveActionsTaken in the event output, press CTRL+C to exit the watch command.

Task 7: Verify the Volume Expansion

Now let’s take a look at our PVC - note the automatic expansion of the volume occurred with no human interaction and no application interruption:

oc get pvc -n vmtest
You should now see the data disk size has doubled.

Autopilot expanded our PersistentVolume, just as our rule specified. If this were a container running in a pod, this would be the last step and the process would be fully unattended. However, a virtual machine has its own operating system and its own partition table and filesystem, so we need to grow the partition and filesystem inside the VM to use the new space.

Let’s expand the virtual machine’s filesystem:

virtctl ssh cloud-user@vm/centos-stream9-autopilot -n vmtest \
  -i ~/.ssh/id_rsa -t "-o StrictHostKeyChecking=no" \
  -c 'yes Fix | sudo parted --script --fix /dev/vdb print ; yes | sudo parted ---pretend-input-tty /dev/vdb resizepart 1 100% ; sudo resize2fs /dev/vdb1'

We can now observe the free space in our virtual machine by running:

virtctl ssh cloud-user@vm/centos-stream9-autopilot -n vmtest -i ~/.ssh/id_rsa -t "-o StrictHostKeyChecking=no" -c 'df -h'

Notice the size of the data disk mounted at /data

You can also see the expansion in the OpenShift Console. Navigate to Virtualization > VirtualMachines, click on centos-stream9-autopilot, and select the Portworx tab to see the VM’s volumes, their capacity, and recent events.

It can take a few minutes for the new capacity to appear in the console. If you don’t see it right away, wait a minute and refresh the page.

You’ve just configured Portworx Autopilot and observed how it can perform automated capacity management based on rules you configure, and be able to "right-size" your underlying persistent storage as it is needed!

In the next lab, we will see how Autopilot can also adjust a volume’s performance limits as its size changes.