Kernel Tracing (ebpf)

This unit deals with system performance observability.

eBPF (The Extended Berkeley Packet Filter) is an in-kernel virtual machine that allows code execution in the kernel space, in a restricted sandbox environment with access to a limited set of functions. There are numerous components shipped by Red Hat that utilize the eBPF virtual machine. Each component is in a different development phase, and thus not all components are currently fully supported. Those that are not supported are available as a Technology Preview and are not intended for production use. For information on Red Hat scope of support for Technology Preview features, see: Technology Preview Features Support Scope.

In this unit, we will be focusing on the kernel tracing offering shipped as bcc-tools (BPF Compiler Collection Tools). We will also touch on bpftrace which is a tool to compile scripts to BPF-bytecode.

For these exercises, you will primarily be using the host node2 as user root. However, these exercises do REQUIRE a second terminal session to run commands on other hosts. Please pay careful attention to what commands to run on different hosts.

From host bastion, ssh to node2.

Use sudo to elevate your privileges.

Verify that you are on the right host for these exercises.

You’ll need your second terminal for these exercises. So just keep your eyes on the instructions for what to run where.

You are now ready to proceed with these exercises.

There are just over 100 tools shipped in the bcc-tools package. A few things of note:

The execsnoop script monitors all calls to execve() and catches all processes that follow the fork→exec sequence, as well as processes that re-exec() themselves. Processes that fork() but do not exec() won’t be caught by this script.

Continuing on host node2, to run execsnoop as follows:

In your second terminal having established yourself as root, run the following command:

Again, this command should run for about 10 seconds and then exit. Before porceeding to the results you can either hit Ctrl-C in the primary terminal OR send a sigkill to the process to terminate execsnoop.

In the primary terminal, you should see output similar to:

This shows you all the processes that ran exec() during that ssh login, their PID, their parent PID, their return code, and the arguments that were sent to the process. You could keep monitoring this for quite some time to catch potential bad actors on the system.

Similar in nature to execsnoop, mountsnoop traces the mount() and umount() syscalls which show processes that are attempting to mount (or unmount) filesystems.

To run this tool, execute the following and give it a couple of seconds to get set up:

In your second terminal, having established yourself as root, let’s try to unmount something we know can NOT be unmounted. For this, we’ll pick the root filesystem '/'.

Taking a look at the terminal running mountsnoop, we see:

This shows us that the mount is busy and cannot be unmounted.

Now let’s try to unmount a filesystem that we should be able to unmount. But before doing so, look at the mount options to ensure we can restore it correctly. On node2 run the following:

Now proceed to umount /dev/shm on node2

Back to the primary terminal and you should see the following:

The umount command succeeded.

Proceed to restore the /dev/shm mount as follows:

This is the last step related to mountsnoop. Before proceeding to the results you can either hit Ctrl-C in the primary terminal OR send a sigkill to the process to terminate mountsnoop.

Finally, back to the primary terminal you should see the following:

This shows us that the mount succeeded and all the options that were passed into the system call.

As you can see, the mountsnoop tool is very useful for seeing what processes are calling the mount and umount system calls and what the results of those calls are.

The cachestat tool traces kernel page cache functions and prints a summary every five seconds to aid you in workload characterization.

To run this tool, execute the following and give it a couple of seconds to get set up:

Let’s now run a little exercise that will generate some i/o. This script flushes the cache and then runs a series of dd commands.

This command should run for a few seconds and then exit. Before porceeding to the results you can either hit Ctrl-C in the primary terminal OR send a sigkill to the process to terminate cachestat.

In the cachestat window, you should output similar to:

This shows that we had 1216 page cache misses during a five second period while running the above loop but, during that same period, there were 16016 hits, indicating great performance from the page cache.

This tool is a swiss army knife allowing you to specify functions to trace and messages to be printed when certain conditions are met.

bpftrace makes use of BCC for interacting with the Linux BPF system as well as existing Linux tracing capabilities:

Let us start this educational journey by examining what system calls a simple command executes.

To begin, we need a test file to use as our target.

The command we will evaluate is cat /tmp/workshop.tmp and we will use strace to list the system calls made and with what frequency (count).

What we see is that our command used the 'openat' systemcall 31 times. Among those calls, 13 resulted in errors but are likely harmless attempts to open files that do not exist.

This concludes the exercises related to EBPF.

Time to finish this unit and return the shell to its home position.

You can find more information: