How to Observe a Linux System with eBPF

eBPF (extended Berkeley Packet Filter) lets you run sandboxed programs inside the Linux kernel without changing kernel source or loading modules. For performance work that means you can trace syscalls, scheduler events, network packets, and hardware counters with near-zero overhead—while the system runs production workloads. This guide covers the two most practical entry points: BCC tools (a curated library of Python-fronted scripts) and bpftrace (a high-level tracing language for one-liners and short scripts). You need a kernel ≥ 5.8 for the best experience; anything ≥ 4.18 will run most examples.

Prerequisites and Installation

Debian / Ubuntu

sudo apt update
sudo apt install -y bpftrace bpfcc-tools linux-headers-$(uname -r)

BCC tools install as /sbin/opensnoop-bpfcc, /sbin/execsnoop-bpfcc, and so on—note the -bpfcc suffix. Some Ubuntu versions also package them without the suffix under bpfcc-tools; check ls /sbin/*bpf*.

Fedora / RHEL 9 / Rocky 9

sudo dnf install -y bpftrace bcc-tools kernel-devel-$(uname -r)

On RHEL 9 you may need to enable the CodeReady Builder (CRB) repo first: sudo subscription-manager repos --enable codeready-builder-for-rhel-9-x86_64-rpms. BCC tools land in /usr/share/bcc/tools/.

Arch Linux

sudo pacman -S bpf bpftrace bcc bcc-tools

What to Look at First: the USE Method, eBPF-style

Before reaching for a specific tool, pick a resource category—CPU, memory, storage, network—and ask: Utilization, Saturation, Errors. The tools below map directly onto those questions. Start broad (system-wide latency histograms), then narrow (per-process, per-function).

bpftrace One-Liners

bpftrace compiles a small program, attaches it to a probe, and tears it down when you interrupt it. All one-liners below run as root (or with CAP_BPF + CAP_PERFMON on kernels ≥ 5.8).

Syscall counts by process

sudo bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }'

Press Ctrl-C after a few seconds. Output is a sorted map of process names to syscall counts. A process dominating the list is worth investigating further.

Block I/O latency histogram

sudo bpftrace -e '
tracepoint:block:block_rq_issue   { @start[args->sector] = nsecs; }
tracepoint:block:block_rq_complete {
  @us = hist((nsecs - @start[args->sector]) / 1000);
  delete(@start[args->sector]);
}'

This produces a log2 histogram of block request latency in microseconds. Latency above ~1 ms for a supposedly fast NVMe drive indicates queue depth problems or thermal throttling.

New process execution

sudo bpftrace -e 'tracepoint:syscalls:sys_enter_execve { printf("%s → %s\n", comm, str(args->filename)); }'

Useful for catching short-lived processes that never appear in top or ps.

TCP connection latency (connect time)

sudo bpftrace -e '
kprobe:tcp_v4_connect       { @start[tid] = nsecs; }
kretprobe:tcp_v4_connect /retval == 0/ {
  @ms = hist((nsecs - @start[tid]) / 1000000);
  delete(@start[tid]);
}'

CPU scheduler off-CPU time

sudo bpftrace -e '
tracepoint:sched:sched_switch /prev_state == TASK_INTERRUPTIBLE/ {
  @off[prev_comm] = hist(nsecs);
}'

High off-CPU time in a process that should be compute-bound points to lock contention or unexpected I/O waits.

Page fault rate

sudo bpftrace -e 'software:page-fault:1 { @[comm] = count(); } interval:s:5 { print(@); clear(@); }'

BCC Tools: the Pre-Built Toolkit

BCC ships ~70 production-ready tools. These are the ones to reach for first.

opensnoop — file opens in real time

# Debian/Ubuntu
sudo opensnoop-bpfcc

# Fedora / Arch
sudo /usr/share/bcc/tools/opensnoop

Add -p <PID> to limit to one process. Watching what a misbehaving daemon opens often reveals misconfiguration faster than strace.

execsnoop — catch short-lived processes

sudo execsnoop-bpfcc

Output includes PID, PPID, exit code, and the full argv. Far lower overhead than auditd for this specific task.

biolatency — block I/O latency histogram

sudo biolatency-bpfcc -D 10 1

The -D flag breaks latency down per disk. Compare the p99 bucket against your storage spec sheet.

tcplife — TCP session summary

sudo tcplife-bpfcc

Prints one line per closed TCP connection: PID, process name, local and remote address/port, duration in ms, and bytes transferred. Invaluable for finding connections that open and close rapidly (connection pool exhaustion, DNS storms).

profile — CPU flame graph data

sudo profile-bpfcc -F 99 -f 30 > stacks.txt

Samples stack traces at 99 Hz for 30 seconds and outputs folded stacks. Feed this directly to Brendan Gregg's flamegraph.pl:

git clone --depth 1 https://github.com/brendangregg/FlameGraph
./FlameGraph/flamegraph.pl stacks.txt > cpu.svg

Open cpu.svg in a browser. Wide plateaus in the flame graph are where CPU time is actually spent.

funclatency — latency inside any kernel or user function

# Kernel function — e.g., VFS read
sudo funclatency-bpfcc vfs_read

# User function — e.g., malloc in libc
sudo funclatency-bpfcc c:malloc

llcstat — last-level cache hit rate by process

sudo llcstat-bpfcc 5

A hit rate below ~85% on a compute workload suggests cache thrashing; consider CPU pinning or NUMA awareness.

Verifying eBPF is Working

sudo bpftrace -e 'BEGIN { printf("eBPF OK, kernel %s\n", system("uname -r")); exit(); }'

A clean output (no errors about missing BTF or locked memory limits) means you are ready. Also check:

ls /sys/kernel/btf/vmlinux   # must exist for CO-RE tools
ulimit -l                    # locked memory; raise if tools fail with EPERM

If /sys/kernel/btf/vmlinux is absent, your kernel was built without CONFIG_DEBUG_INFO_BTF. BCC tools that compile BPF at load time still work; bpftrace CO-RE one-liners will not.

Troubleshooting

• "failed to load BTF from kernel" — kernel lacks CONFIG_DEBUG_INFO_BTF=y. Use BCC (runtime compilation) instead of bpftrace CO-RE mode, or boot a distro kernel (Ubuntu HWE, Fedora, Arch kernels all have it).
• "cannot allocate memory in static TLS block" on older glibc — prepend LD_PRELOAD= (empty) or upgrade to glibc ≥ 2.33.
• Tools hang without output — the tracepoint or kprobe you are targeting may not fire. Confirm the event exists: sudo bpftrace -l 'tracepoint:block:*'.
• Permission denied even as root — check /proc/sys/kernel/perf_event_paranoid. Set it to -1 temporarily: echo -1 | sudo tee /proc/sys/kernel/perf_event_paranoid. In containers, eBPF requires CAP_BPF and CAP_PERFMON at minimum, plus an unrestricted seccomp profile.
• Missing kernel headers — BCC compiles BPF C at load time and needs headers matching the running kernel. After a kernel update, reinstall linux-headers-$(uname -r) before rebooting or right after.