Use bpftrace for Live Kernel Tracing

bpftrace is a high-level tracing language for Linux that compiles to eBPF bytecode and runs safely inside the kernel. It lets you answer real production questions—which syscalls are slowest, which kernel functions are hammered, where allocations are coming from—without restarting anything or loading a kernel module. Think of it as a supercharged combination of strace, perf, and SystemTap, with a syntax you can actually remember.

Installing bpftrace

Debian / Ubuntu (22.04+)

sudo apt install bpftrace

Fedora / RHEL 9+ / Rocky 9+

sudo dnf install bpftrace

On RHEL 8, enable the CodeReady Linux Builder repo first; bpftrace landed late in that cycle and the packaged version is old—consider building from source or using the upstream container image instead.

Arch Linux

sudo pacman -S bpftrace

You need kernel 4.9+ for basic eBPF; 5.8+ unlocks ring buffers and CO-RE (Compile Once, Run Everywhere). Check with uname -r. Any current LTS kernel is fine.

bpftrace --version
bpftrace -l 'tracepoint:syscalls:*' | head -20

The second command lists available tracepoints. If it returns output, your kernel BTF headers are in place and you are ready to go.

Core Concepts in Two Minutes

• Probes — attachment points: kprobe, kretprobe, tracepoint, uprobe, usdt, profile, interval.
• Predicates — filter clauses in /condition/ that gate whether the action body runs.
• Built-in variables — pid, tid, comm, nsecs, cpu, retval (in ret probes), args (structured access to tracepoint fields).
• Maps — kernel-side hash tables prefixed with @. They aggregate without copying every event to userspace.
• hist() / lhist() — power-of-2 and linear histograms, the right tool for latency distribution.

Essential One-Liners

Count syscalls by process name

sudo bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }'

Let it run for a few seconds, then press Ctrl-C. The map prints automatically on exit, ranked by call count.

Trace open() calls with filenames

sudo bpftrace -e '
tracepoint:syscalls:sys_enter_openat
{ printf("%s %s\n", comm, str(args->filename)); }'

args->filename is a kernel pointer; str() copies it safely to userspace. Without the cast bpftrace prints a raw address.

Measure read() latency per process

sudo bpftrace -e '
tracepoint:syscalls:sys_enter_read { @start[tid] = nsecs; }
tracepoint:syscalls:sys_exit_read  /@start[tid]/
{
  @latency_us[comm] = hist((nsecs - @start[tid]) / 1000);
  delete(@start[tid]);
}'

The predicate /@start[tid]/ prevents a spurious subtraction if bpftrace attached mid-syscall. The histogram buckets are in microseconds.

Profile CPU stacks at 99 Hz

sudo bpftrace -e '
profile:hz:99 /pid > 0/
{ @[kstack, ustack, comm] = count(); }'

Filter with /comm == "nginx"/ to target a single process. Pipe the output into FlameGraph scripts for a visual breakdown.

Watch block I/O latency

sudo bpftrace -e '
tracepoint:block:block_rq_issue   { @start[args->sector] = nsecs; }
tracepoint:block:block_rq_complete
/@start[args->sector]/
{
  @io_lat_us = hist((nsecs - @start[args->sector]) / 1000);
  delete(@start[args->sector]);
}'

Working with kprobes and kretprobes

Tracepoints are stable ABI; kprobes attach directly to kernel function entry or return and can break between kernel versions. Use them when no tracepoint exists for what you need.

Trace VFS read entry

sudo bpftrace -e '
kprobe:vfs_read
{ printf("%s calling vfs_read, count=%d\n", comm, (int)arg2); }'

Arguments are positional (arg0–argN) rather than named. Cross-reference with the kernel source or bpftrace -lv 'kprobe:vfs_read' to confirm the signature.

Measure vfs_read return latency

sudo bpftrace -e '
kprobe:vfs_read    { @ts[tid] = nsecs; }
kretprobe:vfs_read /@ts[tid]/
{
  @vfs_lat_us = hist((nsecs - @ts[tid]) / 1000);
  delete(@ts[tid]);
}'

retval in a kretprobe holds the function return value; negative values are errno codes. Add /retval >= 0/ to exclude errors from the latency histogram.

Syscall Latency Analysis in Practice

The pattern is always the same: record nsecs at entry keyed by tid, compute the delta at exit, bucket it with hist(). Here is a complete script you can save and reuse:

cat > /usr/local/bin/syscall-lat.bt <<'EOF'
#!/usr/bin/env bpftrace
/*
 * syscall-lat.bt  -  latency histogram for a single syscall
 * Usage: syscall-lat.bt <syscall_name>
 * Example: syscall-lat.bt write
 */

BEGIN { printf("Tracing %s latency... Ctrl-C to stop.\n", $1); }

tracepoint:syscalls:sys_enter_$1
/comm != "bpftrace"/
{ @enter[tid] = nsecs; }

tracepoint:syscalls:sys_exit_$1
/@enter[tid]/
{
  @latency_us = hist((nsecs - @enter[tid]) / 1000);
  delete(@enter[tid]);
}
EOF
chmod +x /usr/local/bin/syscall-lat.bt
sudo bpftrace /usr/local/bin/syscall-lat.bt write

Sample output after 10 seconds (actual values will vary by workload):

@latency_us:
[0, 1)             8431 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[1, 2)              912 |@@@@
[2, 4)              233 |@
[4, 8)               47 |
[8, 16)              11 |
[256, 512)            2 |

The long tail at 256–512 µs is worth investigating—those could be scheduler delays or device latency spikes.

Listing and Discovering Probes

# All tracepoints
bpftrace -l 'tracepoint:*'

# Tracepoints matching a keyword
bpftrace -l 'tracepoint:*sched*'

# Show structured argument names for a tracepoint
bpftrace -lv 'tracepoint:syscalls:sys_enter_openat'

# Available kprobes matching a function name
bpftrace -l 'kprobe:tcp_*'

The -lv flag (verbose list) is essential before writing scripts—it shows the exact field names so you are not guessing at arg0 offsets.

Verification

Run the syscall count one-liner in one terminal while running stress-ng --io 4 --timeout 30s in another. You should see stress-ng near the top of the map with heavy read/write counts. If nothing appears, confirm debugfs is mounted:

mount | grep debugfs
# expect: debugfs on /sys/kernel/debug type debugfs (rw,relatime)

# If missing:
sudo mount -t debugfs debugfs /sys/kernel/debug

Troubleshooting

• "Could not open BTF data" — your kernel was built without CONFIG_DEBUG_INFO_BTF=y. On Ubuntu kernels earlier than 20.04 HWE, install the linux-image-generic-hwe-* variant or use DWARF-based headers via apt install linux-headers-$(uname -r).
• Permission denied — bpftrace requires CAP_BPF and CAP_PERFMON (kernel 5.8+) or CAP_SYS_ADMIN on older kernels. Always run with sudo or grant capabilities explicitly.
• Tracepoint not found — the syscall name varies by architecture. On ARM64, openat may appear as sys_enter_openat2. Use bpftrace -l to confirm the exact name on your kernel.
• Map memory warnings — default map size is 64 entries. Add @map[key] = count() with high-cardinality keys (e.g., raw filenames) and bpftrace will warn. Use -Bsize or widen the key type.
• Lost events — if the probe fires faster than the ring buffer can drain, bpftrace prints a lost-event count. Reduce overhead by adding stricter predicates or sampling with profile:hz:N instead of per-event probes.