The Linux CPU Scheduler Explained

The Linux CPU scheduler is one of the most performance-critical pieces of the kernel, yet most administrators treat it as a black box. Understanding how it works — and how to tune it — lets you reduce latency for interactive workloads, improve throughput on batch jobs, and even load custom scheduling logic via the new sched_ext framework. This guide covers the Completely Fair Scheduler (CFS), the emerging BPF-extensible scheduler, nice values and real-time policies, multi-queue SMP scheduling, and the tools to observe all of it.

How CFS Works

The Completely Fair Scheduler has been the default Linux process scheduler since kernel 2.6.23. Its goal is to approximate an "ideal, precise multi-tasking CPU" by distributing CPU time proportionally. Instead of a fixed time-slice round-robin, CFS tracks virtual runtime (vruntime) — a per-task counter that advances at a rate inversely proportional to the task's weight. The task with the lowest vruntime is always next to run.

Tasks are stored in a per-CPU red-black tree keyed by vruntime, so selecting the next task is an O(log n) operation. When a task sleeps and wakes, its vruntime is clamped to no more than one sched_latency_ns behind the current minimum, preventing it from monopolising the CPU after a long sleep.

Key tunables live under /proc/sys/kernel/:

• sched_latency_ns (default ~6 ms on recent kernels): the target scheduling period — all runnable tasks get one slot within this window.
• sched_min_granularity_ns: minimum runtime before a task can be preempted; scales up with CPU count.
• sched_wakeup_granularity_ns: prevents newly woken tasks from immediately preempting the current task unless their vruntime advantage exceeds this threshold.

sysctl kernel.sched_latency_ns kernel.sched_min_granularity_ns kernel.sched_wakeup_granularity_ns

For desktop responsiveness, lower sched_latency_ns (e.g., 3000000) and sched_min_granularity_ns (e.g., 300000). For throughput servers, raise them to reduce context-switch overhead.

Nice Values and Priority

Nice is the traditional UNIX interface for adjusting a process's CPU weight. The range is -20 (highest priority) to +19 (lowest). Nice maps to an internal kernel weight via a non-linear table: each step of 1 nice unit changes weight by roughly 10%, so nice -20 gets about 1024 weight units and nice +19 gets about 15.

Setting and Inspecting Nice

# Launch a command at a specific nice level
nice -n 10 make -j$(nproc)

# Change a running process
renice -n -5 -p 12345

# Or by name (lowers whole process group)
renice -n 5 -g $(pgrep -g myapp)

Only root (or a process with CAP_SYS_NICE) can set negative nice values. Unprivileged users can only raise their own process's nice (i.e., be nicer, not ruder).

Inspecting Weights in the Scheduler

# See nice, priority, and vruntime for running tasks
ps -eo pid,ni,pri,cls,comm --sort=ni | head -20

# Per-task scheduler statistics (vruntime, nr_switches, etc.)
cat /proc/$(pgrep myapp)/sched

Real-Time Scheduling Policies

CFS is a time-sharing scheduler — it cannot guarantee bounded latency. For tasks that need hard deadlines (audio servers, industrial control, low-latency trading), Linux provides three real-time policies that bypass CFS entirely.

• SCHED_FIFO: A runnable RT task runs until it blocks or a higher-priority RT task preempts it. No time slicing at the same priority.
• SCHED_RR: Like FIFO but adds a round-robin time quantum between equal-priority RT tasks.
• SCHED_DEADLINE: Implements Earliest Deadline First (EDF). Each task declares runtime, deadline, and period. The kernel uses CBS (Constant Bandwidth Server) admission control to prevent overcommit. This is the most principled policy for real-time workloads.

# Set a running process to SCHED_FIFO priority 50 (requires root)
chrt -f -p 50 $(pgrep jackd)

# Set SCHED_RR
chrt -r -p 30 $(pgrep myprocess)

# Set SCHED_DEADLINE (runtime=5ms, deadline=10ms, period=10ms)
chrt --deadline --sched-runtime 5000000 --sched-deadline 10000000 --sched-period 10000000 -p 0 $(pgrep myapp)

# Inspect the policy and priority of a process
chrt -p $(pgrep jackd)

Warning: A runaway SCHED_FIFO process at priority 99 can lock up the system. The kernel's RT throttling (/proc/sys/kernel/sched_rt_runtime_us, default 950 ms per second) limits total RT time to 95% of wall clock. Do not set this to -1 (unlimited) on production machines unless running an RT kernel with careful admission control.

Multi-Queue SMP Scheduling

On multi-core systems, each CPU has its own run queue. The scheduler must balance load across them without causing cache thrashing from excessive task migrations. This involves two complementary mechanisms:

• Load balancing: Periodically, a softirq-driven rebalancing pass steals tasks from overloaded CPUs. The scheduler domain hierarchy (SMT siblings → physical cores → NUMA nodes) controls how aggressively it balances across each boundary.
• Wake-up affinity: When a task wakes, the scheduler tries to place it on the last CPU it ran on (cache warm) or the CPU of the task that woke it (to minimise pipe latency).

CPU Affinity

# Pin a process to CPUs 0 and 1
taskset -cp 0,1 $(pgrep myapp)

# Launch with affinity
taskset -c 4-7 ./my_latency_sensitive_daemon

# View current affinity mask
taskset -p $(pgrep myapp)

NUMA Awareness

# Run a workload on NUMA node 0's CPUs, with memory also on node 0
numactl --cpunodebind=0 --membind=0 ./myapp

# Check NUMA topology
numactl --hardware

Mismatched NUMA placement — a task on node 1 accessing memory homed on node 0 — is a common hidden performance problem. Use numastat -p $(pgrep myapp) to check for cross-node memory hits.

sched_ext: BPF-Extensible Scheduling

sched_ext (SCX) landed in the mainline kernel at 6.12. It lets you write a custom scheduler in BPF (or C compiled to BPF) that replaces task selection at runtime, without patching the kernel. This is a major shift: schedulers that previously required out-of-tree patches (like BORE, Lavd, or gaming-oriented schedulers) can now ship as userspace daemons.

Checking sched_ext Support

# Requires kernel >= 6.12 and CONFIG_SCHED_CLASS_EXT=y
grep CONFIG_SCHED_CLASS_EXT /boot/config-$(uname -r)

# Check available SCX schedulers (scx_utils package)
scx_lavd --help 2>/dev/null || echo "scx_utils not installed"

Installing scx_utils

# Fedora 41+ / RHEL 10+
dnf install scx-scheds

# Arch Linux (AUR)
yay -S scx-scheds-git

# Ubuntu 25.04+ (kernel 6.14+)
apt install scx-scheds

Running a sched_ext Scheduler

# Run the LAVD scheduler (good for interactive/gaming loads)
sudo scx_lavd

# Run rusty (a multi-domain, load-balancing scheduler)
sudo scx_rusty

# The system reverts to CFS automatically when the SCX process exits

SCX schedulers run as privileged userspace processes. If they crash or are killed, the kernel falls back to CFS within the configurable stall_timeout. This makes experimentation safe on a running system.

Observing the Scheduler

Guessing without measurement is cargo-culting. These tools give you real data.

# High-level runqueue and CPU usage
vmstat 1

# Per-CPU runqueue depth (r column = runnable tasks)
sar -q 1 5

# Scheduler-specific perf events: context switches, migrations
perf stat -e context-switches,cpu-migrations,sched:sched_switch -p $(pgrep myapp) sleep 5

# Flame graph of where time is actually spent
perf record -g -p $(pgrep myapp) sleep 10
perf report

# Real-time per-task scheduler stats (vruntime, wait_sum, etc.)
watch -n1 "cat /proc/\$(pgrep myapp)/sched"

Verification

After any tuning, verify the change had the intended effect. For latency-sensitive workloads, use cyclictest from the rt-tests package:

# Install rt-tests
apt install rt-tests   # Debian/Ubuntu
dnf install rt-tests   # Fedora/RHEL
pacman -S rt-tests     # Arch

# Measure scheduling latency (run for 60 seconds, histogram output)
sudo cyclictest --mlockall --smp --priority=80 --interval=200 --distance=0 --duration=60 --histogram=400

Lower maximum and 99th-percentile latency numbers indicate better real-time behaviour. Compare results before and after policy or tunable changes. For throughput workloads, a CPU-bound benchmark like sysbench cpu or stress-ng --cpu $(nproc) provides a controlled baseline.

Troubleshooting

• High involuntary context switches: Seen in pidstat -w as cswch/s vs nvcswch/s. High involuntary switches mean the task is being preempted — check if a higher-priority or more resource-hungry task is competing.
• Unexpectedly low priority taking CPU: Check if the process has been promoted via SCHED_FIFO by a framework (PulseAudio, pipewire, and systemd services can set RT priority). Use chrt -p <pid> to confirm.
• NUMA-related slowdowns: Use perf stat -e cache-misses,LLC-load-misses and compare with numastat. Bind the workload with numactl.
• sched_ext scheduler not loading: Confirm kernel ≥ 6.12, CONFIG_SCHED_CLASS_EXT=y, and that you are running as root. Check dmesg | grep scx for errors.
• RT process starving the system: If sched_rt_runtime_us has been set to -1 and an RT task loops, the system may become unresponsive. Boot to a recovery shell (or use SysRq), then reset: sysctl kernel.sched_rt_runtime_us=950000.