NUMA and CPU Pinning on Linux

Modern multi-socket servers and high-core-count desktop CPUs are almost always NUMA (Non-Uniform Memory Access) systems. When a thread runs on a CPU core and its memory allocation lives on a different NUMA node, every memory access crosses an interconnect—adding latency and saturating cross-node bandwidth. CPU pinning and NUMA-aware placement eliminate that penalty. This guide covers the tools: taskset for quick CPU affinity masks, numactl for NUMA-aware launch, and cgroup v2 cpusets for persistent, systemd-managed placement.

Understanding Your NUMA Topology

Before pinning anything, map the hardware.

numactl --hardware

Output shows node counts, CPU lists per node, memory per node, and cross-node distance matrix (e.g., local=10, remote=21). Lower distance means lower latency. Distances above 20 signal meaningful penalties.

lscpu --extended

This lists every logical CPU with its socket, core, NUMA node, and cache topology. Save it—you'll reference it throughout. Also check:

cat /sys/devices/system/node/node*/cpulist

Each file shows the CPU list for that NUMA node in kernel range notation (e.g., 0-23,48-71).

taskset: Fast CPU Affinity Without NUMA Awareness

taskset sets or reads the CPU affinity mask of a process. It doesn't know about NUMA nodes—it just restricts scheduling to specified logical CPUs. It's useful for isolating a workload to one socket's CPUs when you know the mapping.

Launch a new process pinned to specific CPUs

# Pin to CPUs 0-7 (range notation)
taskset -c 0-7 ./my_application

# Pin to CPUs 0, 2, 4, 6 (comma-separated)
taskset -c 0,2,4,6 ./my_application

Re-pin a running process

# Find the PID first
pgrep my_application

# Apply new affinity mask (hex bitmask: 0xff = CPUs 0-7)
taskset -p 0xff 12345

# Or use list notation
taskset -cp 0-7 12345

Read current affinity

taskset -cp 12345

Limitation: taskset pins the CPU but doesn't control memory allocation. A process pinned to node-0 CPUs can still allocate memory from node-1, causing remote memory accesses. Use numactl to control both.

numactl: NUMA-Aware Process Launch

numactl sets CPU affinity and memory policy together, which is the correct approach for latency-sensitive workloads.

Install numactl

# Debian/Ubuntu
sudo apt install numactl

# Fedora / RHEL family
sudo dnf install numactl

# Arch
sudo pacman -S numactl

Pin to a NUMA node (CPUs + local memory)

# Run on node 0, allocate memory from node 0
numactl --cpunodebind=0 --membind=0 ./my_application

# Shorthand: --localalloc means "allocate memory on whichever node the thread is running"
numactl --cpunodebind=0 --localalloc ./my_application

Interleave memory across nodes (throughput workloads)

# Useful for workloads that need aggregate bandwidth more than low latency
numactl --interleave=all ./my_application

Pin to specific CPU list with local memory

# Bind to CPUs 0-11 on node 0, memory from node 0
numactl --physcpubind=0-11 --membind=0 ./my_application

Inspect NUMA statistics for a running process

# Check memory mapping and NUMA stats
numastat -p $(pgrep my_application)

The numa_miss column in numastat output shows allocations that landed on the wrong node. A high miss rate confirms misconfigured placement.

cgroup v2 cpusets: Persistent Pinning via systemd

For production services, embedding numactl or taskset in launch scripts is fragile. cgroup v2 cpusets—managed through systemd—survive restarts and apply to every thread the service spawns.

Verify cgroup v2 is active

mount | grep cgroup2
# Should show: cgroup2 on /sys/fs/cgroup type cgroup2 ...

All current LTS Ubuntu (22.04+), Fedora (31+), and Arch systems use unified cgroup v2 by default.

Configure CPU and NUMA pinning in a systemd unit

Edit or create a drop-in override for your service:

sudo systemctl edit myservice.service

Add the following in the editor:

[Service]
# Pin threads to CPUs 0-11 (NUMA node 0 on a typical dual-socket system)
CPUAffinity=0-11

# Restrict memory allocations to NUMA node 0
NUMAPolicy=bind
NUMAMask=0

# Optional: prevent CPU frequency scaling interference
CPUSchedulingPolicy=fifo
CPUSchedulingPriority=50

NUMAPolicy accepts bind, preferred, interleave, and local—matching the numactl semantics exactly. NUMAMask is the node list.

sudo systemctl daemon-reload
sudo systemctl restart myservice.service

Verify cpuset assignment

# Find the service's cgroup path
systemctl show myservice.service -p ControlGroup

# Read effective cpuset
cat /sys/fs/cgroup/system.slice/myservice.service/cpuset.cpus.effective

# Read effective memory nodes
cat /sys/fs/cgroup/system.slice/myservice.service/cpuset.mems.effective

Isolating CPUs from the Scheduler (isolcpus)

For the most demanding use cases—real-time audio, network packet processing, HPC—you can remove CPUs from the general Linux scheduler entirely using the isolcpus kernel parameter. Processes must be explicitly moved onto isolated CPUs with taskset or numactl.

# Add to GRUB_CMDLINE_LINUX in /etc/default/grub (example: isolate CPUs 8-15)
# Debian/Ubuntu/Fedora
sudo grep GRUB_CMDLINE_LINUX /etc/default/grub
# Add: isolcpus=8-15 nohz_full=8-15 rcu_nocbs=8-15

# Debian/Ubuntu
sudo update-grub

# Fedora/RHEL
sudo grub2-mkconfig -o /boot/grub2/grub.cfg

# Arch (systemd-boot)
sudo nano /boot/loader/entries/arch.conf
# Append to options line: isolcpus=8-15 nohz_full=8-15 rcu_nocbs=8-15

After reboot, confirm isolation:

cat /sys/devices/system/cpu/isolated

Warning: isolcpus is a kernel boot parameter change. Test on non-production systems first. Incorrect CPU ranges can leave the system unable to schedule work properly.

Real-World Impact: Measuring the Difference

Benchmark before and after using perf and a memory-bandwidth tool like mbw or stream.

# Install mbw (Debian/Ubuntu)
sudo apt install mbw

# Without NUMA pinning (default scheduler placement)
mbw 512

# With NUMA pinning to node 0
numactl --cpunodebind=0 --membind=0 mbw 512

On a dual-socket EPYC system, pinned runs typically show 15–40% higher memory bandwidth and 10–25% lower p99 latency for memory-bound workloads. Exact gains depend on the NUMA distance matrix and workload access patterns.

# Check for remote NUMA hits on a running process (replace PID)
perf stat -e numa:numa_miss_pinned,numa:numa_miss_not_pinned -p 12345 sleep 5

Troubleshooting

• numactl reports "numactl: error: no nodes available" — Your kernel wasn't built with NUMA support, or the hardware is a single-socket system. Check dmesg | grep -i numa.
• CPUAffinity in systemd has no effect — The service may be forking a subprocess that resets affinity. Wrap the binary with ExecStart=/usr/bin/taskset -c 0-11 /usr/bin/myapp as a belt-and-suspenders measure.
• High numa_miss after pinning — A memory allocator (e.g., glibc malloc arena) may pre-allocate from another thread. Use LD_PRELOAD with a NUMA-aware allocator like libnuma or tune MALLOC_ARENA_MAX.
• Hyperthreading siblings on different nodes — Unlikely but possible on some Threadripper configurations. Always confirm sibling placement with lscpu --extended before assuming a core range maps cleanly to one node.
• isolcpus not showing isolated CPUs after reboot — Verify the grub config was actually regenerated and the correct config file is being booted. On EFI systems with multiple bootloaders, the wrong config may be in use.