Diagnose and Fix "No Space Left" When Disk Is Empty (Inodes)

You get "No space left on device" errors, but df -h shows gigabytes free. The culprit is almost always inode exhaustion. Every filesystem has a fixed pool of inodes — one per file, directory, and symlink — and when that pool runs dry, no new files can be created regardless of how much block space remains. This guide walks through diagnosing the problem, finding the offending directories, and applying lasting fixes.

What Are Inodes?

An inode stores the metadata for a file: permissions, owner, timestamps, and pointers to data blocks. The inode count for most filesystems (ext3, ext4, xfs with certain configurations) is set at format time and cannot be increased without reformatting. A filesystem formatted with default settings gets roughly one inode per 16 KB of capacity. On a 20 GB partition that sounds like plenty — but if a mail spool, a package cache, or a build system generates millions of tiny files, you can exhaust inodes while barely touching disk capacity.

Step 1: Confirm Inode Exhaustion

Check inode usage across all mounted filesystems with the -i flag:

df -i

Look for any filesystem where IUse% is at or near 100%. Output will vary, but a saturated filesystem looks like:

# Example — your numbers will differ
Filesystem      Inodes  IUsed   IFree IUse% Mounted on
/dev/sda1      1310720 1310720      0  100% /
tmpfs           499712     621 499091    1% /run

Confirm from the application side as well:

touch /tmp/testfile && echo "OK" || echo "FAILED"

If you see "cannot touch: No space left on device" while df -h shows free space, inodes are the issue.

Step 2: Find Which Directory Holds the Most Files

The kernel only tells you inodes are gone — you have to find the pile yourself. The most reliable method is find combined with sorting. Run this on the affected mount point (replace / with the specific mount if it is not root):

find / -xdev -printf '%h\n' | sort | uniq -c | sort -rn | head -20

-xdev prevents crossing into other mounted filesystems. %h prints the parent directory of each file. The result ranks directories by file count. On a busy system this may take a minute or two.

A faster alternative uses du in inode-counting mode (available on Linux du from GNU coreutils 8.25+):

du --inodes -xS / | sort -rn | head -20

-S excludes subdirectory counts so each line reflects only files directly inside that directory, making the hot spots obvious.

Step 3: Identify the Common Culprits

Once you have a ranked list, cross-reference these frequent offenders:

• Mail spools — /var/mail or /var/spool/mail; each message is a separate file in Maildir format.
• Session files — PHP applications often write one file per session to /var/lib/php/sessions.
• Apt / dpkg cache — /var/cache/apt/archives and /var/lib/dpkg/info accumulate over years.
• Systemd journal — /var/log/journal can balloon if rate-limiting is not configured.
• Node.js / npm — a single node_modules tree can contain 200 000+ files.
• Temporary files — rogue processes writing to /tmp without cleanup.
• Container layers — Docker/Podman overlay storage in /var/lib/docker or /var/lib/containers.
• Build artifacts — Rust and Go build caches, .ccache, Maven .m2 repositories.

Step 4: Clean Up the Inode Hog

Package Caches

Debian/Ubuntu:

sudo apt clean
sudo apt autoclean

Fedora/RHEL family:

sudo dnf clean all

Arch:

sudo pacman -Sc

PHP Session Files

find /var/lib/php/sessions -type f -mtime +7 -delete

Systemd Journal

sudo journalctl --vacuum-time=14d

Docker / Podman Dangling Layers

# Docker
sudo docker system prune -f

# Podman
podman system prune -f

Mass-Delete Millions of Files Safely

Regular rm -rf can crash or hang when argument lists overflow. Use find ... -delete instead, which avoids shell argument limits entirely:

find /path/to/directory -type f -mtime +30 -delete

If you need to wipe an entire directory tree as fast as possible, rsync with an empty source is faster than rm for very large file counts:

mkdir /tmp/empty
rsync -a --delete /tmp/empty/ /path/to/large-directory/

Step 5: Add Automated Prevention

Cleaning up once is not enough. Put a systemd timer in place for recurring cleanup rather than relying on cron (though cron works fine too).

Create the service unit:

sudo tee /etc/systemd/system/session-cleanup.service <<'EOF'
[Unit]
Description=Clean stale PHP session files

[Service]
Type=oneshot
ExecStart=find /var/lib/php/sessions -type f -mtime +7 -delete
EOF

Create the timer unit:

sudo tee /etc/systemd/system/session-cleanup.timer <<'EOF'
[Unit]
Description=Run PHP session cleanup daily

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now session-cleanup.timer

Step 6: Long-Term — Restructure or Reformat

If the workload genuinely requires millions of files (a mail server, an object store), the correct long-term solution is a filesystem designed for high inode density or one without a fixed inode limit.

• XFS — allocates inodes dynamically; it will not run out of inodes unless it runs out of disk space. Ideal for mail spools and object storage. Reformat requires a backup and restore.
• ext4 with a smaller inode ratio — format with mke2fs -t ext4 -i 4096 /dev/sdXN to get one inode per 4 KB instead of the default 16 KB, quadrupling inode density. Only possible at format time.
• tmpfs for sessions — mount a dedicated tmpfs for volatile small-file directories so they never consume persistent inodes:

  echo "tmpfs /var/lib/php/sessions tmpfs defaults,size=256m,mode=1777 0 0" | sudo tee -a /etc/fstab
  sudo mount -a

• Object storage patterns — if an application writes millions of tiny files, consider migrating to SQLite, a key-value store (Redis, LMDB), or an object store (MinIO) so the filesystem sees fewer entries.

Verification

After cleanup, confirm inodes are freed:

df -i /

Then confirm file creation works again:

touch /tmp/inode-test && echo "Inodes free" && rm /tmp/inode-test

Troubleshooting

• df -i shows low usage but errors persist — the problem may be on a specific bind-mount or overlay filesystem. Run df -i for all mounts and check /proc/mounts. Container runtimes can exhaust inodes on their own overlay layer independently of the host root.
• find runs for hours — use -xdev to avoid scanning network mounts (NFS, CIFS). Also try targeting /var first since that is where most runtime garbage lives.
• Cannot delete files as root — check for immutable flags: lsattr /path/to/file. If you see ----i----, remove the flag with chattr -i /path/to/file.
• Inode count does not drop after deletion — open file descriptors from running processes hold inodes allocated. Check lsof +L1 for deleted-but-open files. The count normalises once those processes close or restart.