Linux Server Disaster Recovery Checklist

A server failure without a tested recovery plan is a career-defining moment—in the wrong direction. This checklist covers what to back up, how to store it safely, how to verify it actually works, and how to hit your recovery targets when something goes wrong. Follow it proactively, not at 2 AM.

Define Your RTO and RPO First

Before touching a single backup tool, write down two numbers:

• RPO (Recovery Point Objective) — how much data loss is acceptable? One hour? One day? This determines backup frequency.
• RTO (Recovery Time Objective) — how long can the service be down? This determines your restore strategy and tooling.

A blog can tolerate an RPO of 24 hours and an RTO of a few hours. A payment service might need an RPO of zero (synchronous replication) and an RTO under five minutes. Every backup decision flows from these two figures. Put them in a document that survives the server—a wiki, a shared drive, a printed sheet in the server room.

What to Back Up

System Configuration

The operating system is reproducible; your custom configuration is not. At minimum, back up:

• /etc — all system and service configuration
• /var/lib — persistent application state (databases, containers)
• /var/log — logs (useful post-incident; lower priority)
• /home and /root — user data, SSH keys, shell configs
• Any custom scripts in /usr/local/bin or /opt
• systemd unit overrides: /etc/systemd/system/
• Cron jobs: /etc/cron* and crontab -l output per user

Track /etc in Git using etckeeper. It commits automatically on package changes and gives you a full audit trail with zero extra effort.

# Debian/Ubuntu
sudo apt install etckeeper
sudo etckeeper init
sudo etckeeper commit "initial commit"

# Fedora/RHEL
sudo dnf install etckeeper
sudo etckeeper init
sudo etckeeper commit "initial commit"

# Arch
sudo pacman -S etckeeper
sudo etckeeper init
sudo etckeeper commit "initial commit"

Databases

Never back up live database files with a filesystem snapshot unless the database is stopped or the tool supports consistent snapshots (like LVM or ZFS). Use native dump tools:

# PostgreSQL — consistent logical dump
sudo -u postgres pg_dumpall --clean --if-exists | gzip > /backup/pg_all_$(date +%F).sql.gz

# MySQL/MariaDB
mysqldump --all-databases --single-transaction --quick \
  -u root -p | gzip > /backup/mysql_all_$(date +%F).sql.gz

The --single-transaction flag for MySQL takes a consistent snapshot without locking tables, which matters for InnoDB on live systems.

Application Data and Secrets

Back up any directory your application writes to: uploaded files, certificates, signing keys, .env files. Secrets need encryption at rest—never push unencrypted keys to a remote backup target.

Choosing a Backup Tool

Restic is the current standard for server backups: it deduplicates, encrypts, and supports dozens of backends (local, S3, SFTP, Backblaze B2). It is available on all major distros.

# Initialize a local repository
restic init --repo /mnt/backup-drive/myserver

# Back up critical directories
restic -r /mnt/backup-drive/myserver backup \
  /etc /home /var/lib /usr/local/bin \
  --exclude /var/lib/docker/overlay2

Store the repository password in a password manager and separately in a sealed envelope. Losing the password means losing all backups. The RESTIC_PASSWORD_FILE environment variable lets you automate without embedding passwords in scripts:

echo 'supersecretpassword' > /etc/restic-password
chmod 600 /etc/restic-password
export RESTIC_PASSWORD_FILE=/etc/restic-password

Off-Site and Immutable Copies

On-site backups protect against accidental deletion and hardware failure. They do not protect against fire, theft, or ransomware that reaches the backup server. You need at least one off-site copy, and it should be immutable.

Immutable backups cannot be altered or deleted for a defined retention period, even by the account that wrote them. Most object storage providers support object lock (S3-compatible, Backblaze B2, Wasabi). Enable it when creating the bucket—it cannot be added retroactively on most platforms.

# Send backups to a Backblaze B2 bucket via restic
export B2_ACCOUNT_ID=youraccountid
export B2_ACCOUNT_KEY=yourappkey
restic -r b2:mybucket-backups:/myserver backup /etc /home /var/lib

Follow the 3-2-1 rule: three copies, on two different media types, with one off-site. A practical setup: local disk + NAS on a separate VLAN + object storage with object lock.

Automate with systemd Timers

Use systemd timers rather than cron for backup jobs. They log to journald, handle missed runs on next boot, and integrate with systemctl status.

# /etc/systemd/system/restic-backup.service
[Unit]
Description=Restic backup
After=network-online.target

[Service]
Type=oneshot
EnvironmentFile=/etc/restic-env
ExecStart=/usr/bin/restic -r b2:mybucket-backups:/myserver backup /etc /home /var/lib
ExecStartPost=/usr/bin/restic -r b2:mybucket-backups:/myserver forget \
  --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune

# /etc/systemd/system/restic-backup.timer
[Unit]
Description=Run restic backup daily

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true

[Install]
WantedBy=timers.target

sudo systemctl daemon-reload
sudo systemctl enable --now restic-backup.timer
sudo systemctl list-timers restic-backup.timer

Restore Drills

A backup you have never restored is a hypothesis. Perform a full restore drill at least quarterly and document the result. Specifically:

1. Spin up a clean VM or cloud instance of the same OS version.
2. Install only the base OS and your backup tool.
3. Restore from the off-site backup—not the local one—to simulate a total site loss.
4. Verify the application starts, data is intact, and TLS certificates are valid.
5. Record the actual time taken and compare it to your RTO.

# Verify snapshot integrity without restoring files
restic -r b2:mybucket-backups:/myserver check

# List available snapshots
restic -r b2:mybucket-backups:/myserver snapshots

# Restore a specific snapshot to a staging directory
restic -r b2:mybucket-backups:/myserver restore latest \
  --target /mnt/restore-test

If the drill fails or takes longer than your RTO, you have found a real problem cheaply. Fix the process, document what broke, and re-run within two weeks.

Verification and Monitoring

Automate backup verification and alert on failure. Services like Healthchecks.io (self-hostable) accept a ping URL after a successful run; silence triggers an alert.

# Add to restic-backup.service after a successful backup
ExecStartPost=/usr/bin/curl -fsS --retry 3 \
  https://hc-ping.com/your-uuid-here > /dev/null

Also run restic check weekly as a separate timer to detect repository corruption early, before you need to use the backup under pressure.

Troubleshooting

Backup job runs but produces no alert ping

Check the service exit code: systemctl status restic-backup.service. The ExecStartPost line only runs if the preceding command exits 0. If restic is failing silently, add --verbose and check journalctl -u restic-backup.service.

Restore is slower than RTO allows

Large repositories over slow links are the most common cause. Consider keeping a recent snapshot on a local attached drive in addition to the off-site target, or moving to block-level replication (DRBD, ZFS send/receive) for low-RTO requirements.

Repository password lost

Restic repositories are unrecoverable without the password by design. Store the password in at minimum two separate physical locations. A hardware password manager and a sealed printed copy in a fireproof safe is a reasonable baseline.

etckeeper not committing on package changes

On systems using dnf or pacman, the package manager hooks may not be installed automatically. Run etckeeper init again after installation and confirm with cat /etc/etckeeper/etckeeper.conf that the correct VCS and package manager are set.