Debug systemd Units that Won't Start

A service that refuses to start is one of the most common systemd headaches. The failure message in systemctl status is often terse, and the real cause is buried in the journal or in a unit file syntax error you typed at 2 AM. This guide walks through a repeatable diagnostic workflow: read the status, dig into the journal, verify the unit file statically, and apply targeted overrides without touching the vendor unit.

Step 1: Read the Status Output Properly

Most people glance at systemctl status and miss half of what it says. Run it with enough lines to see the last log burst:

systemctl status myapp.service -n 50

The output sections that matter most are:

• Loaded: — shows the unit file path and whether it is enabled. If it says vendor preset: disabled, the service won't start at boot unless explicitly enabled.
• Active: — failed, activating (auto-restart), or inactive (dead) each point to different root causes.
• Main PID / code=exited, status= — the exit code. A non-zero exit code (e.g., status=1/FAILURE, status=203/EXEC) narrows the problem immediately. Code 203 specifically means the binary was not found or not executable.
• The log tail — the last lines printed by the service itself before it died.

If the active state is failed, note the result field. result=exit-code means the process started and then exited badly. result=start-limit-hit means systemd gave up after repeated restart attempts — you must reset it before the service can try again:

systemctl reset-failed myapp.service

Step 2: Mine the Journal

journalctl -u gives you the full log for a single unit across all its invocations, not just the last burst shown in status.

Show all logs for the unit

journalctl -u myapp.service

Follow live output while you attempt a start

journalctl -u myapp.service -f &
systemctl start myapp.service

Limit to the current boot

journalctl -u myapp.service -b

Show kernel + service messages together (useful for hardware or cgroup issues)

journalctl -b -p err..alert

Look for lines tagged with the service name or its PID. Common tells:

• "No such file or directory" on the ExecStart binary path — the path in the unit is wrong or the package isn't installed.
• "Permission denied" — a missing capability, SELinux/AppArmor denial, or wrong file ownership.
• "Failed to set up mount namespace" — a PrivateTmp or ProtectSystem sandboxing option conflicting with the app's runtime paths.
• D-Bus or socket activation errors — the companion .socket or .target unit is not running.

If journald itself has lost logs (persistent logging not configured), enable it once and retry:

mkdir -p /var/log/journal
systemd-tmpfiles --create --prefix /var/log/journal
systemctl restart systemd-journald

Step 3: Validate the Unit File Statically

systemd-analyze verify parses unit files and reports syntax errors, unknown options, missing dependencies, and dangerous combinations — without actually running anything. This is especially useful after you edit a unit.

systemd-analyze verify myapp.service

For units not yet installed system-wide, pass the file path directly:

systemd-analyze verify /etc/systemd/system/myapp.service

Sample output you might see (will vary):

/etc/systemd/system/myapp.service:12: Unknown key name 'ExecReload' in section 'Service', ignoring.
myapp.service: Unit configured to use KillMode=none. This is unsafe.

Warnings don't prevent the unit from loading, but errors do. Fix every error line before moving on. After any edit to a unit file, reload the daemon so systemd picks up the changes:

systemctl daemon-reload

Also useful: check what systemd actually resolved the unit to after all includes and dropins are merged:

systemctl cat myapp.service

Step 4: Apply Targeted Fixes with Override.conf

Editing the vendor-installed unit file in /lib/systemd/system/ is a bad idea — package upgrades overwrite it. Instead, use a dropin override. systemctl edit handles this automatically: it creates /etc/systemd/system/myapp.service.d/override.conf and opens your editor.

systemctl edit myapp.service

To override the full unit rather than just patch it (useful when the vendor unit is severely broken):

systemctl edit --full myapp.service

Common override patterns:

Fix an environment variable or binary path

[Service]
Environment="NODE_ENV=production"
ExecStart=
ExecStart=/usr/local/bin/myapp --config /etc/myapp/config.yaml

Note the blank ExecStart= line — it clears the original value before setting the new one. Without it, systemd appends to a list, which is usually not what you want.

Relax a sandboxing option that blocks a path

[Service]
ProtectSystem=false

Increase the start timeout for a slow service

[Service]
TimeoutStartSec=120

Add a missing capability

[Service]
AmbientCapabilities=CAP_NET_BIND_SERVICE

After saving, verify and reload:

systemd-analyze verify myapp.service
systemctl daemon-reload
systemctl restart myapp.service

Step 5: Verify the Service Is Running Correctly

systemctl is-active myapp.service

The command exits 0 and prints active on success. Pair it with a quick status check:

systemctl status myapp.service

For services that expose a port, confirm the socket is listening:

ss -tlnp | grep myapp

If you fixed a start-limit situation, also confirm the failure counter is clear:

systemctl show myapp.service | grep NRestarts

Troubleshooting Specific Error Patterns

SELinux / AppArmor denials

On Fedora, RHEL, Rocky Linux, and derivatives with SELinux enforcing, check the audit log:

ausearch -m avc -ts recent

On Ubuntu/Debian with AppArmor:

journalctl -b | grep apparmor

Use audit2allow (SELinux) or edit the AppArmor profile rather than disabling enforcement entirely.

Service starts manually but not at boot

The unit is probably not enabled, or it depends on a target that runs too early:

systemctl enable myapp.service
systemctl list-dependencies myapp.service

If the service needs the network, verify it declares After=network-online.target and Wants=network-online.target in its [Unit] section.

User services (--user scope)

Services running under a user slice use a separate journal and separate unit directories. Append --user to every command:

systemctl --user status myapp.service
journalctl --user -u myapp.service

Binary path issues on Arch Linux

Arch packages sometimes install to /usr/bin while the unit file references /usr/sbin. Confirm with:

which myapp
type myapp