Analyse Linux Logs with journalctl, lnav and grep

Log analysis is one of the most immediate skills separating a guessing sysadmin from a diagnosing one. Linux gives you three complementary tools: journalctl for structured systemd journal data, lnav for interactive multi-file browsing, and grep (with its siblings awk and sed) for fast pattern extraction. Used together, they cover the vast majority of day-to-day incident investigation without reaching for a full observability stack.

Understanding Your Log Sources

Modern systemd-based systems write logs to the journal by default. Some daemons also write plain-text files under /var/log/. Know which you are dealing with before reaching for a tool.

• systemd journal — binary, structured, queryable via journalctl. Includes kernel messages, service stdout/stderr, and audit events.
• /var/log/ files — plain text: syslog, auth.log, nginx/access.log, dnf.log, and application-specific logs. These are readable by grep and lnav directly.
• Persistent vs. volatile journal — by default on some distros the journal is lost on reboot. Check /etc/systemd/journald.conf and set Storage=persistent if you need history across reboots.

journalctl Essentials

Basic Filtering

Without arguments, journalctl dumps everything from the oldest entry. You almost never want that. Use filters from the start.

# Follow live output (like tail -f)
journalctl -f

# Last 100 lines
journalctl -n 100

# All logs for a specific service
journalctl -u nginx.service

# Combine: follow a specific service
journalctl -fu sshd.service

Time-Based Queries

The --since and --until flags accept human-readable timestamps, which is far faster than grepping for a date pattern.

# Logs from the last 30 minutes
journalctl --since "30 min ago"

# Specific window around an incident
journalctl --since "2025-06-01 14:00:00" --until "2025-06-01 14:30:00"

# Since last boot
journalctl -b

# Previous boot (useful after a crash)
journalctl -b -1

Priority Filtering

Systemd log levels mirror syslog priorities 0–7. Filtering by priority cuts noise dramatically during an incident.

# Show only errors and worse (emerg, alert, crit, err)
journalctl -p err

# Range: warnings and above
journalctl -p warning..emerg

# Errors from a specific unit
journalctl -u postgresql.service -p err --since "1 hour ago"

Output Formats

The -o flag controls output format. short-iso gives grep-friendly timestamps; json feeds pipelines and log shippers.

# ISO timestamps — easier to read and grep
journalctl -u nginx -o short-iso --since "today"

# JSON for scripting or shipping
journalctl -u nginx -o json | jq '.MESSAGE' | head -20

# Compact JSON (one object per line) — best for pipelines
journalctl -o json-seq | head -5

Kernel and Boot Messages

# Kernel ring buffer (replaces dmesg for older-style viewing)
journalctl -k

# Hardware errors since last boot
journalctl -k -p err -b

grep, awk, and Practical Pipelines

Plain-text log files and journalctl text output both respond well to classic UNIX pipelines. The key is to be specific — broad greps on large logs waste time.

Essential grep Patterns

# Case-insensitive search for auth failures
grep -i "failed password" /var/log/auth.log

# Show 3 lines of context around each match
grep -i "oom" /var/log/syslog -A 3 -B 1

# Count occurrences per SSH source IP
grep "Failed password" /var/log/auth.log \
  | awk '{print $(NF-3)}' \
  | sort | uniq -c | sort -rn | head -20

# Find all unique HTTP 5xx errors in nginx logs
grep -E '" 5[0-9]{2} ' /var/log/nginx/access.log \
  | awk '{print $9}' | sort | uniq -c | sort -rn

Combining journalctl with grep

# Search the journal for OOM events
journalctl -k --since "today" | grep -i "out of memory"

# Find all segfaults across all services this boot
journalctl -b | grep -i segfault

# Extract just timestamps and messages for a service error stream
journalctl -u myapp.service -o short-iso -p err \
  | grep -v "^--" \
  | awk '{print $1, $2, substr($0, index($0,$5))}'

Watching Log Rate (detecting log storms)

# Count log lines per minute for a service
journalctl -u myapp.service --since "10 min ago" -o short \
  | awk '{print $1, $2}' \
  | cut -d: -f1,2 \
  | uniq -c

lnav: Interactive Log Navigation

lnav (Log file Navigator) is a terminal UI that understands dozens of log formats automatically, highlights log levels with colour, and lets you search, filter, and run SQL queries across log files without writing a pipeline.

Installation

# Debian / Ubuntu
sudo apt install lnav

# Fedora / RHEL 9+ (EPEL required on RHEL/Rocky)
sudo dnf install lnav

# Arch
sudo pacman -S lnav

Opening Logs

# Open multiple log files at once (lnav merges them by timestamp)
lnav /var/log/nginx/access.log /var/log/nginx/error.log

# Pipe journalctl output into lnav
journalctl -u nginx --since "today" | lnav

# Open all logs in a directory
lnav /var/log/apache2/

Key lnav Shortcuts

Key	Action
/pattern	Search forward (regex)
n / N	Next / previous match
e / E	Jump to next / previous error
w / W	Jump to next / previous warning
i	Toggle log histogram view
;	Enter SQL query mode
q	Quit

SQL Queries Inside lnav

Press ; to open the SQL prompt. lnav exposes every parsed log field as a column in a virtual table called log.

# Inside lnav SQL prompt — top 10 client IPs in access log
SELECT cs_ip, COUNT(*) AS hits FROM access_log GROUP BY cs_ip ORDER BY hits DESC LIMIT 10;

# Count errors per hour
SELECT strftime('%Y-%m-%d %H', log_time) AS hour, COUNT(*) AS errors
FROM syslog WHERE log_level >= 'error'
GROUP BY hour ORDER BY hour;

Verification: Confirming Your Findings

After identifying a pattern, verify it is real and not a parsing artifact.

# Cross-check a journal finding against the raw syslog file
journalctl -u sshd --since "1 hour ago" | grep "Invalid user" | wc -l
grep "Invalid user" /var/log/auth.log | wc -l

# Confirm a process crash by checking the service status
systemctl status myapp.service --full --no-pager

# Verify OOM kill with kernel audit
journalctl -k | grep -E "(oom_kill|Killed process)"

When to Ship Logs to ELK or Loki

Local tools are fast and sufficient for single-host troubleshooting. Reach for a centralised stack when:

• You manage more than a handful of hosts — correlating an incident across 20 servers with SSH and grep is painful.
• You need log retention beyond disk capacity — journald keeps logs until the disk limit is hit; ELK/Loki store compressed, indexed data cheaply at scale.
• You want alerting on log patterns — Loki + Grafana Alertmanager or Elasticsearch Watcher handles this; journalctl does not.
• Non-technical stakeholders need dashboards — Kibana or Grafana give them read access without SSH.
• You have compliance or audit requirements requiring tamper-evident, centralised log storage.

For shipping, consider Promtail (Loki agent), Filebeat (Elasticsearch), or systemd-journal-upload for journal-native forwarding. The local tools covered here remain useful for on-host triage even after you have a central stack.

Troubleshooting Common Issues

Journal shows no logs after reboot

The journal is volatile by default on some distros. Enable persistence:

sudo mkdir -p /var/log/journal
sudo systemctl restart systemd-journald
# Then set Storage=persistent in /etc/systemd/journald.conf

journalctl output is truncated

Long lines are wrapped, not truncated. If messages appear cut off, check for a SystemMaxUse limit in journald.conf, or pipe through cat to disable paging and line wrapping.

journalctl -u myapp --no-pager | cat

lnav does not recognise a log format

lnav ships with many built-in formats. For custom application logs, create a format file under ~/.lnav/formats/ — the lnav documentation covers the JSON format schema. As a quick fallback, raw unknown files still display in lnav; you just lose field-aware features.

grep returns too many results

Add time constraints at the journalctl stage, not the grep stage. Filtering 10 minutes of logs rather than 30 days is faster and the results are more actionable. Alternatively, use grep -c first to gauge volume before reading all matches.