A Practical Guide to sed and awk

sed and awk are two of the most durable tools in a Linux engineer's toolkit. sed (stream editor) surgically edits text line by line; awk is a full pattern-action language built around fields and records. Together they cover the vast majority of text-processing tasks you'll face on the command line or in a script — log parsing, config editing, report generation, and more. This guide teaches both by working through realistic examples.

sed: Stream Editing Fundamentals

sed reads input line by line, applies your editing commands, and writes the result to stdout. The original file is untouched unless you explicitly ask otherwise.

Basic Substitution

The s command is the workhorse. Its syntax is s/pattern/replacement/flags.

# Replace the first occurrence on each line
sed 's/foo/bar/' file.txt

# Replace ALL occurrences on each line (g flag)
sed 's/foo/bar/g' file.txt

# Case-insensitive match (I flag, GNU sed)
sed 's/error/ERROR/gI' file.txt

By default sed prints every line. The -n flag suppresses that; pair it with the p command to print only matching lines — essentially a grep-with-substitution.

# Print only lines where substitution occurred
sed -n 's/foo/bar/gp' file.txt

In-place Editing

The -i flag writes changes back to the file. Always supply a backup suffix on production files.

# Edit in place, keeping original as file.txt.bak
sed -i.bak 's/OldHostname/newhost/g' /etc/hosts

# GNU sed: -i '' is NOT valid — use -i with no space for no backup
sed -i 's/127\.0\.0\.1/0.0.0.0/g' app.conf

Caution: On macOS BSD sed, -i '' (empty string, space-separated) is required for no backup. GNU sed uses -i alone. A script meant to run on both needs a guard or to use perl -pi -e instead.

Address Ranges

Prefix any command with a line number, regex, or range to limit which lines it applies to.

# Apply substitution only on line 5
sed '5s/foo/bar/' file.txt

# Lines 10 through 20
sed '10,20s/debug/DEBUG/g' app.log

# From the line matching START to the line matching END
sed '/^\[SERVICE\]/,/^\[/s/Restart=no/Restart=on-failure/' unit.service

# Delete blank lines
sed '/^$/d' file.txt

# Print lines matching a pattern (like grep)
sed -n '/ERROR/p' app.log

Append, Insert, and Delete

# Insert a line BEFORE line 3
sed '3i\# Added by script' file.conf

# Append a line AFTER every line matching a pattern
sed '/^\[Unit\]/a\Description=My Service' unit.service

# Delete lines 1-5
sed '1,5d' file.txt

# Delete lines matching a pattern
sed '/^#/d' file.conf   # strip comment lines

Multiple Expressions

# Chain commands with -e
sed -e 's/foo/bar/g' -e '/^#/d' -e 's/  */ /g' file.txt

# Or use a semicolon inside a single expression
sed 's/foo/bar/g; /^#/d' file.txt

awk: Field Processing and Pattern Actions

awk sees each line of input as a record split into numbered fields ($1, $2, … $NF for the last). The full line is $0. Programs take the form pattern { action }; both are optional.

On Debian/Ubuntu the default awk is mawk; install gawk for extended features used below:

# Debian / Ubuntu
sudo apt install gawk

# Fedora / RHEL family
sudo dnf install gawk

# Arch
sudo pacman -S gawk

Printing Fields

# Print the first and third fields (whitespace-delimited by default)
awk '{ print $1, $3 }' /var/log/auth.log

# Use a custom field separator (-F)
awk -F: '{ print $1, $3 }' /etc/passwd   # username and UID

# OFS controls the output separator
awk -F: 'BEGIN { OFS="\t" } { print $1, $3, $7 }' /etc/passwd

Pattern Matching

# Print lines where field 3 equals "root"
awk -F: '$3 == 0 { print $1 }' /etc/passwd

# Regex match against the whole line
awk '/FAILED/' /var/log/auth.log

# Regex match against a specific field
awk -F: '$7 ~ /bash$/ { print $1 }' /etc/passwd

# Negate with !~
awk -F: '$7 !~ /nologin/ { print $1, $7 }' /etc/passwd

BEGIN and END Blocks

BEGIN runs before the first line is read; END runs after the last. Use them for setup and summaries.

# Count lines matching a pattern
awk '/ERROR/ { count++ } END { print count, "errors found" }' app.log

# Sum a numeric column
awk '{ total += $5 } END { printf "Total bytes: %d\n", total }' access.log

# Print a header, data, and a footer
awk 'BEGIN { print "User\tUID" } \
     -F: '{ print $1 "\t" $3 }' \
     END { print "---" }' /etc/passwd

Variables, Conditionals, and Loops

# Conditional: flag high-traffic lines (field 10 > 1MB)
awk '$10 > 1048576 { print "LARGE:", $0 }' access.log

# Ternary in printf
awk '{ status = ($9 >= 400) ? "ERR" : "OK"; print status, $7 }' access.log

# Accumulate a per-user count into an associative array
awk '{ hits[$1]++ } END { for (ip in hits) print ip, hits[ip] }' access.log | sort -k2 -rn | head -10

Multi-file and NR/FNR

# NR = total record number across all files
# FNR = record number within the current file
awk 'FNR == 1 { print "--- File:", FILENAME }' file1.log file2.log

# Print line numbers alongside output
awk '{ print NR": "$0 }' file.txt

Combining sed and awk in Pipelines

Real-world tasks often chain both tools. Here are two practical examples.

Extract Active systemd Service Names

systemctl list-units --type=service --state=running --no-legend \
  | awk '{ print $1 }' \
  | sed 's/\.service$//'

Parse an Nginx Access Log for 5xx Errors

awk '$9 ~ /^5/ { print $1, $9, $7 }' /var/log/nginx/access.log \
  | sed 's|/api/v[0-9]*/|/api/VERSION/|g' \
  | sort | uniq -c | sort -rn | head -20

Verification

Always test destructive sed -i commands without the -i flag first, redirecting to a temp file:

# Dry run
sed 's/OldValue/NewValue/g' /etc/app.conf | diff /etc/app.conf -

# If the diff looks right, apply
sed -i.bak 's/OldValue/NewValue/g' /etc/app.conf

Verify an awk pipeline by running it on a small sample first:

head -100 /var/log/nginx/access.log | awk '$9 ~ /^5/ { print $1, $9, $7 }'

Troubleshooting

• sed: -i on macOS fails — BSD sed requires a space between -i and the extension argument (sed -i '' ...). GNU sed does not. Use gsed (via Homebrew) on macOS for GNU behaviour.
• awk prints nothing — Check your field separator. If the file uses commas, pass -F,. Tabs need -F'\t' or -F$'\t' in bash.
• Regex special characters not matching — In sed, characters like ., *, [, \ must be escaped in the pattern. In awk, regex literals go between / slashes and follow ERE rules — no need to double-escape.
• sed address range never closes — If your closing address pattern never appears, the range applies to the rest of the file. Test patterns with grep -n first to confirm they match.
• awk: uninitialized variable — Variables in awk default to zero/empty string, which is usually fine. Use gawk --lint to catch unintended usage.