Diagnose Routing with mtr and traceroute

When a connection misbehaves — high latency, intermittent drops, or a path that looks wrong — traceroute and mtr are your first instruments. Both tools expose the hop-by-hop path packets take through the network, but they behave differently depending on probe protocol, direction, and how routers handle ICMP. Understanding those differences separates useful diagnosis from misleading conclusions.

How Path Tracing Works

Both tools exploit the IP TTL (Time To Live) field. Each probe starts with TTL=1. The first router decrements it to zero, drops the packet, and (optionally) sends back an ICMP Time Exceeded message, revealing its address. The tool increments TTL and repeats, walking hop by hop to the destination.

The critical word above is optionally. Many routers deprioritize or outright discard ICMP Time Exceeded generation, especially for transit traffic. That produces the infamous * * * lines — not necessarily a broken hop.

Installing the Tools

Debian / Ubuntu

sudo apt install mtr-tiny traceroute iputils-tracepath

Fedora / RHEL 9 / Rocky Linux

sudo dnf install mtr traceroute

Arch Linux

sudo pacman -S mtr traceroute

mtr-tiny on Debian/Ubuntu is the curses-only build; install mtr if you want the GTK GUI. Either works identically from the command line.

ICMP vs UDP vs TCP Probes

This is the most misunderstood part of path tracing. The probe protocol changes what firewalls and routers will pass, and therefore what you see.

• UDP (traceroute default on Linux): Sends UDP packets to high-numbered ports (33434+). The destination returns ICMP Port Unreachable when it receives them, confirming arrival. Firewalls commonly block outbound UDP to unknown ports, so you may see stars at the final hop even when the host is up.
• ICMP Echo (traceroute -I, mtr default): Sends ICMP Echo Requests just like ping. Widely permitted outbound, but many ISP routers rate-limit ICMP, causing artificial loss at mid-path hops. If a router shows 20% loss but every hop after it is clean, the loss is cosmetic — the router is rate-limiting probes, not dropping your traffic.
• TCP SYN (traceroute -T, mtr --tcp): Sends SYN packets to a real port (default 80 or 443). This is the most realistic test for web traffic because it traverses the same firewall rules as an actual connection. Requires root or CAP_NET_RAW.

Switching Probe Protocol in traceroute

# Default UDP
sudo traceroute example.com

# ICMP Echo
sudo traceroute -I example.com

# TCP SYN to port 443
sudo traceroute -T -p 443 example.com

Switching Probe Protocol in mtr

# ICMP (default)
sudo mtr example.com

# UDP
sudo mtr --udp example.com

# TCP SYN to port 443
sudo mtr --tcp --port 443 example.com

Reading mtr Output

Run mtr in report mode (-r) for a fixed 10-cycle summary you can paste into a ticket:

sudo mtr -r -c 100 --show-ips example.com

Output will vary, but the columns to focus on are:

• Loss%: Sustained loss (especially at the final hop) is real. Loss only at one intermediate hop with clean hops after it is almost always ICMP rate-limiting.
• Avg / Best / Wrst: A large Wrst relative to Avg signals jitter or occasional congestion at that hop.
• StDev: High standard deviation on a hop confirms intermittent congestion rather than a steady slow link.

Interactive mtr Key Bindings

• d — cycle display modes (latency, drop counts)
• n — toggle DNS resolution
• p — pause/resume
• r — reset counters

Asymmetric Routing: Why the Path Back Matters

Traceroute and mtr only show the forward path — packets from your machine to the target. The return path is often completely different. This is asymmetric routing, and it is normal on the public internet. BGP policy, load balancing, and geographic topology all affect each direction independently.

Why does this matter? Suppose you see high latency at hop 8 in your trace. That latency could be caused by the return ICMP Time Exceeded packet taking a long cross-continental path back to you, not by any problem in the forward direction. The forward link at that hop may be perfectly fast.

To expose the return path, run a traceroute from the remote side. If you have access to a server at the destination, SSH in and trace back to your IP. Services like Hurricane Electric's Looking Glass or RIPE NCC's network tools let you run traces from remote vantage points without needing your own remote server.

# From the remote server, trace the return path to your public IP
sudo mtr -r -c 50 YOUR_PUBLIC_IP

When the Tools Lie (or at Least Mislead)

MPLS and Tunneled Paths

Many carrier networks run MPLS internally. MPLS label-switched paths can be completely invisible to TTL-based probing — you see only the ingress and egress routers. A route that appears to jump from Chicago to London in one hop may have four intermediate nodes you cannot see.

Load Balancing (Per-Flow ECMP)

Equal-Cost Multi-Path routing sends different flows over different links. Because each traceroute probe may be treated as a new flow, successive probes can take different paths, producing apparently inconsistent hops. The --udp mode is especially susceptible because each probe goes to a different destination port. Use --tcp with a fixed source port (--sport) to keep probes on one flow:

sudo mtr --tcp --port 443 --sport 50000 example.com

Firewalls Blocking Probes but Not Traffic

A firewall may pass TCP port 443 traffic but block ICMP Time Exceeded replies or UDP probes. Stars at the final hop with a working connection do not mean the path is broken — the application traffic is getting through on a path the probe cannot see. Always correlate path data with actual application behaviour.

Practical Diagnosis Workflow

1. Start with a plain ICMP mtr to get a quick picture of the forward path and loss pattern.
2. If you see stars or unexpected loss, repeat with --tcp --port 443 (or the relevant service port) to rule out ICMP filtering.
3. Run 100+ cycles (-c 100) before concluding any loss is real. Short runs are noisy.
4. Note whether loss accumulates at a specific hop or only at the destination. Loss only at the destination is a red flag; loss only at one mid-path hop with clean hops after is usually benign.
5. Initiate a reverse trace from the target side. If round-trip latency spikes at a hop, check whether the spike is present in both directions.
6. If load balancing is suspected, lock the flow with --sport and retry.

Verifying Your Results

Cross-check mtr findings with a simple ping flood to confirm actual packet loss to the destination:

ping -c 200 -i 0.2 example.com | tail -3

Compare the loss percentage ping reports against what mtr shows at the final hop. They should be close. A large discrepancy means some probes are being treated differently than normal traffic — switch probe protocol and repeat.

For sustained monitoring, record a baseline mtr report during normal operation so you have something to compare against during incidents:

sudo mtr -r -c 200 --json example.com > baseline-$(date +%F).json

Troubleshooting

• Permission denied running mtr/traceroute: Raw socket operations require root or the CAP_NET_RAW capability. Both tools are usually installed setuid or capability-enabled. If not, prefix with sudo.
• All hops show * * *: Your outbound firewall or the path is blocking all probe types. Try TCP on port 80 or 443 (-T -p 443). If still blocked, you likely have an egress firewall rule preventing raw sockets entirely.
• First hop is missing: Some home routers suppress ICMP Time Exceeded. Start your analysis from the second hop; the first is your gateway.
• mtr not found after install on RHEL/Rocky: The binary may be in /usr/sbin. Add it to your PATH or use the full path: /usr/sbin/mtr.
• Results differ between runs dramatically: Strong sign of ECMP load balancing. Use fixed source port as shown above.