Essential Network Debugging Tools: ping, traceroute, mtr, and tcpdump

The Network Debugging Toolkit

Network problems fall into a handful of categories: complete connectivity loss, packet loss, high latency, path routing issues, or protocol-level failures. Each category has the right tool for diagnosis. This guide covers the essential command-line tools every developer and sysadmin should know.

ping — Basic Connectivity and Latency

ping sends ICMP Echo Request packets and waits for Echo Replies. It answers the most fundamental question: can I reach this host at all?

# Basic ping (send 10 packets):
ping -c 10 example.com

# Output:
# PING example.com (93.184.216.34): 56 data bytes
# 64 bytes from 93.184.216.34: icmp_seq=0 ttl=55 time=12.345 ms
# 64 bytes from 93.184.216.34: icmp_seq=1 ttl=55 time=11.987 ms
# ...
# --- example.com ping statistics ---
# 10 packets transmitted, 10 received, 0% packet loss
# round-trip min/avg/max/stddev = 11.987/12.210/13.100/0.341 ms

Interpreting ping Output

Value	What it tells you
`time=12ms`	Round-trip latency — expect 1ms (LAN), 10ms (same region), 100ms+ (cross-continent)
`0% packet loss`	Network path is clean
`10% packet loss`	Degraded path — investigate with mtr
`100% packet loss`	Host unreachable, firewalled, or down
`ttl=55`	IP TTL — each router decrements by 1; lower TTL = more hops
High stddev	Inconsistent latency — jitter (problematic for real-time apps)

ICMP Unreachable Codes

When ping fails, the error message reveals the cause:

# Network unreachable — no route to destination:
ping 192.0.2.1
# From 10.0.0.1 icmp_seq=0 Destination Net Unreachable

# Host unreachable — route exists but host not responding:
# Request timeout for icmp_seq 0

# Filtered by firewall — silence (no ICMP response):
# No output — connection just times out

# Port unreachable (for UDP ping):
# From host icmp_seq=0 Destination Port Unreachable

Ping Variations

# Flood ping (requires root) — stress test:
sudo ping -f -c 1000 example.com

# Large packet ping — test MTU issues:
ping -s 1472 -c 5 example.com
# (1472 bytes data + 8 ICMP header + 20 IP header = 1500 MTU)

# Ping IPv6:
ping6 example.com

# Ping with specific source interface:
ping -I eth0 example.com

traceroute — Hop-by-Hop Path Analysis

traceroute reveals the path packets take to reach a destination by sending packets with incrementally increasing TTL values. When a router receives a packet with TTL=1, it decrements to zero and sends an ICMP Time Exceeded message back — revealing its IP address:

# Basic traceroute:
traceroute example.com

# Output:
# traceroute to example.com (93.184.216.34), 30 hops max
#  1  192.168.1.1 (192.168.1.1)    1.234 ms   1.100 ms   1.089 ms
#  2  10.50.0.1 (10.50.0.1)        5.678 ms   5.432 ms   5.611 ms
#  3  72.14.232.1 (72.14.232.1)    8.901 ms   9.123 ms   8.987 ms
#  4  * * *
#  5  93.184.216.34 (93.184.216.34) 12.345 ms 12.100 ms 12.234 ms

# Stars (* * *) mean the router does not respond to ICMP probes
# (firewalled but packets still pass through)

Traceroute Variants

# TCP traceroute — bypasses firewalls that block ICMP/UDP:
sudo traceroute -T -p 443 example.com

# Numeric output only (skip reverse DNS lookups — faster):
traceroute -n example.com

# ICMP traceroute (like Windows tracert):
sudo traceroute -I example.com

# macOS:
traceroute example.com   # UDP by default
sudo traceroute -P ICMP example.com

Asymmetric Routing

A critical insight: traceroute only shows the forward path. The return path may be completely different. Latency spikes at a specific hop may be due to that router deprioritizing ICMP traffic, not actual congestion. Use mtr for sustained monitoring that distinguishes transient from persistent problems.

mtr — Real-Time Path Analysis

mtr (Matt's Traceroute) combines ping and traceroute into a real-time display. It continuously sends probes and shows statistics for every hop:

# Interactive mode:
mtr example.com

# Non-interactive report (send 100 packets, then display):
mtr -r -c 100 example.com

# Output:
# HOST: myserver.example.com          Loss%   Snt   Last   Avg  Best  Wrst StDev
#   1.|-- 192.168.1.1                  0.0%   100   1.1    1.2   0.9   2.1   0.2
#   2.|-- 10.50.0.1                    0.0%   100   5.4    5.5   5.0   6.1   0.2
#   3.|-- 72.14.232.1                  0.0%   100   8.9    9.0   8.5  10.2   0.3
#   4.|-- ???                          100.0%  100   0.0    0.0   0.0   0.0   0.0
#   5.|-- 93.184.216.34                0.0%   100  12.3   12.4  11.9  13.1   0.3

Reading mtr Output

Column	Meaning
`Loss%`	Packet loss percentage at this hop
`Snt`	Packets sent
`Last`	Last round-trip time (ms)
`Avg`	Average RTT
`Best`	Best (minimum) RTT
`Wrst`	Worst (maximum) RTT — look for outliers
`StDev`	Jitter — high values indicate unstable path

Key diagnostic patterns:

Loss at hop N but not N+1: Hop N is rate-limiting ICMP — not real loss
Loss at hop N and all subsequent hops: Real packet loss at hop N
Latency spike at hop N: Network congestion or suboptimal routing at that segment
All ???: Complete path — but destination is reachable: router doesn't respond to ICMP probes

tcpdump — Packet Capture and Analysis

tcpdump captures raw network packets. It is the most powerful and versatile network debugging tool — and the steepest learning curve.

Capture Basics

# Capture on all interfaces, show IP addresses (not hostnames):
sudo tcpdump -n -i any

# Capture and save to file for Wireshark analysis:
sudo tcpdump -n -i eth0 -w capture.pcap

# Read saved file:
tcpdump -r capture.pcap

# Capture with timestamps:
sudo tcpdump -n -tttt -i eth0

BPF Filters

Berkeley Packet Filter (BPF) syntax lets you capture only relevant traffic:

# By host:
sudo tcpdump -n host 203.0.113.10

# By port:
sudo tcpdump -n port 443

# By protocol:
sudo tcpdump -n icmp
sudo tcpdump -n udp port 53

# Combined filters:
sudo tcpdump -n 'host 203.0.113.10 and port 443'
sudo tcpdump -n 'src 203.0.113.10 and dst port 80'

# TCP flags — capture only SYN packets (new connections):
sudo tcpdump -n 'tcp[tcpflags] & tcp-syn != 0 and tcp[tcpflags] & tcp-ack == 0'

# Capture HTTP traffic (unencrypted):
sudo tcpdump -n -A port 80 | grep -E 'GET|POST|HTTP'

# DNS queries and responses:
sudo tcpdump -n port 53 -v

TLS/HTTPS Dissection

# tcpdump cannot decrypt TLS, but can capture the handshake:
sudo tcpdump -n -w https_capture.pcap port 443

# To decrypt in Wireshark:
# 1. Set SSLKEYLOGFILE environment variable:
export SSLKEYLOGFILE=~/ssl-keys.log
curl https://example.com  # curl will log TLS session keys

# 2. In Wireshark: Edit → Preferences → TLS → (Pre)-Master-Secret log filename
# → browse to ~/ssl-keys.log
# Now Wireshark decrypts the captured HTTPS traffic

DNS Tools

dig — DNS Query Tool

# Basic A record lookup:
dig example.com

# Specific record type:
dig MX example.com
dig TXT _dmarc.example.com
dig AAAA example.com

# Short output (just the answer):
dig +short example.com

# Trace resolution chain from root:
dig +trace example.com

# Query specific resolver:
dig @8.8.8.8 example.com        # Google
dig @1.1.1.1 example.com        # Cloudflare
dig @9.9.9.9 example.com        # Quad9

# Check DNSSEC validation:
dig +dnssec example.com

# Reverse DNS lookup:
dig -x 203.0.113.10

nslookup and host

# nslookup — interactive or single-shot:
nslookup example.com
nslookup -type=MX example.com

# host — simpler than dig:
host example.com
host -t MX example.com
host 203.0.113.10   # Reverse lookup

whois — Domain Registration Info

# Check domain registrar, expiry, nameservers:
whois example.com

# Check IP ownership (useful for identifying mystery source IPs):
whois 203.0.113.10

ss and netstat — Local Connection State

For investigating connections on the local machine, ss (socket statistics) is the modern replacement for netstat:

# All TCP connections (with process names):
ss -tnp

# Listening ports:
ss -tlnp

# Connections to a specific host:
ss -tn dst 203.0.113.10

# Show TCP detailed info (RTT, congestion window, etc.):
ss -tin dst 203.0.113.10

# Count connections by state:
ss -ant | awk 'NR>1{print $1}' | sort | uniq -c | sort -rn
# Output example:
# 1234 ESTABLISHED
#  456 TIME-WAIT
#   89 LISTEN

Practical Debugging Workflow

When a service is unreachable, follow this systematic approach:

# Step 1: Can we reach the host at all?
ping -c 5 problematic-server.com

# Step 2: Where does the path break?
mtr -r -c 20 problematic-server.com

# Step 3: Is the service listening on the expected port?
nc -zv problematic-server.com 443
# 'Connection succeeded' = port open
# 'Connection refused' = nothing listening on that port
# Timeout = firewall blocking

# Step 4: Is DNS resolving correctly?
dig +short problematic-server.com
dig +trace problematic-server.com  # Check full resolution chain

# Step 5: Capture the actual packets:
sudo tcpdump -n -w debug.pcap host problematic-server.com
# Then attempt the connection in another terminal
# Ctrl+C tcpdump, open debug.pcap in Wireshark

Summary

The four core tools cover the network debugging stack: ping for basic reachability and latency, mtr for persistent path analysis with per-hop statistics, tcpdump for packet-level inspection, and dig for DNS verification. Add ss for local connection state, nc for port testing, and whois for IP ownership lookups. Systematic use of these tools — working from layer 3 upward — resolves the vast majority of production network issues without requiring proprietary monitoring tools.