iproute2 · Software Engineering Wiki

Cheatsheet#

Task	Command
Interfaces and state	`ip -br link`
Addresses, brief	`ip -br addr`
Routing table	`ip route`
Which route a packet takes	`ip route get 198.51.100.7`
Neighbours (ARP/NDP)	`ip neigh`
Add an address (root)	`ip addr add 192.0.2.5/24 dev eth0`
Default route (root)	`ip route add default via 192.0.2.1`
Bring a link up (root)	`ip link set dev eth0 up`
Change MTU (root)	`ip link set dev eth0 mtu 1450`
Listening TCP sockets with process	`ss -ltnp`
Established, with timers	`ss -tno`
Socket internals (rtt, cwnd)	`ss -ti`
Interface counters	`ip -s link show eth0`
Policy rules	`ip rule`
Run a command in a namespace	`ip netns exec ns1 ip addr`
Watch changes live	`ip monitor`
Bridge ports and MAC table	`bridge link`, `bridge fdb show`
Queue discipline	`tc qdisc show dev eth0`

-br (brief) and -j (JSON) work with most ip objects: ip -j addr | jq is the scriptable form. -d (details) adds link-type parameters, -s adds statistics, -4/-6 restrict the family. Verified against iproute2 6.17. Reference: ip(8), ss(8), tc(8).

Warning

Every ip change applies to the running kernel only. It is lost at reboot, and NetworkManager, systemd-networkd or netplan may revert it sooner. Make persistent changes in the network manager’s configuration. Over SSH, ip link set ... down, ip addr flush or a wrong ip route replace default can cut your own session; have console access or schedule a rollback first.

A connectivity problem#

Work up the stack. Each step tells you whether to continue.

ip -br link                      # is the link UP, does it have carrier
ip -br addr                      # does it have the expected address and prefix
ip route get 198.51.100.7        # which route, source address and device the kernel picks
ip neigh show 192.0.2.1          # does the gateway resolve to a MAC
ping -c3 192.0.2.1               # layer 3 to the gateway
ss -ltnp                         # on the server: is anything listening on the port
tracepath -n 198.51.100.7        # where the path stops, and path MTU

198.51.100.7 via 192.0.2.1 dev eth0 src 192.0.2.5 uid 1000
    cache

Symptom	Next check
`state DOWN`	`ip link set dev eth0 up`; if it stays down, cable or driver: `dmesg`, `ethtool eth0`
`NO-CARRIER`	Physical link, or the far-end switch port is disabled
Address missing after reboot	Added with `ip`, not persisted in the network manager
`ip route get` picks the wrong device or source	A more specific route or a policy rule; check `ip rule` and `ip route show table all`
Neighbour `FAILED` or `INCOMPLETE`	No ARP/NDP reply: wrong VLAN, wrong subnet, or filtering
Small packets work, large ones hang	MTU or blocked PMTU discovery: `ping -M do -s 1472 host` (1500-byte path)
Connects, then stalls	Asymmetric routing, or a stateful firewall dropping the return path
`Connection refused`	Host reachable, nothing listening on that address and port; `ss -ltnp`
Timeout with no reply	Firewall dropping, or routing black hole

Name resolution problems are covered in DNS.

Links#

ip -br link                                     # name, state, MAC
ip -s link show eth0                            # RX/TX packets, errors, dropped, overruns
ip -d link show eth0                            # driver-level type details (vlan id, bond mode)
ip link set dev eth0 up
ip link set dev eth0 down                       # drops traffic on eth0
ip link set dev eth0 mtu 9000
ip link set dev eth0 address 02:00:00:00:00:02  # locally administered MAC
ip link add link eth0 name eth0.100 type vlan id 100
ip link add br0 type bridge && ip link set dev eth0 master br0
ip link add bond0 type bond mode 802.3ad
ip link add veth0 type veth peer name veth1     # a pair: packets in one end come out the other
ip link del eth0.100

In ip -s link, RX dropped usually means the host did not keep up (ring buffers, softirq CPU, no socket for the protocol), while RX errors points at the physical layer: cable, optic or switch port. See Linux performance for driver counters.

Addresses#

ip -br addr
ip addr add 192.0.2.5/24 dev eth0
ip addr add 192.0.2.6/24 dev eth0                 # second address in the same subnet becomes "secondary"
ip addr del 192.0.2.5/24 dev eth0
ip addr flush dev eth0                            # removes every address on eth0
ip -6 addr add 2001:db8::5/64 dev eth0
ip addr add 203.0.113.9/32 dev lo                 # service address on loopback (anycast, DSR)

Adding an address with a prefix also adds a connected route for that subnet. A /32 adds no subnet route, which is intended on loopback and a mistake on a physical interface. Deleting the primary address also deletes its secondaries unless net.ipv4.conf.<if>.promote_secondaries=1.

Routes#

ip route                                          # main table
ip route show table all                           # every table, including local
ip route get 198.51.100.7                         # the decision the kernel makes, with source address
ip route get 198.51.100.7 from 192.0.2.5 iif eth0 # as if the packet arrived on eth0
ip route add 198.51.100.0/24 via 192.0.2.1 dev eth0
ip route add default via 192.0.2.1 metric 100
ip route add 10.2.0.0/16 dev wg0                  # on-link through a tunnel, no gateway
ip route replace default via 192.0.2.254          # add or replace in one operation
ip route del 198.51.100.0/24
ip route add blackhole 203.0.113.0/24             # discard silently
ip route add 198.51.100.0/24 nexthop via 192.0.2.1 weight 1 nexthop via 192.0.2.2 weight 1   # ECMP

Within one table the longest matching prefix wins; among routes with the same prefix, the lowest metric wins. Policy rules decide which table is consulted first. ip route get applies rules, tables and source address selection together, so it answers “where will this packet go” more reliably than reading the tables.

Policy routing#

Rules pick a routing table per packet by source, destination, firewall mark, incoming interface and more. A host with two uplinks uses them to answer on the interface a request arrived on.

ip rule                                               # evaluated by priority, lowest number first
ip route add default via 198.51.100.1 dev eth1 table 100
ip rule add from 198.51.100.10 table 100 priority 1000
ip rule add fwmark 0x1 table 100 priority 1001        # mark set by nftables or iptables
ip rule add to 10.9.0.0/16 table 100 priority 1002
ip rule del priority 1000

0:      from all lookup local
1000:   from 198.51.100.10 lookup 100
32766:  from all lookup main
32767:  from all lookup default

Tables can have names in /etc/iproute2/rt_tables (or a file under /etc/iproute2/rt_tables.d/). Since iproute2 6.5 the default copy lives in /usr/share/iproute2/rt_tables and /etc/iproute2 may be empty; a file in /etc takes precedence.

Without the from rule, replies sourced from the second uplink’s address leave through the main default route. The upstream or the receiving host’s reverse path filter then drops them.

sysctl net.ipv4.conf.all.rp_filter net.ipv4.conf.eth1.rp_filter   # 0 off, 1 strict, 2 loose

The kernel uses the higher of the all and per-interface values, so setting only eth1 to 2 has no effect while all is 1.

Neighbours#

ip neigh
ip neigh show dev eth0 nud reachable
ip neigh add 192.0.2.50 lladdr 02:00:00:00:00:32 dev eth0 nud permanent   # static entry
ip neigh flush dev eth0                                  # forces re-resolution
ip -s neigh show 192.0.2.1                               # with usage statistics

State	Meaning
`REACHABLE`	Confirmed recently
`STALE`	Cached, not confirmed recently; verified on next use
`DELAY`, `PROBE`	Verification in progress
`INCOMPLETE`	Request sent, no reply yet
`FAILED`	No reply after all probes
`PERMANENT`, `NOARP`	Static, never verified

A FAILED gateway entry means layer 2 to the gateway is broken, whatever ip link says.

Sockets with ss#

ss -ltnp                                  # listening TCP with owning process (root to see other users')
ss -tunap                                 # TCP and UDP, all states, with process
ss -tn state established '( dport = :443 )'
ss -tn dst 198.51.100.0/24
ss -ti                                    # rtt, cwnd, retrans, delivery rate per socket
ss -tno                                   # timers: retransmit, keepalive, timewait
ss -s                                     # counts by state
ss -tm                                    # socket memory
ss -K dst 198.51.100.7                    # forcibly close matching sockets (root)

ss -K closes only sockets the kernel supports destroying (IPv4 and IPv6, CONFIG_INET_DIAG_DESTROY) and silently skips the rest. It interrupts live connections.

ss -ti separates a slow network from a slow endpoint. High rtt with growing retrans points at the network. A small cwnd without retransmits, or a large Send-Q, points at a receiver that is not reading.

On a listener, Recv-Q is the number of connections waiting for accept() and Send-Q is the backlog limit. With state filters the State column is omitted, so the peer address is column 4, not 5.

Namespaces#

ip netns list
ip netns add ns1
ip link add veth0 type veth peer name veth1
ip link set dev veth1 netns ns1
ip addr add 10.10.0.1/24 dev veth0 && ip link set dev veth0 up
ip netns exec ns1 ip addr add 10.10.0.2/24 dev veth1
ip netns exec ns1 ip link set dev veth1 up
ip netns exec ns1 ip route add default via 10.10.0.1
nsenter -t "$(pgrep -o -f my-app)" -n ss -ltnp    # a container's network namespace, host tools
ip netns del ns1

Any process’s network namespace is reachable through its PID with nsenter -n, so ss, ip or tcpdump from the host can inspect a container image that has none of them. ip netns list only shows namespaces named under /run/netns, not those created by container runtimes.

Bridges#

bridge link                          # ports and their STP state
bridge fdb show br br0               # learned MAC addresses and the port behind each
bridge vlan show                     # VLAN membership per port (vlan_filtering bridges)
ip -d link show br0                  # bridge settings: stp_state, vlan_filtering, ageing_time

A MAC missing from bridge fdb means the bridge has not seen traffic from it on any port. A port stuck in blocking or listening is held by STP.

Tunnels#

ip link add wg0 type wireguard                       # keys and peers are configured with wg(8)
ip link add gre1 type gre local 203.0.113.1 remote 198.51.100.1 ttl 255
ip link add vxlan0 type vxlan id 100 dev eth0 dstport 4789 remote 198.51.100.1
ip link add ipip0 type ipip local 203.0.113.1 remote 198.51.100.1
ip -d link show vxlan0                               # -d prints tunnel parameters

Each encapsulation adds headers: GRE over IPv4 24 bytes, VXLAN over IPv4 50, WireGuard 60 over IPv4 or 80 over IPv6. Set the tunnel MTU at least that much below the underlay MTU. Otherwise large packets are dropped while pings and small requests work, which looks like “some sites load, others hang”.

ip link set dev vxlan0 mtu 1450
ping -M do -s 1422 10.0.5.7          # 1450 minus 20 (IPv4) and 8 (ICMP) header bytes

Traffic control#

tc qdisc show dev eth0
tc -s qdisc show dev eth0                                   # sent, dropped, overlimits, backlog
tc qdisc replace dev eth0 root fq_codel                     # fair queueing with AQM, limits bufferbloat
tc qdisc replace dev eth0 root tbf rate 100mbit burst 32kbit latency 400ms   # shape egress to 100 Mbit/s
tc qdisc replace dev eth0 root netem delay 100ms 10ms loss 0.1%              # add delay, jitter and loss
tc qdisc del dev eth0 root                                  # back to the default qdisc

tc shapes egress only; ingress needs an ifb device or a policer. netem on a test host reproduces latency and loss well enough to find timeout bugs. It applies to all traffic on the interface, including your SSH session, so never leave it on a production interface.

Classful tc: HTB, filters and ingress#

A classless qdisc (fq_codel, tbf, netem) treats all packets the same. To give one kind of traffic a guaranteed share or to impair only one destination, attach a classful qdisc as root, create classes under it and steer packets into classes with filters. Handles are major:minor; the root is 1:, its classes 1:10, 1:20 and so on, and a child qdisc under a class gets its own major number.

tc qdisc add dev eth0 root handle 1: htb default 30                        # unclassified traffic goes to 1:30
tc class add dev eth0 parent 1: classid 1:1 htb rate 1gbit ceil 1gbit      # parent class: the link
tc class add dev eth0 parent 1:1 classid 1:10 htb rate 300mbit ceil 1gbit prio 0   # guaranteed 300, may borrow to 1 Gbit
tc class add dev eth0 parent 1:1 classid 1:20 htb rate 500mbit ceil 1gbit prio 1
tc class add dev eth0 parent 1:1 classid 1:30 htb rate 200mbit ceil 600mbit prio 2
tc qdisc add dev eth0 parent 1:10 fq_codel                                 # AQM inside each leaf, otherwise a FIFO
tc qdisc add dev eth0 parent 1:20 fq_codel
tc qdisc add dev eth0 parent 1:30 fq_codel
tc filter add dev eth0 parent 1: protocol ip prio 1 u32 match ip dport 22 0xffff flowid 1:10        # SSH to the top class
tc filter add dev eth0 parent 1: protocol ip prio 2 u32 match ip dst 198.51.100.0/24 flowid 1:20    # by destination
tc filter add dev eth0 parent 1: protocol ip prio 3 handle 0x5 fw flowid 1:20                       # by nftables/iptables mark 5
tc -s class show dev eth0                  # bytes, packets, dropped, overlimits, borrowed per class
tc filter show dev eth0
tc qdisc del dev eth0 root                 # removes classes and filters with it

rate is the guaranteed share, ceil the maximum when siblings are idle; the sum of children’s rate should not exceed the parent’s. u32 matches on header offsets and is exact but verbose; the fw filter matches a firewall mark, so classification logic can live in nftables with the rest of the policy. flower is the modern alternative to u32 (tc filter add ... flower ip_proto tcp dst_port 443 action ...) and is what hardware offload understands.

Impair one flow instead of the whole interface by hanging netem under a class:

tc qdisc add dev eth0 root handle 1: prio                                  # 3 bands by default, 1:1 1:2 1:3
tc qdisc add dev eth0 parent 1:3 handle 30: netem delay 200ms 20ms loss 1%  # only band 3 is impaired
tc filter add dev eth0 parent 1: protocol ip prio 1 u32 match ip dport 5432 0xffff flowid 1:3   # PostgreSQL traffic only

Ingress shaping redirects incoming packets to an ifb device and shapes its egress:

ip link add ifb0 type ifb && ip link set dev ifb0 up
tc qdisc add dev eth0 handle ffff: ingress
tc filter add dev eth0 parent ffff: protocol all u32 match u32 0 0 action mirred egress redirect dev ifb0
tc qdisc add dev ifb0 root tbf rate 50mbit burst 64kbit latency 400ms       # inbound now limited to 50 Mbit/s

cake (kernel 4.19+) replaces most HTB and fq_codel combinations for a single link: tc qdisc replace dev eth0 root cake bandwidth 100mbit shapes, applies AQM and per-host fairness in one qdisc with sensible defaults; tc -s qdisc show dev eth0 prints its per-tin statistics.

Two uplinks with policy routing#

The full recipe for a host with a management interface (eth0, 192.0.2.5/24, gateway 192.0.2.1) and a second uplink (eth1, 198.51.100.10/24, gateway 198.51.100.1) that must answer on the interface a connection arrived on.

echo '100 uplink2' >> /etc/iproute2/rt_tables.d/local.conf              # name the table (optional)
ip route add 198.51.100.0/24 dev eth1 src 198.51.100.10 table uplink2     # connected route in the table too
ip route add default via 198.51.100.1 dev eth1 table uplink2
ip rule add from 198.51.100.10/32 table uplink2 priority 1000            # replies from that address use eth1
ip rule add iif eth1 table uplink2 priority 1001                          # forwarded traffic that arrived on eth1
ip route get 8.8.8.8 from 198.51.100.10                                   # must show dev eth1 via 198.51.100.1
ip route show table uplink2

Other selectors worth knowing: to (destination), fwmark/mark, ipproto tcp dport 443 (kernel 4.17+), uidrange 1000-1000 (route one user’s traffic, useful for VPN split by account), oif for locally generated traffic bound to a device, and suppress_prefixlength 0, which makes a rule skip a table’s default route while honouring its specific routes. WireGuard’s wg-quick uses that last one: it puts a default route in table 51820 and a not from all fwmark 51820 lookup 51820 rule, so only unmarked traffic enters the tunnel.

ip rule add not fwmark 0xca6c table 51820 priority 5000   # everything except the tunnel's own packets
ip rule add table main suppress_prefixlength 0 priority 4999   # but let specific main-table routes win over the tunnel default
ip -6 rule                                                # rules are per family; IPv6 needs its own set

Rules and per-table routes are not shown by plain ip route; ip route show table all and ip rule together are the whole picture. ip route get remains the fastest check because it applies both.

VLANs, bonds and bridges#

# VLAN sub-interface: tagged frames with id 100 on eth0 appear untagged on eth0.100
ip link add link eth0 name eth0.100 type vlan id 100
ip link set dev eth0.100 up && ip addr add 192.0.2.5/24 dev eth0.100
ip -d link show eth0.100                       # vlan protocol 802.1Q id 100 <REORDER_HDR>
ip link add link eth0 name eth0.200 type vlan id 200 protocol 802.1ad   # QinQ outer tag

# Bond: LACP with fast timers, hash on L3+L4 so one flow stays on one member
ip link add bond0 type bond mode 802.3ad miimon 100 lacp_rate fast xmit_hash_policy layer3+4
ip link set dev eth0 down && ip link set dev eth0 master bond0     # a member must be down when it is added
ip link set dev eth1 down && ip link set dev eth1 master bond0
ip link set dev bond0 up
cat /proc/net/bonding/bond0                    # per-member state, LACP partner, link failure count
ip -d link show bond0                          # mode, active member, hash policy
ip link set dev eth1 nomaster                  # remove a member

# Bridge with VLAN filtering: one bridge, many VLANs, like a managed switch
ip link add br0 type bridge vlan_filtering 1 stp_state 0
ip link set dev eth0 master br0 && ip link set dev eth0 up
ip link set dev vnet0 master br0 && ip link set dev vnet0 up
bridge vlan add dev eth0 vid 100 && bridge vlan add dev eth0 vid 200          # trunk port: tagged 100 and 200
bridge vlan add dev vnet0 vid 100 pvid untagged                              # access port in VLAN 100
bridge vlan del dev vnet0 vid 1                                              # drop the default VLAN
bridge vlan add dev br0 vid 100 self                                         # let the host itself talk on VLAN 100...
ip link add link br0 name br0.100 type vlan id 100                           # ...through a sub-interface on the bridge
bridge vlan show
bridge fdb show br br0 vlan 100
bridge -s link show                            # per-port statistics
bridge link set dev vnet0 learning off flood off   # for ports with a known, single MAC
ip link set dev br0 type bridge ageing_time 30000 forward_delay 1500          # centiseconds

# macvlan and ipvlan: give a container or VM its own address on the parent's segment without a bridge
ip link add link eth0 name mv0 type macvlan mode bridge
ip link add link eth0 name iv0 type ipvlan mode l2

A bond in 802.3ad mode needs LACP configured on the switch; active-backup works against any switch and is the safe default for management links. balance-rr splits one flow across members and causes reordering. miimon polls link state every N ms; without it a failed member is never detected. Check Partner Mac Address and Aggregator ID in /proc/net/bonding/bond0 when a member joins but carries no traffic.

Without vlan_filtering, a bridge floods every VLAN tag to every port and VLAN separation has to be done with one bridge per VLAN and eth0.N sub-interfaces as members. With it, the bridge behaves as a VLAN-aware switch: pvid is the VLAN untagged ingress frames join, untagged strips the tag on egress. Frames from the host stack enter through the bridge device itself, hence the self entry. macvlan and ipvlan interfaces cannot talk to the parent interface’s own addresses; that is a kernel design choice, not a misconfiguration.

Namespaces for testing#

A network namespace has its own links, addresses, routes, rules, sockets, nftables ruleset and /proc/sys/net. Two namespaces joined by a veth pair are enough to reproduce a routing, MTU or firewall problem on a laptop, and netem inside one adds latency without touching a real link.

ip netns add client && ip netns add server
ip link add c0 type veth peer name s0
ip link set c0 netns client && ip link set s0 netns server
ip -n client addr add 10.0.0.1/24 dev c0 && ip -n client link set c0 up && ip -n client link set lo up   # -n runs one ip command in a namespace
ip -n server addr add 10.0.0.2/24 dev s0 && ip -n server link set s0 up && ip -n server link set lo up
ip netns exec server python3 -m http.server 8080 &
ip netns exec client curl -s http://10.0.0.2:8080/ | head -3
ip netns exec client tc qdisc add dev c0 root netem delay 150ms 30ms loss 2%   # impair only the lab
ip netns exec server tcpdump -ni s0 -c 20 tcp port 8080
ip -all netns exec ip -br addr                 # run a command in every named namespace
ip netns pids server                           # processes inside it
ip netns identify "$$"                         # which namespace this shell is in (empty for the root namespace)
ip netns attach ctr "$(podman inspect -f '{{.State.Pid}}' my-app)"   # name a container's namespace so ip -n works (iproute2 5.6+)
ip netns del client; ip netns del server       # veth ends are destroyed with their namespaces

ip netns exec bind-mounts /etc/netns/NAME/ over /etc/ for the command, so a namespace can have its own resolv.conf. Forwarding between a namespace and the outside needs sysctl net.ipv4.ip_forward=1 in the namespace that routes, plus a route or NAT on the host side. systemd services can start inside a named namespace with NetworkNamespacePath=/run/netns/NAME, which is a clean way to pin one daemon to a VPN or a second uplink without policy routing.

JSON output and scripting#

-j returns structured output for ip, bridge and tc, and -p pretty-prints it; ss has no JSON mode, so parse its columns or use -H to drop the header. Field names follow the text output, so ip -j -d link exposes linkinfo.info_kind and linkinfo.info_data.

ip -j addr | jq -r '.[] | .ifname as $i | .addr_info[] | select(.family == "inet") | "\($i) \(.local)/\(.prefixlen)"'
ip -j link | jq -r '.[] | select(.operstate != "UP" and .ifname != "lo") | .ifname'         # links not up
ip -j -d link | jq -r '.[] | select(.linkinfo.info_kind == "vlan") | "\(.ifname) id=\(.linkinfo.info_data.id) on \(.link)"'
ip -j route show table all | jq -r 'group_by(.table // "main") | map("\(.[0].table // "main"): \(length) routes") | .[]'
ip -j neigh | jq -r '.[] | select(.state | index("FAILED")) | "\(.dst) dev \(.dev)"'
tc -j -s qdisc show dev eth0 | jq -r '.[] | "\(.kind) handle \(.handle) drops=\(.drops) backlog=\(.backlog)"'
bridge -j vlan show | jq -r '.[] | "\(.ifname): \([.vlans[].vlan] | join(","))"'
ip -j route get 198.51.100.7 | jq -e '.[0].dev == "eth1"' >/dev/null || echo 'wrong uplink'
ip -ts monitor route                           # timestamped route changes, for a log during an incident

ip -o (one line per object) is the pre-JSON scripting form and still useful with awk; ip -br is for humans. JSON field names are stable across releases in a way column positions are not, so prefer -j in anything that runs unattended.

Oneliners#

# Interfaces that are up, with addresses
ip -br addr | awk '$2 == "UP"'

# Gateway, device and source address for the default path
ip -j route get 198.51.100.1 | jq -r '.[0] | "\(.gateway // "on-link") \(.dev) \(.prefsrc)"'

# Routes on one device as TSV
ip -j route | jq -r '.[] | select(.dev == "eth0") | [.dst, .gateway // "-"] | @tsv'

# Error and drop counters for every interface
ip -s -j link | jq -r '.[] | [.ifname, .stats64.rx.errors, .stats64.rx.dropped, .stats64.tx.errors, .stats64.tx.dropped] | @tsv'

# Top remote addresses by established connection count (IPv4 and IPv6)
ss -tn state established | awk 'NR > 1 {sub(/:[^:]*$/, "", $4); print $4}' | sort | uniq -c | sort -rn | head

# Which process owns a port
ss -ltnp 'sport = :8080'

# Sockets with retransmissions right now
ss -ti | grep -B1 retrans | head -20

# Watch routing, link and address changes
ip monitor route link addr

# Path MTU to a destination
tracepath -n 198.51.100.7 | tail -3

# tcpdump inside a Docker container's network namespace
nsenter -t "$(docker inspect -f '{{.State.Pid}}' my-app)" -n tcpdump -ni any -c 50

# Addresses in every named namespace
ip -br addr; for ns in $(ip netns list | awk '{print $1}'); do echo "== $ns"; ip netns exec "$ns" ip -br addr; done

# Default gateway reachability at layer 2 (state should be REACHABLE or STALE, never FAILED)
ip neigh show "$(ip -j route show default | jq -r '.[0].gateway')"

# Interfaces without carrier
ip -j link | jq -r '.[] | select(.flags | index("NO-CARRIER")) | .ifname'

# MTU of every link, smallest first (tunnels and containers should be below the underlay)
ip -j link | jq -r '.[] | "\(.mtu) \(.ifname)"' | sort -n

# Every route with its protocol, to spot ones added by hand (proto boot or static) versus a daemon (bgp, dhcp, kernel)
ip -j route | jq -r '.[] | [.dst, .dev, .protocol // "-", .gateway // "-"] | @tsv' | column -t

# Delete every route a routing daemon left behind after it crashed (destructive: verify the list first)
ip route show proto bgp; ip route flush proto bgp

# Listening sockets bound to all interfaces rather than a specific address
ss -Hltn | awk '$4 ~ /^(0\.0\.0\.0|\*|\[::\]):/ {print $4}'

# Connections per local port, busiest first
ss -Htan state established | awk '{sub(/.*:/, "", $3); print $3}' | sort | uniq -c | sort -rn | head

# UDP sockets with receive drops (Recv-Q growing means the application is not reading fast enough)
ss -Hlun | awk '$2 > 0'

# Kill every connection from one client address (root; interrupts them mid-flight)
ss -K dst 198.51.100.7

# Connections to a service sorted by round-trip time
ss -Htin dport = :5432 | awk '/rtt:/ {for (i = 1; i <= NF; i++) if ($i ~ /^rtt:/) print $i}' | sort -t: -k2 -rn | head

# Re-resolve a stale or wrong ARP entry for one host
sudo ip neigh del 192.0.2.50 dev eth0 && ping -c1 -W1 192.0.2.50

# Temporarily add a second address and remove it in 10 minutes if you get cut off
sudo ip addr add 192.0.2.99/24 dev eth0 && (sleep 600 && sudo ip addr del 192.0.2.99/24 dev eth0) &

# Route a single test flow through the second uplink without touching the main table
sudo ip rule add to 198.51.100.7 table uplink2 priority 900 && ip route get 198.51.100.7

# Is IP forwarding on, per interface
sysctl -a 2>/dev/null | grep -E 'net.ipv4.conf.(all|eth0).forwarding'

# Which VLANs a trunk port carries on a filtering bridge
bridge -j vlan show dev eth0 | jq -r '.[0].vlans[] | "\(.vlan) \(.flags // [] | join(","))"'

# Bond members and which one is active
awk '/^Slave Interface|^MII Status|^Currently Active/ {print}' /proc/net/bonding/bond0

# Bytes per second on an interface over 5 seconds
a=$(cat /sys/class/net/eth0/statistics/rx_bytes); sleep 5; b=$(cat /sys/class/net/eth0/statistics/rx_bytes); echo "$(( (b - a) / 5 )) B/s"

# Dropped packets per qdisc across all interfaces
tc -s qdisc show | awk '/^qdisc/ {q = $2 " " $5} /dropped/ {print q, $7}' | sort -k3 -rn | head

# Watch for link flaps with timestamps during an incident
ip -ts monitor link

# Namespace of a Kubernetes pod from the node (containerd)
nsenter -t "$(crictl inspect -o go-template --template '{{.info.pid}}' "$(crictl ps -q --name my-app)")" -n ip -br addr

Scripts#

Build a two-namespace lab joined by a veth pair, optionally with netem impairment, and tear it down on exit. Handy for reproducing MTU, timeout and firewall behaviour without touching real interfaces; needs root.

#!/usr/bin/env bash
# usage: netns-lab.sh [--delay 100ms] [--loss 1%] [--mtu 1400] -- command args...
# Runs COMMAND in the "client" namespace with a "server" namespace reachable at 10.0.0.2.
set -euo pipefail

delay='' loss='' mtu=1500
while [[ $# -gt 0 ]]; do
  case $1 in
    --delay) delay=$2; shift 2 ;;
    --loss)  loss=$2; shift 2 ;;
    --mtu)   mtu=$2; shift 2 ;;
    --)      shift; break ;;
    *)       echo "unknown option: $1" >&2; exit 2 ;;
  esac
done
(( $# )) || { echo 'no command given' >&2; exit 2; }
(( EUID == 0 )) || { echo 'run as root' >&2; exit 2; }

cleanup() { ip netns del lab-client 2>/dev/null || true; ip netns del lab-server 2>/dev/null || true; }
trap cleanup EXIT
cleanup                                          # remove leftovers from an interrupted run

ip netns add lab-client && ip netns add lab-server
ip link add lc0 mtu "$mtu" type veth peer name ls0 mtu "$mtu"
ip link set lc0 netns lab-client && ip link set ls0 netns lab-server
for ns in lab-client lab-server; do ip -n "$ns" link set lo up; done
ip -n lab-client addr add 10.0.0.1/24 dev lc0 && ip -n lab-client link set lc0 up
ip -n lab-server addr add 10.0.0.2/24 dev ls0 && ip -n lab-server link set ls0 up
if [[ -n $delay || -n $loss ]]; then
  ip netns exec lab-client tc qdisc add dev lc0 root netem ${delay:+delay "$delay"} ${loss:+loss "$loss"}
fi
ip netns exec lab-server python3 -m http.server --bind 10.0.0.2 8080 >/dev/null 2>&1 &
sleep 0.5
ip netns exec lab-client "$@"

Snapshot the network configuration of a host into one directory so it can be diffed against a later snapshot or another host. Read-only; JSON where the tool supports it.

#!/usr/bin/env bash
# usage: net-snapshot.sh [outdir]
set -euo pipefail

out=${1:-net-snapshot-$(hostname -s)-$(date +%Y%m%dT%H%M%S)}
mkdir -p "$out"

ip -j -d link            > "$out/links.json"
ip -j addr               > "$out/addrs.json"
ip -j route show table all > "$out/routes.json"
ip -j -6 route show table all > "$out/routes6.json"
ip -j rule               > "$out/rules.json"
ip -j -6 rule            > "$out/rules6.json"
ip -j neigh              > "$out/neigh.json"
tc -j -s qdisc show      > "$out/qdisc.json"
bridge -j link show      > "$out/bridge-links.json" 2>/dev/null || true
bridge -j vlan show      > "$out/bridge-vlans.json" 2>/dev/null || true
ss -Hltnup               > "$out/listening.txt"
sysctl -a 2>/dev/null | grep -E '^net\.(ipv4|ipv6|core)\.' | sort > "$out/sysctl.txt"
[[ -d /proc/net/bonding ]] && cat /proc/net/bonding/* > "$out/bonding.txt" 2>/dev/null || true
for ns in $(ip netns list 2>/dev/null | awk '{print $1}'); do
  ip -n "$ns" -j addr > "$out/netns-$ns-addrs.json"
  ip -n "$ns" -j route > "$out/netns-$ns-routes.json"
done
# Normalise volatile fields so two snapshots diff cleanly
for f in "$out"/*.json; do jq -S 'walk(if type == "object" then del(.stats64, .stats, .valid_life_time, .preferred_life_time, .used) else . end)' "$f" > "$f.tmp" && mv "$f.tmp" "$f"; done
printf 'snapshot in %s; compare with: diff -r OLD %s\n' "$out" "$out"

Verify that a host’s routing does what a change intended, before and after applying it: a list of destination expected-device [expected-gateway] lines is checked with ip route get and the script exits non-zero on the first mismatch.

#!/usr/bin/env bash
# usage: route-check.sh expectations.txt
# expectations.txt lines: DEST DEV [GATEWAY]   e.g.  8.8.8.8 eth1 198.51.100.1
set -euo pipefail

file=${1:?expectations file}
rc=0
while read -r dst dev gw _; do
  [[ -z $dst || $dst == \#* ]] && continue
  got=$(ip -j route get "$dst" 2>/dev/null | jq -r '.[0] | "\(.dev) \(.gateway // "-")"') || got='unreachable -'
  read -r gdev ggw <<< "$got"
  if [[ $gdev != "$dev" || ( -n ${gw:-} && $ggw != "$gw" ) ]]; then
    printf 'FAIL %-18s want dev=%s gw=%s  got dev=%s gw=%s\n' "$dst" "$dev" "${gw:--}" "$gdev" "$ggw"; rc=1
  else
    printf 'ok   %-18s dev=%s gw=%s\n' "$dst" "$gdev" "$ggw"
  fi
done < "$file"
exit "$rc"

Troubleshooting#

Error	Cause	Fix
`RTNETLINK answers: Operation not permitted`	Not root, or missing `CAP_NET_ADMIN`	`sudo`
`RTNETLINK answers: File exists`	Address or route already present	`ip addr show` / `ip route show`; use `ip route replace`
`Error: Nexthop has invalid gateway.`	Gateway not on any connected subnet	Add the address first, or add `onlink` if the gateway is reachable anyway
`Cannot find device "eth0"`	Interface name differs (predictable names such as `enp1s0`)	`ip -br link`
`RTNETLINK answers: No such process` on delete	Route or rule does not match exactly	Copy the line from `ip route` or `ip rule` including table and metric
`ss -p` shows no process	Socket owned by another user	Run as root
`Object "..." is unknown`	Old iproute2 or typo in the object name	`ip help`, `ip -V`
`ip netns exec` fails with `No such file or directory`	Namespace not in `/run/netns`	Use `nsenter -t PID -n`, or `ip netns attach NAME PID`
`RTNETLINK answers: Device or resource busy` when enslaving to a bond	Member interface is up or already has a master	`ip link set dev eth1 down nomaster`, then `master bond0`
Bond member shows `MII Status: down` but `ip link` says UP	`miimon` not set, or the switch port is not in the LACP group	Add `miimon 100`; check `Partner Mac Address` in `/proc/net/bonding/bond0`
VLAN sub-interface passes no traffic	Parent is down, switch port is not a trunk for that VLAN, or `vlan_filtering` bridge lacks the VID	`ip -br link` for the parent; `bridge vlan show`; check the switch side
Bridge forwards nothing for 15 s after a port joins	STP `forward_delay` on a new port	Expected; `ip link set br0 type bridge stp_state 0` on a bridge with no loops
Container on macvlan cannot reach the host	macvlan design: no traffic between a sub-interface and the parent	Add a macvlan interface on the host too, or use a bridge
`Error: Exclusivity flag on, cannot modify.` from `tc`	A root qdisc already exists	`tc qdisc replace`, or `tc qdisc del dev eth0 root` first
`tc filter` accepted but class receives nothing	Wrong `parent`, `protocol` or byte offset in `u32`	`tc -s filter show dev eth0` and `tc -s class show dev eth0` to see hit counters; prefer `flower` or `fw`
Policy rule present but `ip route get` ignores it	Table has no matching route, so lookup falls through; or the rule’s `from` address is not the packet’s source	Add the connected route to the table; test with `ip route get DST from SRC`
`ip route get` works but real traffic fails	Reverse path filter or a firewall on the return path	`sysctl net.ipv4.conf.all.rp_filter`; `nft list ruleset`; `tcpdump` on both interfaces
Route disappears seconds after adding it	NetworkManager or systemd-networkd reasserted its configuration	Configure it in the network manager (`nmcli connection modify ... +ipv4.routes`)
`netem` impairs the SSH session too	Root qdisc applies to all egress	Use a `prio` qdisc with a filter for the target port, or run the test in a namespace