Software Engineering WikiSE Wiki

Cisco IOS

Diagnose and configure Cisco IOS and IOS XE devices: show commands, interfaces, VLANs, OSPF, BGP, ACLs and changes that roll back if they lock you out.

Reviewed MarkdownEdit

On this page

Cheatsheet#

TaskCommand
Stop paging for this sessionterminal length 0
Running configshow running-config
One interface’s configshow running-config interface Gi1/0/1
One section of the configshow running-config | section router ospf
Interface summaryshow ip interface brief
Switch port statusshow interfaces status
Interface detail and countersshow interfaces Gi1/0/1
Errors only, quicklyshow interfaces | include line protocol|error|drops
MAC tableshow mac address-table
ARPshow ip arp
Neighboursshow cdp neighbors detail, show lldp neighbors detail
Routing tableshow ip route
Route used for one addressshow ip route 192.0.2.7
VLANsshow vlan brief
Trunk statusshow interfaces trunk
Logshow logging
Uptime, image, last reload reasonshow version
Savecopy running-config startup-config
Safety net before a changereload in 10 then reload cancel

Interface names differ by platform: Gi0/1 on older IOS, Gi1/0/1 on Catalyst 9000 stacks (switch/module/port), Gi0/0/0 on IOS XE routers. Examples below use Gi1/0/1.

How IOS configuration works#

IOS keeps two configurations. running-config is in memory and every configure terminal command changes it immediately; there is no candidate or commit. startup-config is in NVRAM and is what the device loads after a reload. A change you have not saved disappears on reboot, which is both a risk and the basis of the reload in safety net.

The CLI has modes: user EXEC (>), privileged EXEC (#, after enable), global config ((config)#) and sub-modes such as (config-if)#. show commands run from privileged EXEC; from config mode prefix them with do, as in do show ip interface brief.

Getting oriented#

enable
terminal length 0
show version
show inventory
show running-config | include hostname|ip route|username
show processes cpu sorted | exclude 0.00%
show memory statistics
show logging | begin <Mon DD HH:MM>

show tech-support collects everything for a TAC case and is too large to read; use targeted show commands with output filters instead.

FilterEffect
| include XLines matching regex X
| exclude XLines not matching X
| section XWhole configuration blocks whose header matches X
| begin XEverything from the first match onward
| count XNumber of lines matching X
| redirect flash:out.txtWrite output to a file instead of the screen

The filter argument is a regular expression. | inside it means OR, _ matches a space or line boundary.

Show commands by question#

QuestionCommand
Is the hardware healthyshow environment all, show platform (IOS XE), show power inline (PoE budget per port)
What optic is in the port and what light levelshow interfaces Gi1/0/1 transceiver detail
Which stack member is master, stack healthshow switch, show switch stack-ports
Port-channel members and their stateshow etherchannel summary (flags: P bundled, D down, s suspended, I individual)
First-hop redundancyshow standby brief, show vrrp brief, show glbp brief
Which routing protocols run, with timers and networksshow ip protocols
OSPF interface cost, timers, DR/BDRshow ip ospf interface Gi1/0/1
BGP table for one prefix, best path and whyshow ip bgp 192.0.2.0/24
What CEF will do with a packetshow ip cef 192.0.2.7 detail, show ip cef exact-route 10.0.0.1 192.0.2.7
DHCP server stateshow ip dhcp binding, show ip dhcp pool, show ip dhcp conflict
NAT translations and hit countsshow ip nat translations, show ip nat statistics
Port security state and violationsshow port-security, show port-security interface Gi1/0/2
Who is logged in and from whereshow users, show line
Time, NTP sync and driftshow clock detail, show ntp status, show ntp associations
Syslog destinations and levelsshow logging | section Logging
Files on flash, free spacedir flash:, show file systems
Boot image and install modeshow boot, show version | include image|mode, show install summary (IOS XE)
IP SLA probes and resultsshow ip sla summary, show ip sla statistics 1
Multicast groups and IGMP snoopingshow ip igmp snooping groups, show ip mroute
TCAM and hardware resource usageshow platform hardware fed switch active fwd-asic resource tcam utilization (Catalyst 9000)
Reason for the last crashshow version | include reload, dir crashinfo:, show logging onboard

show ip cef is the answer when show ip route says one thing and packets do another: the FIB is what the hardware forwards on, and a stale adjacency or a recursive route that fails to resolve shows here first.

EtherChannel, first-hop redundancy and port security#

Bundle links with LACP (active) rather than PAgP or on; both ends must agree on speed, duplex, trunk mode and allowed VLANs or the member is suspended.

interface range GigabitEthernet1/0/23 - 24
 channel-group 1 mode active            ! creates interface Port-channel1
interface Port-channel1
 switchport mode trunk
 switchport trunk allowed vlan 10,20,30
 switchport nonegotiate
show etherchannel summary
show etherchannel 1 detail | include Protocol|State|Port-channel
show lacp neighbor

HSRP puts a virtual gateway address on a VLAN across two switches. preempt lets the higher-priority router take over when it returns, and tracking an uplink lowers the priority when the path upstream is gone so the standby with a working uplink takes over.

interface Vlan20
 ip address 192.0.2.2 255.255.255.0
 standby version 2
 standby 20 ip 192.0.2.1
 standby 20 priority 110
 standby 20 preempt delay minimum 60
 standby 20 track 1 decrement 20
!
track 1 interface GigabitEthernet1/0/1 line-protocol
show standby brief                       ! P = preempt, state Active/Standby, virtual IP
show track brief

Port security limits which MAC addresses may send on an access port. sticky learns and writes them into the running config; violation restrict drops offending frames and logs, while the default shutdown err-disables the port.

interface GigabitEthernet1/0/2
 switchport port-security
 switchport port-security maximum 2
 switchport port-security mac-address sticky
 switchport port-security violation restrict
!
errdisable recovery cause psecure-violation
errdisable recovery interval 300

Management hardening#

The baseline every device should have before it carries traffic: SSH only, a management ACL on the VTY lines, NTP with authentication, syslog to a collector, AAA with local fallback, and no plaintext services.

hostname sw-01
ip domain name example.com
crypto key generate rsa modulus 4096
ip ssh version 2
ip ssh server algorithm encryption aes256-gcm aes256-ctr
!
no ip http server
no ip http secure-server                 ! unless RESTCONF or a web GUI is required
no service pad
no ip source-route
service password-encryption
service timestamps log datetime msec localtime show-timezone
!
username admin privilege 15 secret <secret>       ! secret hashes it (type 9 scrypt on current IOS XE); password does not
enable secret <secret>
!
ip access-list standard MGMT-HOSTS
 permit 192.0.2.0 0.0.0.255
!
line vty 0 15
 transport input ssh
 access-class MGMT-HOSTS in
 exec-timeout 10 0
 logging synchronous
!
ntp authentication-key 1 hmac-sha2-256 <key>
ntp authenticate
ntp trusted-key 1
ntp server 192.0.2.123 key 1
clock timezone AEST 10 0
!
logging host 192.0.2.50
logging trap informational
logging buffered 64000 informational
logging source-interface Vlan999
!
aaa new-model
tacacs server ise-01
 address ipv4 192.0.2.60
 key <secret>
aaa group server tacacs+ ISE
 server name ise-01
aaa authentication login default group ISE local
aaa authorization exec default group ISE local if-authenticated
aaa accounting commands 15 default start-stop group ISE

Enable aaa new-model from a session you are sure of, with reload in 10 running and a second session open: it applies to the VTY lines the moment it is entered, and the local fallback only works if a local user already exists. show aaa servers and test aaa group ISE admin <password> new-code prove the TACACS path before you depend on it.

Configuration backup#

The archive block from the change-safety section doubles as a local backup, and path can point at a remote server so every write memory uploads a copy. Log every configuration command while you are at it; show archive log config all then answers who changed what.

archive
 path scp://backup@192.0.2.40/cisco/$h-$t
 write-memory                            ! archive on every save
 time-period 1440                        ! and daily regardless
 log config
  logging enable
  logging size 500
  hidekeys
  notify syslog contenttype plaintext
copy running-config scp://backup@192.0.2.40/cisco/sw-01.cfg      ! one-off, prompts for the password
copy running-config tftp://192.0.2.40/sw-01.cfg                   ! plaintext transport; lab or isolated management network only
copy running-config flash:pre-change.cfg                          ! local copy before a change; flash survives a reload
show archive
show archive log config all
show archive config differences flash:archive/sw-01-3 system:running-config
configure replace flash:archive/sw-01-3 list                      ! restore a backup, printing each command

Prefer pulling from a collector over pushing from the device: a script that runs show running-config over SSH on a schedule and commits the result to Git gives history, diffs and review for free, and does not need credentials for the backup server on every switch. See Network automation for the scripted form, and the backup option of the Ansible cisco.ios.ios_config module. Whichever way, back up before every change, restore into a lab or a spare device occasionally to prove the copies are usable, and treat the files as secrets: they contain type 7 passwords, SNMP communities and keys.

Interfaces#

Routed port on a Layer 3 switch:

configure terminal
interface GigabitEthernet1/0/1
 description uplink to core-01
 no switchport
 ip address 198.51.100.2 255.255.255.252
 no shutdown
end

Access port:

interface GigabitEthernet1/0/2
 switchport mode access
 switchport access vlan 20
 spanning-tree portfast
 spanning-tree bpduguard enable

portfast skips the listening and learning delay, which is only safe where no switch will ever connect. bpduguard err-disables the port if a BPDU arrives, which catches the case where someone plugs a switch in anyway.

Trunk port:

interface GigabitEthernet1/0/24
 switchport mode trunk
 switchport nonegotiate
 switchport trunk allowed vlan 10,20,30
 switchport trunk native vlan 999

nonegotiate turns off DTP so the port cannot be talked into or out of trunking. An unused native VLAN keeps untagged frames out of production VLANs.

Caution

switchport trunk allowed vlan 40 replaces the list. To add a VLAN use switchport trunk allowed vlan add 40. Forgetting add on an uplink drops every other VLAN.

show interfaces status
show interfaces Gi1/0/1 | include errors|drops|duplex|rate
show interfaces counters errors
show interfaces status err-disabled
clear counters GigabitEthernet1/0/1
CounterMeaning
input errors / CRCPhysical layer: cable, optic, dirty fibre, or duplex mismatch
late collisionsDuplex mismatch, almost always
output dropsEgress congestion: more traffic is queued for the interface than it can send
input queue dropsThe CPU is not keeping up with traffic punted to it
interface resetsLink flapping or keepalive failures; check the far end and the optic

Counters accumulate since boot or the last clear counters. Clear them, wait, and read them again to see whether errors are still increasing.

An err-disabled port stays down until shutdown then no shutdown, or until errdisable recovery cause <cause> re-enables it automatically. show interfaces status err-disabled names the cause (for example bpduguard or psecure-violation).

VLANs and spanning tree#

vlan 20
 name servers
exit
interface Vlan20
 ip address 192.0.2.1 255.255.255.0
 no shutdown

An SVI (interface Vlan20) comes up only when the VLAN exists and at least one port in it, or a trunk carrying it, is up.

show vlan brief
show spanning-tree vlan 20
show spanning-tree root
show spanning-tree inconsistentports
spanning-tree vlan 20 root primary

Set the root bridge explicitly. Without it the switch with the lowest MAC address wins, often the oldest one, and a re-election triggers topology changes that flush MAC tables across the VLAN. show interfaces trunk shows which VLANs actually forward on a trunk, which can differ from the allowed list once spanning tree blocking and VTP pruning apply.

Routing#

Static routes:

ip route 203.0.113.0 255.255.255.0 198.51.100.1 name to-dc2
show ip route static
show ip route 203.0.113.20

show ip route <address> shows the entry that wins by longest prefix match, which is the one that matters when two routes overlap.

OSPF:

key chain OSPF-KEYS
 key 1
  key-string <secret>
  cryptographic-algorithm hmac-sha-256
!
router ospf 1
 router-id 198.51.100.2
 passive-interface default
 no passive-interface GigabitEthernet1/0/1
 network 198.51.100.0 0.0.0.3 area 0
!
interface GigabitEthernet1/0/1
 ip ospf authentication key-chain OSPF-KEYS

passive-interface default stops hellos on every interface except the ones you name, so OSPF only forms adjacencies where intended. Key-chain authentication with HMAC-SHA is configured per interface on IOS XE; see OSPFv2 cryptographic authentication.

show ip ospf neighbor
show ip ospf interface brief
show ip ospf database
Neighbour stateMeaning
DOWNNo hellos received
INITHellos received, but they do not list this router yet (one-way)
2WAYBidirectional. Normal final state between two DROTHERs on a broadcast segment
EXSTART/EXCHANGEDatabase exchange. Stuck here usually means an MTU mismatch
LOADINGRequesting LSAs it is missing
FULLAdjacency complete

No neighbour at all usually means mismatched hello/dead timers, area, subnet, authentication, or a passive interface.

BGP:

router bgp 65001
 bgp log-neighbor-changes
 neighbor 203.0.113.1 remote-as 64511
 neighbor 203.0.113.1 password <secret>
 address-family ipv4
  network 192.0.2.0 mask 255.255.255.0
  neighbor 203.0.113.1 activate
  neighbor 203.0.113.1 prefix-list TO-PEER out
  neighbor 203.0.113.1 maximum-prefix 1000 90 restart 15

network only advertises a prefix that exists in the routing table with exactly that mask; add a static route to Null0 for an aggregate. maximum-prefix 1000 90 restart 15 warns at 90 % and tears the session down above 1000 prefixes, retrying after 15 minutes. An outbound prefix list plus an inbound maximum-prefix on every external peer stops a local mistake from leaking routes to the internet.

show ip bgp summary
show ip bgp neighbors 203.0.113.1 advertised-routes
show ip bgp neighbors 203.0.113.1 routes
show ip bgp 192.0.2.0/24
clear ip bgp 203.0.113.1 soft in

show bgp ipv4 unicast ... is the address-family-aware form of the same commands. clear ip bgp ... soft re-applies policy without resetting the session; a hard clear ip bgp <peer> drops it and withdraws its routes.

ACLs#

ip access-list extended MGMT-IN
 permit tcp 192.0.2.0 0.0.0.255 any eq 22
 permit icmp any any echo-reply
 deny   ip any any log
!
interface GigabitEthernet1/0/1
 ip access-group MGMT-IN in
show access-lists MGMT-IN
show ip interface Gi1/0/1 | include access list

Entries are evaluated top down and the first match wins. Wildcard masks are inverted subnet masks: 0.0.0.255 matches a /24. Every ACL ends with an implicit deny ip any any that does not log or count, so add an explicit one to see hit counts. To restrict SSH to the device itself, apply a standard or extended ACL to the VTY lines with access-class rather than to an interface.

An ACL on your management path ends your session

Add the permit for your own source first, schedule reload in 10, apply the ACL, verify from a second session, then reload cancel.

Recovery and change safety#

Scheduled reload: if the change cuts you off, the device reboots into the saved startup-config. Do not save until you have verified.

reload in 10
! make the change, then verify from a new session
reload cancel

The reload drops all traffic for the reboot time, so on production gear prefer a confirmed change, which reverts only the configuration. It needs a configuration archive:

archive
 path flash:archive/$h-
 write-memory
 maximum 14
configure replace flash:intended.cfg time 5
! verify from a new session within 5 minutes
configure confirm

configure replace ... time 5 swaps in the whole file and reverts automatically unless configure confirm arrives in time. configure revert now rolls back immediately and configure revert timer 15 resets the timer to 15 minutes. See Configuration Rollback Confirmed Change.

For line-by-line edits, configure terminal revert timer 5 starts a revertible session that is confirmed the same way.

Unverified

Cisco’s feature history lists configure terminal as modified by this feature, but the revert timer syntax was not confirmed on a current command reference page. Test it on a lab device first.

show archive
show archive config differences nvram:startup-config system:running-config
configure replace nvram:startup-config list   ! roll running config back to the saved one, printing each command

Password recovery needs console access and a reload into ROMMON to bypass the startup config. Arrange console or out-of-band access before changing AAA or management ACLs.

Troubleshooting#

ping 203.0.113.7 source Vlan20 repeat 100 size 1400 df-bit
traceroute 203.0.113.7 source Vlan20
show ip arp 203.0.113.7
show mac address-table address 0011.2233.4455
show processes cpu history

ping ... size 1400 df-bit finds MTU problems: if smaller sizes pass and this fails, something on the path has a lower MTU.

SymptomWhere to look
Intermittent loss on one portshow interfaces counters: CRC, late collisions, resets
Port down, will not come upshow interfaces status err-disabled, then the far end and optic
Works locally, fails across a trunkshow interfaces trunk: allowed, forwarding and pruned VLANs
Hosts in the same VLAN cannot reach each otherPort security, private VLAN, protected port, or wrong access VLAN
Traffic takes an unexpected pathshow ip route <dest>, then routing protocol metrics and administrative distance
High CPUshow processes cpu sorted; traffic punted to the CPU (ARP storms, TTL expiry, logging ACLs) is a common cause
OSPF stuck in EXSTARTInterface MTU differs between neighbours
BGP stuck in Active or IdleTCP 179 not reachable, wrong remote-as, wrong source address, or MD5 password mismatch (the log shows it)
Neighbour flappingPhysical layer first, then MTU, timers and authentication
Port-channel member s (suspended) or I (individual)Mismatched trunk/VLAN/speed settings, or the far end is not running LACP; show etherchannel summary, show lacp neighbor
Both HSRP routers ActiveThey cannot see each other’s hellos: VLAN not carried on the trunk between them, or an ACL blocking 224.0.0.102 (v2) / 224.0.0.2 (v1)
Hosts lose the gateway after a failoverPreempt without a delay, or the standby has no working uplink; show standby brief, show track brief
Port err-disabled with psecure-violationMore MACs than maximum, often a hub, phone or VM host; show port-security interface
Logs show %SYS-5-CONFIG_I from an unknown sourceSomeone (or an automation account) changed config; show archive log config all, show users
show ntp status says unsynchronizedServer unreachable, authentication mismatch, or stratum 16; show ntp associations (* marks the selected peer)
Locked out after aaa new-modelNo local user or the server group is unreachable; console in, or wait for reload in
show ip cef shows a different next hop from show ip routeRecursive route unresolved or adjacency incomplete; show ip cef <prefix> detail, show adjacency
Optic shows RX power below the thresholdDirty or damaged fibre, wrong optic type for the distance; show interfaces transceiver detail
PoE device does not power onBudget exhausted or port limited; show power inline, show power inline Gi1/0/5 detail

debug output goes to the CPU and the log. On a busy device it can overwhelm the control plane.

access-list 100 permit ip host 192.0.2.10 host 203.0.113.7
debug ip packet 100 detail
undebug all

Warning

Never run debug ip packet without an ACL on a production device. It shows only packets handled by the CPU (process switched), not CEF-switched transit traffic, so an empty result does not prove traffic is absent. Have undebug all ready before you start. Prefer show counters when they answer the question.

Oneliners#

! Interfaces with protocol down, excluding admin down
show ip interface brief | exclude up|administratively

! Ports with errors
show interfaces counters errors

! Which port a MAC is on, then what that port is
show mac address-table address 0011.2233.4455
show interfaces Gi1/0/7 status

! Configuration differences since the last save
show archive config differences nvram:startup-config system:running-config

! Uptime, image and last reload reason
show version | include uptime|System image|Last reload

! Prefixes received from each BGP peer (PfxRcd column)
show ip bgp summary

! CPU-heavy processes right now
show processes cpu sorted | exclude 0.00%

! Log entries from a point in time
show logging | begin Sep 15 09:

! Confirm an ACL is matching
show ip access-lists MGMT-IN | include matches

! Count configured interfaces
show running-config | count ^interface

! Ports that are up but have no description (undocumented ports)
show interfaces description | include ^Gi.*up +up *$

! Ports that have been down for a long time (candidates to reclaim)
show interfaces | include line protocol is down|Last input

! Trunks and the VLANs actually forwarding on each
show interfaces trunk | begin forwarding

! Every VLAN's SVI state
show ip interface brief | include Vlan

! Port-channels with a member that is not bundled
show etherchannel summary | include \(SU\)|\(SD\)|\(s\)|\(I\)|\(D\)

! Top MAC counts per VLAN (a VLAN with thousands is a candidate for a loop or a flat network)
show mac address-table count

! Spanning-tree root for every VLAN and whether this switch is it
show spanning-tree root

! Ports currently blocking or in a transitional STP state
show spanning-tree | include BLK|LRN|LIS

! Recent topology changes and where the last one came from
show spanning-tree detail | include ieee|occurr|from|is exec

! OSPF neighbours that are not FULL (2WAY on a DR segment is fine)
show ip ospf neighbor | exclude FULL

! BGP peers not Established (State/PfxRcd column shows a word, not a number)
show ip bgp summary | include Idle|Active|Connect|OpenSent

! Routes learned from a BGP peer, count only
show ip bgp neighbors 203.0.113.1 routes | include Total

! Routes by source (connected, static, OSPF, BGP)
show ip route summary

! Default route and where it comes from
show ip route 0.0.0.0

! ARP entries for a subnet, to find who is live
show ip arp 192.0.2.0 255.255.255.0

! Which switch port an IP address is on: ARP for the MAC, then the MAC table
show ip arp 192.0.2.10
show mac address-table address 0011.2233.4455

! DHCP snooping bindings on an access switch (IP to port map without a scan)
show ip dhcp snooping binding

! Interface with the most output drops
show interfaces | include ^[A-Z].*is up|Total output drops

! Half-duplex or 10/100 ports on a gigabit switch (cabling or negotiation problems)
show interfaces status | include a-half|a-100|a-10 |10 |100

! Config lines that will be a problem: telnet, http, plaintext SNMP communities
show running-config | include transport input|ip http|snmp-server community

! Unsaved changes: a non-empty diff means "write memory" is pending
show archive config differences nvram:startup-config system:running-config

! What changed in the last archive interval, with the user
show archive log config all | tail 20

! Type 7 passwords still present (weak encoding; migrate to secret)
show running-config | include password 7

! Free flash before an image copy
dir flash: | include bytes free

! NTP status in one line
show ntp status | include synchronized|stratum

! Environment alarms only
show environment all | include FAULT|Alarm|NOT PRESENT|Critical

! Reload reason and uptime for a stack of switches
show version | include uptime|Last reload|System image

! IOS XE: install-mode packages and whether a reload is pending
show install summary | include IMG|SMU

! Save and archive in one line
write memory

Scripts#

Back up every device’s running configuration over SSH into a Git repository and commit only when something changed, so history is a diff per device per change.

#!/usr/bin/env bash
# ios-backup.sh HOSTS_FILE REPO_DIR: pull running-config from each device and commit changes
# Uses public-key SSH; the device needs "ip ssh pubkey-chain" configured for the backup user.
set -euo pipefail
hosts=$1; repo=$2
cd "$repo"
failed=0
while read -r host; do
  [ -n "$host" ] && [ "${host#\#}" = "$host" ] || continue
  if ! ssh -o BatchMode=yes -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new "$host" 'terminal length 0
show running-config' 2>/dev/null \
      | sed -E '/^(Building configuration|Current configuration|! Last configuration change|! NVRAM config last updated|ntp clock-period)/d' > "$host.cfg.tmp"; then
    echo "FAILED $host" >&2; rm -f "$host.cfg.tmp"; failed=$((failed + 1)); continue
  fi
  grep -q '^hostname ' "$host.cfg.tmp" || { echo "FAILED $host: no hostname line, output incomplete" >&2; rm -f "$host.cfg.tmp"; failed=$((failed + 1)); continue; }
  mv "$host.cfg.tmp" "$host.cfg"
done < "$hosts"
git add -- '*.cfg'
if git diff --cached --quiet; then echo "no changes"; else
  git diff --cached --stat
  git commit -qm "Config backup $(date -u +%FT%TZ)" && echo "committed"
fi
exit $(( failed > 0 ))

Poll interface error counters across a fleet twice, a minute apart, and report only the interfaces whose CRC, input error or output drop counters increased, which separates live faults from historical noise.

#!/usr/bin/env python3
"""Report interfaces with increasing error counters.

Usage: iface-errors.py hosts.txt [interval_seconds]
Credentials from NET_USER, NET_PASS (and NET_ENABLE if enable is needed).
"""
import os
import sys
import time
from concurrent.futures import ThreadPoolExecutor

from netmiko import ConnectHandler

hosts = [h.strip() for h in open(sys.argv[1]) if h.strip() and not h.startswith("#")]
interval = int(sys.argv[2]) if len(sys.argv) > 2 else 60
WATCH = ("input_errors", "crc", "output_errors", "interface_resets")   # field names from the ntc-templates "show interfaces" template

def snapshot(host):
    dev = {"device_type": "cisco_ios", "host": host, "username": os.environ["NET_USER"],
           "password": os.environ["NET_PASS"], "secret": os.environ.get("NET_ENABLE", ""), "conn_timeout": 10}
    with ConnectHandler(**dev) as c:
        if dev["secret"]:
            c.enable()
        rows = c.send_command("show interfaces", use_textfsm=True)
    if not isinstance(rows, list):
        raise RuntimeError("no TextFSM template match")
    return {r["interface"]: {k: int(r.get(k) or 0) for k in WATCH} for r in rows}

def collect():
    out, errors = {}, {}
    with ThreadPoolExecutor(max_workers=10) as pool:
        for host, res in zip(hosts, pool.map(lambda h: _safe(snapshot, h), hosts)):
            (errors if isinstance(res, Exception) else out)[host] = res
    return out, errors

def _safe(fn, arg):
    try:
        return fn(arg)
    except Exception as exc:  # report per host rather than abort the sweep
        return exc

first, err1 = collect()
time.sleep(interval)
second, err2 = collect()
for host, exc in {**err1, **err2}.items():
    print(f"{host}: ERROR {exc}", file=sys.stderr)
found = False
for host in sorted(set(first) & set(second)):
    for iface, before in first[host].items():
        after = second[host].get(iface)
        if not after:
            continue
        delta = {k: after[k] - before[k] for k in WATCH if after[k] > before[k]}
        if delta:
            found = True
            print(f"{host}\t{iface}\t" + " ".join(f"{k}+{v}" for k, v in delta.items()))
if not found:
    print(f"no counters increased in {interval}s across {len(second)} devices")

Further reading#

For scripting any of this across many devices, see Network automation.