Linux performance · Software Engineering Wiki

Cheatsheet #

Question	Command
Overall pressure	`cat /proc/pressure/{cpu,io,memory}`
Load and run queue	`uptime`, `vmstat 1 5`
Per-CPU utilisation	`mpstat -P ALL 1`
Top processes	`top -o %CPU`, `top -o %MEM`
Memory truth	`free -h`, `cat /proc/meminfo`
Disk saturation	`iostat -xz 1`
Per-process I/O	`pidstat -d 1`, `iotop -o`
Network throughput	`sar -n DEV 1`, `nstat -az`
Retransmits and drops	`nstat \| grep -iE 'retrans\|drop'`, `ss -ti`
Who owns this port	`ss -ltnp`
System calls a process makes	`strace -c -p PID`
Where a process is stuck	`cat /proc/PID/stack`, `cat /proc/PID/wchan`
Live flame data	`perf top`, `perf record -F 99 -a -g -- sleep 30`
Per-cgroup usage	`systemd-cgtop`

The first minute #

uptime                              # load average: runnable + uninterruptible
dmesg -T | tail -20                 # OOM kills, disk errors, link flaps
vmstat 1 5                          # r, b, si/so, us/sy/id/wa
mpstat -P ALL 1 3                   # a single hot CPU hides in the average
pidstat -u 1 3                      # per-process CPU
iostat -xz 1 3                      # %util, await, aqu-sz per device
free -h                             # available, not free
sar -n DEV 1 3                      # interface throughput
ss -s                               # socket summary
top -o %CPU

Load average counts processes in uninterruptible sleep as well as running ones, so a load of 30 on an idle-looking box usually means disk or NFS waits, not CPU.

cat /proc/pressure/cpu /proc/pressure/io /proc/pressure/memory

Pressure stall information is the most direct answer available: some avg10 is the share of the last ten seconds during which at least one task was stalled on that resource. Anything consistently above ~10% is a real bottleneck.

USE, per resource #

For each resource, check utilisation, saturation and errors — in that order, because a resource can be saturated while utilisation still looks acceptable.

Resource	Utilisation	Saturation	Errors
CPU	`mpstat -P ALL 1`	run queue `r` in `vmstat`, PSI cpu	throttling in cgroup stats
Memory	`free -h`, `/proc/meminfo`	swapping `si/so`, PSI memory	`dmesg` OOM kills
Disk	`iostat -x` `%util`	`await`, `aqu-sz`, PSI io	`smartctl -a`, `dmesg`
Network	`sar -n DEV`	`nstat` retransmits, `ss -ti` cwnd	`ip -s link`, `netstat -i` errors

CPU #

mpstat -P ALL 1                  # %usr, %sys, %iowait, %steal per core
pidstat -u 1                     # per-process
perf top -F 99                   # live symbol-level profile
perf record -F 99 -a -g -- sleep 30 && perf report --stdio | head -40
cat /proc/PID/status | grep -E 'voluntary|Threads'
taskset -cp PID                  # CPU affinity

Symptom	Meaning
High `%usr`	Application work; profile it
High `%sys`	Kernel work: syscall storms, context switching, network stack
High `%iowait`	Blocked on disk, not a CPU problem
High `%steal`	The hypervisor is giving CPU to someone else
High involuntary context switches	More runnable threads than cores
One core at 100%, rest idle	Single-threaded bottleneck, or interrupt affinity

In containers, CPU limits throttle rather than kill: check cpu.stat for nr_throttled and throttled_usec. Throttling shows up as latency with plenty of apparent headroom.

cat /sys/fs/cgroup/cpu.stat                                    # cgroup v2
kubectl exec pod -- cat /sys/fs/cgroup/cpu.max

Memory #

free -h
cat /proc/meminfo | grep -E 'MemAvailable|Dirty|Writeback|Slab|Committed_AS'
ps -eo pid,comm,rss,vsz --sort=-rss | head
smem -rs uss | head                     # unique set size: what would be freed
slabtop -o | head                       # kernel object caches
dmesg -T | grep -i 'killed process'

free reports page cache as used; MemAvailable is the number that predicts whether an allocation will succeed. Cache is not waste — it is why the second read is fast.

Swapping (si/so in vmstat) is the failure mode, not swap being allocated. A little swap in use with no paging activity is fine.

cat /sys/fs/cgroup/memory.current /sys/fs/cgroup/memory.max
cat /sys/fs/cgroup/memory.events           # oom, oom_kill counters

An OOM kill inside a container reports exit code 137 to the orchestrator and may leave no application log at all — memory.events is the evidence.

Disk #

iostat -xz 1                    # await = latency, %util = busy time, aqu-sz = queue depth
pidstat -d 1                    # per-process read/write
iotop -oPa                      # accumulated I/O by process
biolatency-bpfcc 10 1           # block layer latency histogram
df -h; df -i                    # space and inodes
lsof +L1                        # deleted files still held open, consuming space

%util near 100% on an SSD does not mean saturated — NVMe devices handle many requests in parallel. await and queue depth are the honest signals.

Filesystem full with df -h showing space free means inodes (df -i) or a deleted-but-open file. The latter frees only when the process closes it or restarts.

Network #

sar -n DEV 1                    # per-interface throughput
sar -n TCP,ETCP 1               # active/passive opens, retransmits
nstat -az | grep -E 'TcpRetransSegs|TcpExtListenDrops|TcpExtTCPSynRetrans'
ss -ti                          # per-socket: rtt, cwnd, retrans, pacing
ss -ltn '( sport = :8080 )'
ip -s link show eth0            # errors, drops, carrier changes
tcpdump -ni eth0 -c 100 'port 443 and tcp[tcpflags] & tcp-syn != 0'
mtr -rwzbc 100 api.example.com  # latency and loss per hop

Signal	Meaning
Retransmits rising	Loss somewhere in the path, or a saturated link
`ListenDrops`/`ListenOverflows`	Accept backlog full: application not accepting fast enough
`TIME_WAIT` in the tens of thousands	Normal for a busy client; only a problem if ports exhaust
RX drops on the interface	Ring buffer or CPU cannot keep up — check `ethtool -S`
High rtt variance in `ss -ti`	Queueing in the path, or a distant peer

ethtool -S eth0 | grep -E 'drop|error|miss'
ethtool -g eth0                 # ring buffer sizes
sysctl net.core.somaxconn net.ipv4.tcp_max_syn_backlog

Tracing #

strace -c -f -p PID             # syscall counts and time, quick and blunt
strace -f -e trace=openat,connect -p PID
ltrace -p PID
execsnoop-bpfcc                 # every exec on the box
opensnoop-bpfcc -p PID          # files being opened
tcpconnect-bpfcc                # outbound connections as they happen
biosnoop-bpfcc                  # per-I/O latency with process attribution
funclatency-bpfcc 'vfs_read'    # latency histogram for a kernel function
profile-bpfcc -F 99 30          # sampled stacks, low overhead

strace stops the process at every syscall and can slow it by an order of magnitude — safe for a quick count on a spare replica, not on a hot production process. The bcc/eBPF tools sample instead and cost a few percent.

Where to look first #

Symptom	Start with
Everything is slow, load is high	`vmstat 1`, then `%iowait` versus `%sys` versus `%usr`
One service is slow, box looks fine	Its own latency breakdown; then `ss -ti` to its dependencies
Latency spikes at intervals	GC, log rotation, cron, `bgsave`, or CPU throttling
Slow after a deploy	New version’s CPU or allocation profile; compare `perf` output
Slow only under load	Queueing: check backlog, pool sizes, thread counts
Memory grows steadily	Leak, or page cache — confirm with `smem` USS, not RSS
Disk full but files deleted	`lsof +L1`

Oneliners #

# Top 10 processes by resident memory
ps -eo pid,comm,rss --sort=-rss | head -11

# CPU time by process over 10 seconds
pidstat -u 1 10 | awk '/Average/ && $8 > 1 {print $8, $NF}' | sort -rn | head

# Which process is writing to disk right now
pidstat -d 1 3 | awk '$5 > 0 {print $5, $NF}' | sort -rn | head

# Established connections by remote address
ss -tn state established | awk 'NR>1 {split($4,a,":"); print a[1]}' | sort | uniq -c | sort -rn | head

# Sockets in each state
ss -tan | awk 'NR>1 {print $1}' | sort | uniq -c | sort -rn

# Largest directories under a path
du -x --max-depth=1 /var | sort -h | tail

# Deleted files still consuming space
lsof +L1 | awk '{print $7, $9}' | sort -rn | head

# Context switches and interrupts per second
vmstat 1 5 | awk 'NR>2 {print "cs="$12, "in="$11}'

# Threads of a process, sorted by CPU
top -H -b -n1 -p "$PID" | head -20

# Per-cgroup CPU and memory, live
systemd-cgtop -m --depth=3

# What a stuck process is waiting on
cat /proc/$PID/wchan; echo; cat /proc/$PID/stack 2>/dev/null | head

# Quick flame graph data
perf record -F 99 -a -g -- sleep 20 && perf script > out.perf