# Linux performance

> Find whether CPU, memory, disk or network is the bottleneck on a Linux host with a first-minute checklist, the USE method and per-resource tools.

Canonical: https://www.wiki.jodisand.me/linux-performance/
Reviewed: 2026-09-24
Related: [systemd](https://www.wiki.jodisand.me/systemd/index.md), [iproute2](https://www.wiki.jodisand.me/iproute2/index.md), [Kubernetes](https://www.wiki.jodisand.me/kubernetes/index.md), [Prometheus](https://www.wiki.jodisand.me/prometheus/index.md)


## Cheatsheet

| Question | Command |
| --- | --- |
| Is any resource stalling tasks | `cat /proc/pressure/{cpu,io,memory}` |
| Load and run queue | `uptime`, `vmstat 1 5` |
| Per-CPU utilisation | `mpstat -P ALL 1` |
| Top processes | `top -o %CPU`, `top -o %MEM` |
| Memory that is really available | `free -h` (the `available` column) |
| Disk latency and queue | `iostat -xz 1` |
| Per-process I/O | `pidstat -d 1`, `iotop -o` |
| Network throughput | `sar -n DEV 1` |
| Retransmits and drops | `nstat -az \| grep -iE 'retrans\|drop'`, `ss -ti` |
| Who owns a port | `ss -ltnp` |
| Syscalls a process makes | `strace -c -f -p PID` |
| Where a process is blocked | `cat /proc/PID/wchan`, `sudo cat /proc/PID/stack` |
| CPU profile | `perf top`, `perf record -F 99 -a -g -- sleep 30` |
| Usage per systemd unit | `systemd-cgtop` |

`mpstat`, `pidstat`, `iostat` and `sar` come from the `sysstat` package, which is often not installed by default. The bcc tools are named `biolatency-bpfcc` and so on on Debian and Ubuntu, and installed under `/usr/share/bcc/tools/` on Fedora and RHEL.

## The first minute

Brendan Gregg's 60-second checklist, run in order on a host you know nothing about. Each line either rules a resource out or points at the next section.

```sh
uptime                     # 1, 5, 15 minute load averages: rising or falling
dmesg -T | tail -20        # OOM kills, disk errors, link flaps, hung tasks (may need root)
vmstat 1 5                 # r (runnable), b (blocked), si/so (swapping), us/sy/id/wa/st
mpstat -P ALL 1 3          # one hot CPU hides in the average
pidstat 1 3                # per-process CPU over time, unlike top's snapshot
iostat -xz 1 3             # per device: r/s, w/s, await, aqu-sz, %util
free -h                    # read "available", not "free"
sar -n DEV 1 3             # per-interface rxkB/s and txkB/s against link speed
sar -n TCP,ETCP 1 3        # new connections per second, retransmits per second
top                        # cross-check the picture
```

Ignore the first line of `vmstat` and `iostat` output: it is the average since boot, not the current interval.

Linux load average counts tasks that are runnable plus tasks in uninterruptible sleep (state `D`, usually waiting on disk or NFS). A load of 30 on a host with idle CPUs therefore means blocked I/O, not CPU demand. Compare load against the CPU count (`nproc`) and look at `b` in `vmstat`.

```sh
cat /proc/pressure/cpu /proc/pressure/io /proc/pressure/memory
```

```text
some avg10=12.40 avg60=8.10 avg300=3.02 total=160383532
full avg10=0.00 avg60=0.00 avg300=0.00 total=0
```

Pressure stall information (PSI, kernel 4.20+) measures lost time directly. `some` is the share of time at least one task was stalled on the resource; `full` is the share when all non-idle tasks were stalled at once, meaning no work progressed. `avg10`, `avg60` and `avg300` are percentages over 10, 60 and 300 seconds. System-wide `cpu full` is always zero. Non-zero `memory full` or `io full` is a strong sign of a bottleneck. Every cgroup v2 directory has the same `cpu.pressure`, `memory.pressure` and `io.pressure` files for per-service or per-container views. See the [kernel PSI documentation](https://docs.kernel.org/accounting/psi.html).

If `/proc/pressure` does not exist, the kernel was built without `CONFIG_PSI` or with PSI disabled by default; the `psi=1` kernel parameter enables it in the latter case.

## USE, per resource

The USE method checks every resource for **U**tilisation (how busy), **S**aturation (work queued because it is busy) and **E**rrors. Check saturation even when utilisation looks acceptable: a resource averaging 60% busy can still be saturated in bursts shorter than the sampling interval. Source: Gregg's [USE Linux checklist](https://www.brendangregg.com/USEmethod/use-linux.html).

| Resource | Utilisation | Saturation | Errors |
| --- | --- | --- | --- |
| CPU | `mpstat -P ALL 1`; `vmstat` `us`+`sy`+`st` | `vmstat` `r` above CPU count; PSI cpu; cgroup `nr_throttled` | Machine-check events in `dmesg`, `ras-mc-ctl --errors` |
| Memory | `free -h`, `vmstat` `free` | `vmstat` `si`/`so`; `sar -B` `pgscank`/`pgscand`; PSI memory; OOM kills in `dmesg` | Hardware errors (ECC/EDAC) in `dmesg` |
| Disk | `iostat -xz 1` `%util` | `aqu-sz`, rising `r_await`/`w_await`; PSI io | `dmesg` I/O errors, `smartctl -a /dev/sda` |
| Network | `sar -n DEV 1` against link speed; `ip -s link` | Drops and overruns in `ip -s link`; retransmits in `nstat` | `ip -s link` errors; `ethtool -S eth0` |

## CPU

```sh
mpstat -P ALL 1                  # %usr, %sys, %iowait, %steal, %irq, %soft per CPU
pidstat -u 1                     # per-process %usr and %system
pidstat -w 1                     # voluntary (cswch/s) and involuntary (nvcswch/s) context switches
perf top -F 99                   # live profile by function
perf record -F 99 -a -g -- sleep 30 && perf report --stdio | head -40   # 30 s system-wide with stacks
taskset -cp PID                  # CPU affinity of a process
```

`perf` needs root, or `kernel.perf_event_paranoid` lowered, to profile other users' processes and the kernel. Stacks need frame pointers or debug info; without them `perf report` shows `[unknown]` frames.

| Symptom | Meaning |
| --- | --- |
| High `%usr` | Application work; profile it with `perf` |
| High `%sys` | Kernel work: syscall-heavy code, context switching, network stack, page faults |
| High `%iowait` | CPU idle while I/O is outstanding. A disk symptom, not CPU load, and it falls when other work keeps the CPU busy |
| High `%steal` | The hypervisor ran another guest on this vCPU |
| High `%soft` on one CPU | Network receive processing pinned to one queue; check RSS and IRQ affinity |
| High involuntary context switches | More runnable threads than CPUs, or CPU throttling |
| One CPU at 100%, rest idle | Single-threaded bottleneck, or interrupts on one CPU |

A container or systemd CPU limit (`cpu.max`, `CPUQuota=`) throttles the cgroup for the rest of each period once it uses its quota. Throttling shows as latency while the host looks idle.

```sh
cat /sys/fs/cgroup/system.slice/my-app.service/cpu.stat   # nr_periods, nr_throttled, throttled_usec
kubectl exec my-pod -- cat /sys/fs/cgroup/cpu.max /sys/fs/cgroup/cpu.stat   # inside a pod
```

`nr_throttled` rising between two reads means the limit is too low for the burst pattern, even if average usage is under the limit.

## Memory

```sh
free -h
grep -E 'MemAvailable|Dirty|Writeback|Slab|SUnreclaim|Committed_AS' /proc/meminfo
ps -eo pid,comm,rss,vsz --sort=-rss | head       # resident set size in KiB
smem -rs uss | head                               # unique set size: memory freed if the process exits
slabtop -o | head                                 # kernel object caches (root)
vmstat 1 5                                        # si/so: pages swapped in/out per second
dmesg -T | grep -iE 'out of memory|killed process'
```

`free` counts page cache in `buff/cache`. `available` (`MemAvailable`) estimates how much can be allocated without swapping, including reclaimable cache; that is the number to watch. A full page cache is normal and makes repeat reads fast.

Swap in use is not a problem by itself. Ongoing swap activity (`si`/`so` non-zero every second) is.

```sh
cat /sys/fs/cgroup/system.slice/my-app.service/memory.current
cat /sys/fs/cgroup/system.slice/my-app.service/memory.max
cat /sys/fs/cgroup/system.slice/my-app.service/memory.events   # high, max, oom, oom_kill counters
```

An OOM kill inside a cgroup sends `SIGKILL`, so the process gets no chance to log. Orchestrators report exit code 137 (128 + 9). The `oom_kill` counter in `memory.events` and the `dmesg` line are the evidence. `memory.events` `high` rising means the cgroup is being throttled at `memory.high` / `MemoryHigh=`.

## Disk

```sh
iostat -xz 1                    # r_await/w_await: ms per I/O including queueing; aqu-sz: average queue length
pidstat -d 1                    # kB_rd/s, kB_wr/s, iodelay per process
iotop -oPa                      # accumulated I/O by process (root)
biolatency 10 1                 # block I/O latency histogram (bcc, root)
df -h; df -i                    # space and inodes
lsof +L1                        # deleted files still held open
```

`%util` is the share of time the device had at least one request in flight. For SSDs, NVMe and RAID, which serve many requests in parallel, 100% does not mean saturated; the `iostat` man page says so. Use `await` and `aqu-sz` instead. Older sysstat releases print `avgqu-sz` for `aqu-sz`, and a `svctm` column that should be ignored.

A filesystem that reports full while `df -h` shows free space has run out of inodes (`df -i`) or is `df` on a different mount. Space that `du` cannot account for is usually a deleted file still held open by a process; it is freed only when the process closes it or restarts.

## Network

```sh
sar -n DEV 1                     # rxkB/s, txkB/s, %ifutil per interface
sar -n TCP,ETCP 1                # active/s (outbound), passive/s (inbound), retrans/s
nstat -az | grep -E 'TcpRetransSegs|TcpExtListenDrops|TcpExtListenOverflows|TcpExtTCPSynRetrans'
ss -ti                           # per-socket rtt, cwnd, retrans, delivery rate
ss -ltn 'sport = :8080'          # Recv-Q on a listener = connections waiting for accept()
ip -s link show eth0             # RX/TX errors, dropped, overruns
ethtool -S eth0 | grep -iE 'drop|err|miss|fifo'   # driver-level counters
mtr -rwzbc 100 host.example.com  # per-hop loss and latency
```

`nstat` prints counter deltas since its last run (it keeps history per user); `-a` shows absolute values and `-z` includes zero counters.

| Signal | Meaning |
| --- | --- |
| Retransmits rising | Loss in the path or a congested link; `ss -ti` shows which peers |
| `ListenOverflows` / `ListenDrops` rising | Accept queue full: the application is not calling `accept()` fast enough, or the backlog is small |
| Large `TIME_WAIT` count | Normal for a busy client; a problem only if ephemeral ports run out |
| RX `dropped` on the interface | Host not draining ring buffers fast enough; check `ethtool -g` and softirq CPU |
| RX `errors` | Physical layer: cable, optic, switch port, duplex |
| High or variable `rtt` in `ss -ti` | Queueing in the path, or a distant peer |

```sh
ethtool -g eth0                  # ring buffer sizes, current and maximum
sysctl net.core.somaxconn net.ipv4.tcp_max_syn_backlog
```

For links, routes and sockets see [iproute2](https://www.wiki.jodisand.me/iproute2/#a-connectivity-problem).

## Tracing

```sh
strace -c -f -p PID               # syscall counts and time; Ctrl-C to print the summary
strace -f -tt -e trace=openat,connect -p PID   # specific syscalls with timestamps
execsnoop                         # every exec on the host, with arguments (bcc)
opensnoop -p PID                  # files a process opens (bcc)
tcpconnect                        # outbound TCP connections as they happen (bcc)
biosnoop                          # every block I/O with process and latency (bcc)
funclatency vfs_read              # latency histogram for a kernel function (bcc)
profile -F 99 30                  # sampled stacks for 30 s (bcc)
```

`strace` uses ptrace, stopping the target on every syscall; syscall-heavy processes can slow by an order of magnitude or more. Use it on a replica or for a few seconds. The bcc and `bpftrace` tools run in the kernel via eBPF and summarise there, so overhead is usually low, although tracing very frequent events still costs CPU. All need root.

## perf

`perf` samples or counts hardware and software events. `perf stat` answers "how much" (cycles, instructions, faults, context switches) with almost no overhead; `perf record` answers "where" by sampling stacks at a fixed rate; `perf trace` is an strace replacement that does not stop the target.

```sh
perf list | head -40                                     # events this kernel and CPU expose
perf stat -- ./my-app --once                             # task-clock, context switches, migrations, page faults, cycles, IPC
perf stat -e task-clock,page-faults,context-switches -p "$PID" -- sleep 10   # counters for a running process
perf stat -a -e cycles,instructions,cache-misses -- sleep 5   # whole machine; IPC well under 1 means memory-bound
perf record -F 99 -g -p "$PID" -- sleep 30               # 99 Hz with stacks for one process
perf record -F 99 -a -g --call-graph dwarf -- sleep 10   # unwind without frame pointers; larger perf.data
perf report --stdio --no-children --percent-limit 1      # self time per symbol, top-down
perf report --sort comm,dso                              # which processes and libraries own the samples
perf annotate --stdio -s my_hot_function                 # instruction-level breakdown of one function
perf script | stackcollapse-perf.pl | flamegraph.pl > flame.svg   # FlameGraph scripts, github.com/brendangregg/FlameGraph
perf trace -p "$PID" -- sleep 5                          # syscalls with durations, via tracepoints rather than ptrace
perf trace -s -p "$PID" -- sleep 5                       # per-syscall summary: count, total, min, avg, max
perf record -e sched:sched_switch -a -- sleep 5 && perf sched latency --sort max   # scheduler run-queue delay per task
perf top -e cache-misses                                 # live view for one event
```

Sampling at 99 Hz rather than 100 avoids lockstep with timers. `-g` records the kernel stack and, if the binary has frame pointers, the user stack; Go, Rust and most distribution C libraries keep frame pointers, JIT runtimes and `-fomit-frame-pointer` builds do not, hence `--call-graph dwarf`. `perf.data` is written to the current directory and can be hundreds of megabytes; `perf record -o` puts it elsewhere. `perf sched` needs `CONFIG_SCHEDSTATS` and records every context switch, so keep the window short on busy hosts.

## bpftrace

`bpftrace` attaches short programs to kernel tracepoints, kprobes, user-space probes and timers, aggregates in the kernel and prints maps on exit. It replaces one-off C for the questions that `perf` and the bcc tools do not already answer.

```sh
bpftrace -l 'tracepoint:syscalls:sys_enter_*' | wc -l    # list probes; -lv shows a probe's arguments
bpftrace -lv tracepoint:block:block_rq_issue
bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%-16s %s\n", comm, str(args.filename)); }'
bpftrace -e 'tracepoint:syscalls:sys_enter_* { @[probe] = count(); } interval:s:5 { print(@); clear(@); }'   # syscall rate table every 5 s
bpftrace -e 'kprobe:vfs_read { @bytes[comm] = hist(arg2); }'                    # requested read sizes per command
bpftrace -e 'kretprobe:vfs_read /retval > 0/ { @returned = hist(retval); }'     # actual bytes returned
bpftrace -e 'tracepoint:syscalls:sys_enter_read { @s[tid] = nsecs; }
             tracepoint:syscalls:sys_exit_read /@s[tid]/ { @us[comm] = hist((nsecs - @s[tid]) / 1000); delete(@s[tid]); }'   # read() latency per command
bpftrace -e 'profile:hz:99 /pid == $1/ { @[ustack] = count(); }' "$PID"          # user stacks sampled at 99 Hz, $1 is the first argument
bpftrace -e 'tracepoint:sched:sched_process_exit { printf("%d %s exited\n", pid, comm); }'
bpftrace -e 'kprobe:tcp_retransmit_skb { @[kstack] = count(); }'                # who is retransmitting
bpftrace -e 'uprobe:/usr/lib64/libc.so.6:malloc /pid == $1/ { @sz = hist(arg0); }' "$PID"   # allocation sizes in one process
bpftrace -e 'tracepoint:oom:mark_victim { printf("OOM killed pid %d\n", args.pid); }'
```

Filters go in `/.../` before the block. `@name[key] = hist(x)` builds a power-of-two histogram, `count()`, `sum()`, `avg()`, `min()`, `max()` and `lhist()` (linear) are the other aggregations; `printf` fires per event and floods on hot probes, so aggregate instead. Tracepoints are stable across kernel versions; kprobes attach to any kernel function but the function and its argument order can change, and the argument names from `-lv` need BTF (`/sys/kernel/btf/vmlinux`). `args.field` syntax is bpftrace 0.20+; older releases used `args->field`. The `bpftrace` package ships the classic bcc tools as `.bt` scripts under `/usr/share/bpftrace/tools/`.

## cgroup accounting

Every systemd unit, container and Kubernetes pod is a cgroup v2 directory under `/sys/fs/cgroup`, and the kernel accounts CPU, memory, I/O and PIDs per directory. When the host looks idle but one workload is slow, its cgroup files hold the evidence.

```sh
cat /sys/fs/cgroup/cgroup.controllers          # controllers enabled on this host: cpuset cpu io memory pids ...
cat /proc/"$PID"/cgroup                        # 0::/system.slice/my-app.service: the path under /sys/fs/cgroup
systemctl show -p ControlGroup my-app.service
systemd-cgls --no-pager system.slice/my-app.service   # processes in the unit's cgroup
systemd-cgtop -m -d 2 --depth 4                # live table sorted by memory, refreshed every 2 s

cd /sys/fs/cgroup/system.slice/my-app.service
cat cpu.stat          # usage_usec user_usec system_usec nr_periods nr_throttled throttled_usec nr_bursts burst_usec
cat cpu.max           # "max 100000" is unlimited; "50000 100000" is half a CPU per 100 ms period
cat cpu.weight        # 1..10000, default 100; relative share only when CPUs are contended
cat memory.current memory.max memory.high memory.peak     # memory.peak needs kernel 5.19+
grep -E '^(anon|file|kernel|slab|file_dirty|workingset_refault_file|pgscan|pgsteal) ' memory.stat
cat memory.events     # low high max oom oom_kill: counters, not gauges; diff two reads
cat io.stat           # per device: rbytes wbytes rios wios dbytes dios
cat io.pressure memory.pressure cpu.pressure   # PSI scoped to this cgroup
cat pids.current pids.max
```

Set limits with systemd rather than writing the files, so they survive a restart and appear in the unit's properties. `systemctl set-property` writes a drop-in under `/etc/systemd/system.control/` and applies immediately; `--runtime` keeps it until reboot.

```sh
sudo systemctl set-property my-app.service CPUQuota=200% MemoryHigh=1500M MemoryMax=2G IOWeight=50
sudo systemctl set-property --runtime my-app.service CPUWeight=500     # experiment first, persist later
systemd-run --scope -p MemoryMax=512M -p CPUQuota=50% -- ./batch-job    # one-off command inside a transient cgroup
systemctl show -p CPUQuotaPerSecUSec,MemoryMax,MemoryHigh my-app.service
```

On a Kubernetes node the pod's cgroup is `kubepods.slice/kubepods-<qos>.slice/kubepods-<qos>-pod<uid>.slice/` with one `cri-containerd-<id>.scope` (or `crio-<id>.scope`) per container; `kubectl get pod my-pod -o jsonpath='{.metadata.uid}'` gives the UID with dashes, the slice name replaces them with underscores. Inside a container, `/sys/fs/cgroup` is the container's own cgroup, so `cat /sys/fs/cgroup/cpu.stat` works without knowing the path.

`memory.current` includes page cache the cgroup touched, so it can sit near `memory.max` indefinitely on a healthy I/O-heavy service. The `anon` line in `memory.stat` and `workingset_refault_file` are the better signals: growing anon is the application, growing refaults mean the cache is too small for the working set and reads are going back to disk.

## Memory pressure

Pressure develops in stages: reclaim of clean page cache (cheap), writeback of dirty pages and swap-out (costly, shows in `sar -B` and PSI), then OOM. PSI `memory` `some` above a few percent for a minute means tasks are already waiting on reclaim; `full` means the whole cgroup or host is stalled.

```sh
sar -B 1 5                       # pgscank/s and pgscand/s: kswapd and direct reclaim scanning; pgsteal/s: pages reclaimed; %vmeff
sar -r 1 3                       # kbmemfree, kbavail, %memused, kbdirty
sar -W 1 3                       # pswpin/s, pswpout/s
grep -E 'workingset_refault|pgmajfault|allocstall' /proc/vmstat   # refaults: cache evicted then read again; allocstall: direct reclaim
watch -n1 'cat /proc/pressure/memory'
swapon --show; zramctl          # swap devices; Fedora uses a zram device by default
sysctl vm.swappiness vm.dirty_ratio vm.dirty_background_ratio vm.min_free_kbytes
cat /sys/kernel/mm/transparent_hugepage/enabled   # [always] causes khugepaged CPU and latency spikes for some workloads; madvise is the safe setting
for p in /proc/[0-9]*; do printf '%s %s %s\n' "$(cat "$p/oom_score" 2>/dev/null)" "${p#/proc/}" "$(cat "$p/comm" 2>/dev/null)"; done | sort -rn | head   # most likely OOM victims
```

`MemoryHigh=` (`memory.high`) throttles a cgroup's allocations by forcing reclaim before it reaches the limit, turning a would-be OOM kill into slowness; `MemoryMax=` (`memory.max`) is the hard line where the kernel kills. Set both: `MemoryHigh` somewhat below `MemoryMax` gives a warning window in `memory.events` `high` and PSI before anything dies. `memory.oom.group=1` (`OOMPolicy=kill` in systemd) kills the whole cgroup instead of one process, which keeps a multi-process service consistent.

`systemd-oomd` watches PSI and swap per cgroup and kills the offending slice before the kernel OOM killer runs. `oomctl` shows what it monitors; `ManagedOOMMemoryPressure=kill` and `ManagedOOMMemoryPressureLimit=` on a slice enable it. A service that disappears with `systemd-oomd killed` in the journal was killed by user-space policy, not by the kernel, and `dmesg` will show nothing.

PSI files accept triggers: writing `some 150000 1000000` to `memory.pressure` and polling the descriptor wakes the reader when tasks were stalled for 150 ms within any 1 s window. That is how oomd and container runtimes react within seconds rather than at the next scrape.

> [!WARNING] drop_caches
> `echo 3 > /proc/sys/vm/drop_caches` discards clean page cache and slab objects. It is not a fix for anything: the kernel already frees cache on demand, and the next reads go to disk. Use it only to reset a benchmark.

## I/O latency

`iostat` gives averages per device and interval; a p99 of 200 ms hides inside an `await` of 5 ms when most requests are fast. Latency histograms and per-request traces show the distribution and who caused it.

```sh
iostat -xz 1                     # r_await w_await d_await f_await (discard, flush); aqu-sz; rareq-sz wareq-sz (average request size)
biolatency -D 10 1               # histogram per disk over 10 s (bcc); -Q includes time queued in the block layer
biosnoop                         # each I/O: time, comm, pid, disk, direction, sector, bytes, latency ms
biotop 5                         # top processes by block I/O
ext4slower 10                    # ext4 operations slower than 10 ms with file name; xfsslower, nfsslower, btrfsslower
fileslower 10                    # synchronous file reads and writes above 10 ms, any filesystem
filetop -C                       # busiest files by reads and writes
cat /proc/"$PID"/io              # rchar wchar read_bytes write_bytes: what this process has done since it started
awk '{print "run=" $1/1e6 "ms wait=" $2/1e6 "ms"}' /proc/"$PID"/schedstat   # run-queue wait time for a task
cat /sys/block/nvme0n1/queue/scheduler         # [none] for NVMe, mq-deadline or bfq for SATA and virtual disks
cat /sys/block/sda/queue/nr_requests /sys/block/sda/queue/rotational
ioping -c 10 /var/lib/postgresql               # request latency to a directory's filesystem, like ping
fio --name=randread --filename=/var/tmp/fio.test --size=1G --rw=randread --bs=4k --iodepth=1 --direct=1 --runtime=20 --time_based --group_reporting   # single-queue random read latency; creates a 1 GiB file
lsblk -o NAME,SIZE,TYPE,MOUNTPOINTS,ROTA,SCHED,RQ-SIZE
```

`fio` with `--iodepth=1 --direct=1` measures the latency a database's fsync-bound path sees; raise `iodepth` to measure throughput instead. Delete the test file afterwards. `d_await` and `f_await` (sysstat 12+) separate discards and flushes, which on some virtual disks and consumer SSDs dominate write latency; `fstrim` timers and `fsync`-heavy workloads show there. Steady `aqu-sz` above the device's effective queue depth with rising `await` is saturation; `await` rising while `aqu-sz` stays low points at the device or the layer below it (the hypervisor, the SAN).

## Where to look first

| Symptom | Start with |
| --- | --- |
| Everything slow, load high | `vmstat 1`: compare `r` with `b`, then `us`/`sy`/`wa` |
| One service slow, host looks fine | Its cgroup PSI and `cpu.stat` throttling; `ss -ti` to its dependencies |
| Latency spikes at intervals | GC pauses, log rotation, cron or timers, snapshotting, CPU throttling |
| Slow after a deploy | Compare `perf` profiles before and after |
| Slow only under load | Queueing: accept backlog, connection pool size, thread count |
| Memory grows steadily | Leak or page cache; compare `smem` USS over time, not `free` |
| Disk full but files deleted | `lsof +L1` |
| Process in state `D` for a long time | `cat /proc/PID/wchan`, `dmesg` for `hung_task` warnings, NFS or storage errors |

## Oneliners

```sh
# Top 10 processes by resident memory
ps -eo pid,comm,rss --sort=-rss | head -11

# Processes averaging over 1% CPU across 10 seconds
LC_ALL=C pidstat -u 1 10 | awk '/^Average/ && $8 > 1 {print $8, $NF}' | sort -rn | head

# Processes writing to disk now (kB_wr/s)
LC_ALL=C pidstat -d 1 3 | awk '$5 ~ /^[0-9.]+$/ && $5 > 0 {print $5, $NF}' | sort -rn | head

# Established connections by remote address
ss -tn state established | awk 'NR > 1 {sub(/:[^:]*$/, "", $4); print $4}' | sort | uniq -c | sort -rn | head

# Sockets in each TCP state
ss -tan | awk 'NR > 1 {print $1}' | sort | uniq -c | sort -rn

# Largest directories on one filesystem
du -x -d 1 /var 2>/dev/null | sort -h | tail

# Deleted files still consuming space: size, command, path
sudo lsof +L1 | awk 'NR > 1 {print $7, $1, $10}' | sort -rn | head

# Interrupts and context switches per second
vmstat 1 5 | awk 'NR > 2 {print "in=" $11, "cs=" $12}'

# Threads of a process by CPU
top -H -b -n 1 -p "$PID" | head -20

# Live CPU and memory by cgroup
systemd-cgtop -m --depth=3

# Record 20 s of stacks for a flame graph
sudo perf record -F 99 -a -g -- sleep 20 && sudo perf script > out.perf

# PSI for every resource on one line, refreshed each second
watch -n1 'for r in cpu io memory; do printf "%-7s" $r; head -1 /proc/pressure/$r; done'

# Services with the highest memory PSI over the last minute
for f in /sys/fs/cgroup/system.slice/*/memory.pressure; do awk -v u="${f%/memory.pressure}" '/^some/ {split($3,a,"="); print a[2], u}' "$f"; done | sort -rn | head

# Units currently being CPU-throttled (nr_throttled rising is the real test; this shows any throttling since start)
grep -H '^nr_throttled' /sys/fs/cgroup/system.slice/*/cpu.stat | awk -F'[:/ ]' '$NF > 0 {print $NF, $(NF-2)}' | sort -rn | head

# Memory usage per unit, largest first
for d in /sys/fs/cgroup/system.slice/*/; do [ -f "$d/memory.current" ] && printf '%s %s\n' "$(numfmt --to=iec < "$d/memory.current")" "$(basename "$d")"; done | sort -h | tail

# Who was OOM-killed, from the journal, with cgroup path
journalctl -k -g 'oom-kill\|Killed process' --no-pager -n 20

# Kills by systemd-oomd (user-space OOM) in the last day
journalctl -u systemd-oomd --since -1d --no-pager

# Processes in uninterruptible sleep and what they are waiting on
ps -eo pid,stat,wchan:32,comm | awk '$2 ~ /D/'

# Major page faults per process (reads from disk to satisfy a fault)
ps -eo pid,maj_flt,comm --sort=-maj_flt | head

# Per-CPU softirq distribution: one column much higher than the rest means IRQ imbalance
awk '/NET_RX/ {for (i = 2; i <= NF; i++) printf "%s ", $i; print ""}' /proc/softirqs

# IRQs and the CPUs handling them
grep -E 'eth0|nvme' /proc/interrupts | awk '{s = 0; for (i = 2; i < NF - 2; i++) s += $i; print $1, s, $NF}'

# Count syscalls by name for 10 s, no ptrace
sudo bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); } interval:s:10 { exit(); }'

# Files being opened right now, by process
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%-6d %-16s %s\n", pid, comm, str(args.filename)); }'

# Off-CPU time by user stack for one process, 10 s (why it is waiting, not what it is running)
sudo offcputime -p "$PID" -u 10 | head -60

# Block I/O latency histogram per disk for 30 s
sudo biolatency -D 30 1

# fsync latency of a process (database checkpoints, log flushes)
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_fsync /pid == $1/ { @s[tid] = nsecs; } tracepoint:syscalls:sys_exit_fsync /@s[tid]/ { @ms = hist((nsecs - @s[tid]) / 1000000); delete(@s[tid]); }' "$PID"

# Cycles, instructions and IPC for one process over 10 s
sudo perf stat -e cycles,instructions,task-clock -p "$PID" -- sleep 10

# Scheduler run-queue latency: time from wakeup to running, as a histogram
sudo runqlat 10 1

# Hottest kernel and user functions, whole host, 10 s, text output
sudo perf record -F 99 -a -g -- sleep 10 >/dev/null && sudo perf report --stdio --no-children --percent-limit 2 | grep -v '^#' | head -40

# CPU frequency per core (throttled or power-saving cores show low values)
grep MHz /proc/cpuinfo | sort | uniq -c

# Container's own CPU throttling from inside the container
kubectl exec my-pod -- sh -c 'cat /sys/fs/cgroup/cpu.stat | grep -E "nr_throttled|throttled_usec"'
```

## Scripts

Snapshot of the health of every systemd service's cgroup: memory against its limit, CPU throttling and 60-second PSI, sorted by memory pressure. Read-only; run as root for units that restrict access.

```sh
#!/usr/bin/env bash
# usage: cgroup-report.sh [slice]   default system.slice
set -euo pipefail

slice=${1:-system.slice}
base=/sys/fs/cgroup/$slice
[[ -d $base ]] || { echo "no cgroup at $base" >&2; exit 2; }

psi60() { awk '/^some/ {split($3, a, "="); print a[2]}' "$1" 2>/dev/null || echo 0; }
val() { [[ -r $1 ]] && cat "$1" || echo -; }

printf '%-40s %10s %10s %8s %8s %8s\n' UNIT MEM MAX THROTTLED MEMPSI IOPSI
for d in "$base"/*/; do
  unit=$(basename "$d")
  [[ -f $d/memory.current ]] || continue
  mem=$(numfmt --to=iec < "$d/memory.current")
  max=$(val "$d/memory.max"); [[ $max == max ]] || max=$(numfmt --to=iec <<< "$max")
  thr=$(awk '/^nr_throttled/ {print $2}' "$d/cpu.stat" 2>/dev/null || echo -)
  printf '%-40s %10s %10s %8s %8s %8s\n' "$unit" "$mem" "$max" "$thr" "$(psi60 "$d/memory.pressure")" "$(psi60 "$d/io.pressure")"
done | sort -k5 -rn
```

Alert when PSI on any resource stays above a threshold for a sustained window, logging a snapshot of the top consumers when it fires. Suitable as a systemd service for hosts without a metrics pipeline.

```sh
#!/usr/bin/env bash
# usage: psi-watch.sh [threshold-percent] [interval-seconds]
set -euo pipefail

threshold=${1:-10} interval=${2:-10}
[[ -d /proc/pressure ]] || { echo 'PSI not available (boot with psi=1)' >&2; exit 2; }

snapshot() {
  {
    printf '=== %s: %s avg60=%s%% above %s%%\n' "$(date -Is)" "$1" "$2" "$threshold"
    uptime
    head -1 /proc/pressure/{cpu,io,memory}
    ps -eo pid,pcpu,rss,stat,comm --sort=-pcpu | head -6
    ps -eo pid,pcpu,rss,stat,comm --sort=-rss | head -6
    systemd-cgtop -b -n 1 --depth 2 -m | head -12
  } | logger -t psi-watch
}

while :; do
  for r in cpu io memory; do
    avg60=$(awk '/^some/ {split($3, a, "="); print a[2]}' /proc/pressure/"$r")
    if (( $(printf '%.0f' "$avg60") >= threshold )); then snapshot "$r" "$avg60"; fi
  done
  sleep "$interval"
done
```

Collect a 30-second performance bundle before restarting a misbehaving service, so the evidence survives the restart. Writes into a timestamped directory under `/var/tmp` and needs root for `perf` and the cgroup files.

```sh
#!/usr/bin/env bash
# usage: perf-bundle.sh my-app.service [seconds]
set -euo pipefail

unit=${1:?unit required} secs=${2:-30}
out=/var/tmp/perf-bundle-${unit%.service}-$(date +%Y%m%dT%H%M%S)
mkdir -p "$out"
cg=$(systemctl show -p ControlGroup --value "$unit")
[[ -n $cg ]] || { echo "$unit has no cgroup (not running?)" >&2; exit 1; }
mainpid=$(systemctl show -p MainPID --value "$unit")

{ uptime; free -h; cat /proc/pressure/{cpu,io,memory}; } > "$out/host.txt"
for f in cpu.stat memory.current memory.max memory.stat memory.events io.stat cpu.pressure memory.pressure io.pressure; do
  cp "/sys/fs/cgroup$cg/$f" "$out/cgroup-$f" 2>/dev/null || true
done
systemctl status --no-pager -l "$unit" > "$out/status.txt" || true
journalctl -u "$unit" -n 500 --no-pager > "$out/journal.txt" || true
ss -tanpi > "$out/sockets.txt"
vmstat 1 "$secs" > "$out/vmstat.txt" &
iostat -xz 1 "$secs" > "$out/iostat.txt" &
pidstat -u -r -d -p "$mainpid" 1 "$secs" > "$out/pidstat.txt" &
perf record -F 99 -g -p "$mainpid" -o "$out/perf.data" -- sleep "$secs" 2> "$out/perf-record.log" || true
wait
perf report -i "$out/perf.data" --stdio --no-children --percent-limit 1 > "$out/perf-report.txt" 2>/dev/null || true
tar -C "$(dirname "$out")" -czf "$out.tar.gz" "$(basename "$out")" && rm -rf "$out"
printf 'bundle: %s.tar.gz\n' "$out"
```

## Troubleshooting the tools

| Symptom | Cause | Fix |
| --- | --- | --- |
| `mpstat: command not found` | `sysstat` not installed | Install `sysstat` |
| `sar` reports "Cannot open /var/log/sa/..." | History collection not enabled | Use the interval form (`sar -n DEV 1`), or enable `sysstat` collection |
| `perf` shows `[unknown]` frames | No frame pointers or symbols | Install debug info, or `perf record --call-graph dwarf` |
| `perf: Permission denied` | `kernel.perf_event_paranoid` too high | Run as root |
| bcc tool fails to compile | Missing kernel headers or BTF | Install headers for the running kernel, or use `bpftrace`/libbpf-tools builds |
| `/proc/pressure` missing | PSI disabled | Boot with `psi=1` |
| `/proc/PID/stack` permission denied | Needs root | `sudo` |
| Container shows host-wide numbers | `/proc` is not namespaced for most counters | Read the cgroup files under `/sys/fs/cgroup` |
| `bpftrace: ERROR: ... args.filename` or `no BTF found` | Kernel lacks BTF, so tracepoint argument names are unknown | Install `kernel-debuginfo` or use `args->filename` on old bpftrace; check `ls /sys/kernel/btf/vmlinux` |
| `kprobe` attaches but never fires | Function was inlined or renamed in this kernel | `bpftrace -l 'kprobe:*name*'` to find the real symbol; prefer a tracepoint |
| `perf record` writes gigabytes | Too high a frequency or too many events on a busy host | `-F 99`, limit with `-p PID` or `-C cpu`, shorten the `sleep` |
| `perf report` shows only `[kernel.kallsyms]` and hex | `kptr_restrict` or missing user symbols | `sysctl kernel.kptr_restrict=0` as root; install debuginfo for the binary |
| `cpu.stat` has no `nr_throttled` line | Unit has no `CPUQuota=`, so the cpu controller is not enforcing a bandwidth limit | Expected; look at the parent slice or set a quota |
| `memory.peak: No such file` | Kernel older than 5.19 | Sample `memory.current` over time instead |
| `systemd-cgtop` shows `-` for CPU or I/O | Accounting for that controller is off for the slice | `systemctl set-property SLICE CPUAccounting=yes IOAccounting=yes` (default on cgroup v2 for most distributions) |
| `ethtool -S` empty or `Operation not supported` | Virtual NIC without driver statistics | Use `ip -s link` and the guest driver's counters (`virtio`, `ena`) |
| `fio` numbers far better than the application sees | Test hit the page cache or a different device | `--direct=1`, `--fsync=1` for durability paths, check `df` of the test path |
| `systemd-oomd` killed a service that looked fine | Slice-level `ManagedOOMMemoryPressureLimit` exceeded by a sibling's swap or PSI | `oomctl`, `journalctl -u systemd-oomd`; raise the limit or move the service to its own slice |

Tool map and background: Brendan Gregg's [Linux performance](https://www.brendangregg.com/linuxperf.html) page. Field definitions: `man 5 proc`.

## Further reading

- [Kernel: PSI - Pressure Stall Information](https://docs.kernel.org/accounting/psi.html): the semantics of `some`, `full` and triggers.
- [Kernel: Control Group v2](https://docs.kernel.org/admin-guide/cgroup-v2.html): every `cpu.*`, `memory.*`, `io.*` file and how limits interact.
- [perf wiki tutorial](https://perf.wiki.kernel.org/index.php/Tutorial): `perf stat`, `record`, `report` and event selection.
- [bpftrace reference guide](https://github.com/bpftrace/bpftrace/blob/master/docs/reference_guide.md): probe types, builtins and aggregation functions.
- [bcc tools reference](https://github.com/iovisor/bcc/blob/master/README.md): one page per tool with example output.
- [systemd.resource-control(5)](https://www.freedesktop.org/software/systemd/man/latest/systemd.resource-control.html): `CPUQuota=`, `MemoryHigh=`, `IOWeight=`, `ManagedOOMMemoryPressure=` and how they map to cgroup files.