Software Engineering WikiSE Wiki

Operations scripts

Complete Bash and Python scripts for disk alerts, certificate expiry, health checks, backups, cleanup, audits, metrics export and parallel SSH, ready to drop into cron or a timer.

Reviewed MarkdownEdit

On this page

Cheatsheet#

TaskScript
Alert when a filesystem passes a thresholddisk-usage-alert
Report certificates about to expirecert-expiry
Health check with retries and meaningful exit codeshttp-health
Compress and expire old log fileslog-rotate
Incremental rsync backups with retentionbackup-rsync
Remove finished Jobs and dead pods (dry run by default)k8s-stale-cleanup
Report or prune unused container imagesimage-prune-report
Sweep hosts and ports for reachabilityport-sweep
Failed units with their last log linessystemd-failures
Fast-forward every repository under a directorygit-bulk-update
Sync a directory to S3 and verifys3-sync
Audit local accountsuser-audit
Block until a dependency answerswait-for
Export shell-gathered metrics to Prometheustextfile-metrics
Run a command on many hosts in parallelparallel-ssh

Every Bash script targets Bash 5 with GNU coreutils on a Linux host, starts with set -euo pipefail, validates its arguments, reads secrets from the environment or files rather than flags, and passes shellcheck -s bash. The Python scripts need only the standard library on Python 3.11 or later. Exit codes follow the Nagios convention where it fits: 0 OK, 1 warning, 2 critical, 3 usage or unknown. The building blocks are explained in Bash and Python; shorter single commands live in Shell one-liners.

Install a script under /usr/local/bin, keep it owned by root and mode 0755, and run it from a systemd timer rather than cron so the journal captures its output and systemctl list-timers shows when it last ran.

Disk usage alert#

Reads df once, compares each real filesystem’s space and inode usage against a threshold, prints the offenders and exits 1 if any crosses the warning level or 2 for the critical level. The output is one line per filesystem, so a timer can pipe it to logger or a chat webhook without further parsing. Pseudo-filesystems, overlay mounts and /snap are skipped.

usage: disk-usage-alert [-w PERCENT] [-c PERCENT] [-x MOUNT]...
#!/usr/bin/env bash
# Alert when any local filesystem's space or inode usage exceeds a threshold.
set -euo pipefail

warn=80
crit=90
exclude=()

usage() {
  cat >&2 <<'EOF'
usage: disk-usage-alert [-w PERCENT] [-c PERCENT] [-x MOUNT]...
  -w  warning threshold, percent used (default 80)
  -c  critical threshold, percent used (default 90)
  -x  mountpoint to skip; repeatable
exit: 0 ok, 1 warning, 2 critical, 3 usage
EOF
  exit 3
}

while getopts ':w:c:x:h' opt; do
  case $opt in
    w) warn=$OPTARG ;;
    c) crit=$OPTARG ;;
    x) exclude+=("$OPTARG") ;;
    *) usage ;;
  esac
done
shift $((OPTIND - 1))
[[ $# -eq 0 ]] || usage
[[ $warn =~ ^[0-9]+$ && $crit =~ ^[0-9]+$ && $warn -le 100 && $crit -le 100 ]] || usage
(( warn <= crit )) || { printf 'warning threshold must not exceed critical\n' >&2; exit 3; }

skip() {
  local m
  for m in "${exclude[@]+"${exclude[@]}"}"; do [[ $m == "$1" ]] && return 0; done
  return 1
}

status=0
check() {                       # check <label> <mount> <percent>
  local label=$1 mount=$2 pct=$3
  if (( pct >= crit )); then
    printf 'CRITICAL %s %s%% %s\n' "$mount" "$pct" "$label"; status=2
  elif (( pct >= warn )); then
    printf 'WARNING %s %s%% %s\n' "$mount" "$pct" "$label"; (( status < 1 )) && status=1
  fi
}

# -P gives POSIX single-line output; -x excludes pseudo filesystems.
while read -r _ _ _ _ pct mount; do
  skip "$mount" && continue
  check space "$mount" "${pct%\%}"
done < <(df -P -l -x tmpfs -x devtmpfs -x overlay -x squashfs -x efivarfs | tail -n +2)

while read -r _ _ _ _ pct mount; do
  skip "$mount" && continue
  [[ $pct == '-' ]] && continue       # filesystems without inode accounting (btrfs, xfs report a value)
  check inodes "$mount" "${pct%\%}"
done < <(df -P -l -i -x tmpfs -x devtmpfs -x overlay -x squashfs -x efivarfs | tail -n +2)

(( status == 0 )) && printf 'OK all filesystems below %s%%\n' "$warn"
exit "$status"

Certificate expiry checker#

Connects to every host[:port] in a file, reads the served certificate’s notAfter, and prints days remaining per host sorted soonest first. Unreachable hosts are reported separately and make the script exit 2, so a monitoring job distinguishes “expiring” from “cannot tell”. The port defaults to 443 and SNI is always sent, which is what most virtual-hosted endpoints require. Set STARTTLS=smtp in the environment for a mail server on 25 or 587.

usage: cert-expiry [-d DAYS] [-t SECONDS] HOSTS_FILE
#!/usr/bin/env bash
# Report TLS certificates expiring within DAYS for each host[:port] in a file.
set -euo pipefail

days=30
connect_timeout=5

usage() {
  cat >&2 <<'EOF'
usage: cert-expiry [-d DAYS] [-t SECONDS] HOSTS_FILE
  -d  warn when fewer than DAYS remain (default 30)
  -t  connect timeout per host in seconds (default 5)
HOSTS_FILE has one host or host:port per line; # comments and blank lines are ignored.
STARTTLS=smtp|imap|pop3|ftp|postgres in the environment enables STARTTLS.
exit: 0 ok, 1 something expires within DAYS, 2 a host was unreachable, 3 usage
EOF
  exit 3
}

while getopts ':d:t:h' opt; do
  case $opt in
    d) days=$OPTARG ;;
    t) connect_timeout=$OPTARG ;;
    *) usage ;;
  esac
done
shift $((OPTIND - 1))
[[ $# -eq 1 && -r $1 ]] || usage
[[ $days =~ ^[0-9]+$ && $connect_timeout =~ ^[0-9]+$ ]] || usage
hosts_file=$1

starttls_args=()
[[ -n ${STARTTLS:-} ]] && starttls_args=(-starttls "$STARTTLS")

now=$(date +%s)
status=0
results=()
failures=()

while IFS= read -r line || [[ -n $line ]]; do
  line=${line%%#*}; line=${line//[[:space:]]/}
  [[ -z $line ]] && continue
  host=${line%%:*}
  port=${line#*:}; [[ $port == "$line" ]] && port=443

  if ! cert=$(timeout "$connect_timeout" openssl s_client -connect "$host:$port" -servername "$host" \
        "${starttls_args[@]+"${starttls_args[@]}"}" </dev/null 2>/dev/null); then
    failures+=("$host:$port connect failed"); continue
  fi
  if ! not_after=$(printf '%s' "$cert" | openssl x509 -noout -enddate 2>/dev/null); then
    failures+=("$host:$port no certificate in response"); continue
  fi
  expiry=$(date -d "${not_after#notAfter=}" +%s)
  remaining=$(( (expiry - now) / 86400 ))
  issuer=$(printf '%s' "$cert" | openssl x509 -noout -issuer -nameopt RFC2253 | sed 's/^issuer=//; s/,.*//')
  results+=("$remaining $host:$port $issuer")
  (( remaining < days )) && status=1
done < "$hosts_file"

if (( ${#results[@]} )); then
  printf '%s\n' "${results[@]}" | sort -n | while read -r rem hp iss; do
    flag=OK; (( rem < days )) && flag=EXPIRING; (( rem < 0 )) && flag=EXPIRED
    printf '%-9s %5d days  %-40s %s\n' "$flag" "$rem" "$hp" "$iss"
  done
fi
if (( ${#failures[@]} )); then
  printf 'UNREACHABLE %s\n' "${failures[@]}" >&2
  status=2
fi
exit "$status"

HTTP health check#

Probes one or more URLs with a timeout, retries with exponential backoff, and exits 0 only when every URL returns the expected status. Any other outcome is classified: 1 when a URL returned a wrong status after all retries, 2 when a URL never answered. Python’s urllib is used rather than curl so the retry logic, per-attempt timing and body check live in one place with no external dependency. A bearer token is read from HEALTH_TOKEN if set; it never appears on the command line.

usage: http-health [--expect 200] [--retries 3] [--timeout 5] [--contains TEXT] URL...
#!/usr/bin/env python3
"""Check URLs with retries; exit 0 ok, 1 wrong status or body, 2 unreachable, 3 usage."""
import argparse
import os
import sys
import time
import urllib.error
import urllib.request


def probe(url: str, expect: int, timeout: float, contains: str | None, token: str | None) -> tuple[bool, str]:
    req = urllib.request.Request(url, headers={"User-Agent": "http-health/1"})
    if token:
        req.add_header("Authorization", f"Bearer {token}")
    start = time.monotonic()
    try:
        with urllib.request.urlopen(req, timeout=timeout) as resp:  # noqa: S310 - URL comes from the operator
            status = resp.status
            body = resp.read(65536).decode("utf-8", "replace")
    except urllib.error.HTTPError as exc:  # non-2xx still counts as a response
        status = exc.code
        body = exc.read(65536).decode("utf-8", "replace")
    except (urllib.error.URLError, TimeoutError, OSError) as exc:
        return False, f"unreachable ({exc.reason if hasattr(exc, 'reason') else exc})"
    elapsed = time.monotonic() - start
    if status != expect:
        return False, f"status {status}, expected {expect} ({elapsed:.2f}s)"
    if contains and contains not in body:
        return False, f"body missing {contains!r} ({elapsed:.2f}s)"
    return True, f"status {status} ({elapsed:.2f}s)"


def main() -> int:
    ap = argparse.ArgumentParser(description="HTTP health check with retries")
    ap.add_argument("urls", nargs="+", metavar="URL")
    ap.add_argument("--expect", type=int, default=200, help="expected HTTP status (default 200)")
    ap.add_argument("--retries", type=int, default=3, help="attempts per URL (default 3)")
    ap.add_argument("--timeout", type=float, default=5.0, help="seconds per attempt (default 5)")
    ap.add_argument("--contains", help="text the body must contain")
    args = ap.parse_args()
    if args.retries < 1 or args.timeout <= 0 or not 100 <= args.expect <= 599:
        ap.error("retries must be >= 1, timeout > 0, expect a valid HTTP status")
    for url in args.urls:
        if not url.startswith(("http://", "https://")):
            ap.error(f"not an HTTP URL: {url}")

    token = os.environ.get("HEALTH_TOKEN")
    worst = 0
    for url in args.urls:
        ok, detail = False, ""
        for attempt in range(1, args.retries + 1):
            ok, detail = probe(url, args.expect, args.timeout, args.contains, token)
            if ok:
                break
            if attempt < args.retries:
                time.sleep(min(2 ** (attempt - 1), 30))
        label = "OK" if ok else "FAIL"
        print(f"{label:4} {url} {detail}")
        if not ok:
            worst = max(worst, 2 if detail.startswith("unreachable") else 1)
    return worst


if __name__ == "__main__":
    sys.exit(main())

Log rotation and compression#

For applications that write dated log files themselves, or where logrotate is not available: compresses files matching a glob that have not been modified for a day, then deletes compressed files older than the retention period. It never touches the file the application is currently writing (modified within the last 24 hours), and compression is skipped for a file that is still open, checked with fuser. zstd is used when installed, gzip otherwise.

usage: log-rotate [-k DAYS] [-p GLOB] [-n] DIRECTORY
#!/usr/bin/env bash
# Compress inactive log files in a directory and delete compressed ones past retention.
set -euo pipefail

keep=30
pattern='*.log'
dry_run=0

usage() {
  cat >&2 <<'EOF'
usage: log-rotate [-k DAYS] [-p GLOB] [-n] DIRECTORY
  -k  delete compressed logs older than DAYS (default 30)
  -p  glob for log files (default '*.log')
  -n  dry run: print what would happen
Files modified in the last 24 hours or still open by a process are left alone.
EOF
  exit 3
}

while getopts ':k:p:nh' opt; do
  case $opt in
    k) keep=$OPTARG ;;
    p) pattern=$OPTARG ;;
    n) dry_run=1 ;;
    *) usage ;;
  esac
done
shift $((OPTIND - 1))
[[ $# -eq 1 && -d $1 && -w $1 ]] || usage
[[ $keep =~ ^[0-9]+$ ]] || usage
dir=$1

if command -v zstd >/dev/null; then
  compress=(zstd -q --rm -T0); ext=zst
else
  compress=(gzip); ext=gz
fi

run() {                          # run <cmd...>: print in dry-run mode, execute otherwise
  if (( dry_run )); then printf '+ %s\n' "$*"; else "$@"; fi
}

compressed=0 skipped=0 removed=0
while IFS= read -r -d '' f; do
  if fuser -s -- "$f" 2>/dev/null; then
    printf 'skip (open): %s\n' "$f"; (( skipped++ )) || true; continue
  fi
  run "${compress[@]}" -- "$f"
  (( compressed++ )) || true
done < <(find "$dir" -maxdepth 1 -type f -name "$pattern" -mtime +0 -print0)

while IFS= read -r -d '' f; do
  run rm -f -- "$f"
  (( removed++ )) || true
done < <(find "$dir" -maxdepth 1 -type f \( -name "$pattern.gz" -o -name "$pattern.zst" -o -name "$pattern.$ext" \) -mtime +"$keep" -print0)

printf 'compressed=%d skipped=%d removed=%d dir=%s\n' "$compressed" "$skipped" "$removed" "$dir"

rsync backup with retention#

Creates one dated snapshot per run under a destination, using --link-dest so unchanged files are hard links to the previous snapshot and each snapshot costs only the changed bytes while still being a complete tree. The destination can be local or user@host:/path. A lock prevents overlapping runs, the snapshot is written to an .incomplete name and renamed only after rsync succeeds, and snapshots older than the retention period are removed. Exit status 24 from rsync (files vanished during transfer) is tolerated because it is normal on a live system.

usage: backup-rsync [-k DAYS] [-e FILE] [-n] SOURCE DEST
#!/usr/bin/env bash
# Dated hard-linked rsync snapshots of SOURCE under DEST with retention.
set -euo pipefail

keep=14
exclude_file=''
dry_run=0

usage() {
  cat >&2 <<'EOF'
usage: backup-rsync [-k DAYS] [-e FILE] [-n] SOURCE DEST
  -k  keep snapshots for DAYS (default 14)
  -e  rsync --exclude-from file
  -n  dry run (passes --dry-run to rsync, no pruning)
DEST is a local directory or user@host:/path. Snapshots are DEST/YYYY-MM-DDTHHMMSS.
EOF
  exit 3
}

while getopts ':k:e:nh' opt; do
  case $opt in
    k) keep=$OPTARG ;;
    e) exclude_file=$OPTARG ;;
    n) dry_run=1 ;;
    *) usage ;;
  esac
done
shift $((OPTIND - 1))
[[ $# -eq 2 ]] || usage
[[ $keep =~ ^[0-9]+$ ]] || usage
src=${1%/}/                      # trailing slash: copy contents, not the directory itself
dest=${2%/}
[[ -d $src ]] || { printf 'source is not a directory: %s\n' "$src" >&2; exit 3; }
[[ -z $exclude_file || -r $exclude_file ]] || usage

remote=''
dest_path=$dest
if [[ $dest == *:* && $dest != /* ]]; then
  remote=${dest%%:*}; dest_path=${dest#*:}
fi

# Run a shell command on the destination, locally or over SSH.
on_dest() {
  if [[ -n $remote ]]; then ssh -o BatchMode=yes -o ConnectTimeout=10 "$remote" "$@"
  else bash -c "$@"; fi
}

exec 9>"/run/lock/backup-rsync-$(printf '%s' "$src$dest" | sha256sum | cut -c1-16).lock"
flock -n 9 || { printf 'another backup of %s is running\n' "$src" >&2; exit 1; }

on_dest "mkdir -p -- '$dest_path'"
latest=$(on_dest "ls -1d -- '$dest_path'/[0-9]*T[0-9]* 2>/dev/null | sort | tail -n1" || true)
stamp=$(date +%Y-%m-%dT%H%M%S)
target="$dest_path/$stamp.incomplete"

args=(-aHAX --delete --numeric-ids --partial --info=stats1 --timeout=600)
[[ -n $latest ]] && args+=(--link-dest="$latest")
[[ -n $exclude_file ]] && args+=(--exclude-from="$exclude_file")
[[ -n $remote ]] && args+=(-e 'ssh -o BatchMode=yes -o ConnectTimeout=10')
(( dry_run )) && args+=(--dry-run)

rc=0
rsync "${args[@]}" -- "$src" "${remote:+$remote:}$target" || rc=$?
if (( rc != 0 && rc != 24 )); then
  printf 'rsync failed with status %d; leaving %s for inspection\n' "$rc" "$target" >&2
  exit 2
fi
(( dry_run )) && { printf 'dry run complete (link-dest %s)\n' "${latest:-none}"; exit 0; }

on_dest "mv -- '$target' '$dest_path/$stamp'"
# Prune: snapshot names sort chronologically, so compare against a cutoff name.
cutoff=$(date -d "$keep days ago" +%Y-%m-%dT%H%M%S)
on_dest "cd -- '$dest_path' && for d in [0-9]*T[0-9]*; do [ -d \"\$d\" ] && [ \"\$d\" \\< '$cutoff' ] && rm -rf -- \"\$d\"; done; true"
printf 'snapshot %s/%s complete (rsync status %d)\n' "$dest" "$stamp" "$rc"

--delete and hard links

--delete removes files from the new snapshot that no longer exist in the source; older snapshots keep their copies. Hard-linked snapshots share inodes, so editing a file inside an old snapshot changes every snapshot that links it. Treat the destination as read-only except through this script.

Kubernetes stale resource cleanup#

Finds Jobs that completed or failed more than a given age ago, pods that were evicted or reached Failed or Succeeded, and ReplicaSets with zero replicas that a Deployment no longer references. It prints everything it would delete and only deletes with --apply. The context and namespace are always shown before any action, because the most expensive mistake is a cleanup against the wrong cluster. Jobs owned by a CronJob are kept, since the CronJob’s own history limits govern them.

usage: k8s-stale-cleanup [-n NAMESPACE|-A] [-a HOURS] [--apply]
#!/usr/bin/env bash
# List (and with --apply delete) finished Jobs, dead pods and orphaned ReplicaSets.
set -euo pipefail

ns_args=(-A)
age_hours=24
apply=0

usage() {
  cat >&2 <<'EOF'
usage: k8s-stale-cleanup [-n NAMESPACE | -A] [-a HOURS] [--apply]
  -n       limit to one namespace (default: all namespaces)
  -a       minimum age in hours for Jobs (default 24)
  --apply  actually delete; without it the script only prints
EOF
  exit 3
}

while (( $# )); do
  case $1 in
    -n) [[ -n ${2:-} ]] || usage; ns_args=(-n "$2"); shift 2 ;;
    -A) ns_args=(-A); shift ;;
    -a) [[ ${2:-} =~ ^[0-9]+$ ]] || usage; age_hours=$2; shift 2 ;;
    --apply) apply=1; shift ;;
    *) usage ;;
  esac
done
command -v kubectl >/dev/null && command -v jq >/dev/null || { printf 'kubectl and jq are required\n' >&2; exit 3; }

ctx=$(kubectl config current-context)
printf 'context=%s scope=%s mode=%s\n' "$ctx" "${ns_args[*]}" "$( (( apply )) && echo DELETE || echo dry-run )" >&2
cutoff=$(date -u -d "$age_hours hours ago" +%FT%TZ)

# Each finder prints "kind namespace name" lines.
finished_jobs() {
  kubectl get jobs "${ns_args[@]}" -o json | jq -r --arg cutoff "$cutoff" '
    .items[]
    | select((.metadata.ownerReferences // []) | all(.kind != "CronJob"))
    | select(.status.completionTime != null and .status.completionTime < $cutoff
             or ((.status.conditions // [])[] | select(.type == "Failed" and .status == "True" and .lastTransitionTime < $cutoff)))
    | "job \(.metadata.namespace) \(.metadata.name)"' | sort -u
}
dead_pods() {
  kubectl get pods "${ns_args[@]}" -o json | jq -r '
    .items[]
    | select(.status.reason == "Evicted" or .status.phase == "Failed" or .status.phase == "Succeeded")
    | select((.metadata.ownerReferences // []) | all(.kind != "Job"))     # Job pods go with their Job
    | "pod \(.metadata.namespace) \(.metadata.name)"'
}
orphan_replicasets() {
  kubectl get rs "${ns_args[@]}" -o json | jq -r '
    .items[]
    | select(.spec.replicas == 0 and .status.replicas == 0)
    | select((.metadata.ownerReferences // []) | length == 0)
    | "replicaset \(.metadata.namespace) \(.metadata.name)"'
}

mapfile -t targets < <(finished_jobs; dead_pods; orphan_replicasets)
if (( ${#targets[@]} == 0 )); then printf 'nothing to clean\n'; exit 0; fi

printf '%s\n' "${targets[@]}"
(( apply )) || { printf '%d objects; rerun with --apply to delete\n' "${#targets[@]}" >&2; exit 0; }

rc=0
for t in "${targets[@]}"; do
  read -r kind ns name <<<"$t"
  kubectl delete "$kind" "$name" -n "$ns" --ignore-not-found --wait=false || rc=1
done
exit "$rc"

Container image prune report#

Lists images that no container (running or stopped) references, with their age and size, and totals what a prune would reclaim. Works with Docker or Podman, picking whichever is installed unless ENGINE is set. With --prune it removes those images, but only ones older than the given age so a freshly pulled image for a deployment in progress is not taken away. Tagged and dangling images are treated the same; the criterion is “nothing uses it”.

usage: image-prune-report [-a DAYS] [--prune]
#!/usr/bin/env bash
# Report container images unused by any container; --prune removes those older than DAYS.
set -euo pipefail

min_age_days=7
prune=0

usage() {
  cat >&2 <<'EOF'
usage: image-prune-report [-a DAYS] [--prune]
  -a       only prune images created more than DAYS ago (default 7)
  --prune  remove the images instead of only reporting
ENGINE=docker|podman overrides autodetection.
EOF
  exit 3
}

while (( $# )); do
  case $1 in
    -a) [[ ${2:-} =~ ^[0-9]+$ ]] || usage; min_age_days=$2; shift 2 ;;
    --prune) prune=1; shift ;;
    *) usage ;;
  esac
done

engine=${ENGINE:-}
if [[ -z $engine ]]; then
  for e in docker podman; do command -v "$e" >/dev/null && { engine=$e; break; }; done
fi
[[ -n $engine ]] || { printf 'no container engine found\n' >&2; exit 3; }
[[ $engine == docker || $engine == podman ]] || usage

# Image IDs referenced by any container, running or not.
mapfile -t in_use < <("$engine" ps -a --no-trunc --format '{{.ImageID}}' | sort -u)
in_use_re=$(IFS='|'; printf '%s' "${in_use[*]:-^$}")

now=$(date +%s)
cutoff=$(( now - min_age_days * 86400 ))
total=0 count=0
to_remove=()

# Inspect gives a parseable timestamp and byte size for every image ID.
while IFS= read -r id; do
  [[ $id =~ ^($in_use_re)$ ]] && continue
  IFS=$'\t' read -r created size tags < <("$engine" image inspect --format \
    $'{{.Created}}\t{{.Size}}\t{{join .RepoTags ","}}' "$id")
  created_s=$(date -d "$created" +%s)
  age=$(( (now - created_s) / 86400 ))
  printf '%-14s %5dd %8s  %s\n' "${id#sha256:}" "$age" "$(numfmt --to=iec "$size")" "${tags:-<none>}"
  total=$(( total + size )); (( count++ )) || true
  (( created_s < cutoff )) && to_remove+=("$id")
done < <("$engine" images -q --no-trunc | sort -u)

printf 'unused: %d images, %s\n' "$count" "$(numfmt --to=iec "$total")"
(( prune )) || exit 0
if (( ${#to_remove[@]} == 0 )); then printf 'nothing older than %d days to prune\n' "$min_age_days"; exit 0; fi
"$engine" rmi "${to_remove[@]}"
printf 'removed %d images\n' "${#to_remove[@]}"

Port and service reachability sweep#

Tests every combination of host and TCP port from two lists in parallel and prints a tab-separated result per pair, using Bash’s /dev/tcp so nothing beyond timeout is needed on the running host. A failing pair does not stop the sweep. The exit status is 1 if any pair was closed, which makes the script usable as a pre-deployment gate (“can every app node reach the database and the registry”). Output is sorted so successive runs can be diffed.

usage: port-sweep [-t SECONDS] [-j JOBS] -p PORT[,PORT...] HOSTS_FILE
#!/usr/bin/env bash
# Check TCP reachability of every host x port pair, in parallel, one line per pair.
set -euo pipefail

timeout_s=3
jobs=16
ports=''

usage() {
  cat >&2 <<'EOF'
usage: port-sweep [-t SECONDS] [-j JOBS] -p PORT[,PORT...] HOSTS_FILE
  -t  connect timeout per pair (default 3)
  -j  parallel probes (default 16)
  -p  comma-separated TCP ports (required)
Output: host<TAB>port<TAB>open|closed. Exit 1 if any pair is closed.
EOF
  exit 3
}

while getopts ':t:j:p:h' opt; do
  case $opt in
    t) timeout_s=$OPTARG ;;
    j) jobs=$OPTARG ;;
    p) ports=$OPTARG ;;
    *) usage ;;
  esac
done
shift $((OPTIND - 1))
[[ $# -eq 1 && -r $1 && -n $ports ]] || usage
[[ $timeout_s =~ ^[0-9]+$ && $jobs =~ ^[1-9][0-9]*$ && $ports =~ ^[0-9]+(,[0-9]+)*$ ]] || usage
hosts_file=$1

# Bash needs to open /dev/tcp itself, so the probe is a bash -c invocation run by xargs.
probe='h=$1; p=$2; if timeout "$3" bash -c "exec 3<>/dev/tcp/$h/$p" 2>/dev/null; then echo open; else echo closed; fi'

results=$(
  grep -vE '^\s*(#|$)' "$hosts_file" | tr -d '[:space:]\r' | while IFS= read -r h; do
    IFS=, read -ra plist <<<"$ports"
    for p in "${plist[@]}"; do printf '%s\t%s\n' "$h" "$p"; done
  done | xargs -P "$jobs" -L1 sh -c 'printf "%s\t%s\t%s\n" "$1" "$2" "$(bash -c "$0" _ "$1" "$2" "$3")"' "$probe" "$@" 2>/dev/null
) || true
# xargs -L1 passes each line's fields as $1 $2 to sh; the probe string is $0 and the timeout the third field.

printf '%s\n' "$results" | sort
if printf '%s\n' "$results" | grep -q $'\tclosed$'; then exit 1; fi

The xargs invocation is easier to read than to write: each line of host<TAB>port becomes $1 and $2 of a sh -c command, which runs the probe under a fresh bash with timeout and prints one result line. $@ after "$probe" is empty and is only there so the sweep command line does not depend on positional arguments of the outer script. Pass the timeout via the environment if you change the shape of the input.

systemd unit failure report#

Prints every failed unit with its result, its restart count and the last lines of its journal, then exits 1 if anything failed. Intended for a daily timer whose output lands in the journal or an email, and for the first minute of an incident, when “what is broken on this box” needs one command. Timers that missed their last run are listed too, because a silently dead timer is the usual reason a backup or certificate renewal stopped happening.

usage: systemd-failures [-l LINES] [-s SINCE]
#!/usr/bin/env bash
# Report failed units and their recent journal lines; exit 1 if any unit has failed.
set -euo pipefail

lines=20
since='-1d'

usage() {
  cat >&2 <<'EOF'
usage: systemd-failures [-l LINES] [-s SINCE]
  -l  journal lines per unit (default 20)
  -s  journalctl --since expression (default -1d)
EOF
  exit 3
}

while getopts ':l:s:h' opt; do
  case $opt in
    l) lines=$OPTARG ;;
    s) since=$OPTARG ;;
    *) usage ;;
  esac
done
shift $((OPTIND - 1))
[[ $# -eq 0 && $lines =~ ^[0-9]+$ ]] || usage

mapfile -t failed < <(systemctl list-units --state=failed --plain --no-legend --all | awk '{print $1}')
printf '%s: %d failed units\n' "$(hostname -f 2>/dev/null || hostname)" "${#failed[@]}"

for u in "${failed[@]+"${failed[@]}"}"; do
  printf '\n== %s\n' "$u"
  systemctl show "$u" -p Result -p NRestarts -p ExecMainStatus -p ActiveEnterTimestamp -p InactiveEnterTimestamp 2>/dev/null \
    | sed 's/^/   /'
  journalctl -u "$u" --since "$since" -n "$lines" --no-pager -o short-iso 2>/dev/null | sed 's/^/   /'
done

# Timers whose last trigger is in the past but that have no next trigger are stuck or disabled.
stuck=$(systemctl list-timers --all --no-legend --plain | awk '$1 == "-" && $5 != "-" {print $NF}' || true)
if [[ -n $stuck ]]; then
  printf '\n== timers with no next run\n'
  printf '   %s\n' "$stuck"
fi

(( ${#failed[@]} == 0 ))

Git repository bulk update#

Fetches and fast-forwards every Git repository found beneath a directory, in parallel, and reports one line per repository: updated, already current, dirty (skipped), diverged (skipped) or failed. Nothing is ever merged or rebased; a repository that cannot fast-forward is reported for a human to look at. GIT_TERMINAL_PROMPT=0 stops a missing credential from hanging the whole run.

usage: git-bulk-update [-j JOBS] [-d DEPTH] ROOT
#!/usr/bin/env bash
# Fast-forward every Git repository under ROOT and report per-repository status.
set -euo pipefail

jobs=8
depth=3

usage() {
  cat >&2 <<'EOF'
usage: git-bulk-update [-j JOBS] [-d DEPTH] ROOT
  -j  repositories updated in parallel (default 8)
  -d  how deep below ROOT to look for .git (default 3)
Reports: updated, current, dirty, diverged, detached, failed. Exit 1 if any failed.
EOF
  exit 3
}

while getopts ':j:d:h' opt; do
  case $opt in
    j) jobs=$OPTARG ;;
    d) depth=$OPTARG ;;
    *) usage ;;
  esac
done
shift $((OPTIND - 1))
[[ $# -eq 1 && -d $1 && $jobs =~ ^[1-9][0-9]*$ && $depth =~ ^[1-9][0-9]*$ ]] || usage
root=$1
export GIT_TERMINAL_PROMPT=0

update_one() {                    # prints "<status> <path> [detail]"
  local repo=$1 branch upstream head remote base
  cd "$repo" || { printf 'failed %s cannot cd\n' "$repo"; return; }
  if [[ -n $(git status --porcelain --untracked-files=no) ]]; then
    printf 'dirty %s\n' "$repo"; return
  fi
  branch=$(git symbolic-ref --short -q HEAD || true)
  [[ -n $branch ]] || { printf 'detached %s\n' "$repo"; return; }
  if ! git fetch -q --prune 2>/dev/null; then printf 'failed %s fetch\n' "$repo"; return; fi
  upstream=$(git rev-parse --abbrev-ref -q '@{upstream}' 2>/dev/null || true)
  [[ -n $upstream ]] || { printf 'current %s no upstream\n' "$repo"; return; }
  head=$(git rev-parse HEAD); remote=$(git rev-parse "$upstream"); base=$(git merge-base HEAD "$upstream")
  if [[ $head == "$remote" ]]; then printf 'current %s\n' "$repo"
  elif [[ $head == "$base" ]]; then
    if git merge -q --ff-only "$upstream" 2>/dev/null; then
      printf 'updated %s %s..%s\n' "$repo" "${head:0:7}" "${remote:0:7}"
    else printf 'failed %s ff-merge\n' "$repo"; fi
  elif [[ $remote == "$base" ]]; then printf 'current %s ahead of %s\n' "$repo" "$upstream"
  else printf 'diverged %s\n' "$repo"; fi
}
export -f update_one

report=$(find "$root" -maxdepth "$depth" -type d -name .git -printf '%h\0' \
  | xargs -0 -P "$jobs" -I{} bash -c 'update_one "$1"' _ {} | sort -k2)
printf '%s\n' "$report"
printf '\n'; printf '%s\n' "$report" | awk '{n[$1]++} END {for (s in n) printf "%s=%d ", s, n[s]; print ""}'
! printf '%s\n' "$report" | grep -q '^failed '

S3 bucket sync with checks#

Syncs a local directory to an S3 prefix and then verifies the result rather than trusting aws s3 sync’s exit status alone: it confirms the bucket is reachable with the current identity, refuses to run with --delete unless explicitly asked, compares local and remote object counts after the sync, and spot-checks a sample of objects by size. Credentials come from the normal AWS provider chain (environment, profile, instance role); nothing is passed on the command line. Dry run is the default.

usage: s3-sync [-p PROFILE] [-r REGION] [--delete] [--apply] LOCAL_DIR s3://BUCKET/PREFIX
#!/usr/bin/env bash
# Sync a directory to S3 with pre-flight checks and post-sync verification.
set -euo pipefail

profile='' region='' delete=0 apply=0

usage() {
  cat >&2 <<'EOF'
usage: s3-sync [-p PROFILE] [-r REGION] [--delete] [--apply] LOCAL_DIR s3://BUCKET/PREFIX
  -p        AWS profile
  -r        AWS region
  --delete  remove remote objects that no longer exist locally
  --apply   perform the sync; default is --dryrun
exit: 0 ok, 1 verification mismatch, 2 sync or access failure, 3 usage
EOF
  exit 3
}

while (( $# )); do
  case $1 in
    -p) [[ -n ${2:-} ]] || usage; profile=$2; shift 2 ;;
    -r) [[ -n ${2:-} ]] || usage; region=$2; shift 2 ;;
    --delete) delete=1; shift ;;
    --apply) apply=1; shift ;;
    -*) usage ;;
    *) break ;;
  esac
done
bucket_re='^s3://[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]{1}(/.*)?$'
[[ $# -eq 2 && -d $1 && $2 =~ $bucket_re ]] || usage
src=${1%/}; dest=${2%/}
bucket=${dest#s3://}; bucket=${bucket%%/*}
command -v aws >/dev/null || { printf 'aws CLI is required\n' >&2; exit 3; }

aws_args=(--output json --cli-connect-timeout 10 --cli-read-timeout 60)
[[ -n $profile ]] && aws_args+=(--profile "$profile")
[[ -n $region ]] && aws_args+=(--region "$region")

identity=$(aws "${aws_args[@]}" sts get-caller-identity --query Arn --output text) \
  || { printf 'no usable AWS credentials\n' >&2; exit 2; }
aws "${aws_args[@]}" s3api head-bucket --bucket "$bucket" >/dev/null \
  || { printf 'bucket %s not accessible as %s\n' "$bucket" "$identity" >&2; exit 2; }
printf 'identity=%s bucket=%s mode=%s delete=%d\n' "$identity" "$bucket" "$( (( apply )) && echo apply || echo dry-run )" "$delete" >&2

sync_args=(s3 sync "$src" "$dest" --only-show-errors --exact-timestamps)
(( delete )) && sync_args+=(--delete)
(( apply )) || sync_args+=(--dryrun)
aws "${aws_args[@]}" "${sync_args[@]}" || { printf 'sync failed\n' >&2; exit 2; }
(( apply )) || { printf 'dry run complete\n'; exit 0; }

# Verification: object counts, then a sample of sizes. Count remote keys under the prefix only.
local_count=$(find "$src" -type f | wc -l)
remote_count=$(aws "${aws_args[@]}" s3 ls "$dest/" --recursive | wc -l)
printf 'local=%d remote=%d\n' "$local_count" "$remote_count"
status=0
(( delete )) && (( local_count != remote_count )) && { printf 'object count mismatch\n' >&2; status=1; }
(( delete )) || (( remote_count >= local_count )) || { printf 'remote has fewer objects than local\n' >&2; status=1; }

while IFS= read -r -d '' f; do
  rel=${f#"$src"/}
  lsize=$(stat -c %s -- "$f")
  rsize=$(aws "${aws_args[@]}" s3api head-object --bucket "$bucket" --key "${dest#s3://"$bucket"/}/$rel" --query ContentLength --output text 2>/dev/null || echo missing)
  [[ $lsize == "$rsize" ]] || { printf 'size mismatch: %s local=%s remote=%s\n' "$rel" "$lsize" "$rsize" >&2; status=1; }
done < <(find "$src" -type f -print0 | shuf -z -n 20)
exit "$status"

User account audit#

Reports the local accounts that matter for security review: UID 0 accounts other than root, accounts with an empty or missing password hash, human users with no password expiry, members of privileged groups, accounts with a login shell that have never logged in, and home directories or authorized_keys files with permissive modes. Reads /etc/passwd, /etc/shadow and /etc/group directly (the spwd module was removed in Python 3.13) so it must run as root. Output is one finding per line prefixed by severity, and the exit status is 1 when anything with severity HIGH is found.

usage: sudo user-audit [--min-uid 1000] [--groups wheel,sudo,docker]
#!/usr/bin/env python3
"""Audit local accounts; exit 0 clean, 1 HIGH findings, 3 usage or permission error."""
import argparse
import os
import pwd
import stat
import sys
from pathlib import Path

PRIVILEGED_DEFAULT = "wheel,sudo,root,docker,adm"


def read_shadow() -> dict[str, tuple[str, int]]:
    """name -> (hash, max_days). Requires root."""
    out: dict[str, tuple[str, int]] = {}
    for line in Path("/etc/shadow").read_text(encoding="utf-8").splitlines():
        f = line.split(":")
        if len(f) < 5:
            continue
        max_days = int(f[4]) if f[4].isdigit() else -1
        out[f[0]] = (f[1], max_days)
    return out


def read_groups() -> dict[str, set[str]]:
    out: dict[str, set[str]] = {}
    for line in Path("/etc/group").read_text(encoding="utf-8").splitlines():
        f = line.split(":")
        if len(f) == 4:
            out[f[0]] = {m for m in f[3].split(",") if m}
    return out


def main() -> int:
    ap = argparse.ArgumentParser(description="Local user account audit")
    ap.add_argument("--min-uid", type=int, default=1000, help="first UID treated as a human user")
    ap.add_argument("--groups", default=PRIVILEGED_DEFAULT, help="comma-separated privileged groups")
    args = ap.parse_args()
    if os.geteuid() != 0:
        print("must run as root to read /etc/shadow", file=sys.stderr)
        return 3

    try:
        shadow = read_shadow()
    except OSError as exc:
        print(f"cannot read /etc/shadow: {exc}", file=sys.stderr)
        return 3
    groups = read_groups()
    findings: list[tuple[str, str]] = []
    add = findings.append
    nologin = ("/sbin/nologin", "/usr/sbin/nologin", "/bin/false", "/usr/bin/false")

    for u in pwd.getpwall():
        interactive = u.pw_shell not in nologin and u.pw_shell != ""
        if u.pw_uid == 0 and u.pw_name != "root":
            add(("HIGH", f"{u.pw_name}: UID 0"))
        pw_hash, max_days = shadow.get(u.pw_name, ("?", -1))
        if interactive and pw_hash in ("", "?"):
            add(("HIGH", f"{u.pw_name}: interactive account with empty or missing password hash"))
        if interactive and u.pw_uid >= args.min_uid and not pw_hash.startswith(("!", "*")) and max_days in (-1, 99999):
            add(("MEDIUM", f"{u.pw_name}: password never expires"))
        if u.pw_uid < args.min_uid and u.pw_uid != 0 and interactive:
            add(("MEDIUM", f"{u.pw_name}: system account with login shell {u.pw_shell}"))
        home = Path(u.pw_dir)
        if interactive and u.pw_uid >= args.min_uid:
            if not home.is_dir():
                add(("LOW", f"{u.pw_name}: home {home} missing"))
            else:
                mode = stat.S_IMODE(home.stat().st_mode)
                if mode & 0o002:
                    add(("HIGH", f"{u.pw_name}: home {home} is world-writable ({mode:o})"))
                keys = home / ".ssh" / "authorized_keys"
                if keys.exists():
                    kmode = stat.S_IMODE(keys.stat().st_mode)
                    if kmode & 0o022 or keys.stat().st_uid != u.pw_uid:
                        add(("HIGH", f"{u.pw_name}: {keys} writable by others or not owned by user"))

    for g in args.groups.split(","):
        for member in sorted(groups.get(g, set())):
            add(("INFO", f"{member}: member of {g}"))

    for sev, msg in sorted(findings, key=lambda f: ("HIGH", "MEDIUM", "LOW", "INFO").index(f[0])):
        print(f"{sev:6} {msg}")
    print(f"{len(findings)} findings", file=sys.stderr)
    return 1 if any(s == "HIGH" for s, _ in findings) else 0


if __name__ == "__main__":
    sys.exit(main())

Wait for dependency#

Blocks until every listed dependency answers or a deadline passes: a host:port is checked with a TCP connect, an http:// or https:// URL must return a 2xx or 3xx status, and a plain path must exist. Meant for container entrypoints and ExecStartPre= lines, where the alternative is a fixed sleep that is either too short on a slow day or wastes time on every other day. Exit 0 when everything is up, 1 on timeout, with the still-missing dependencies named.

usage: wait-for [-t SECONDS] [-i SECONDS] TARGET... [-- COMMAND...]
#!/usr/bin/env bash
# Wait until host:port, URL and path targets are available, then optionally exec a command.
set -euo pipefail

deadline=60
interval=2

usage() {
  cat >&2 <<'EOF'
usage: wait-for [-t SECONDS] [-i SECONDS] TARGET... [-- COMMAND...]
  -t  give up after SECONDS (default 60)
  -i  poll interval (default 2)
TARGET is host:port, http(s)://url, or a filesystem path. After --, exec COMMAND once all are ready.
EOF
  exit 3
}

while getopts ':t:i:h' opt; do
  case $opt in
    t) deadline=$OPTARG ;;
    i) interval=$OPTARG ;;
    *) usage ;;
  esac
done
shift $((OPTIND - 1))
[[ $deadline =~ ^[0-9]+$ && $interval =~ ^[0-9]+$ && $interval -gt 0 ]] || usage

targets=(); cmd=()
while (( $# )); do
  if [[ $1 == -- ]]; then shift; cmd=("$@"); break; fi
  targets+=("$1"); shift
done
(( ${#targets[@]} )) || usage

ready() {                          # ready <target>: 0 when available
  local t=$1 h p
  case $t in
    http://*|https://*)
      curl -fsS -o /dev/null --max-time 5 "$t" ;;
    *:*)
      h=${t%:*}; p=${t##*:}
      [[ $p =~ ^[0-9]+$ ]] || return 1
      timeout 3 bash -c "exec 3<>/dev/tcp/$h/$p" 2>/dev/null ;;
    *)
      [[ -e $t ]] ;;
  esac
}

start=$SECONDS
pending=("${targets[@]}")
while (( ${#pending[@]} )); do
  still=()
  for t in "${pending[@]}"; do ready "$t" || still+=("$t"); done
  pending=("${still[@]+"${still[@]}"}")
  (( ${#pending[@]} )) || break
  if (( SECONDS - start >= deadline )); then
    printf 'timed out after %ds waiting for: %s\n' "$deadline" "${pending[*]}" >&2
    exit 1
  fi
  sleep "$interval"
done
printf 'all %d targets ready after %ds\n' "${#targets[@]}" "$((SECONDS - start))" >&2
(( ${#cmd[@]} )) && exec "${cmd[@]}"
exit 0

Prometheus textfile collector#

Gathers values that no exporter provides (backup age, certificate days remaining, a queue directory’s file count, whatever the host knows) and writes them in the Prometheus text format to the node_exporter textfile directory. The file is written to a temporary name and renamed so node_exporter never reads a half-written file, and every metric carries HELP and TYPE lines. Run it from a timer; the _last_run_timestamp_seconds metric it emits lets an alert fire when the timer itself stops. See Prometheus for the query side.

usage: textfile-metrics [-o DIR] [-b BACKUP_DIR] [-q QUEUE_DIR] [-c CERT_FILE]...
#!/usr/bin/env bash
# Write host-specific metrics for node_exporter's textfile collector.
set -euo pipefail

out_dir=/var/lib/node_exporter/textfile_collector
backup_dir=''
queue_dir=''
certs=()

usage() {
  cat >&2 <<'EOF'
usage: textfile-metrics [-o DIR] [-b BACKUP_DIR] [-q QUEUE_DIR] [-c CERT_FILE]...
  -o  textfile collector directory (default /var/lib/node_exporter/textfile_collector)
  -b  directory of dated backups; emits age of the newest entry
  -q  directory whose file count is a queue depth
  -c  PEM certificate file; emits its expiry as a timestamp; repeatable
EOF
  exit 3
}

while getopts ':o:b:q:c:h' opt; do
  case $opt in
    o) out_dir=$OPTARG ;;
    b) backup_dir=$OPTARG ;;
    q) queue_dir=$OPTARG ;;
    c) certs+=("$OPTARG") ;;
    *) usage ;;
  esac
done
shift $((OPTIND - 1))
[[ $# -eq 0 && -d $out_dir && -w $out_dir ]] || usage
[[ -z $backup_dir || -d $backup_dir ]] || usage
[[ -z $queue_dir || -d $queue_dir ]] || usage

out="$out_dir/host_custom.prom"
tmp=$(mktemp "$out_dir/.host_custom.XXXXXX")
trap 'rm -f -- "$tmp"' EXIT

# escape <label value>: backslash, quote and newline must be escaped in label values.
escape() { local s=${1//\\/\\\\}; s=${s//\"/\\\"}; printf '%s' "${s//$'\n'/\\n}"; }

{
  if [[ -n $backup_dir ]]; then
    newest=$(find "$backup_dir" -mindepth 1 -maxdepth 1 -printf '%T@\n' | sort -n | tail -n1)
    printf '# HELP host_backup_newest_timestamp_seconds Modification time of the newest entry in the backup directory.\n'
    printf '# TYPE host_backup_newest_timestamp_seconds gauge\n'
    printf 'host_backup_newest_timestamp_seconds{dir="%s"} %d\n' "$(escape "$backup_dir")" "${newest%.*}"
  fi

  if [[ -n $queue_dir ]]; then
    depth=$(find "$queue_dir" -mindepth 1 -maxdepth 1 -type f | wc -l)
    printf '# HELP host_queue_depth Files waiting in the queue directory.\n'
    printf '# TYPE host_queue_depth gauge\n'
    printf 'host_queue_depth{dir="%s"} %d\n' "$(escape "$queue_dir")" "$depth"
  fi

  if (( ${#certs[@]} )); then
    printf '# HELP host_cert_expiry_timestamp_seconds Certificate notAfter as a Unix timestamp.\n'
    printf '# TYPE host_cert_expiry_timestamp_seconds gauge\n'
    for c in "${certs[@]}"; do
      [[ -r $c ]] || { printf 'unreadable certificate: %s\n' "$c" >&2; continue; }
      not_after=$(openssl x509 -in "$c" -noout -enddate | cut -d= -f2)
      printf 'host_cert_expiry_timestamp_seconds{path="%s"} %d\n' "$(escape "$c")" "$(date -d "$not_after" +%s)"
    done
  fi

  printf '# HELP host_custom_last_run_timestamp_seconds When this collector last ran.\n'
  printf '# TYPE host_custom_last_run_timestamp_seconds gauge\n'
  printf 'host_custom_last_run_timestamp_seconds %d\n' "$(date +%s)"
} > "$tmp"

chmod 0644 "$tmp"
mv -f -- "$tmp" "$out"
trap - EXIT

Alert on time() - host_custom_last_run_timestamp_seconds > 2 * 3600 and on host_cert_expiry_timestamp_seconds - time() < 14 * 86400. node_exporter needs --collector.textfile.directory=/var/lib/node_exporter/textfile_collector and the file must end in .prom.

Parallel SSH command runner#

Runs one command on every host in a file with a bounded number of concurrent SSH sessions, captures each host’s output to its own file, and prints a summary with per-host exit status. BatchMode=yes means a host that prompts for a password fails immediately instead of hanging the batch. The command is passed as a single string to the remote shell, so quote it once on the command line and it runs unchanged everywhere. Output directories are timestamped and never overwritten, which keeps evidence from successive runs.

usage: parallel-ssh [-j JOBS] [-t SECONDS] [-u USER] [-o DIR] HOSTS_FILE 'COMMAND'
#!/usr/bin/env bash
# Run a command on many hosts over SSH in parallel, keeping per-host output and status.
set -euo pipefail

jobs=10
timeout_s=60
user=''
out_base=${XDG_STATE_HOME:-$HOME/.local/state}/parallel-ssh

usage() {
  cat >&2 <<'EOF'
usage: parallel-ssh [-j JOBS] [-t SECONDS] [-u USER] [-o DIR] HOSTS_FILE 'COMMAND'
  -j  concurrent sessions (default 10)
  -t  per-host timeout in seconds (default 60)
  -u  remote user (default: ssh config or current user)
  -o  base directory for output (default ~/.local/state/parallel-ssh)
Output goes to DIR/<timestamp>/<host>.out and .rc. Exit 1 if any host failed.
EOF
  exit 3
}

while getopts ':j:t:u:o:h' opt; do
  case $opt in
    j) jobs=$OPTARG ;;
    t) timeout_s=$OPTARG ;;
    u) user=$OPTARG ;;
    o) out_base=$OPTARG ;;
    *) usage ;;
  esac
done
shift $((OPTIND - 1))
[[ $# -eq 2 && -r $1 && -n $2 ]] || usage
[[ $jobs =~ ^[1-9][0-9]*$ && $timeout_s =~ ^[1-9][0-9]*$ ]] || usage
hosts_file=$1; command=$2

run_dir="$out_base/$(date +%Y%m%dT%H%M%S)"
mkdir -p "$run_dir"

run_host() {                       # run_host <host>: writes <host>.out and <host>.rc
  local host=$1 rc=0
  local target=${REMOTE_USER:+$REMOTE_USER@}$host
  timeout "$TIMEOUT_S" ssh -o BatchMode=yes -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new \
    -n "$target" -- "$REMOTE_COMMAND" >"$RUN_DIR/$host.out" 2>&1 || rc=$?
  printf '%d\n' "$rc" > "$RUN_DIR/$host.rc"
  printf '%-40s rc=%d\n' "$host" "$rc"
}
export -f run_host
export REMOTE_USER=$user TIMEOUT_S=$timeout_s REMOTE_COMMAND=$command RUN_DIR=$run_dir

grep -vE '^\s*(#|$)' "$hosts_file" | tr -d ' \t\r' \
  | xargs -P "$jobs" -I{} bash -c 'run_host "$1"' _ {} | sort > "$run_dir/summary.txt"

cat "$run_dir/summary.txt"
failed=$(grep -cv 'rc=0$' "$run_dir/summary.txt" || true)
total=$(wc -l < "$run_dir/summary.txt")
printf '%d/%d ok; output in %s\n' "$((total - failed))" "$total" "$run_dir" >&2
(( failed == 0 ))

Exit status 124 in the summary means the per-host timeout fired; 255 is SSH itself failing (unreachable, key rejected, host key mismatch); anything else is the remote command’s own status.

Troubleshooting#

SymptomCauseFix
unbound variable on an empty arrayBash before 4.4 with set -uThe scripts use "${arr[@]+"${arr[@]}"}"; keep that form when editing
Script works by hand, fails from a timerPATH under systemd lacks /usr/local/bin, no HOMESet Environment=PATH=... in the unit, use absolute paths; see systemd
timeout: failed to run commandWrapped command not on PATH, or timeout given a non-numeric durationCheck command -v, quote the duration
ssh: Host key verification failed under BatchModeUnknown host key and no TTY to accept itssh-keyscan the hosts into known_hosts first, or StrictHostKeyChecking=accept-new for first contact only
xargs: ... terminated by signal 13The consumer closed the pipe early (head)Expected when truncating; otherwise add -r and check the consumer
Certificate checker reports every host unreachableEgress firewall, or openssl too old to send SNI by defaultTest one host with openssl s_client -servername; the script always sends SNI
Cleanup script deletes nothing with --applykubectl context or namespace differs from what was listedRead the context= line printed on stderr before every run
rsync: failed to set times or Operation not permittedRemote user cannot chown, or -A/-X on a filesystem without ACL or xattr supportDrop -AX, or --no-owner --no-group for a non-root destination
S3 sync says up to date but files differSame size and older timestamp; sync compares size and mtime, not contentUse --exact-timestamps (already set) or --size-only; check with s3api head-object ETag
Python script prints ModuleNotFoundError: spwdPython 3.13 removed spwdThe audit reads /etc/shadow directly; do not import spwd
Metrics never appear in PrometheusWrong textfile directory flag, file lacks .prom suffix, or a parse errorcurl -s localhost:9100/metrics | grep node_textfile_scrape_error
flock: cannot open lock file/run/lock not writable by the userPoint the lock at $XDG_RUNTIME_DIR or a directory the user owns

Further reading#