# Operations scripts

> Complete Bash and Python scripts for disk alerts, certificate expiry, health checks, backups, cleanup, audits, metrics export and parallel SSH, ready to drop into cron or a timer.

Canonical: https://www.wiki.jodisand.me/scripts/
Reviewed: 2026-09-24
Related: [Bash](https://www.wiki.jodisand.me/bash/index.md), [Shell one-liners](https://www.wiki.jodisand.me/oneliners/index.md), [Python](https://www.wiki.jodisand.me/python/index.md), [systemd](https://www.wiki.jodisand.me/systemd/index.md), [Kubernetes](https://www.wiki.jodisand.me/kubernetes/index.md), [Docker](https://www.wiki.jodisand.me/docker/index.md), [SSH](https://www.wiki.jodisand.me/ssh/index.md), [TLS and certificates](https://www.wiki.jodisand.me/tls/index.md), [AWS](https://www.wiki.jodisand.me/aws/index.md), [Prometheus](https://www.wiki.jodisand.me/prometheus/index.md)


## Cheatsheet

| Task | Script |
| --- | --- |
| Alert when a filesystem passes a threshold | [disk-usage-alert](#disk-usage-alert) |
| Report certificates about to expire | [cert-expiry](#certificate-expiry-checker) |
| Health check with retries and meaningful exit codes | [http-health](#http-health-check) |
| Compress and expire old log files | [log-rotate](#log-rotation-and-compression) |
| Incremental rsync backups with retention | [backup-rsync](#rsync-backup-with-retention) |
| Remove finished Jobs and dead pods (dry run by default) | [k8s-stale-cleanup](#kubernetes-stale-resource-cleanup) |
| Report or prune unused container images | [image-prune-report](#container-image-prune-report) |
| Sweep hosts and ports for reachability | [port-sweep](#port-and-service-reachability-sweep) |
| Failed units with their last log lines | [systemd-failures](#systemd-unit-failure-report) |
| Fast-forward every repository under a directory | [git-bulk-update](#git-repository-bulk-update) |
| Sync a directory to S3 and verify | [s3-sync](#s3-bucket-sync-with-checks) |
| Audit local accounts | [user-audit](#user-account-audit) |
| Block until a dependency answers | [wait-for](#wait-for-dependency) |
| Export shell-gathered metrics to Prometheus | [textfile-metrics](#prometheus-textfile-collector) |
| Run a command on many hosts in parallel | [parallel-ssh](#parallel-ssh-command-runner) |

Every Bash script targets Bash 5 with GNU coreutils on a Linux host, starts with `set -euo pipefail`, validates its arguments, reads secrets from the environment or files rather than flags, and passes `shellcheck -s bash`. The Python scripts need only the standard library on Python 3.11 or later. Exit codes follow the Nagios convention where it fits: 0 OK, 1 warning, 2 critical, 3 usage or unknown. The building blocks are explained in [Bash](https://www.wiki.jodisand.me/bash/) and [Python](https://www.wiki.jodisand.me/python/); shorter single commands live in [Shell one-liners](https://www.wiki.jodisand.me/oneliners/).

Install a script under `/usr/local/bin`, keep it owned by root and mode 0755, and run it from a [systemd timer](https://www.wiki.jodisand.me/systemd/#timers) rather than cron so the journal captures its output and `systemctl list-timers` shows when it last ran.

## Disk usage alert

Reads `df` once, compares each real filesystem's space and inode usage against a threshold, prints the offenders and exits 1 if any crosses the warning level or 2 for the critical level. The output is one line per filesystem, so a timer can pipe it to `logger` or a chat webhook without further parsing. Pseudo-filesystems, overlay mounts and `/snap` are skipped.

```text
usage: disk-usage-alert [-w PERCENT] [-c PERCENT] [-x MOUNT]...
```

```sh
#!/usr/bin/env bash
# Alert when any local filesystem's space or inode usage exceeds a threshold.
set -euo pipefail

warn=80
crit=90
exclude=()

usage() {
  cat >&2 <<'EOF'
usage: disk-usage-alert [-w PERCENT] [-c PERCENT] [-x MOUNT]...
  -w  warning threshold, percent used (default 80)
  -c  critical threshold, percent used (default 90)
  -x  mountpoint to skip; repeatable
exit: 0 ok, 1 warning, 2 critical, 3 usage
EOF
  exit 3
}

while getopts ':w:c:x:h' opt; do
  case $opt in
    w) warn=$OPTARG ;;
    c) crit=$OPTARG ;;
    x) exclude+=("$OPTARG") ;;
    *) usage ;;
  esac
done
shift $((OPTIND - 1))
[[ $# -eq 0 ]] || usage
[[ $warn =~ ^[0-9]+$ && $crit =~ ^[0-9]+$ && $warn -le 100 && $crit -le 100 ]] || usage
(( warn <= crit )) || { printf 'warning threshold must not exceed critical\n' >&2; exit 3; }

skip() {
  local m
  for m in "${exclude[@]+"${exclude[@]}"}"; do [[ $m == "$1" ]] && return 0; done
  return 1
}

status=0
check() {                       # check <label> <mount> <percent>
  local label=$1 mount=$2 pct=$3
  if (( pct >= crit )); then
    printf 'CRITICAL %s %s%% %s\n' "$mount" "$pct" "$label"; status=2
  elif (( pct >= warn )); then
    printf 'WARNING %s %s%% %s\n' "$mount" "$pct" "$label"; (( status < 1 )) && status=1
  fi
}

# -P gives POSIX single-line output; -x excludes pseudo filesystems.
while read -r _ _ _ _ pct mount; do
  skip "$mount" && continue
  check space "$mount" "${pct%\%}"
done < <(df -P -l -x tmpfs -x devtmpfs -x overlay -x squashfs -x efivarfs | tail -n +2)

while read -r _ _ _ _ pct mount; do
  skip "$mount" && continue
  [[ $pct == '-' ]] && continue       # filesystems without inode accounting (btrfs, xfs report a value)
  check inodes "$mount" "${pct%\%}"
done < <(df -P -l -i -x tmpfs -x devtmpfs -x overlay -x squashfs -x efivarfs | tail -n +2)

(( status == 0 )) && printf 'OK all filesystems below %s%%\n' "$warn"
exit "$status"
```

## Certificate expiry checker

Connects to every `host[:port]` in a file, reads the served certificate's `notAfter`, and prints days remaining per host sorted soonest first. Unreachable hosts are reported separately and make the script exit 2, so a monitoring job distinguishes "expiring" from "cannot tell". The port defaults to 443 and SNI is always sent, which is what most virtual-hosted endpoints require. Set `STARTTLS=smtp` in the environment for a mail server on 25 or 587.

```text
usage: cert-expiry [-d DAYS] [-t SECONDS] HOSTS_FILE
```

```sh
#!/usr/bin/env bash
# Report TLS certificates expiring within DAYS for each host[:port] in a file.
set -euo pipefail

days=30
connect_timeout=5

usage() {
  cat >&2 <<'EOF'
usage: cert-expiry [-d DAYS] [-t SECONDS] HOSTS_FILE
  -d  warn when fewer than DAYS remain (default 30)
  -t  connect timeout per host in seconds (default 5)
HOSTS_FILE has one host or host:port per line; # comments and blank lines are ignored.
STARTTLS=smtp|imap|pop3|ftp|postgres in the environment enables STARTTLS.
exit: 0 ok, 1 something expires within DAYS, 2 a host was unreachable, 3 usage
EOF
  exit 3
}

while getopts ':d:t:h' opt; do
  case $opt in
    d) days=$OPTARG ;;
    t) connect_timeout=$OPTARG ;;
    *) usage ;;
  esac
done
shift $((OPTIND - 1))
[[ $# -eq 1 && -r $1 ]] || usage
[[ $days =~ ^[0-9]+$ && $connect_timeout =~ ^[0-9]+$ ]] || usage
hosts_file=$1

starttls_args=()
[[ -n ${STARTTLS:-} ]] && starttls_args=(-starttls "$STARTTLS")

now=$(date +%s)
status=0
results=()
failures=()

while IFS= read -r line || [[ -n $line ]]; do
  line=${line%%#*}; line=${line//[[:space:]]/}
  [[ -z $line ]] && continue
  host=${line%%:*}
  port=${line#*:}; [[ $port == "$line" ]] && port=443

  if ! cert=$(timeout "$connect_timeout" openssl s_client -connect "$host:$port" -servername "$host" \
        "${starttls_args[@]+"${starttls_args[@]}"}" </dev/null 2>/dev/null); then
    failures+=("$host:$port connect failed"); continue
  fi
  if ! not_after=$(printf '%s' "$cert" | openssl x509 -noout -enddate 2>/dev/null); then
    failures+=("$host:$port no certificate in response"); continue
  fi
  expiry=$(date -d "${not_after#notAfter=}" +%s)
  remaining=$(( (expiry - now) / 86400 ))
  issuer=$(printf '%s' "$cert" | openssl x509 -noout -issuer -nameopt RFC2253 | sed 's/^issuer=//; s/,.*//')
  results+=("$remaining $host:$port $issuer")
  (( remaining < days )) && status=1
done < "$hosts_file"

if (( ${#results[@]} )); then
  printf '%s\n' "${results[@]}" | sort -n | while read -r rem hp iss; do
    flag=OK; (( rem < days )) && flag=EXPIRING; (( rem < 0 )) && flag=EXPIRED
    printf '%-9s %5d days  %-40s %s\n' "$flag" "$rem" "$hp" "$iss"
  done
fi
if (( ${#failures[@]} )); then
  printf 'UNREACHABLE %s\n' "${failures[@]}" >&2
  status=2
fi
exit "$status"
```

## HTTP health check

Probes one or more URLs with a timeout, retries with exponential backoff, and exits 0 only when every URL returns the expected status. Any other outcome is classified: 1 when a URL returned a wrong status after all retries, 2 when a URL never answered. Python's `urllib` is used rather than `curl` so the retry logic, per-attempt timing and body check live in one place with no external dependency. A bearer token is read from `HEALTH_TOKEN` if set; it never appears on the command line.

```text
usage: http-health [--expect 200] [--retries 3] [--timeout 5] [--contains TEXT] URL...
```

```python
#!/usr/bin/env python3
"""Check URLs with retries; exit 0 ok, 1 wrong status or body, 2 unreachable, 3 usage."""
import argparse
import os
import sys
import time
import urllib.error
import urllib.request


def probe(url: str, expect: int, timeout: float, contains: str | None, token: str | None) -> tuple[bool, str]:
    req = urllib.request.Request(url, headers={"User-Agent": "http-health/1"})
    if token:
        req.add_header("Authorization", f"Bearer {token}")
    start = time.monotonic()
    try:
        with urllib.request.urlopen(req, timeout=timeout) as resp:  # noqa: S310 - URL comes from the operator
            status = resp.status
            body = resp.read(65536).decode("utf-8", "replace")
    except urllib.error.HTTPError as exc:  # non-2xx still counts as a response
        status = exc.code
        body = exc.read(65536).decode("utf-8", "replace")
    except (urllib.error.URLError, TimeoutError, OSError) as exc:
        return False, f"unreachable ({exc.reason if hasattr(exc, 'reason') else exc})"
    elapsed = time.monotonic() - start
    if status != expect:
        return False, f"status {status}, expected {expect} ({elapsed:.2f}s)"
    if contains and contains not in body:
        return False, f"body missing {contains!r} ({elapsed:.2f}s)"
    return True, f"status {status} ({elapsed:.2f}s)"


def main() -> int:
    ap = argparse.ArgumentParser(description="HTTP health check with retries")
    ap.add_argument("urls", nargs="+", metavar="URL")
    ap.add_argument("--expect", type=int, default=200, help="expected HTTP status (default 200)")
    ap.add_argument("--retries", type=int, default=3, help="attempts per URL (default 3)")
    ap.add_argument("--timeout", type=float, default=5.0, help="seconds per attempt (default 5)")
    ap.add_argument("--contains", help="text the body must contain")
    args = ap.parse_args()
    if args.retries < 1 or args.timeout <= 0 or not 100 <= args.expect <= 599:
        ap.error("retries must be >= 1, timeout > 0, expect a valid HTTP status")
    for url in args.urls:
        if not url.startswith(("http://", "https://")):
            ap.error(f"not an HTTP URL: {url}")

    token = os.environ.get("HEALTH_TOKEN")
    worst = 0
    for url in args.urls:
        ok, detail = False, ""
        for attempt in range(1, args.retries + 1):
            ok, detail = probe(url, args.expect, args.timeout, args.contains, token)
            if ok:
                break
            if attempt < args.retries:
                time.sleep(min(2 ** (attempt - 1), 30))
        label = "OK" if ok else "FAIL"
        print(f"{label:4} {url} {detail}")
        if not ok:
            worst = max(worst, 2 if detail.startswith("unreachable") else 1)
    return worst


if __name__ == "__main__":
    sys.exit(main())
```

## Log rotation and compression

For applications that write dated log files themselves, or where `logrotate` is not available: compresses files matching a glob that have not been modified for a day, then deletes compressed files older than the retention period. It never touches the file the application is currently writing (modified within the last 24 hours), and compression is skipped for a file that is still open, checked with `fuser`. zstd is used when installed, gzip otherwise.

```text
usage: log-rotate [-k DAYS] [-p GLOB] [-n] DIRECTORY
```

```sh
#!/usr/bin/env bash
# Compress inactive log files in a directory and delete compressed ones past retention.
set -euo pipefail

keep=30
pattern='*.log'
dry_run=0

usage() {
  cat >&2 <<'EOF'
usage: log-rotate [-k DAYS] [-p GLOB] [-n] DIRECTORY
  -k  delete compressed logs older than DAYS (default 30)
  -p  glob for log files (default '*.log')
  -n  dry run: print what would happen
Files modified in the last 24 hours or still open by a process are left alone.
EOF
  exit 3
}

while getopts ':k:p:nh' opt; do
  case $opt in
    k) keep=$OPTARG ;;
    p) pattern=$OPTARG ;;
    n) dry_run=1 ;;
    *) usage ;;
  esac
done
shift $((OPTIND - 1))
[[ $# -eq 1 && -d $1 && -w $1 ]] || usage
[[ $keep =~ ^[0-9]+$ ]] || usage
dir=$1

if command -v zstd >/dev/null; then
  compress=(zstd -q --rm -T0); ext=zst
else
  compress=(gzip); ext=gz
fi

run() {                          # run <cmd...>: print in dry-run mode, execute otherwise
  if (( dry_run )); then printf '+ %s\n' "$*"; else "$@"; fi
}

compressed=0 skipped=0 removed=0
while IFS= read -r -d '' f; do
  if fuser -s -- "$f" 2>/dev/null; then
    printf 'skip (open): %s\n' "$f"; (( skipped++ )) || true; continue
  fi
  run "${compress[@]}" -- "$f"
  (( compressed++ )) || true
done < <(find "$dir" -maxdepth 1 -type f -name "$pattern" -mtime +0 -print0)

while IFS= read -r -d '' f; do
  run rm -f -- "$f"
  (( removed++ )) || true
done < <(find "$dir" -maxdepth 1 -type f \( -name "$pattern.gz" -o -name "$pattern.zst" -o -name "$pattern.$ext" \) -mtime +"$keep" -print0)

printf 'compressed=%d skipped=%d removed=%d dir=%s\n' "$compressed" "$skipped" "$removed" "$dir"
```

## rsync backup with retention

Creates one dated snapshot per run under a destination, using `--link-dest` so unchanged files are hard links to the previous snapshot and each snapshot costs only the changed bytes while still being a complete tree. The destination can be local or `user@host:/path`. A lock prevents overlapping runs, the snapshot is written to an `.incomplete` name and renamed only after rsync succeeds, and snapshots older than the retention period are removed. Exit status 24 from rsync (files vanished during transfer) is tolerated because it is normal on a live system.

```text
usage: backup-rsync [-k DAYS] [-e FILE] [-n] SOURCE DEST
```

```sh
#!/usr/bin/env bash
# Dated hard-linked rsync snapshots of SOURCE under DEST with retention.
set -euo pipefail

keep=14
exclude_file=''
dry_run=0

usage() {
  cat >&2 <<'EOF'
usage: backup-rsync [-k DAYS] [-e FILE] [-n] SOURCE DEST
  -k  keep snapshots for DAYS (default 14)
  -e  rsync --exclude-from file
  -n  dry run (passes --dry-run to rsync, no pruning)
DEST is a local directory or user@host:/path. Snapshots are DEST/YYYY-MM-DDTHHMMSS.
EOF
  exit 3
}

while getopts ':k:e:nh' opt; do
  case $opt in
    k) keep=$OPTARG ;;
    e) exclude_file=$OPTARG ;;
    n) dry_run=1 ;;
    *) usage ;;
  esac
done
shift $((OPTIND - 1))
[[ $# -eq 2 ]] || usage
[[ $keep =~ ^[0-9]+$ ]] || usage
src=${1%/}/                      # trailing slash: copy contents, not the directory itself
dest=${2%/}
[[ -d $src ]] || { printf 'source is not a directory: %s\n' "$src" >&2; exit 3; }
[[ -z $exclude_file || -r $exclude_file ]] || usage

remote=''
dest_path=$dest
if [[ $dest == *:* && $dest != /* ]]; then
  remote=${dest%%:*}; dest_path=${dest#*:}
fi

# Run a shell command on the destination, locally or over SSH.
on_dest() {
  if [[ -n $remote ]]; then ssh -o BatchMode=yes -o ConnectTimeout=10 "$remote" "$@"
  else bash -c "$@"; fi
}

exec 9>"/run/lock/backup-rsync-$(printf '%s' "$src$dest" | sha256sum | cut -c1-16).lock"
flock -n 9 || { printf 'another backup of %s is running\n' "$src" >&2; exit 1; }

on_dest "mkdir -p -- '$dest_path'"
latest=$(on_dest "ls -1d -- '$dest_path'/[0-9]*T[0-9]* 2>/dev/null | sort | tail -n1" || true)
stamp=$(date +%Y-%m-%dT%H%M%S)
target="$dest_path/$stamp.incomplete"

args=(-aHAX --delete --numeric-ids --partial --info=stats1 --timeout=600)
[[ -n $latest ]] && args+=(--link-dest="$latest")
[[ -n $exclude_file ]] && args+=(--exclude-from="$exclude_file")
[[ -n $remote ]] && args+=(-e 'ssh -o BatchMode=yes -o ConnectTimeout=10')
(( dry_run )) && args+=(--dry-run)

rc=0
rsync "${args[@]}" -- "$src" "${remote:+$remote:}$target" || rc=$?
if (( rc != 0 && rc != 24 )); then
  printf 'rsync failed with status %d; leaving %s for inspection\n' "$rc" "$target" >&2
  exit 2
fi
(( dry_run )) && { printf 'dry run complete (link-dest %s)\n' "${latest:-none}"; exit 0; }

on_dest "mv -- '$target' '$dest_path/$stamp'"
# Prune: snapshot names sort chronologically, so compare against a cutoff name.
cutoff=$(date -d "$keep days ago" +%Y-%m-%dT%H%M%S)
on_dest "cd -- '$dest_path' && for d in [0-9]*T[0-9]*; do [ -d \"\$d\" ] && [ \"\$d\" \\< '$cutoff' ] && rm -rf -- \"\$d\"; done; true"
printf 'snapshot %s/%s complete (rsync status %d)\n' "$dest" "$stamp" "$rc"
```

> [!WARNING] `--delete` and hard links
> `--delete` removes files from the new snapshot that no longer exist in the source; older snapshots keep their copies. Hard-linked snapshots share inodes, so editing a file inside an old snapshot changes every snapshot that links it. Treat the destination as read-only except through this script.

## Kubernetes stale resource cleanup

Finds Jobs that completed or failed more than a given age ago, pods that were evicted or reached `Failed` or `Succeeded`, and ReplicaSets with zero replicas that a Deployment no longer references. It prints everything it would delete and only deletes with `--apply`. The context and namespace are always shown before any action, because the most expensive mistake is a cleanup against the wrong cluster. Jobs owned by a CronJob are kept, since the CronJob's own history limits govern them.

```text
usage: k8s-stale-cleanup [-n NAMESPACE|-A] [-a HOURS] [--apply]
```

```sh
#!/usr/bin/env bash
# List (and with --apply delete) finished Jobs, dead pods and orphaned ReplicaSets.
set -euo pipefail

ns_args=(-A)
age_hours=24
apply=0

usage() {
  cat >&2 <<'EOF'
usage: k8s-stale-cleanup [-n NAMESPACE | -A] [-a HOURS] [--apply]
  -n       limit to one namespace (default: all namespaces)
  -a       minimum age in hours for Jobs (default 24)
  --apply  actually delete; without it the script only prints
EOF
  exit 3
}

while (( $# )); do
  case $1 in
    -n) [[ -n ${2:-} ]] || usage; ns_args=(-n "$2"); shift 2 ;;
    -A) ns_args=(-A); shift ;;
    -a) [[ ${2:-} =~ ^[0-9]+$ ]] || usage; age_hours=$2; shift 2 ;;
    --apply) apply=1; shift ;;
    *) usage ;;
  esac
done
command -v kubectl >/dev/null && command -v jq >/dev/null || { printf 'kubectl and jq are required\n' >&2; exit 3; }

ctx=$(kubectl config current-context)
printf 'context=%s scope=%s mode=%s\n' "$ctx" "${ns_args[*]}" "$( (( apply )) && echo DELETE || echo dry-run )" >&2
cutoff=$(date -u -d "$age_hours hours ago" +%FT%TZ)

# Each finder prints "kind namespace name" lines.
finished_jobs() {
  kubectl get jobs "${ns_args[@]}" -o json | jq -r --arg cutoff "$cutoff" '
    .items[]
    | select((.metadata.ownerReferences // []) | all(.kind != "CronJob"))
    | select(.status.completionTime != null and .status.completionTime < $cutoff
             or ((.status.conditions // [])[] | select(.type == "Failed" and .status == "True" and .lastTransitionTime < $cutoff)))
    | "job \(.metadata.namespace) \(.metadata.name)"' | sort -u
}
dead_pods() {
  kubectl get pods "${ns_args[@]}" -o json | jq -r '
    .items[]
    | select(.status.reason == "Evicted" or .status.phase == "Failed" or .status.phase == "Succeeded")
    | select((.metadata.ownerReferences // []) | all(.kind != "Job"))     # Job pods go with their Job
    | "pod \(.metadata.namespace) \(.metadata.name)"'
}
orphan_replicasets() {
  kubectl get rs "${ns_args[@]}" -o json | jq -r '
    .items[]
    | select(.spec.replicas == 0 and .status.replicas == 0)
    | select((.metadata.ownerReferences // []) | length == 0)
    | "replicaset \(.metadata.namespace) \(.metadata.name)"'
}

mapfile -t targets < <(finished_jobs; dead_pods; orphan_replicasets)
if (( ${#targets[@]} == 0 )); then printf 'nothing to clean\n'; exit 0; fi

printf '%s\n' "${targets[@]}"
(( apply )) || { printf '%d objects; rerun with --apply to delete\n' "${#targets[@]}" >&2; exit 0; }

rc=0
for t in "${targets[@]}"; do
  read -r kind ns name <<<"$t"
  kubectl delete "$kind" "$name" -n "$ns" --ignore-not-found --wait=false || rc=1
done
exit "$rc"
```

## Container image prune report

Lists images that no container (running or stopped) references, with their age and size, and totals what a prune would reclaim. Works with Docker or Podman, picking whichever is installed unless `ENGINE` is set. With `--prune` it removes those images, but only ones older than the given age so a freshly pulled image for a deployment in progress is not taken away. Tagged and dangling images are treated the same; the criterion is "nothing uses it".

```text
usage: image-prune-report [-a DAYS] [--prune]
```

```sh
#!/usr/bin/env bash
# Report container images unused by any container; --prune removes those older than DAYS.
set -euo pipefail

min_age_days=7
prune=0

usage() {
  cat >&2 <<'EOF'
usage: image-prune-report [-a DAYS] [--prune]
  -a       only prune images created more than DAYS ago (default 7)
  --prune  remove the images instead of only reporting
ENGINE=docker|podman overrides autodetection.
EOF
  exit 3
}

while (( $# )); do
  case $1 in
    -a) [[ ${2:-} =~ ^[0-9]+$ ]] || usage; min_age_days=$2; shift 2 ;;
    --prune) prune=1; shift ;;
    *) usage ;;
  esac
done

engine=${ENGINE:-}
if [[ -z $engine ]]; then
  for e in docker podman; do command -v "$e" >/dev/null && { engine=$e; break; }; done
fi
[[ -n $engine ]] || { printf 'no container engine found\n' >&2; exit 3; }
[[ $engine == docker || $engine == podman ]] || usage

# Image IDs referenced by any container, running or not.
mapfile -t in_use < <("$engine" ps -a --no-trunc --format '{{.ImageID}}' | sort -u)
in_use_re=$(IFS='|'; printf '%s' "${in_use[*]:-^$}")

now=$(date +%s)
cutoff=$(( now - min_age_days * 86400 ))
total=0 count=0
to_remove=()

# Inspect gives a parseable timestamp and byte size for every image ID.
while IFS= read -r id; do
  [[ $id =~ ^($in_use_re)$ ]] && continue
  IFS=$'\t' read -r created size tags < <("$engine" image inspect --format \
    $'{{.Created}}\t{{.Size}}\t{{join .RepoTags ","}}' "$id")
  created_s=$(date -d "$created" +%s)
  age=$(( (now - created_s) / 86400 ))
  printf '%-14s %5dd %8s  %s\n' "${id#sha256:}" "$age" "$(numfmt --to=iec "$size")" "${tags:-<none>}"
  total=$(( total + size )); (( count++ )) || true
  (( created_s < cutoff )) && to_remove+=("$id")
done < <("$engine" images -q --no-trunc | sort -u)

printf 'unused: %d images, %s\n' "$count" "$(numfmt --to=iec "$total")"
(( prune )) || exit 0
if (( ${#to_remove[@]} == 0 )); then printf 'nothing older than %d days to prune\n' "$min_age_days"; exit 0; fi
"$engine" rmi "${to_remove[@]}"
printf 'removed %d images\n' "${#to_remove[@]}"
```

## Port and service reachability sweep

Tests every combination of host and TCP port from two lists in parallel and prints a tab-separated result per pair, using Bash's `/dev/tcp` so nothing beyond `timeout` is needed on the running host. A failing pair does not stop the sweep. The exit status is 1 if any pair was closed, which makes the script usable as a pre-deployment gate ("can every app node reach the database and the registry"). Output is sorted so successive runs can be diffed.

```text
usage: port-sweep [-t SECONDS] [-j JOBS] -p PORT[,PORT...] HOSTS_FILE
```

```sh
#!/usr/bin/env bash
# Check TCP reachability of every host x port pair, in parallel, one line per pair.
set -euo pipefail

timeout_s=3
jobs=16
ports=''

usage() {
  cat >&2 <<'EOF'
usage: port-sweep [-t SECONDS] [-j JOBS] -p PORT[,PORT...] HOSTS_FILE
  -t  connect timeout per pair (default 3)
  -j  parallel probes (default 16)
  -p  comma-separated TCP ports (required)
Output: host<TAB>port<TAB>open|closed. Exit 1 if any pair is closed.
EOF
  exit 3
}

while getopts ':t:j:p:h' opt; do
  case $opt in
    t) timeout_s=$OPTARG ;;
    j) jobs=$OPTARG ;;
    p) ports=$OPTARG ;;
    *) usage ;;
  esac
done
shift $((OPTIND - 1))
[[ $# -eq 1 && -r $1 && -n $ports ]] || usage
[[ $timeout_s =~ ^[0-9]+$ && $jobs =~ ^[1-9][0-9]*$ && $ports =~ ^[0-9]+(,[0-9]+)*$ ]] || usage
hosts_file=$1

# Bash needs to open /dev/tcp itself, so the probe is a bash -c invocation run by xargs.
probe='h=$1; p=$2; if timeout "$3" bash -c "exec 3<>/dev/tcp/$h/$p" 2>/dev/null; then echo open; else echo closed; fi'

results=$(
  grep -vE '^\s*(#|$)' "$hosts_file" | tr -d '[:space:]\r' | while IFS= read -r h; do
    IFS=, read -ra plist <<<"$ports"
    for p in "${plist[@]}"; do printf '%s\t%s\n' "$h" "$p"; done
  done | xargs -P "$jobs" -L1 sh -c 'printf "%s\t%s\t%s\n" "$1" "$2" "$(bash -c "$0" _ "$1" "$2" "$3")"' "$probe" "$@" 2>/dev/null
) || true
# xargs -L1 passes each line's fields as $1 $2 to sh; the probe string is $0 and the timeout the third field.

printf '%s\n' "$results" | sort
if printf '%s\n' "$results" | grep -q $'\tclosed$'; then exit 1; fi
```

The `xargs` invocation is easier to read than to write: each line of `host<TAB>port` becomes `$1` and `$2` of a `sh -c` command, which runs the probe under a fresh `bash` with `timeout` and prints one result line. `$@` after `"$probe"` is empty and is only there so the sweep command line does not depend on positional arguments of the outer script. Pass the timeout via the environment if you change the shape of the input.

## systemd unit failure report

Prints every failed unit with its result, its restart count and the last lines of its journal, then exits 1 if anything failed. Intended for a daily timer whose output lands in the journal or an email, and for the first minute of an incident, when "what is broken on this box" needs one command. Timers that missed their last run are listed too, because a silently dead timer is the usual reason a backup or certificate renewal stopped happening.

```text
usage: systemd-failures [-l LINES] [-s SINCE]
```

```sh
#!/usr/bin/env bash
# Report failed units and their recent journal lines; exit 1 if any unit has failed.
set -euo pipefail

lines=20
since='-1d'

usage() {
  cat >&2 <<'EOF'
usage: systemd-failures [-l LINES] [-s SINCE]
  -l  journal lines per unit (default 20)
  -s  journalctl --since expression (default -1d)
EOF
  exit 3
}

while getopts ':l:s:h' opt; do
  case $opt in
    l) lines=$OPTARG ;;
    s) since=$OPTARG ;;
    *) usage ;;
  esac
done
shift $((OPTIND - 1))
[[ $# -eq 0 && $lines =~ ^[0-9]+$ ]] || usage

mapfile -t failed < <(systemctl list-units --state=failed --plain --no-legend --all | awk '{print $1}')
printf '%s: %d failed units\n' "$(hostname -f 2>/dev/null || hostname)" "${#failed[@]}"

for u in "${failed[@]+"${failed[@]}"}"; do
  printf '\n== %s\n' "$u"
  systemctl show "$u" -p Result -p NRestarts -p ExecMainStatus -p ActiveEnterTimestamp -p InactiveEnterTimestamp 2>/dev/null \
    | sed 's/^/   /'
  journalctl -u "$u" --since "$since" -n "$lines" --no-pager -o short-iso 2>/dev/null | sed 's/^/   /'
done

# Timers whose last trigger is in the past but that have no next trigger are stuck or disabled.
stuck=$(systemctl list-timers --all --no-legend --plain | awk '$1 == "-" && $5 != "-" {print $NF}' || true)
if [[ -n $stuck ]]; then
  printf '\n== timers with no next run\n'
  printf '   %s\n' "$stuck"
fi

(( ${#failed[@]} == 0 ))
```

## Git repository bulk update

Fetches and fast-forwards every Git repository found beneath a directory, in parallel, and reports one line per repository: updated, already current, dirty (skipped), diverged (skipped) or failed. Nothing is ever merged or rebased; a repository that cannot fast-forward is reported for a human to look at. `GIT_TERMINAL_PROMPT=0` stops a missing credential from hanging the whole run.

```text
usage: git-bulk-update [-j JOBS] [-d DEPTH] ROOT
```

```sh
#!/usr/bin/env bash
# Fast-forward every Git repository under ROOT and report per-repository status.
set -euo pipefail

jobs=8
depth=3

usage() {
  cat >&2 <<'EOF'
usage: git-bulk-update [-j JOBS] [-d DEPTH] ROOT
  -j  repositories updated in parallel (default 8)
  -d  how deep below ROOT to look for .git (default 3)
Reports: updated, current, dirty, diverged, detached, failed. Exit 1 if any failed.
EOF
  exit 3
}

while getopts ':j:d:h' opt; do
  case $opt in
    j) jobs=$OPTARG ;;
    d) depth=$OPTARG ;;
    *) usage ;;
  esac
done
shift $((OPTIND - 1))
[[ $# -eq 1 && -d $1 && $jobs =~ ^[1-9][0-9]*$ && $depth =~ ^[1-9][0-9]*$ ]] || usage
root=$1
export GIT_TERMINAL_PROMPT=0

update_one() {                    # prints "<status> <path> [detail]"
  local repo=$1 branch upstream head remote base
  cd "$repo" || { printf 'failed %s cannot cd\n' "$repo"; return; }
  if [[ -n $(git status --porcelain --untracked-files=no) ]]; then
    printf 'dirty %s\n' "$repo"; return
  fi
  branch=$(git symbolic-ref --short -q HEAD || true)
  [[ -n $branch ]] || { printf 'detached %s\n' "$repo"; return; }
  if ! git fetch -q --prune 2>/dev/null; then printf 'failed %s fetch\n' "$repo"; return; fi
  upstream=$(git rev-parse --abbrev-ref -q '@{upstream}' 2>/dev/null || true)
  [[ -n $upstream ]] || { printf 'current %s no upstream\n' "$repo"; return; }
  head=$(git rev-parse HEAD); remote=$(git rev-parse "$upstream"); base=$(git merge-base HEAD "$upstream")
  if [[ $head == "$remote" ]]; then printf 'current %s\n' "$repo"
  elif [[ $head == "$base" ]]; then
    if git merge -q --ff-only "$upstream" 2>/dev/null; then
      printf 'updated %s %s..%s\n' "$repo" "${head:0:7}" "${remote:0:7}"
    else printf 'failed %s ff-merge\n' "$repo"; fi
  elif [[ $remote == "$base" ]]; then printf 'current %s ahead of %s\n' "$repo" "$upstream"
  else printf 'diverged %s\n' "$repo"; fi
}
export -f update_one

report=$(find "$root" -maxdepth "$depth" -type d -name .git -printf '%h\0' \
  | xargs -0 -P "$jobs" -I{} bash -c 'update_one "$1"' _ {} | sort -k2)
printf '%s\n' "$report"
printf '\n'; printf '%s\n' "$report" | awk '{n[$1]++} END {for (s in n) printf "%s=%d ", s, n[s]; print ""}'
! printf '%s\n' "$report" | grep -q '^failed '
```

## S3 bucket sync with checks

Syncs a local directory to an S3 prefix and then verifies the result rather than trusting `aws s3 sync`'s exit status alone: it confirms the bucket is reachable with the current identity, refuses to run with `--delete` unless explicitly asked, compares local and remote object counts after the sync, and spot-checks a sample of objects by size. Credentials come from the normal AWS provider chain (environment, profile, instance role); nothing is passed on the command line. Dry run is the default.

```text
usage: s3-sync [-p PROFILE] [-r REGION] [--delete] [--apply] LOCAL_DIR s3://BUCKET/PREFIX
```

```sh
#!/usr/bin/env bash
# Sync a directory to S3 with pre-flight checks and post-sync verification.
set -euo pipefail

profile='' region='' delete=0 apply=0

usage() {
  cat >&2 <<'EOF'
usage: s3-sync [-p PROFILE] [-r REGION] [--delete] [--apply] LOCAL_DIR s3://BUCKET/PREFIX
  -p        AWS profile
  -r        AWS region
  --delete  remove remote objects that no longer exist locally
  --apply   perform the sync; default is --dryrun
exit: 0 ok, 1 verification mismatch, 2 sync or access failure, 3 usage
EOF
  exit 3
}

while (( $# )); do
  case $1 in
    -p) [[ -n ${2:-} ]] || usage; profile=$2; shift 2 ;;
    -r) [[ -n ${2:-} ]] || usage; region=$2; shift 2 ;;
    --delete) delete=1; shift ;;
    --apply) apply=1; shift ;;
    -*) usage ;;
    *) break ;;
  esac
done
bucket_re='^s3://[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]{1}(/.*)?$'
[[ $# -eq 2 && -d $1 && $2 =~ $bucket_re ]] || usage
src=${1%/}; dest=${2%/}
bucket=${dest#s3://}; bucket=${bucket%%/*}
command -v aws >/dev/null || { printf 'aws CLI is required\n' >&2; exit 3; }

aws_args=(--output json --cli-connect-timeout 10 --cli-read-timeout 60)
[[ -n $profile ]] && aws_args+=(--profile "$profile")
[[ -n $region ]] && aws_args+=(--region "$region")

identity=$(aws "${aws_args[@]}" sts get-caller-identity --query Arn --output text) \
  || { printf 'no usable AWS credentials\n' >&2; exit 2; }
aws "${aws_args[@]}" s3api head-bucket --bucket "$bucket" >/dev/null \
  || { printf 'bucket %s not accessible as %s\n' "$bucket" "$identity" >&2; exit 2; }
printf 'identity=%s bucket=%s mode=%s delete=%d\n' "$identity" "$bucket" "$( (( apply )) && echo apply || echo dry-run )" "$delete" >&2

sync_args=(s3 sync "$src" "$dest" --only-show-errors --exact-timestamps)
(( delete )) && sync_args+=(--delete)
(( apply )) || sync_args+=(--dryrun)
aws "${aws_args[@]}" "${sync_args[@]}" || { printf 'sync failed\n' >&2; exit 2; }
(( apply )) || { printf 'dry run complete\n'; exit 0; }

# Verification: object counts, then a sample of sizes. Count remote keys under the prefix only.
local_count=$(find "$src" -type f | wc -l)
remote_count=$(aws "${aws_args[@]}" s3 ls "$dest/" --recursive | wc -l)
printf 'local=%d remote=%d\n' "$local_count" "$remote_count"
status=0
(( delete )) && (( local_count != remote_count )) && { printf 'object count mismatch\n' >&2; status=1; }
(( delete )) || (( remote_count >= local_count )) || { printf 'remote has fewer objects than local\n' >&2; status=1; }

while IFS= read -r -d '' f; do
  rel=${f#"$src"/}
  lsize=$(stat -c %s -- "$f")
  rsize=$(aws "${aws_args[@]}" s3api head-object --bucket "$bucket" --key "${dest#s3://"$bucket"/}/$rel" --query ContentLength --output text 2>/dev/null || echo missing)
  [[ $lsize == "$rsize" ]] || { printf 'size mismatch: %s local=%s remote=%s\n' "$rel" "$lsize" "$rsize" >&2; status=1; }
done < <(find "$src" -type f -print0 | shuf -z -n 20)
exit "$status"
```

## User account audit

Reports the local accounts that matter for security review: UID 0 accounts other than root, accounts with an empty or missing password hash, human users with no password expiry, members of privileged groups, accounts with a login shell that have never logged in, and home directories or `authorized_keys` files with permissive modes. Reads `/etc/passwd`, `/etc/shadow` and `/etc/group` directly (the `spwd` module was removed in Python 3.13) so it must run as root. Output is one finding per line prefixed by severity, and the exit status is 1 when anything with severity `HIGH` is found.

```text
usage: sudo user-audit [--min-uid 1000] [--groups wheel,sudo,docker]
```

```python
#!/usr/bin/env python3
"""Audit local accounts; exit 0 clean, 1 HIGH findings, 3 usage or permission error."""
import argparse
import os
import pwd
import stat
import sys
from pathlib import Path

PRIVILEGED_DEFAULT = "wheel,sudo,root,docker,adm"


def read_shadow() -> dict[str, tuple[str, int]]:
    """name -> (hash, max_days). Requires root."""
    out: dict[str, tuple[str, int]] = {}
    for line in Path("/etc/shadow").read_text(encoding="utf-8").splitlines():
        f = line.split(":")
        if len(f) < 5:
            continue
        max_days = int(f[4]) if f[4].isdigit() else -1
        out[f[0]] = (f[1], max_days)
    return out


def read_groups() -> dict[str, set[str]]:
    out: dict[str, set[str]] = {}
    for line in Path("/etc/group").read_text(encoding="utf-8").splitlines():
        f = line.split(":")
        if len(f) == 4:
            out[f[0]] = {m for m in f[3].split(",") if m}
    return out


def main() -> int:
    ap = argparse.ArgumentParser(description="Local user account audit")
    ap.add_argument("--min-uid", type=int, default=1000, help="first UID treated as a human user")
    ap.add_argument("--groups", default=PRIVILEGED_DEFAULT, help="comma-separated privileged groups")
    args = ap.parse_args()
    if os.geteuid() != 0:
        print("must run as root to read /etc/shadow", file=sys.stderr)
        return 3

    try:
        shadow = read_shadow()
    except OSError as exc:
        print(f"cannot read /etc/shadow: {exc}", file=sys.stderr)
        return 3
    groups = read_groups()
    findings: list[tuple[str, str]] = []
    add = findings.append
    nologin = ("/sbin/nologin", "/usr/sbin/nologin", "/bin/false", "/usr/bin/false")

    for u in pwd.getpwall():
        interactive = u.pw_shell not in nologin and u.pw_shell != ""
        if u.pw_uid == 0 and u.pw_name != "root":
            add(("HIGH", f"{u.pw_name}: UID 0"))
        pw_hash, max_days = shadow.get(u.pw_name, ("?", -1))
        if interactive and pw_hash in ("", "?"):
            add(("HIGH", f"{u.pw_name}: interactive account with empty or missing password hash"))
        if interactive and u.pw_uid >= args.min_uid and not pw_hash.startswith(("!", "*")) and max_days in (-1, 99999):
            add(("MEDIUM", f"{u.pw_name}: password never expires"))
        if u.pw_uid < args.min_uid and u.pw_uid != 0 and interactive:
            add(("MEDIUM", f"{u.pw_name}: system account with login shell {u.pw_shell}"))
        home = Path(u.pw_dir)
        if interactive and u.pw_uid >= args.min_uid:
            if not home.is_dir():
                add(("LOW", f"{u.pw_name}: home {home} missing"))
            else:
                mode = stat.S_IMODE(home.stat().st_mode)
                if mode & 0o002:
                    add(("HIGH", f"{u.pw_name}: home {home} is world-writable ({mode:o})"))
                keys = home / ".ssh" / "authorized_keys"
                if keys.exists():
                    kmode = stat.S_IMODE(keys.stat().st_mode)
                    if kmode & 0o022 or keys.stat().st_uid != u.pw_uid:
                        add(("HIGH", f"{u.pw_name}: {keys} writable by others or not owned by user"))

    for g in args.groups.split(","):
        for member in sorted(groups.get(g, set())):
            add(("INFO", f"{member}: member of {g}"))

    for sev, msg in sorted(findings, key=lambda f: ("HIGH", "MEDIUM", "LOW", "INFO").index(f[0])):
        print(f"{sev:6} {msg}")
    print(f"{len(findings)} findings", file=sys.stderr)
    return 1 if any(s == "HIGH" for s, _ in findings) else 0


if __name__ == "__main__":
    sys.exit(main())
```

## Wait for dependency

Blocks until every listed dependency answers or a deadline passes: a `host:port` is checked with a TCP connect, an `http://` or `https://` URL must return a 2xx or 3xx status, and a plain path must exist. Meant for container entrypoints and `ExecStartPre=` lines, where the alternative is a fixed `sleep` that is either too short on a slow day or wastes time on every other day. Exit 0 when everything is up, 1 on timeout, with the still-missing dependencies named.

```text
usage: wait-for [-t SECONDS] [-i SECONDS] TARGET... [-- COMMAND...]
```

```sh
#!/usr/bin/env bash
# Wait until host:port, URL and path targets are available, then optionally exec a command.
set -euo pipefail

deadline=60
interval=2

usage() {
  cat >&2 <<'EOF'
usage: wait-for [-t SECONDS] [-i SECONDS] TARGET... [-- COMMAND...]
  -t  give up after SECONDS (default 60)
  -i  poll interval (default 2)
TARGET is host:port, http(s)://url, or a filesystem path. After --, exec COMMAND once all are ready.
EOF
  exit 3
}

while getopts ':t:i:h' opt; do
  case $opt in
    t) deadline=$OPTARG ;;
    i) interval=$OPTARG ;;
    *) usage ;;
  esac
done
shift $((OPTIND - 1))
[[ $deadline =~ ^[0-9]+$ && $interval =~ ^[0-9]+$ && $interval -gt 0 ]] || usage

targets=(); cmd=()
while (( $# )); do
  if [[ $1 == -- ]]; then shift; cmd=("$@"); break; fi
  targets+=("$1"); shift
done
(( ${#targets[@]} )) || usage

ready() {                          # ready <target>: 0 when available
  local t=$1 h p
  case $t in
    http://*|https://*)
      curl -fsS -o /dev/null --max-time 5 "$t" ;;
    *:*)
      h=${t%:*}; p=${t##*:}
      [[ $p =~ ^[0-9]+$ ]] || return 1
      timeout 3 bash -c "exec 3<>/dev/tcp/$h/$p" 2>/dev/null ;;
    *)
      [[ -e $t ]] ;;
  esac
}

start=$SECONDS
pending=("${targets[@]}")
while (( ${#pending[@]} )); do
  still=()
  for t in "${pending[@]}"; do ready "$t" || still+=("$t"); done
  pending=("${still[@]+"${still[@]}"}")
  (( ${#pending[@]} )) || break
  if (( SECONDS - start >= deadline )); then
    printf 'timed out after %ds waiting for: %s\n' "$deadline" "${pending[*]}" >&2
    exit 1
  fi
  sleep "$interval"
done
printf 'all %d targets ready after %ds\n' "${#targets[@]}" "$((SECONDS - start))" >&2
(( ${#cmd[@]} )) && exec "${cmd[@]}"
exit 0
```

## Prometheus textfile collector

Gathers values that no exporter provides (backup age, certificate days remaining, a queue directory's file count, whatever the host knows) and writes them in the Prometheus text format to the node_exporter textfile directory. The file is written to a temporary name and renamed so node_exporter never reads a half-written file, and every metric carries `HELP` and `TYPE` lines. Run it from a timer; the `_last_run_timestamp_seconds` metric it emits lets an alert fire when the timer itself stops. See [Prometheus](https://www.wiki.jodisand.me/prometheus/) for the query side.

```text
usage: textfile-metrics [-o DIR] [-b BACKUP_DIR] [-q QUEUE_DIR] [-c CERT_FILE]...
```

```sh
#!/usr/bin/env bash
# Write host-specific metrics for node_exporter's textfile collector.
set -euo pipefail

out_dir=/var/lib/node_exporter/textfile_collector
backup_dir=''
queue_dir=''
certs=()

usage() {
  cat >&2 <<'EOF'
usage: textfile-metrics [-o DIR] [-b BACKUP_DIR] [-q QUEUE_DIR] [-c CERT_FILE]...
  -o  textfile collector directory (default /var/lib/node_exporter/textfile_collector)
  -b  directory of dated backups; emits age of the newest entry
  -q  directory whose file count is a queue depth
  -c  PEM certificate file; emits its expiry as a timestamp; repeatable
EOF
  exit 3
}

while getopts ':o:b:q:c:h' opt; do
  case $opt in
    o) out_dir=$OPTARG ;;
    b) backup_dir=$OPTARG ;;
    q) queue_dir=$OPTARG ;;
    c) certs+=("$OPTARG") ;;
    *) usage ;;
  esac
done
shift $((OPTIND - 1))
[[ $# -eq 0 && -d $out_dir && -w $out_dir ]] || usage
[[ -z $backup_dir || -d $backup_dir ]] || usage
[[ -z $queue_dir || -d $queue_dir ]] || usage

out="$out_dir/host_custom.prom"
tmp=$(mktemp "$out_dir/.host_custom.XXXXXX")
trap 'rm -f -- "$tmp"' EXIT

# escape <label value>: backslash, quote and newline must be escaped in label values.
escape() { local s=${1//\\/\\\\}; s=${s//\"/\\\"}; printf '%s' "${s//$'\n'/\\n}"; }

{
  if [[ -n $backup_dir ]]; then
    newest=$(find "$backup_dir" -mindepth 1 -maxdepth 1 -printf '%T@\n' | sort -n | tail -n1)
    printf '# HELP host_backup_newest_timestamp_seconds Modification time of the newest entry in the backup directory.\n'
    printf '# TYPE host_backup_newest_timestamp_seconds gauge\n'
    printf 'host_backup_newest_timestamp_seconds{dir="%s"} %d\n' "$(escape "$backup_dir")" "${newest%.*}"
  fi

  if [[ -n $queue_dir ]]; then
    depth=$(find "$queue_dir" -mindepth 1 -maxdepth 1 -type f | wc -l)
    printf '# HELP host_queue_depth Files waiting in the queue directory.\n'
    printf '# TYPE host_queue_depth gauge\n'
    printf 'host_queue_depth{dir="%s"} %d\n' "$(escape "$queue_dir")" "$depth"
  fi

  if (( ${#certs[@]} )); then
    printf '# HELP host_cert_expiry_timestamp_seconds Certificate notAfter as a Unix timestamp.\n'
    printf '# TYPE host_cert_expiry_timestamp_seconds gauge\n'
    for c in "${certs[@]}"; do
      [[ -r $c ]] || { printf 'unreadable certificate: %s\n' "$c" >&2; continue; }
      not_after=$(openssl x509 -in "$c" -noout -enddate | cut -d= -f2)
      printf 'host_cert_expiry_timestamp_seconds{path="%s"} %d\n' "$(escape "$c")" "$(date -d "$not_after" +%s)"
    done
  fi

  printf '# HELP host_custom_last_run_timestamp_seconds When this collector last ran.\n'
  printf '# TYPE host_custom_last_run_timestamp_seconds gauge\n'
  printf 'host_custom_last_run_timestamp_seconds %d\n' "$(date +%s)"
} > "$tmp"

chmod 0644 "$tmp"
mv -f -- "$tmp" "$out"
trap - EXIT
```

Alert on `time() - host_custom_last_run_timestamp_seconds > 2 * 3600` and on `host_cert_expiry_timestamp_seconds - time() < 14 * 86400`. node_exporter needs `--collector.textfile.directory=/var/lib/node_exporter/textfile_collector` and the file must end in `.prom`.

## Parallel SSH command runner

Runs one command on every host in a file with a bounded number of concurrent SSH sessions, captures each host's output to its own file, and prints a summary with per-host exit status. `BatchMode=yes` means a host that prompts for a password fails immediately instead of hanging the batch. The command is passed as a single string to the remote shell, so quote it once on the command line and it runs unchanged everywhere. Output directories are timestamped and never overwritten, which keeps evidence from successive runs.

```text
usage: parallel-ssh [-j JOBS] [-t SECONDS] [-u USER] [-o DIR] HOSTS_FILE 'COMMAND'
```

```sh
#!/usr/bin/env bash
# Run a command on many hosts over SSH in parallel, keeping per-host output and status.
set -euo pipefail

jobs=10
timeout_s=60
user=''
out_base=${XDG_STATE_HOME:-$HOME/.local/state}/parallel-ssh

usage() {
  cat >&2 <<'EOF'
usage: parallel-ssh [-j JOBS] [-t SECONDS] [-u USER] [-o DIR] HOSTS_FILE 'COMMAND'
  -j  concurrent sessions (default 10)
  -t  per-host timeout in seconds (default 60)
  -u  remote user (default: ssh config or current user)
  -o  base directory for output (default ~/.local/state/parallel-ssh)
Output goes to DIR/<timestamp>/<host>.out and .rc. Exit 1 if any host failed.
EOF
  exit 3
}

while getopts ':j:t:u:o:h' opt; do
  case $opt in
    j) jobs=$OPTARG ;;
    t) timeout_s=$OPTARG ;;
    u) user=$OPTARG ;;
    o) out_base=$OPTARG ;;
    *) usage ;;
  esac
done
shift $((OPTIND - 1))
[[ $# -eq 2 && -r $1 && -n $2 ]] || usage
[[ $jobs =~ ^[1-9][0-9]*$ && $timeout_s =~ ^[1-9][0-9]*$ ]] || usage
hosts_file=$1; command=$2

run_dir="$out_base/$(date +%Y%m%dT%H%M%S)"
mkdir -p "$run_dir"

run_host() {                       # run_host <host>: writes <host>.out and <host>.rc
  local host=$1 rc=0
  local target=${REMOTE_USER:+$REMOTE_USER@}$host
  timeout "$TIMEOUT_S" ssh -o BatchMode=yes -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new \
    -n "$target" -- "$REMOTE_COMMAND" >"$RUN_DIR/$host.out" 2>&1 || rc=$?
  printf '%d\n' "$rc" > "$RUN_DIR/$host.rc"
  printf '%-40s rc=%d\n' "$host" "$rc"
}
export -f run_host
export REMOTE_USER=$user TIMEOUT_S=$timeout_s REMOTE_COMMAND=$command RUN_DIR=$run_dir

grep -vE '^\s*(#|$)' "$hosts_file" | tr -d ' \t\r' \
  | xargs -P "$jobs" -I{} bash -c 'run_host "$1"' _ {} | sort > "$run_dir/summary.txt"

cat "$run_dir/summary.txt"
failed=$(grep -cv 'rc=0$' "$run_dir/summary.txt" || true)
total=$(wc -l < "$run_dir/summary.txt")
printf '%d/%d ok; output in %s\n' "$((total - failed))" "$total" "$run_dir" >&2
(( failed == 0 ))
```

Exit status 124 in the summary means the per-host timeout fired; 255 is SSH itself failing (unreachable, key rejected, host key mismatch); anything else is the remote command's own status.

## Troubleshooting

| Symptom | Cause | Fix |
| --- | --- | --- |
| `unbound variable` on an empty array | Bash before 4.4 with `set -u` | The scripts use `"${arr[@]+"${arr[@]}"}"`; keep that form when editing |
| Script works by hand, fails from a timer | `PATH` under systemd lacks `/usr/local/bin`, no `HOME` | Set `Environment=PATH=...` in the unit, use absolute paths; see [systemd](https://www.wiki.jodisand.me/systemd/#a-failing-service) |
| `timeout: failed to run command` | Wrapped command not on `PATH`, or `timeout` given a non-numeric duration | Check `command -v`, quote the duration |
| `ssh: Host key verification failed` under `BatchMode` | Unknown host key and no TTY to accept it | `ssh-keyscan` the hosts into `known_hosts` first, or `StrictHostKeyChecking=accept-new` for first contact only |
| `xargs: ... terminated by signal 13` | The consumer closed the pipe early (`head`) | Expected when truncating; otherwise add `-r` and check the consumer |
| Certificate checker reports every host unreachable | Egress firewall, or `openssl` too old to send SNI by default | Test one host with `openssl s_client -servername`; the script always sends SNI |
| Cleanup script deletes nothing with `--apply` | `kubectl` context or namespace differs from what was listed | Read the `context=` line printed on stderr before every run |
| `rsync: failed to set times` or `Operation not permitted` | Remote user cannot `chown`, or `-A`/`-X` on a filesystem without ACL or xattr support | Drop `-AX`, or `--no-owner --no-group` for a non-root destination |
| S3 sync says up to date but files differ | Same size and older timestamp; `sync` compares size and mtime, not content | Use `--exact-timestamps` (already set) or `--size-only`; check with `s3api head-object` ETag |
| Python script prints `ModuleNotFoundError: spwd` | Python 3.13 removed `spwd` | The audit reads `/etc/shadow` directly; do not import `spwd` |
| Metrics never appear in Prometheus | Wrong textfile directory flag, file lacks `.prom` suffix, or a parse error | `curl -s localhost:9100/metrics \| grep node_textfile_scrape_error` |
| `flock: cannot open lock file` | `/run/lock` not writable by the user | Point the lock at `$XDG_RUNTIME_DIR` or a directory the user owns |

## Further reading

- [GNU Bash manual: conditional constructs and arrays](https://www.gnu.org/software/bash/manual/bash.html#Bash-Conditional-Expressions)
- [rsync(1)](https://download.samba.org/pub/rsync/rsync.1)
- [OpenSSL s_client](https://docs.openssl.org/master/man1/openssl-s_client/)
- [AWS CLI s3 sync](https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html)
- [node_exporter textfile collector](https://github.com/prometheus/node_exporter#textfile-collector)
- [Python urllib.request](https://docs.python.org/3/library/urllib.request.html)