Bash · Software Engineering Wiki

Cheatsheet#

Task	Snippet
Fail fast	`set -euo pipefail`
Script’s own directory	`cd "$(dirname "${BASH_SOURCE[0]}")"`
Default if unset or empty	`${VAR:-default}`
Abort if unset or empty	`${VAR:?message}`
Strip suffix	`${file%.txt}`
Strip directory	`${path##*/}`
Replace all	`${str//old/new}`
Lowercase	`${str,,}`
Length of string, of array	`${#str}`, `${#array[@]}`
Command output	`out=$(cmd)`
Arithmetic	`(( count++ ))`, `n=$(( a * b ))`
Loop over files safely	`for f in *.log; do [ -e "$f" ] \|\| continue; done`
Read a file line by line	`while IFS= read -r line; do ...; done < file`
File into an array of lines	`mapfile -t lines < file`
Temp file that cleans up	`t=$(mktemp); trap 'rm -f "$t"' EXIT`
Is a command available	`command -v jq >/dev/null`
Timestamped name (Bash 4.2+)	`printf -v f 'dump-%(%Y%m%dT%H%M%S)T.sql' -1`
Quote a value for reuse as shell input	`printf '%q\n' "$value"` or `${value@Q}`
Trace a script	`bash -x script.sh`
Lint	`shellcheck script.sh`

Behaviour below is Bash 5.x unless a version is given. macOS ships Bash 3.2, which lacks associative arrays, mapfile, ${var,,} and most of what follows; check with bash --version. Reference: the GNU Bash manual.

Quoting#

An unquoted expansion goes through word splitting on $IFS and then pathname (glob) expansion. That is the cause of nearly every script bug involving spaces, asterisks or empty values.

rm $file        # two arguments if $file contains a space; every file in the directory if it is "*"
rm "$file"      # always one argument
rm -- "$file"   # also safe when the name starts with "-"

"$@"            # each positional argument, individually quoted: almost always what you want
"$*"            # all arguments joined into one string with the first character of IFS
'$literal'      # single quotes: no expansion at all
"$(cmd)"        # command substitution, quoted

Quote every expansion unless you want splitting. Unquoted is safe inside [[ ]], on the right side of a plain assignment (a=$b) and in case $x in. shellcheck flags the rest.

Strict mode#

#!/usr/bin/env bash
set -euo pipefail

Option	Effect	Trap
`-e` (`errexit`)	Exit when a command returns non-zero	Ignored in `if`/`while` conditions, left of `&&`/`\|\|`, after `!`, and in any function called from those contexts
`-u` (`nounset`)	Expanding an unset variable is an error	Use `${VAR:-}` for optional ones. Before Bash 4.4, `"${arr[@]}"` on an empty array also errors
`-o pipefail`	A pipeline’s status is the last non-zero stage	Without it `false \| true` succeeds
`-E` (`errtrace`)	`ERR` trap is inherited by functions and subshells	Without it an `ERR` trap never fires inside functions
`-x` (`xtrace`)	Print each command after expansion	Use around one block: `set -x; ...; set +x`

Cases where set -e does not stop the script:

local out=$(false)          # status of `local` (0) masks the substitution; declare first, assign second
out=$(false; echo after)    # command substitution runs without errexit unless shopt -s inherit_errexit (Bash 4.4+)
f() { false; echo still here; }
f || echo failed            # errexit is off for the whole body of f because it is tested

For steps with real consequences, test the status explicitly instead of relying on -e:

if ! output=$(risky_command 2>&1); then
  printf 'failed: %s\n' "$output" >&2
  exit 1
fi

IFS=$'\n\t' is often added to “strict mode”. It stops splitting on spaces, which hides missing quotes rather than fixing them and changes how "$*" joins. Quote properly instead.

Parameter expansion#

Expansion edits strings in the shell process. In a loop it is much faster than spawning sed, cut or basename per item.

file=/var/log/nginx/access.log.1

${file##*/}        # access.log.1        remove longest prefix matching */ (basename)
${file%/*}         # /var/log/nginx      remove shortest suffix matching /* (dirname)
${file%.*}         # /var/log/nginx/access.log   drop last extension
${file##*.}        # 1                   last extension only

name="deploy-prod-api"
${name//-/_}       # deploy_prod_api     replace all
${name/prod/stag}  # deploy-stag-api     replace first
${name:0:6}        # deploy              substring (offset:length)
${name: -3}        # api                 from the end; the space stops it being parsed as :-
${#name}           # 15                  length
${name^^}          # DEPLOY-PROD-API     uppercase
${name,,}          # deploy-prod-api     lowercase
${name@Q}          # 'deploy-prod-api'   quoted for reuse as input (Bash 4.4+)

${VAR:-default}    # default if VAR is unset or empty
${VAR:=default}    # same, and assign it to VAR
${VAR:?message}    # abort with message if unset or empty
${VAR:+value}      # value if VAR is set and non-empty, otherwise nothing

Without the colon (${VAR-default}, ${VAR+value}) only unset counts; an empty string is treated as set. Use that form for options that may legitimately be empty.

Tests#

[[ -f "$path" ]]        # regular file exists
[[ -d "$path" ]]        # directory
[[ -s "$path" ]]        # exists and size > 0
[[ -r "$path" ]]        # readable by this user
[[ -n "$str" ]]         # non-empty string
[[ -z "$str" ]]         # empty string
[[ -v VAR ]]            # variable is set (Bash 4.2+), even if empty
[[ "$a" == "$b" ]]      # string equality
[[ "$a" == prefix* ]]   # glob match; the pattern must be unquoted
[[ "$a" =~ ^[0-9]+$ ]]  # regex; unquoted; captures in "${BASH_REMATCH[@]}"
(( n > 5 ))             # arithmetic comparison
[[ "$f" -nt "$g" ]]     # f is newer than g

Use [[ ]] in Bash scripts: it does not word-split or glob its operands and supports =~ and pattern matching. [ ] is the POSIX test command, needed only in /bin/sh scripts, and there every operand must be quoted. Inside [[ ]] quoting the right side of == or =~ makes it a literal string, not a pattern.

Loops#

for f in *.log; do
  [ -e "$f" ] || continue          # no match leaves the literal "*.log"; or use shopt -s nullglob
  printf '%s\n' "$f"
done

while IFS= read -r line; do        # IFS= keeps leading/trailing whitespace, -r keeps backslashes
  printf '%s\n' "$line"
done < input.txt

while IFS=, read -r name port _; do    # split fields; _ absorbs the rest of the line
  printf '%s -> %s\n' "$name" "$port"
done < hosts.csv

find . -name '*.tmp' -print0 | while IFS= read -r -d '' f; do rm -- "$f"; done   # NUL-safe; deletes files

for i in {1..10}; do :; done       # brace expansion runs before variable expansion: {1..$n} does not work
for ((i = 0; i < n; i++)); do :; done

read returns non-zero at end of file without a trailing newline, so the last line is dropped. while IFS= read -r line || [[ -n $line ]] keeps it.

Each stage of a pipeline runs in a subshell, so variables set in cmd | while ... are lost when the loop ends. Feed the loop with redirection or process substitution instead (or shopt -s lastpipe in a non-interactive script):

count=0
while IFS= read -r _; do (( count++ )); done < <(grep 'ERROR' app.log)
printf '%d errors\n' "$count"      # correct here; would print 0 after `grep ... | while`

Arrays#

arr=(one two "three four")
arr+=(five)

"${arr[@]}"        # each element as a separate word
"${arr[*]}"        # one string joined by the first character of IFS
"${#arr[@]}"       # element count
"${arr[2]}"        # three four
"${arr[@]:1:2}"    # slice: two, three four
"${!arr[@]}"       # indices
mapfile -t lines < file              # file into array, newline stripped (-t)
mapfile -t pods < <(kubectl get pods -o name)

declare -A limits=([cpu]=2 [memory]=4Gi)   # associative array, Bash 4.0+
limits[disk]=100Gi
"${!limits[@]}"    # keys (unordered)
"${limits[@]}"     # values
[[ -v limits[cpu] ]] && echo set

Build command lines in arrays, not strings. args=(--name "$name"); cmd "${args[@]}" keeps each argument intact; args="--name $name"; cmd $args splits on any space in $name.

Associative arrays need declare -A first. Without it Bash creates an indexed array and evaluates each subscript arithmetically, so cpu and memory (unset names) both become index 0 and overwrite each other.

Functions#

usage() {
  cat >&2 <<'EOF'
usage: deploy [-n] <env>
  -n  dry run
EOF
  exit 2
}

deploy() {
  local env=${1:?env required}     # without local, the variable is global
  local -r dry=${2:-false}         # -r: read-only for the rest of the function
  printf 'deploying %s\n' "$env"
}

deploy prod || { printf 'failed\n' >&2; exit 1; }

A function returns only an exit status (0 to 255). Return data by printing it and capturing with $(...), or by assigning to a variable name the caller passes (local -n ref=$1, Bash 4.3+). Bash 5.3 adds ${ cmd; }, which captures output without forking a subshell, so variable changes inside it persist.

Traps and cleanup#

tmp=$(mktemp -d)
trap 'rm -rf -- "$tmp"' EXIT       # set immediately after creating the resource
trap 'exit 130' INT                # convert Ctrl-C into a normal exit so the EXIT trap runs
trap 'printf "failed at line %s: %s\n" "$LINENO" "$BASH_COMMAND" >&2' ERR

The EXIT trap runs when the shell exits for any reason other than SIGKILL, including set -e failures and exit. Put cleanup there rather than at the end of the script. A later trap ... EXIT replaces the earlier one; combine commands into a single handler function. Use set -E if the ERR trap must fire inside functions.

Options and input#

while getopts ":n:v" opt; do       # leading ":" enables the \? and : cases below
  case $opt in
    n) name=$OPTARG ;;
    v) verbose=1 ;;
    \?) printf 'unknown option: -%s\n' "$OPTARG" >&2; exit 2 ;;
    :)  printf 'option -%s needs a value\n' "$OPTARG" >&2; exit 2 ;;
  esac
done
shift $((OPTIND - 1))              # "$@" is now the positional arguments

getopts handles short options only. For long options, loop over "$@" with case "$1" in --name) name=$2; shift 2 ;; esac.

read -r -p 'Continue? [y/N] ' reply
[[ $reply == [yY]* ]] || exit 0

cat <<EOF          # unquoted delimiter: variables and $(...) expand
host=$HOSTNAME
EOF

cat <<'EOF'        # quoted delimiter: literal text
cost=$100
EOF

Redirection#

cmd > out.log 2>&1        # stdout to file, then stderr to where stdout now points
cmd &> out.log            # the same, Bash shorthand
cmd 2>&1 | tee out.log    # both streams through a pipe
cmd > /dev/null 2>&1      # discard everything
cmd 2> >(logger -t my-app)   # stderr to another process
exec 3< file              # open fd 3 for reading; exec 3<&- closes it
diff <(sort a) <(sort b)  # process substitution: command output as a file path
printf '%s\n' "$data" | cmd  # use printf, not echo, for data you did not write

Redirections are processed left to right, and 2>&1 copies wherever fd 1 points at that moment. cmd 2>&1 > file therefore sends stderr to the terminal (the old stdout) and only stdout to the file.

Long options#

getopts stops at the first non-option and knows nothing about --name. A manual loop over "$@" handles long options, --opt=value, -- as end of options and bundled short flags well enough for most scripts.

name='' verbose=0 dry_run=0
positional=()
while [[ $# -gt 0 ]]; do
  case $1 in
    -n|--name)     [[ $# -ge 2 ]] || { echo "$1 needs a value" >&2; exit 2; }
                   name=$2; shift 2 ;;
    --name=*)      name=${1#*=}; shift ;;          # strip everything up to the first =
    -v|--verbose)  verbose=1; shift ;;
    --dry-run)     dry_run=1; shift ;;
    -h|--help)     usage ;;
    --)            shift; positional+=("$@"); break ;;   # everything after -- is positional
    -?*)           printf 'unknown option: %s\n' "$1" >&2; exit 2 ;;
    *)             positional+=("$1"); shift ;;
  esac
done
set -- "${positional[@]}"          # restore "$@" as the positional arguments only

The -- case matters when a positional argument can legitimately start with a dash (rm -- -f is the classic). -?* catches anything else beginning with - so a typo like --verbsoe fails loudly instead of becoming a filename. Exit status 2 for usage errors follows the convention of most GNU tools; reserve 1 for runtime failures so callers can tell them apart.

Here-documents and here-strings#

A here-document feeds a block of text to a command’s stdin. The delimiter word controls expansion: unquoted (<<EOF) expands $var, $(cmd) and backslash escapes; quoted (<<'EOF') passes the text through untouched. <<-EOF also strips leading tabs (tabs only, not spaces) so the block can be indented inside a function or loop.

ssh app.example.com bash -s <<'EOF'   # run a multi-line script remotely; quoted so nothing expands locally
set -euo pipefail
systemctl is-active my-app
journalctl -u my-app -n 20 --no-pager
EOF

psql -h db.example.com -U app -v ON_ERROR_STOP=1 <<EOF   # unquoted: $table expands before psql sees it
SELECT count(*) FROM $table WHERE created_at > now() - interval '1 day';
EOF

render() {
	cat <<-EOF                # <<- strips the leading tabs (this file must use real tabs here)
	server {
	  listen 80;
	  server_name $1;
	}
	EOF
}

cat > /etc/my-app/config.ini <<EOF   # redirect the here-doc to a file
[main]
host=$HOSTNAME
EOF

A here-string (<<<) is a one-line here-document. Bash appends a newline to the value, which is usually what a line-oriented reader wants and occasionally surprising (wc -c <<< '' prints 1).

read -r major minor patch <<< "${version//./ }"   # split 1.28.3 into three variables without a subshell
grep -q 'ERROR' <<< "$output"
jq -r '.name' <<< "$json"
while IFS= read -r line; do :; done <<< "$multiline"   # loop runs in the current shell, unlike cmd | while

Process substitution and coproc#

Process substitution turns a command’s output (<(cmd)) or input (>(cmd)) into a path like /dev/fd/63 that another command opens as a file. It runs the inner command asynchronously in a subshell and is the standard way to feed a while read loop without losing variables to a pipeline subshell.

diff <(kubectl get cm my-app -o yaml) <(kubectl get cm my-app -o yaml --context staging)
comm -13 <(sort expected.txt) <(sort actual.txt)     # lines only in actual
paste <(cut -d, -f1 a.csv) <(cut -d, -f3 b.csv)
tee >(gzip > out.gz) >(sha256sum > out.sha) < in.bin >/dev/null   # fan out one stream to several writers
exec > >(tee -a "$log") 2>&1                                       # everything this script prints also goes to a file

Process substitution needs /dev/fd and is not POSIX; /bin/sh scripts must use a named pipe (mkfifo) or a temporary file instead. The inner command’s exit status is not visible to the outer one, so check inputs you care about before, or use wait $! on Bash 5.x, where the last <(...) PID is available as $!.

coproc runs a command in the background with a two-way pipe connected to it, for cases where a script must send several requests to one long-lived process and read each reply, such as an interactive CLI or a database shell.

coproc db { psql -h db.example.com -U app -qAt; }   # ${db[0]} reads from psql, ${db[1]} writes to it
printf 'SELECT 1;\n' >&"${db[1]}"
IFS= read -r -u "${db[0]}" answer                    # read -u: read from that file descriptor
printf 'got %s\n' "$answer"
exec {db[1]}>&-                                       # close psql's stdin so it exits cleanly
wait "$db_PID"

A coprocess is a job like any other; $db_PID holds its PID and the descriptors close when it exits. Only one coprocess with a given name may exist at a time, and Bash warns if you start a second while the first is still running. Reads block, so put a read -t timeout on anything that might not answer.

printf#

printf is the portable, injection-safe way to produce output. echo interprets or ignores -n, -e and backslashes differently between Bash, dash and /bin/echo; printf always does what its format string says and never treats the data as options.

printf '%s\n' "$line"                     # print any string verbatim, including -n and backslashes
printf '%s\n' "${arr[@]}"                 # one element per line; the format repeats for each argument
printf '%-20s %8s %6.2f%%\n' "$host" "$state" "$pct"   # left-justify to 20, right-justify to 8, two decimals
printf '%05d\n' 42                        # 00042: zero-pad to width 5
printf '%x %o %e\n' 255 8 12345.678       # ff 10 1.234568e+04
printf '%b\n' 'a\tb'                      # %b interprets backslash escapes in the argument, %s does not
printf '%q ' rm -rf "$dir"; echo          # shell-quoted for reuse or for logging exactly what will run
printf '%(%Y-%m-%d %H:%M:%S)T\n' -1       # current time via strftime, no fork (Bash 4.2+); -2 is shell start time
printf '%(%s)T\n' -1                      # epoch seconds; $EPOCHSECONDS does the same on Bash 5.0+
printf -v padded '%08.3f' "$value"        # -v: assign the result to a variable instead of printing
printf '%s\0' "${files[@]}" | xargs -0 ls -l   # NUL-separate for tools that accept -0
printf 'Progress: %3d%%\r' "$pct"         # \r overwrites the line; finish with a plain newline
printf '%*s\n' "$width" ''                # * takes the width from an argument; prints $width spaces

%d rejects non-numeric input (invalid number) and prints what it could parse; validate with [[ $n =~ ^-?[0-9]+$ ]] first when the value comes from outside the script. Bash 5.2 adds %Q, which applies a precision before quoting so %.10Q truncates a value then quotes it. With no arguments printf still prints the format once with empty substitutions, so printf '%s\n' "${empty[@]}" emits one blank line rather than nothing; guard with (( ${#empty[@]} )) when that matters.

Shell options#

shopt toggles Bash-specific behaviour; set -o covers the POSIX options plus a few extras. The ones that change how a script behaves:

shopt -s nullglob       # unmatched glob expands to nothing instead of the literal pattern
shopt -s failglob       # unmatched glob is an error (the safer choice for scripts that must find files)
shopt -s dotglob        # * also matches names starting with .
shopt -s globstar       # ** matches recursively (Bash 4.0+): for f in src/**/*.go
shopt -s extglob        # extended patterns: !(*.bak), +([0-9]), @(yes|no)
shopt -s nocasematch    # case-insensitive [[ == ]] and case
shopt -s inherit_errexit  # $(...) inherits set -e (Bash 4.4+)
shopt -s lastpipe       # last stage of a pipeline runs in the current shell (non-interactive only)
shopt -s extdebug       # richer BASH_ARGV/BASH_ARGC and function tracing for debuggers
shopt -p | grep -E 'nullglob|globstar'   # -p prints the current setting in reusable form

With nullglob on, ls *.log with no matches runs ls on the current directory; failglob aborts the command instead, which is usually what a script wants. Set either at the top of the script, not around one loop.

extglob patterns work in case, [[ ]] and parameter expansion, so ${path//+(\/)//} collapses repeated slashes and rm !(*.keep) removes everything except the files you want (a destructive command; test the glob with printf '%s\n' !(*.keep) first).

Oneliners#

# Directory of the running script, symlinks resolved (GNU readlink)
script_dir=$(cd -- "$(dirname -- "$(readlink -f -- "${BASH_SOURCE[0]}")")" && pwd)

# Require commands up front
for c in jq curl kubectl; do command -v "$c" >/dev/null || { echo "need $c" >&2; exit 1; }; done

# Retry with exponential backoff, 5 attempts
for i in {1..5}; do cmd && break; sleep $(( 2 ** i )); done

# Run at most 8 jobs in parallel
printf '%s\n' "${hosts[@]}" | xargs -P8 -I{} ssh {} uptime

# Wait for background jobs and fail if any failed
pids=(); for h in "${hosts[@]}"; do ssh "$h" uptime & pids+=($!); done
rc=0; for p in "${pids[@]}"; do wait "$p" || rc=1; done; exit "$rc"

# Stop a command that may hang (sends TERM after 30s)
timeout 30s curl -fsS https://api.example.com/health

# Allow one instance of a script at a time
exec 9>/run/lock/my-job.lock; flock -n 9 || exit 0

# Trim leading and trailing whitespace without an external command
trim() { local s=$1; s=${s#"${s%%[![:space:]]*}"}; printf '%s' "${s%"${s##*[![:space:]]}"}"; }

# Epoch seconds to local time (GNU date)
date -d "@$epoch" '+%F %T'

# Sum the third column
awk '{s += $3} END {print s}' file

# Most frequent values in the first column
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head

# Bytes as human-readable
numfmt --to=iec-i --suffix=B 1234567

# Is this an interactive shell
[[ $- == *i* ]] && echo interactive

# Colour only when stdout is a terminal
if [[ -t 1 ]]; then red=$'\e[31m' rst=$'\e[0m'; else red='' rst=''; fi
printf '%sfailed%s\n' "$red" "$rst"

# Join array elements with a comma (IFS applies only to this expansion)
(IFS=,; printf '%s\n' "${arr[*]}")

# Split a delimited string into an array without a subshell
IFS=: read -r -a parts <<< "$PATH"

# Read NUL-separated output into an array (Bash 4.4+)
mapfile -d '' files < <(find . -name '*.conf' -print0)

# Wait for whichever background job finishes first, then act (Bash 4.3+)
wait -n && echo 'one job done'

# Time a block without spawning `time` on each command
SECONDS=0; long_task; printf 'took %ds\n' "$SECONDS"

# Random integer in a range (SRANDOM is 32-bit and not seeded from time, Bash 5.1+)
n=$(( SRANDOM % 100 ))

# Indirect reference: the value of the variable whose name is in $name
name=HOME; printf '%s\n' "${!name}"

# All variable names starting with a prefix (for dumping config passed by environment)
for v in "${!MYAPP_@}"; do printf '%s=%s\n' "$v" "${!v}"; done

# Read with a timeout so an unattended run does not hang forever (returns >128 on timeout)
read -r -t 10 -p 'Token: ' token || { echo 'no input' >&2; exit 1; }

# Read a password without echo
read -r -s -p 'Password: ' pass; echo

# Case-insensitive match without changing global shell options
[[ ${answer,,} == y* ]] && echo yes

# Log every line the script prints to both the terminal and the journal
exec > >(tee >(logger -t my-script)) 2>&1

# Pass a function to xargs or find by exporting it (Bash only)
check() { curl -fsS --max-time 5 "https://$1/health" >/dev/null && echo "$1 ok" || echo "$1 FAIL"; }
export -f check; printf '%s\n' "${hosts[@]}" | xargs -P8 -I{} bash -c 'check "$@"' _ {}

# Confirm before a destructive step, defaulting to no
read -r -p "Delete ${#targets[@]} files? [y/N] " a; [[ $a == [yY] ]] || exit 0

# Print a stack trace from inside a function (useful in an ERR trap)
for ((i = 1; i < ${#FUNCNAME[@]}; i++)); do printf '  at %s (%s:%s)\n' "${FUNCNAME[$i]}" "${BASH_SOURCE[$i]}" "${BASH_LINENO[$((i-1))]}"; done

# Replace a file atomically: write to a temp file in the same directory, then rename
tmp=$(mktemp "${target}.XXXXXX") && generate > "$tmp" && mv -f -- "$tmp" "$target"

# Strip ANSI colour codes from captured output
clean=$(sed 's/\x1b\[[0-9;]*m//g' <<< "$raw")

# Check that a variable is a positive integer before using it in arithmetic
[[ $count =~ ^[1-9][0-9]*$ ]] || { echo "count must be a positive integer" >&2; exit 2; }

# Source a config file only if it is a regular file owned by the current user
[[ -f $cfg && -O $cfg ]] && . "$cfg"

Scripts#

Parallel health check over a list of hosts with a per-host timeout, exit status reflecting any failure, and a summary at the end.

#!/usr/bin/env bash
# usage: health-check.sh hosts.txt   (one hostname per line, # comments allowed)
set -euo pipefail

hosts_file=${1:?hosts file required}
jobs=${JOBS:-8}
timeout_s=${TIMEOUT:-5}
results=$(mktemp)
trap 'rm -f -- "$results"' EXIT

check() {                                  # runs in a child bash via xargs; prints one line per host
  local host=$1 code
  code=$(curl -sS -o /dev/null -w '%{http_code}' --max-time "$TIMEOUT" "https://$host/health" 2>/dev/null || true)
  if [[ $code == 200 ]]; then printf 'OK   %s\n' "$host"; else printf 'FAIL %s (%s)\n' "$host" "${code:-timeout}"; fi
}
export -f check
export TIMEOUT=$timeout_s

grep -Ev '^\s*(#|$)' "$hosts_file" \
  | xargs -P "$jobs" -I{} bash -c 'check "$1"' _ {} \
  | tee "$results"

failed=$(grep -c '^FAIL' "$results" || true)
printf '\n%d checked, %d failed\n' "$(wc -l < "$results")" "$failed"
(( failed == 0 ))

Log cleanup that compresses files older than a threshold and deletes compressed files past a retention limit, with --dry-run printing what would happen. Deletes files when run without --dry-run.

#!/usr/bin/env bash
# usage: log-cleanup.sh [--dry-run] [--compress-days N] [--delete-days N] DIR...
set -euo pipefail

dry=0 compress_days=7 delete_days=90 dirs=()
while [[ $# -gt 0 ]]; do
  case $1 in
    --dry-run)           dry=1; shift ;;
    --compress-days)     compress_days=$2; shift 2 ;;
    --delete-days)       delete_days=$2; shift 2 ;;
    --)                  shift; dirs+=("$@"); break ;;
    -*)                  printf 'unknown option: %s\n' "$1" >&2; exit 2 ;;
    *)                   dirs+=("$1"); shift ;;
  esac
done
(( ${#dirs[@]} )) || { echo 'no directories given' >&2; exit 2; }
(( delete_days > compress_days )) || { echo '--delete-days must exceed --compress-days' >&2; exit 2; }

run() { if (( dry )); then printf '[dry-run] %q ' "$@"; echo; else "$@"; fi; }

for d in "${dirs[@]}"; do
  [[ -d $d ]] || { printf 'skip %s: not a directory\n' "$d" >&2; continue; }
  # -mtime +N means strictly more than N whole days old; -type f skips symlinks and directories
  while IFS= read -r -d '' f; do run gzip -9 -- "$f"; done \
    < <(find "$d" -maxdepth 1 -type f -name '*.log' ! -name '*.gz' -mtime +"$compress_days" -print0)
  while IFS= read -r -d '' f; do run rm -f -- "$f"; done \
    < <(find "$d" -maxdepth 1 -type f -name '*.log.gz' -mtime +"$delete_days" -print0)
done

Wrapper that runs a command under a lock so only one copy executes at a time, retrying with exponential backoff and jitter on failure. Meant for cron and systemd timers where overlapping runs are the usual cause of corrupted state.

#!/usr/bin/env bash
# usage: with-retry.sh [-n attempts] [-l lockfile] -- command args...
set -euo pipefail

attempts=5 lock=/run/lock/with-retry.lock
while getopts ':n:l:' opt; do
  case $opt in
    n) attempts=$OPTARG ;;
    l) lock=$OPTARG ;;
    :) printf 'option -%s needs a value\n' "$OPTARG" >&2; exit 2 ;;
    \?) printf 'unknown option: -%s\n' "$OPTARG" >&2; exit 2 ;;
  esac
done
shift $((OPTIND - 1))
(( $# )) || { echo 'no command given' >&2; exit 2; }

exec 9>"$lock"
if ! flock -n 9; then printf 'another run holds %s, exiting\n' "$lock" >&2; exit 0; fi

for (( i = 1; i <= attempts; i++ )); do
  "$@" && exit 0
  rc=$?                                            # status of the && list is the command's status
  (( i < attempts )) || break
  delay=$(( (2 ** i) + RANDOM % 5 ))              # jitter stops synchronised retries across hosts
  printf 'attempt %d/%d failed (rc=%d), retrying in %ds\n' "$i" "$attempts" "$rc" "$delay" >&2
  sleep "$delay"
done
printf 'giving up after %d attempts\n' "$attempts" >&2
exit "${rc:-1}"

Debugging a script#

bash -n script.sh                                    # parse only, no execution
shellcheck script.sh                                 # static analysis: quoting, set -e traps, portability
PS4='+ ${BASH_SOURCE}:${LINENO}:${FUNCNAME[0]:-main}: ' bash -x script.sh   # trace with file and line

Troubleshooting#

Symptom	Cause	Fix
`$'\r': command not found`	Windows (CRLF) line endings	`sed -i 's/\r$//' script.sh`; set `* text=auto eol=lf` in `.gitattributes`
`bad substitution` or `declare: -A: invalid option`	Run by `sh` (dash) or Bash 3.2	Run with `bash`, check the shebang and `bash --version`
`unbound variable`	`set -u` and an optional variable	`${VAR:-}`; for empty arrays on Bash < 4.4 use `${arr[@]+"${arr[@]}"}`
Script exits silently part way	`set -e` and a non-zero command	`bash -x`, or trap `ERR` with `$LINENO`
Error ignored despite `set -e`	Command is in a tested context, a pipeline without `pipefail`, `local x=$(...)` or `$(...)`	Test the status explicitly
Variable empty after a loop	Loop ran in a pipeline subshell	`done < <(cmd)` or `shopt -s lastpipe`
Last line of a file not processed	No trailing newline	`while IFS= read -r l \|\| [[ -n $l ]]`
`Argument list too long`	Glob or `$(...)` exceeds `ARG_MAX`	`find ... -exec cmd {} +` or `xargs -0`
Filenames with spaces break	Unquoted expansion or `for f in $(ls)`	Quote, and iterate with a glob or `find -print0`
Works in terminal, fails from cron or systemd	Different `PATH`, no TTY, different working directory	Use absolute paths, set `PATH`, `cd` explicitly; see systemd
Script exits at `(( i++ ))` when `i` is 0	Post-increment returns the old value; 0 is “false”, so `set -e` fires	Use `(( i += 1 ))`, `(( ++i ))` or `i=$(( i + 1 ))`
Backslashes vanish from lines read from a file	`read` without `-r` treats `\` as an escape	Always `read -r`
`echo -e` prints `-e` or escapes are ignored	`echo` differs between Bash, dash and `/bin/echo`	`printf '%b\n' "$s"` for escapes, `printf '%s\n'` otherwise
`*` skips `.env` and other dotfiles	Globs do not match a leading `.` by default	`shopt -s dotglob`, or `find -name '.*'`
`[[ $x == "$y" ]]` never matches a pattern	Quoted right-hand side is a literal string	Leave the pattern unquoted: `[[ $x == $y ]]`, or use `=~` for regex
`printf: abc: invalid number`	`%d` given non-numeric input	Validate with `[[ $n =~ ^-?[0-9]+$ ]]` before formatting
`wc -c <<< "$s"` is one more than `${#s}`	Here-strings append a newline	Use `printf '%s' "$s" \| cmd` when the trailing newline matters
`ls *.log` lists everything	`nullglob` is on and nothing matched, so `ls` got no arguments	Use `failglob`, or test `[ -e "$f" ]` inside the loop
`command not found` for a function passed to `xargs` or `find`	The child `bash -c` does not see unexported functions	`export -f name`, then `bash -c 'name "$@"' _ {}`