grep, sed and awk · Software Engineering Wiki

Cheatsheet#

Task	Command
Lines matching, case-insensitive, with line numbers	`grep -in 'error' app.log`
Recursive, only filenames, skip `.git`	`grep -rl --exclude-dir=.git 'TODO' .`
Fixed string, not a regex	`grep -F '[main]' config.ini`
Invert match, count	`grep -vc '^#' file`
Only the matched part	`grep -oE '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+' access.log`
Context around a match	`grep -B2 -A5 'panic' app.log`
Replace on every line, in place, with backup	`sed -i.bak 's/old/new/g' file`
Delete lines matching	`sed '/^\s*#/d' file`
Print lines 10 to 20	`sed -n '10,20p' file`
Print between two patterns	`sed -n '/BEGIN/,/END/p' file`
Sum a column	`awk '{s += $3} END {print s}' file`
Rows where a column exceeds a value	`awk -F, '$4 > 500' data.csv`
Count by field	`awk '{c[$1]++} END {for (k in c) print c[k], k}' file \| sort -rn`
Print the last field	`awk '{print $NF}' file`
Frequency table	`sort file \| uniq -c \| sort -rn \| head`
Sort by the second column numerically	`sort -k2,2n file`
Columns 1 and 3 of a CSV	`cut -d, -f1,3 data.csv`
Uppercase, squeeze repeated spaces	`tr 'a-z' 'A-Z' < file`, `tr -s ' '`
Join two files side by side	`paste -d, a.txt b.txt`
Align output into columns	`column -t -s,`
Run a command per line, in parallel	`xargs -P8 -I{} cmd {}`
NUL-safe filename handling	`find . -name '*.log' -print0 \| xargs -0 gzip`
Characters, words, lines	`wc -c`, `wc -w`, `wc -l`

Behaviour below is GNU grep 3.11, GNU sed 4.9, GNU awk (gawk) 5.3 and GNU coreutils 9.x as shipped on Fedora and RHEL 9. macOS and Alpine ship BSD or BusyBox variants with fewer options; the differences that bite are called out. References: the GNU grep, GNU sed and gawk manuals.

Which tool#

grep selects lines. sed edits lines in a single pass with a tiny state machine. awk splits lines into fields and gives you variables, arrays, arithmetic and formatted output. Reach for the simplest one that does the job: a grep | awk pipeline is usually clearer than one awk program that does both, and an awk program with three arrays is usually clearer than the equivalent Python once you know awk. All three read stdin when given no file, so they compose.

All three use regular expressions but not the same flavour. grep defaults to POSIX BRE, -E gives ERE and -P gives PCRE. sed also defaults to BRE with -E for ERE. awk always uses ERE. In BRE the characters + ? | ( ) { } are literal unless escaped, so grep 'a+' matches a literal plus and grep -E 'a+' matches one or more a. See Regular expressions for the full comparison.

grep#

grep prints each line that matches a pattern. With several files it prefixes the filename; -h suppresses that and -H forces it.

grep -n 'timeout' app.log                 # -n: line numbers
grep -i 'warn' app.log                    # -i: case-insensitive
grep -w 'error' app.log                   # -w: whole word; error but not errors
grep -x 'ok' status.txt                   # -x: whole line
grep -v '^$' file                         # -v: invert; drops empty lines
grep -c 'GET' access.log                  # -c: count of matching lines (not matches)
grep -l 'password' -r /etc 2>/dev/null    # -l: filenames with a match; -L: filenames without
grep -o '[0-9]\{3\}' file                 # -o: print only the matched text, one per line
grep -m 5 'ERROR' app.log                 # -m: stop after 5 matches
grep -q 'ready' status && echo up         # -q: no output; exit status only
grep -e '-v' -e 'foo' file                # -e: patterns that start with a dash, or several patterns
grep -f patterns.txt file                 # -f: one pattern per line from a file
grep -F 'a.b[0]' file                     # -F: fixed strings, no regex; fastest
grep -E 'error|fatal' app.log             # -E: extended regex; alternation without backslashes
grep -P '(?<=user=)\w+' -o app.log        # -P: PCRE; lookbehind, \d, non-greedy
grep -z -o 'BEGIN.*END' file              # -z: NUL-separated records, so . spans newlines
grep -A3 -B1 'Exception' app.log          # context after and before; -C3 for both
grep --color=always 'x' file | less -R    # keep highlighting through a pager

Recursive search options:

grep -r 'pattern' src/                    # recurse; follows symlinks given on the command line only (-R follows all)
grep -rn --include='*.go' 'context.TODO' .
grep -rn --exclude='*.min.js' --exclude-dir={.git,node_modules,vendor} 'fetch(' .
grep -rI 'pattern' .                      # -I: skip binary files
grep -ra 'pattern' .                      # -a: treat binary as text (prints matched binary lines)

Exit status is 0 for a match, 1 for none, 2 for an error. -q exits on the first match, so it is also the fastest way to test for presence. -s hides errors about missing or unreadable files.

The -P engine is the only one with lazy quantifiers, lookaround and \d. It refuses to run on files with invalid UTF-8 in a UTF-8 locale unless you add -a or set LC_ALL=C. LC_ALL=C grep is also several times faster on large ASCII logs because byte comparison replaces multibyte character handling.

sed#

sed reads a line into the pattern space, runs every command whose address matches, prints the pattern space unless -n is set, and moves to the next line. That model explains almost everything: commands are cheap, the file is streamed, and anything that needs to look at two lines at once takes extra work.

Substitution#

sed 's/old/new/' file           # first occurrence per line
sed 's/old/new/g' file          # every occurrence
sed 's/old/new/2' file          # second occurrence only
sed 's/old/new/gi' file         # I or i: case-insensitive (GNU)
sed 's|/usr/local|/opt|g' file  # any delimiter; pick one not in the pattern
sed -E 's/([0-9]+)-([0-9]+)/\2-\1/' file    # -E: ERE with unescaped groups; \1 \2 backreferences
sed 's/.*/"&"/' file            # &: the whole match
sed -E 's/\w+/\u&/g' file       # \u: uppercase next character; \U...\E uppercase a span (GNU)
sed 's/x/y/w changed.txt' file  # w: write changed lines to a file
sed 's/[[:space:]]+$//' file    # POSIX class; in BRE + is literal, so this needs -E
sed -E 's/[[:space:]]+$//' file # trailing whitespace removed

Addresses#

An address before a command restricts it to matching lines. Two addresses separated by a comma select a range from the first match to the next match of the second.

sed -n '5p' file                # line 5
sed -n '5,10p' file             # lines 5 to 10
sed -n '$p' file                # last line
sed -n '/start/,/end/p' file    # from a line matching start to the next matching end, inclusive
sed '/^#/d' file                # delete comment lines
sed '1d' file                   # drop the header
sed '/^$/d' file                # drop blank lines
sed -n '/error/!p' file         # !: negate; lines not matching
sed '0~2d' file                 # GNU: first~step; every second line starting at line 0 (even lines)
sed '10,$d' file                # from line 10 to end
sed '/pattern/,+3d' file        # GNU: the match and the three lines after it
sed -n '/BEGIN/{n;p}' file      # n: load the next line; print the line after each BEGIN

Other commands#

sed '3i\inserted before line 3' file
sed '3a\appended after line 3' file
sed '/marker/c\replacement line' file     # c: replace the whole line
sed 'y/abc/xyz/' file                     # transliterate, like tr
sed '=' file | sed 'N;s/\n/\t/'           # =: print line number; N appends the next line; joins them
sed -n 'l' file                           # l: print unambiguously, escapes and $ at line end
sed '$!N;s/\n/ /' file                    # join pairs of lines (N on every line but the last)
sed ':a;N;$!ba;s/\n/,/g' file             # slurp the file into one line, replacing newlines
sed -e 's/a/b/' -e 's/c/d/' file          # several scripts; or separate with ;
sed -f edits.sed file                     # commands from a file
sed -s -n '1p' *.log                      # -s: treat files separately so 1 means line 1 of each
sed 's/a/b/;t;s/c/d/' file                # t: branch to end if the last s succeeded

The pattern space and hold space (h, H, g, G, x) let sed carry state across lines, which is how sed -n '1!G;h;$p' reverses a file. Once a script needs the hold space, awk is usually clearer.

In-place editing: GNU versus BSD#

GNU sed takes an optional suffix attached to -i; BSD sed (macOS, FreeBSD) requires a suffix argument, which may be an empty string.

sed -i 's/old/new/g' file        # GNU: edit in place, no backup
sed -i.bak 's/old/new/g' file    # GNU: backup to file.bak (no space before the suffix)
sed -i '' 's/old/new/g' file     # BSD: empty suffix, no backup (GNU would treat '' as the script)
sed -i.bak 's/old/new/g' file    # BSD: also works, and is the one form portable to both

-i writes a new file and renames it over the original, so it breaks hard links, follows symlinks only with --follow-symlinks (GNU) and resets ownership if you run as a different user than the owner. It does not honour -n in the way you might expect: sed -n -i 'p' file keeps the file, but sed -n -i '/x/p' file deletes every line that does not match. Test without -i first.

BSD sed also lacks \+, \|, \u, \U, I, 0~2, addr,+N and interprets a, i and c more strictly (they need a backslash and newline). For scripts that must run on both, stick to POSIX BRE with -E for ERE and avoid GNU extensions.

awk#

awk splits each input record (a line by default) into fields $1 to $NF on the field separator FS (runs of whitespace by default), then runs every pattern { action } pair whose pattern is true for that record. BEGIN runs before input and END after. A pattern without an action prints the record; an action without a pattern runs on every record.

awk '{print $1, $3}' file                 # comma inserts OFS (a space); print $1 $3 concatenates
awk -F: '{print $1}' /etc/passwd          # -F: field separator; also a regex: -F'[:,]'
awk -F'\t' -v OFS=',' '{$1 = $1; print}' file   # TSV to CSV; assigning a field rebuilds $0 with OFS
awk 'NR == 1' file                        # header only; NR: record number so far
awk 'NR > 1' file                         # skip header
awk 'NF' file                             # NF: number of fields; drops blank lines
awk 'NF > 5' file                         # lines with more than five fields
awk '$3 > 100' file                       # numeric comparison on a field
awk '$1 == "GET" && $9 >= 500' access.log
awk '/error/ && !/timeout/' file          # regex patterns, combined
awk '/start/,/end/' file                  # range pattern, like sed
awk 'length($0) > 120' file               # long lines
awk '$2 ~ /^10\./' file                   # field matches regex; !~ negates
awk 'NR % 100 == 0' file                  # every hundredth line
awk 'END {print NR}' file                 # line count
awk '{print NR": "$0}' file               # number the lines
awk '{print $NF}' file                    # last field; $(NF-1) second last
awk '{$NF = ""; print}' file              # drop the last field (leaves a trailing OFS)
awk 'FNR == 1 && NR != 1 {next} 1' *.csv  # concatenate CSVs keeping only the first header; FNR resets per file
awk -v n=3 '{print $n}' file              # -v: shell value into an awk variable, set before BEGIN
awk '{print $1 > ($2 ".txt")}' file       # split into files named by the second column

Uninitialised variables are "" and 0 at once, so c[$1]++ needs no setup. Strings compare as strings and numbers as numbers; a field that looks numeric compares numerically ("10" > "9" is false as strings, true as strnums). Force a string comparison with $1 "" == "10" or a numeric one with $1 + 0 == 10.

Arrays#

awk arrays are associative with string keys. Iteration order is unspecified; pipe to sort or use gawk’s PROCINFO["sorted_in"].

awk '{c[$1]++} END {for (k in c) print c[k], k}' access.log | sort -rn | head    # count by first field
awk '{s[$1] += $10} END {for (k in s) printf "%s %.1f MiB\n", k, s[k]/1048576}' access.log   # bytes per client
awk '!seen[$0]++' file                    # deduplicate keeping first occurrence and order
awk '!seen[$2]++' file                    # unique by column 2
awk 'NR == FNR {a[$1]; next} $1 in a' keys.txt data.txt    # lines of data.txt whose key is in keys.txt
awk 'NR == FNR {m[$1] = $2; next} {print $0, m[$1]}' lookup.txt data.txt   # join: append a looked-up value
awk '{if ($3 > max[$1]) max[$1] = $3} END {for (k in max) print k, max[k]}' file   # max per group
awk '{n = split($0, parts, ","); print parts[n]}' file     # split returns the count; last element
awk 'BEGIN {PROCINFO["sorted_in"] = "@val_num_desc"} {c[$1]++} END {for (k in c) print c[k], k}' file   # gawk: sorted iteration
awk '{delete c; ...}'                     # delete the whole array (gawk and POSIX 2024)

NR == FNR is true only while reading the first file, which is the idiom for loading a lookup table before processing the second file.

printf and formatting#

printf does not append a newline. Width and precision work as in C.

awk '{printf "%-20s %8d %6.2f%%\n", $1, $2, $3 * 100}' file   # left-justify 20, right-justify 8, two decimals
awk '{printf "%s\t%s\n", toupper($1), substr($2, 1, 8)}' file
awk 'BEGIN {printf "%5.1f\n", 1234567 / 1048576}'             # 1.2
awk '{printf "%08.3f\n", $1}' file                             # zero-padded
awk 'BEGIN {OFMT = "%.2f"} {print $1 / 3}' file                # OFMT: format for print of non-integer numbers
awk '{print strftime("%F %T", $1)}' epochs.txt                 # gawk: epoch to date
awk 'BEGIN {print mktime("2026 09 24 00 00 00")}'              # gawk: date to epoch
awk '{gsub(/"/, ""); print}' file                              # gsub: replace all in $0; sub: first only
awk '{sub(/\r$/, ""); print}' file                             # strip CR
awk 'BEGIN {IGNORECASE = 1} /error/' file                      # gawk: case-insensitive matching
awk -F, 'BEGIN {OFS = ","} {print $2, $1}' file                # swap columns
awk 'BEGIN {FPAT = "([^,]+)|(\"[^\"]+\")"} {print $2}' quoted.csv   # gawk: CSV fields with quoted commas
awk --csv '{print $2}' quoted.csv                              # gawk 5.3+: proper CSV parsing including embedded newlines

String functions: length(s), substr(s, start, len), index(s, t), split(s, arr, sep), sub(re, repl, target), gsub, match(s, re) (sets RSTART and RLENGTH), tolower, toupper, sprintf. gawk adds gensub(re, repl, how, target) with \\1 backreferences, patsplit, strftime, systime, mktime and asort/asorti.

Multi-line records: set RS to a blank line (RS="") to treat paragraphs as records, with each line a field when FS="\n". RS may be a regex in gawk.

awk 'BEGIN {RS = ""; FS = "\n"} {print $1 " -> " NF " lines"}' blocks.txt

sort, uniq and friends#

sort is locale-aware; LC_ALL=C sort is byte order and much faster. uniq only collapses adjacent duplicates, so sort first.

sort file                       # lexical, locale collation
sort -n file                    # numeric
sort -h file                    # human-readable sizes: 1K, 2M, 3G (GNU)
sort -V versions.txt            # version sort: 1.2.10 after 1.2.9 (GNU)
sort -r file                    # reverse
sort -u file                    # unique lines (on the whole line, or the key with -k)
sort -k2,2n -k1,1 file          # by column 2 numerically, then column 1; always give the end field
sort -t, -k3,3nr data.csv       # CSV, column 3, numeric, descending
sort -t$'\t' -k2 file           # tab-separated
sort -s -k1,1 file              # stable: equal keys keep input order
sort -R file                    # random order (shuf is faster and does not group equal lines)
sort -c file                    # check whether sorted; exit 1 and the first offending line if not
sort -m a.sorted b.sorted       # merge already-sorted inputs
sort --parallel=8 -S 2G -T /var/tmp bigfile   # threads, memory buffer, temp directory
sort -z                         # NUL-terminated records for filenames

-k2 without an end means “from field 2 to the end of the line”, which is almost never intended. -k2,2 is one field. -n on a key applies only to that key: -k2,2n.

sort file | uniq                # duplicates removed
sort file | uniq -c             # count each; count first, then the line
sort file | uniq -d             # only lines that appear more than once
sort file | uniq -u             # only lines that appear once
sort file | uniq -c | sort -rn | head -20    # top 20 most frequent
uniq -f1 file                   # ignore the first field when comparing
uniq -i file                    # case-insensitive
uniq -w 10 file                 # compare the first 10 characters only

cut extracts by delimiter or byte or character position. It cannot reorder fields or handle multi-character delimiters; use awk for those.

cut -d: -f1,7 /etc/passwd       # fields 1 and 7
cut -d, -f2- data.csv           # field 2 to end
cut -d, -f-3 data.csv           # fields 1 to 3
cut -c1-10 file                 # characters 1 to 10
cut -b1-4 file                  # bytes
cut -d, -f2 --complement data.csv   # everything except field 2 (GNU)
cut -d' ' -f1 --output-delimiter=, file
cut -f1 file                    # default delimiter is TAB

Lines without the delimiter are printed whole unless -s is given. cut treats each delimiter as a field boundary, so runs of spaces produce empty fields; tr -s ' ' first, or use awk.

tr maps, squeezes or deletes characters. It works on characters, never strings.

tr 'a-z' 'A-Z' < file           # uppercase
tr '[:lower:]' '[:upper:]'      # locale-safe form
tr -d '\r' < dos.txt > unix.txt # delete CR
tr -d '[:punct:]'               # delete punctuation
tr -s ' ' < file                # squeeze repeated spaces to one
tr -s '[:space:]' '\n'          # one word per line
tr ':' '\n' <<< "$PATH"         # split PATH
tr -cd '[:alnum:]\n' < file     # -c: complement; keep only alphanumerics and newlines
tr '\0' '\n' < /proc/1/environ  # NUL-separated to lines

paste joins files line by line, or serialises one file’s lines.

paste a.txt b.txt               # side by side, tab-separated
paste -d, a.txt b.txt           # comma
paste -sd, file                 # -s: serial; join all lines of one file with commas
paste - - < file                # two lines into one, tab-separated (each - is a stdin read)
paste -d'\n' a.txt b.txt        # interleave lines

join merges two sorted files on a common field like a SQL inner join; comm compares two sorted files line by line.

join -t, -1 1 -2 2 <(sort -t, -k1,1 a.csv) <(sort -t, -k2,2 b.csv)   # a.csv col 1 with b.csv col 2
join -a1 -e MISSING -o 0,1.2,2.2 a.txt b.txt   # left join with a placeholder for unmatched
comm -12 <(sort a) <(sort b)    # lines in both
comm -23 <(sort a) <(sort b)    # lines only in a
comm -13 <(sort a) <(sort b)    # lines only in b

column formats into aligned columns. On Fedora and RHEL it is the util-linux version, which also emits JSON.

column -t file                  # align on whitespace
column -t -s, data.csv          # CSV input
column -t -s, -N NAME,SIZE,DATE data.csv   # header names (util-linux)
column -t -s, -J -N name,size data.csv     # JSON output (util-linux)
column -c 80 list.txt           # fill columns to 80 characters wide
mount | column -t

Other coreutils that turn up in pipelines: head -n -5 (all but the last 5), tail -n +2 (from line 2), tail -F (follow across rotation), tac (reverse lines), rev (reverse characters), nl (number lines), fold -w 80 -s (wrap at spaces), fmt, expand/unexpand (tabs), shuf -n 10 (sample), split -l 100000 big.log part- (split into files), numfmt --to=iec (human sizes), seq 1 10, wc -l.

xargs#

xargs reads items from stdin and runs a command with them as arguments, batching as many as fit. Without options it splits on whitespace and interprets quotes, which is wrong for filenames; use -0 with find -print0 or -d '\n' for line-oriented input.

find . -name '*.log' -print0 | xargs -0 gzip              # batches; NUL-safe
xargs -d '\n' -I{} cp {} /backup/ < list.txt              # one line per item; {} placeholder
xargs -n1 echo < list.txt                                 # one item per command
xargs -n 50 rm -f < files.txt                             # 50 per command
xargs -P 8 -I{} curl -fsS -o /dev/null -w '%{http_code} {}\n' {} < urls.txt   # 8 in parallel
xargs -r rm < maybe-empty.txt                             # -r: do nothing if input is empty (GNU; BSD default)
xargs -t cmd < list.txt                                   # -t: print each command before running it
xargs -p rm < list.txt                                    # -p: prompt before each command
echo a b c | xargs -n1 -I{} sh -c 'echo "item: $1"' _ {}  # shell logic per item; $1, not {}, inside the script
cat urls.txt | xargs -P4 -n1 -- wget -q                   # -- ends xargs options

-I implies -L 1 (one item per command) and disables -n. Output of parallel jobs interleaves; add --process-slot-var or write to per-item files when order matters. xargs exits 123 if any invocation returned 1 to 125, 124 if one exited 255, and 125 to 127 for its own failures.

find -exec cmd {} + does the same batching without a pipe and is the portable choice when the input is a find result. GNU parallel adds job logs, retries and output grouping when xargs -P is not enough; it is a separate package.

Log-analysis pipelines#

Start narrow with grep, then aggregate with awk, then rank with sort | uniq -c | sort -rn. Use LC_ALL=C on large files and avoid cat file | grep when grep pattern file does the same thing.

Requests per status code from an nginx or Apache combined log (status is field 9, bytes field 10):

awk '{c[$9]++} END {for (s in c) printf "%s %d\n", s, c[s]}' access.log | sort -k2,2rn

Top 20 client IPs, then the same restricted to 5xx responses:

awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -20
awk '$9 ~ /^5/ {print $1}' access.log | sort | uniq -c | sort -rn | head -20

Requests per minute, to find a spike:

awk -F'[][]' '{print substr($2, 1, 17)}' access.log | uniq -c | awk '{printf "%s %6d %s\n", $2, $1, substr("##################################################", 1, $1 / 50)}'

The -F'[][]' splits on either bracket so the timestamp is field 2; the timestamp’s first 17 characters are 24/Sep/2026:14:03, and because the log is already in time order, uniq -c counts without a sort.

Bytes served per URL path, in MiB, top 10, ignoring the query string:

awk '{split($7, u, "?"); b[u[1]] += $10} END {for (p in b) printf "%10.1f %s\n", b[p] / 1048576, p}' access.log | sort -rn | head

Slowest requests when the log ends with a request time in seconds (a custom $request_time field, here field 11), with the 95th percentile:

awk '{print $11, $7}' access.log | sort -rn | head
awk '{t[NR] = $11} END {n = asort(t); printf "p50 %.3f p95 %.3f p99 %.3f max %.3f\n", t[int(n * .5)], t[int(n * .95)], t[int(n * .99)], t[n]}' access.log   # gawk asort

Errors per hour from the journal or a syslog-format file, and the distinct error messages with numbers and hex stripped so they group:

journalctl -u my-app --since today -o short-iso | grep -i error | cut -c1-13 | uniq -c
grep -i error app.log | sed -E 's/[0-9]+/N/g; s/0x[0-9a-f]+/0xH/g' | sort | uniq -c | sort -rn | head

Pull the block between a start and end marker for the request ID you care about, from a multi-line log:

sed -n '/request_id=abc123/,/^--- end/p' app.log
awk '/BEGIN TRANSACTION/ {buf = ""; keep = 0} {buf = buf $0 "\n"} /ERROR/ {keep = 1} /COMMIT|ROLLBACK/ {if (keep) printf "%s", buf}' db.log   # only blocks containing an error

Unique users who logged in today from auth.log or journalctl -t sshd, and failed password sources ranked:

journalctl -t sshd --since today | grep -oP 'Accepted \S+ for \K\S+' | sort -u
journalctl -t sshd --since today | grep -oP 'Failed password for (invalid user )?\S+ from \K\S+' | sort | uniq -c | sort -rn | head

\K in a PCRE pattern discards everything matched before it from the -o output, which avoids a cut or awk stage.

Follow a log and highlight without losing the rest of the lines:

tail -F app.log | grep --line-buffered -E 'ERROR|WARN|$'     # $ matches every line; only the keywords colour
tail -F app.log | awk '/ERROR/ {print "\033[31m" $0 "\033[0m"; next} 1'

grep block-buffers when writing to a pipe, so tail -F | grep | awk shows nothing for a long time without --line-buffered. awk and sed have fflush() and -u respectively; stdbuf -oL cmd fixes most other tools.

Oneliners#

# Count matching lines across many files and total them
grep -c 'ERROR' *.log | awk -F: '{s += $2; print} END {print "total:", s}'

# Files containing both patterns
grep -lZ 'foo' -r . | xargs -0 grep -l 'bar'

# Lines in file1 not in file2, regardless of order
grep -vxFf file2 file1

# Print the line after each match, without the match
grep -A1 'pattern' file | grep -v -e 'pattern' -e '^--$'

# Extract every email address
grep -oE '[[:alnum:]._%+-]+@[[:alnum:].-]+\.[[:alpha:]]{2,}' file | sort -u

# Extract IPv4 addresses and rank them
grep -oE '\b([0-9]{1,3}\.){3}[0-9]{1,3}\b' file | sort | uniq -c | sort -rn

# Search a compressed log without decompressing to disk
zgrep -c 'ERROR' app.log.1.gz; zstdgrep 'ERROR' app.log.zst

# Replace across a tree, only in files that match (avoids rewriting unchanged files)
grep -rlZ 'old.example.com' src/ | xargs -0 sed -i 's/old\.example\.com/new.example.com/g'

# Replace a path containing slashes using a different delimiter
sed -i 's#/var/www/html#/srv/www#g' /etc/nginx/conf.d/*.conf

# Change a key=value in a config file, only if the key exists uncommented
sed -i -E 's/^(max_connections\s*=\s*).*/\1500/' postgresql.conf

# Append a line after a matching line, once
grep -q '^Include conf.d' sshd_config || sed -i '/^Port /a Include conf.d/*.conf' sshd_config

# Comment out lines matching
sed -i '/^PermitRootLogin/s/^/#/' sshd_config

# Delete from a pattern to end of file
sed -i '/^\[legacy\]/,$d' config.ini

# Print the Nth line quickly (q stops reading)
sed -n '1000{p;q}' bigfile

# Strip ANSI escape sequences
sed -E 's/\x1b\[[0-9;]*[A-Za-z]//g' coloured.log

# Trim leading and trailing whitespace
sed -E 's/^[[:space:]]+|[[:space:]]+$//g' file

# Convert CRLF to LF and back
sed -i 's/\r$//' file; sed -i 's/$/\r/' file

# Sum a column of a CSV, skipping the header
awk -F, 'NR > 1 {s += $3} END {printf "%.2f\n", s}' data.csv

# Average of a column
awk '{s += $1; n++} END {if (n) print s / n}' numbers.txt

# Min and max of a column
awk 'NR == 1 {min = max = $1} $1 < min {min = $1} $1 > max {max = $1} END {print min, max}' numbers.txt

# Group by column 1 and sum column 2
awk '{s[$1] += $2} END {for (k in s) print k, s[k]}' file | sort

# Print lines where a field is in a set
awk 'BEGIN {split("GET POST", a); for (i in a) ok[a[i]]} $6 in ok' access.log

# Transpose rows and columns
awk '{for (i = 1; i <= NF; i++) m[i, NR] = $i; if (NF > nf) nf = NF} END {for (i = 1; i <= nf; i++) {row = ""; for (j = 1; j <= NR; j++) row = row (j > 1 ? " " : "") m[i, j]; print row}}' file

# Print a column of a fixed-width file by character positions
awk '{print substr($0, 10, 8)}' fixed.txt

# Lines longer than 80 characters, with their numbers
awk 'length > 80 {print FILENAME ":" FNR ": " length}' *.md

# Sort a file by line length
awk '{print length, $0}' file | sort -n | cut -d' ' -f2-

# Deduplicate without sorting, keeping the first occurrence
awk '!seen[$0]++' file

# Random sample of 1% of lines
awk 'BEGIN {srand()} rand() < 0.01' bigfile

# Convert epoch timestamps in column 1 to ISO 8601 (gawk)
awk '{$1 = strftime("%FT%T%z", $1)} 1' events.log

# Human-readable sizes from du without -h (for sorting first)
du -sk * | sort -rn | head | numfmt --field=1 --from-unit=1024 --to=iec

# Top 10 processes by RSS, formatted
ps -eo rss,comm --sort=-rss | head -11 | awk 'NR == 1 {print; next} {printf "%8.1f MiB %s\n", $1 / 1024, $2}'

# Parse key=value logs into one field
grep -oP 'duration=\K[0-9.]+' app.log | sort -n | tail -1

# Find the header column index by name, then print that column
awk -F, -v col=status 'NR == 1 {for (i = 1; i <= NF; i++) if ($i == col) c = i; next} {print $c}' data.csv

# Reverse fields on each line
awk '{for (i = NF; i > 0; i--) printf "%s%s", $i, (i > 1 ? OFS : ORS)}' file

# Join lines matching a prefix with the previous line (continuation lines)
awk '/^[[:space:]]/ {printf "%s", $0; next} NR > 1 {print ""} {printf "%s", $0} END {print ""}' file

# Words by frequency in a document
tr -cs '[:alpha:]' '\n' < book.txt | tr 'A-Z' 'a-z' | sort | uniq -c | sort -rn | head -20

# Every second line, odd then even
sed -n 'p;n' file; sed -n 'n;p' file

# Rename files by regex: replace spaces with underscores
find . -maxdepth 1 -name '* *' -print0 | xargs -0 -I{} sh -c 'mv -- "$1" "$(printf %s "$1" | tr " " _)"' _ {}

# Check that every line has the same number of fields
awk -F, '{c[NF]++} END {for (n in c) print n " fields: " c[n] " lines"}' data.csv

# Diff two command outputs side by side
diff -y --suppress-common-lines <(sort a.txt) <(sort b.txt)

Scripts#

Summarise an nginx combined-format access log: request count, error rate, top status codes, top paths and top clients, for one file or a glob including gzipped rotations.

#!/usr/bin/env bash
# usage: access-report.sh access.log [access.log.1 access.log.2.gz ...]
set -euo pipefail
(( $# )) || { echo 'usage: access-report.sh LOG...' >&2; exit 2; }

read_logs() { for f in "$@"; do case $f in *.gz) zcat -- "$f" ;; *.zst) zstdcat -- "$f" ;; *) cat -- "$f" ;; esac; done; }

export LC_ALL=C
read_logs "$@" | awk '
  { total++; status[$9]++; split($7, u, "?"); path[u[1]]++; client[$1]++ }
  $9 ~ /^5/ { err++ }
  END {
    printf "requests: %d   5xx: %d (%.2f%%)\n\n", total, err, total ? 100 * err / total : 0
    print "status:"; for (s in status) printf "  %s %d\n", s, status[s] | "sort -k2,2rn"; close("sort -k2,2rn")
    print "\ntop paths:"; for (p in path) printf "  %7d %s\n", path[p], p | "sort -rn | head -15"; close("sort -rn | head -15")
    print "\ntop clients:"; for (c in client) printf "  %7d %s\n", client[c], c | "sort -rn | head -15"
  }'

Bulk search-and-replace across a tree with a dry-run diff, so the change is reviewed before files are rewritten. Rewrites files only when run with --apply.

#!/usr/bin/env bash
# usage: bulk-replace.sh [--apply] 'regex' 'replacement' DIR [glob]
set -euo pipefail
apply=0; [[ ${1:-} == --apply ]] && { apply=1; shift; }
(( $# >= 3 )) || { echo "usage: bulk-replace.sh [--apply] REGEX REPL DIR [GLOB]" >&2; exit 2; }
re=$1 repl=$2 dir=$3 glob=${4:-*}

mapfile -d '' files < <(grep -rlZ --include="$glob" --exclude-dir=.git -E -- "$re" "$dir")
(( ${#files[@]} )) || { echo 'no files match' >&2; exit 0; }
printf '%d file(s) match\n' "${#files[@]}" >&2

for f in "${files[@]}"; do
  if (( apply )); then
    sed -i -E -- "s/$re/$repl/g" "$f"; printf 'rewrote %s\n' "$f"
  else
    diff -u --label "$f" --label "$f (proposed)" "$f" <(sed -E -- "s/$re/$repl/g" "$f") || true
  fi
done
(( apply )) || echo 'dry run; re-run with --apply to write' >&2

Watch a log and alert when the error rate over a sliding one-minute window exceeds a threshold, for a quick check during a deploy when nothing better is wired up.

#!/usr/bin/env bash
# usage: error-rate.sh /var/log/my-app/app.log [threshold-per-minute]
set -euo pipefail
log=${1:?log file required} threshold=${2:-20}

tail -Fn0 -- "$log" | awk -v thr="$threshold" '
  /ERROR/ {
    now = systime(); t[++n] = now
    while (n && t[1] < now - 60) { for (i = 2; i <= n; i++) t[i - 1] = t[i]; delete t[n--] }   # drop entries older than 60 s
    if (n > thr && now - last_alert > 60) { printf "%s ALERT: %d errors in the last minute\n", strftime("%T"), n; fflush(); last_alert = now }
  }'

Troubleshooting#

Symptom	Cause	Fix
`grep: pattern+` matches a literal plus	BRE treats `+`, `?`, `\|`, `()` and `{}` as literals	`grep -E`, or escape as `\+` in GNU BRE
`sed -i '' 's/a/b/' file` fails with `unknown command` on Linux	GNU `-i` takes the suffix attached; `''` became the script	`sed -i 's/a/b/' file`; portable form is `sed -i.bak`
`sed -i 's/a/b/' file` on macOS complains `undefined label` or eats the script	BSD `-i` requires a suffix argument	`sed -i '' 's/a/b/' file` on BSD
`sed: -e expression #1, char N: unknown option to s`	The replacement or pattern contains the delimiter (usually `/` in a path)	Use another delimiter: `s#/old#/new#`
`\d`, `\w` do nothing in `sed` or `grep`	Not part of BRE or ERE; `\w` is a GNU extension, `\d` is not	`[0-9]`, `[[:digit:]]`, or `grep -P`
Replacement inserts `$1` literally	sed uses `\1`, not `$1`; `-E` still needs `\1`	`sed -E 's/(x)/\1/'`
`awk` prints the whole line when I asked for a field	`print $1 $3` concatenates; `$` missing on a variable name	`print $1, $3`; `print $n` not `print n`
`awk -F,` splits quoted CSV fields containing commas	`-F` is a plain separator	`gawk --csv` (5.3+), `FPAT`, or a CSV-aware tool such as `mlr` or Python `csv`
`sort -k2` sorts wrongly	Key runs from field 2 to end of line	`-k2,2`; add `n` for numeric
`uniq -c` misses duplicates	Input not sorted; `uniq` compares adjacent lines only	`sort \| uniq -c`
`sort` output order differs between hosts	Locale collation (`en_AU.UTF-8` versus `C`)	`LC_ALL=C sort` for byte order, and for speed
`xargs: unterminated quote` or files with spaces split	Default whitespace and quote parsing	`find -print0 \| xargs -0`, or `xargs -d '\n'`
Pipeline through `grep` or `awk` shows nothing until it ends	Block buffering when stdout is a pipe	`grep --line-buffered`, `awk '{...; fflush()}'`, `sed -u`, `stdbuf -oL`
`grep -P` says `invalid UTF-8 byte sequence in input`	Binary or Latin-1 bytes in a UTF-8 locale	`LC_ALL=C grep -P` or `grep -aP`
`grep` reports `Binary file matches`	A NUL byte in the file	`grep -a` to print lines, `-I` to skip binary files
`tr 'a-z' 'A-Z'` mangles non-ASCII text	`tr` works on bytes, not multibyte characters	`sed 's/.*/\U&/'` (GNU) or `awk '{print toupper($0)}'`
`awk` numeric comparison treats `10` less than `9`	Values compared as strings (one operand is a string constant)	Force numeric: `$1 + 0 > 9`
`Argument list too long` from `grep` or `sed` with a glob	Too many filenames for one command	`find ... -exec grep ... {} +` or `xargs`