jq
Select, filter, reshape and aggregate JSON with jq, pass shell values in safely and emit output that other shell tools can consume.
On this page
Cheatsheet#
| Task | Command |
|---|---|
| Pretty print | jq . file.json |
| Top-level keys | jq 'keys' |
| Key and value type of an object | jq 'to_entries | map({key, type: (.value | type)})' |
| One field as plain text | jq -r '.name' |
| Nested, with a fallback | jq -r '.a.b // "none"' |
| Every array element | jq '.items[]' |
| Filter elements | jq '.items[] | select(.status == "ready")' |
| Project to new objects | jq '[.items[] | {name, ip: .addr}]' |
| Count | jq '.items | length' |
| Sum | jq '[.items[].bytes] | add' |
| Group and count | jq 'group_by(.zone) | map({zone: .[0].zone, n: length})' |
| Sort descending | jq 'sort_by(-.size)' |
| Tab-separated rows | jq -r '.[] | [.a, .b] | @tsv' |
| One object per line | jq -c '.[]' |
| Use a shell variable | jq --arg v "$x" '.[] | select(.name == $v)' |
| Exit non-zero when false or null | jq -e '.items | length > 0' |
| Edit a field | jq '.spec.replicas = 3' |
Verified against jq 1.8.1. Features added in 1.7 or 1.8 are marked; check jq --version, since many distributions still ship 1.6. Reference: the jq manual.
How a jq program runs#
A jq program is a filter: it takes one JSON value as input and produces zero, one or many outputs. | feeds each output of the left side into the right side. , produces the outputs of both sides. .items[] produces one output per element, and everything to its right runs once per element.
echo '{"items":[{"n":1},{"n":2}]}' | jq '.items[] | .n'1
2Wrapping a stream in [...] collects it into an array. That is why .items[] | select(...) prints separate values while [.items[] | select(...)] or .items | map(select(...)) prints one array. When a filter produces nothing (empty, a select that fails), later stages never run.
With several input documents (JSON lines, or concatenated objects), jq runs the program once per document. -s (slurp) reads them all into one array first; -n with inputs reads them lazily.
Look at the shape first#
Most jq time is lost guessing at structure. Print the top level, then descend.
jq 'keys' file.json
jq '.items | length' file.json
jq '.items[0]' file.json # one representative element
jq -r 'paths(type | . != "object" and . != "array") | join(".")' file.json | sort -u | head -40 # every leaf path
jq 'map(keys) | add | unique' file.json # union of keys across an array of objectsThe leaf-path filter prints every addressable value in an unfamiliar document, for example items.0.metadata.name. The common shortcut paths(scalars) skips leaves whose value is null or false, because paths(f) keeps a path only when f returns a truthy value.
Selecting#
jq '.name' # field; null if absent
jq '.a.b.c' # nested; null if any level is absent or null
jq '.["odd-key"]' # keys that are not identifiers
jq '.items[]' # stream the elements
jq '.items[2]' # index; .items[-1] is the last
jq '.items[1:3]' # slice: elements 1 and 2
jq '.. | .id? // empty' # every "id" at any depth
jq '.a // "default"' # fallback when .a is null, false or missing
jq '.items[]?' # no error if .items is not iterable? suppresses the error when the input is the wrong type (for example indexing a string). // supplies a value when the left side produces null, false or nothing. They solve different problems and often appear together.
Filtering#
jq '.items[] | select(.status == "ready")'
jq '.items[] | select(.count > 10 and .zone != "a")'
jq '.items[] | select(.name | startswith("prod"))'
jq '.items[] | select(.name | test("^api-[0-9]+$"))' # regex (Oniguruma)
jq '.items[] | select(any(.tags[]; . == "urgent"))' # array contains a value
jq '.items[] | select(.zone | IN("a", "b"))' # value is one of several
jq '.items[] | select(has("error"))' # key present, even if null
jq '.items | map(select(.active)) | length' # count matches
jq 'del(.items[] | select(.deleted))' # remove matching elementsselect(cond) passes its input through unchanged when cond is truthy and produces nothing otherwise. Only false and null are falsy: 0, "" and [] are true.
Reshaping and editing#
jq '{name, ip: .addr}' # {name} is short for {name: .name}
jq '.items | map({name: .metadata.name, images: [.spec.containers[].image]})'
jq 'to_entries | map({k: .key, v: .value})' # object to array of {key, value}
jq 'from_entries' # and back
jq 'with_entries(.value |= ascii_downcase)' # transform every value
jq 'pick(.metadata.name, .spec.replicas)' # keep only these paths (1.7+)
jq '.metadata.labels += {env: "prod"}' # merge into a nested object
jq 'del(.metadata.managedFields)' # drop noise
jq '.a as $x | .b | {x: $x, y: .}' # bind a value for later use
jq -s 'add' # several documents into one (arrays concatenate, objects merge)
jq 'flatten' # nested arrays to one level| Operator | Right-hand side is evaluated against | Example |
|---|---|---|
= | The original input . | .b = .a copies .a into .b |
|= | The current value at the path | .count |= . + 1 |
+=, -=, *= … | The original input, then combined with the current value | .count += 1 |
Assignment creates missing paths: {} | .a.b = 1 gives {"a":{"b":1}}. jq never edits a file in place; write to a temporary file and move it (jq ... f > f.tmp && mv f.tmp f), or use sponge from moreutils. jq . f > f truncates f before jq reads it.
Aggregating#
jq '[.items[].bytes] | add' # sum; add of an empty array is null
jq '[.items[].bytes] | add / length' # mean
jq '[.items[].latency] | sort | .[length / 2 | floor]' # median (upper for even counts)
jq 'group_by(.zone) | map({zone: .[0].zone, count: length, total: (map(.bytes) | add)})'
jq 'map(.status) | group_by(.) | map({status: .[0], n: length})'
jq 'max_by(.age)'
jq 'unique_by(.host)'
jq 'sort_by(.name) | reverse'
jq 'reduce .items[] as $i (0; . + $i.count)' # explicit fold
jq '.sum = add(.xs[])' # add over a generator (1.8+)group_by, unique_by and sort_by sort their input, so they cost O(n log n) and return groups ordered by the key, not by first appearance.
Output for the shell#
jq -r '.name' # raw: no quotes, no JSON escapes
jq -c '.items[]' # compact: one JSON value per line
jq -r '.items[] | [.name, .ip] | @tsv' # tab-separated, tabs and newlines escaped
jq -r '.items[] | "\(.name)=\(.ip)"' # string interpolation
jq -r '.[] | @csv' # CSV with quoting
jq -r '.data | @base64d' # decode base64; @base64 encodes
jq -r '.cmd | @sh' # quote for safe use in a shell command
jq -r 'to_entries[] | "export \(.key)=\(.value | @sh)"'
jq --raw-output0 '.files[]' | xargs -0 rm -- # NUL-separated for names with newlines (1.7+); deletes files
jq --tab . # indent with tabs; --indent n for n spaces@tsv with -r is the bridge to awk, cut and while IFS=$'\t' read -r. @sh is the safe way to interpolate untrusted JSON into a shell command line.
Arguments and exit codes#
jq --arg name "$NAME" '.items[] | select(.name == $name)' # always a string
jq --argjson limit 5 '.items[] | select(.count > $limit)' # parsed as JSON: numbers, booleans, objects
jq --slurpfile extra other.json '. + $extra[0]' # file contents as an array
jq --rawfile body payload.txt '{body: $body}' # file contents as one string
jq -n --args '$ARGS.positional' a b c # remaining arguments as an array
jq -n '{ts: (now | todate), user: $ENV.USER}' # build JSON from nothingNever build a filter by string concatenation (jq ".name == \"$x\""): a quote in $x breaks or changes the program. Pass values with --arg or --argjson.
| Exit status | Meaning |
|---|---|
| 0 | Ran successfully (with -e: last output was neither false nor null) |
| 1 | With -e: last output was false or null |
| 2 | Usage problem or system error, such as a missing file |
| 3 | Compile error in the jq program |
| 4 | With -e: no output was produced |
| 5 | Runtime error, such as indexing a string, or halt_error |
Large inputs#
jq -c '.[]' huge.json | while IFS= read -r line; do process "$line"; done
jq -c --stream 'select(length == 2)' huge.json | head # [path, leaf] events
jq -cn --stream 'fromstream(1 | truncate_stream(inputs))' huge.json # top-level array, one element at a time
curl -sN https://api.example.com/stream | jq -c --unbuffered '.event'Normal jq parses the whole document into memory before running the program, and memory use is several times the file size. --stream emits [path, value] events while parsing, so a multi-gigabyte top-level array can be processed one element at a time. --unbuffered flushes after each output, which matters when the consumer is a live pipe.
Each streaming event is [path, leaf] for a scalar and [path] (length 1) when a container closes; select(length == 2) keeps only the leaves. truncate_stream(depth) drops the first depth path components, and fromstream reassembles the events back into values, which is how the array example above yields whole elements without ever holding the array. --seq reads and writes RFC 7464 JSON text sequences (each text prefixed with an RS byte), which lets a consumer resynchronise after a truncated record; --stream-errors turns parse errors into stream events instead of aborting.
reduce and foreach#
reduce folds a stream into one value: reduce GEN as $x (INIT; UPDATE) runs UPDATE once per output of GEN, with . bound to the accumulator and $x to the current item, and emits the final accumulator. foreach has the same shape but emits the accumulator after every step, and an optional third argument projects each emitted state.
jq 'reduce .items[] as $i ({}; .[$i.zone] += $i.bytes)' # bytes per zone as an object, no sort
jq 'reduce .items[] as $i ({}; .[$i.host] = $i)' # index an array by a field (last wins)
jq '[foreach .items[] as $i (0; . + $i.bytes)]' # running total
jq '[foreach .items[] as $i (0; . + 1; select(. % 100 == 0) | $i)]' # every 100th element
jq -n 'reduce inputs as $l ({}; .[$l.level] += 1)' # count JSON-lines records by a field without slurping
jq -n '[limit(3; inputs)]' # first three documents only
jq -n 'first(inputs | select(.id == 42))' # stop reading at the first match
jq 'until(. > 1000; . * 2)' # loop until a condition holds
jq '[range(0; 10; 2)]' # 0 2 4 6 8
jq '[.[] | select(.ok)] | .[0] // error("no healthy backend")' # abort with a message (exit status 5)reduce over inputs with -n is the memory-efficient alternative to -s for JSON lines: it holds only the accumulator. limit and first stop consuming their generator as soon as they have enough outputs, so first(inputs | select(...)) on a large file finishes without reading the rest. skip(n; f) (1.8+) drops the first n outputs.
Paths#
A path is an array of keys and indices, such as ["items", 0, "name"]. Path functions let a program work on a document generically without knowing its shape in advance.
jq -c 'paths' file.json # every path, containers included
jq -c 'paths(scalars)' file.json # paths whose value is a scalar (skips null and false leaves)
jq -c '[paths(type == "number")]' file.json # paths to numbers only
jq 'getpath(["a", "b"])' # like .a.b, but the path is data
jq 'setpath(["a", "b"]; 1)' # like .a.b = 1
jq 'delpaths([["a"], ["items", 0]])' # delete several paths at once
jq '[paths(scalars) | select(.[-1] | tostring | test("password"; "i"))] as $ps | reduce $ps[] as $p (.; setpath($p; "REDACTED"))' # redact by key name at any depth
jq 'path(.items[0].name)' # ["items",0,"name"]: the path an expression addresses
jq '[paths] | map(join(".")) ' file.json # fails on numeric indices; use map(tostring) first
jq -r '[paths | map(tostring) | join(".")] | .[]' file.json
jq 'to_entries[] | select(.value | type == "object") | .key' # keys whose value is an object
jq 'tostream' # the same [path, leaf] events that --stream producesleaf_paths is the older name for paths(scalars) and behaves the same, including skipping null and false leaves. del(f), to_entries, pick and |= are all defined on top of path(f), so any expression that is valid inside del(...) is one jq can turn into paths; an expression that computes a new value (del(.a + .b)) is not and fails with Invalid path expression.
Strings, regex and dates#
jq -r '.name | ascii_downcase'
jq -r '.name | ltrimstr("prod-") | rtrimstr("-v2")' # strip a prefix or suffix if present
jq -r '.line | trim' # whitespace both ends (1.7.1+); ltrim, rtrim
jq -r '.tags | join(",")' # join converts numbers and booleans, errors on objects
jq '.csv | split(",")' # plain-string split
jq '.path | split("/"; null)' # regex split: second argument is regex flags
jq '.msg | sub("^ERROR: "; "")' # first match; gsub replaces all
jq '.msg | gsub("(?<n>[0-9]+)"; "<\(.n)>")' # named captures usable in the replacement
jq '.line | capture("(?<ip>[0-9.]+) - (?<user>\\S+)")' # object of named groups, null if no match
jq '[.text | scan("[a-z]+@[a-z.]+")]' # every match as a string
jq '.text | test("error"; "i")' # flags: i ignore case, x extended, g global, n ignore empty
jq '.v | tonumber' # "5" to 5; errors on non-numeric strings
jq '.n | tostring'
jq '.n | tojson', jq '.s | fromjson' # encode a value as a JSON string and back
jq -r '.q | @uri' # percent-encode; @urid decodes (1.8+)
jq -r '"\(.name | ascii_upcase): \(.count)"'
jq '.ts | fromdate' # "2026-09-24T10:00:00Z" to epoch seconds
jq '.epoch | todate' # epoch seconds to ISO 8601
jq '.when | strptime("%d/%b/%Y:%H:%M:%S %z") | mktime' # Apache log timestamp to epoch
jq -r '.epoch | strftime("%Y-%m-%d")'
jq 'now - .epoch | . / 86400 | floor' # age in whole days
jq -r '.epoch | localtime | strftime("%H:%M %Z")' # local zone; gmtime for UTCRegex functions use Oniguruma with Perl-style syntax. Backslashes must be doubled inside a jq string ("\\d+"), and the pattern is a string so it can come from --arg. fromdate accepts only the %Y-%m-%dT%H:%M:%SZ form; anything with fractional seconds or an offset needs strptime with an explicit format, and %z parsing depends on the platform’s libc.
Errors, try and defaults#
jq '.items[] | try .config.port catch "unset"' # replace a runtime error with a value
jq '.items[] | (.n | tonumber)? // 0' # ? is try without catch; then default
jq 'try error("boom") catch .' # the caught value is the error message
jq 'try error({code: 3}) catch .code' # error() accepts any value
jq -e '.ok' && echo yes # exit 1 when .ok is false or null, 4 when there is no output
jq 'if .ok then . else halt_error(2) end' # print the input to stderr and exit 2
jq 'if type != "array" then error("expected array, got \(type)") else . end'
jq '.items[] | debug | .name' # print each value to stderr as ["DEBUG:", value], pass it through
jq '.items[] | debug("checking \(.name)") | .name' # message form (1.7+)
jq 'input_filename, $__loc__' file.json # file being read; {file, line} of the expression itself// and try are not the same thing. // reacts to null, false and empty output; try reacts to errors. .a.b // 0 handles a missing key, but (.a | tonumber) // 0 does not catch the error tonumber raises on "abc"; write (.a | tonumber)? // 0 for both. Errors inside try also stop the generator, so try (.items[] | f) yields nothing after the first failing element; put the try around f instead.
Arguments, the environment and modules#
--arg always binds a string, so --arg n 5 gives "5" and .count == $n is false against a number. --argjson parses its value, which is how numbers, booleans, null, arrays and objects get in. Every named argument is also available as $ARGS.named, and --args or --jsonargs collect the remaining command-line words into $ARGS.positional.
jq --arg env prod --argjson min 3 '.items[] | select(.env == $env and .replicas >= $min)'
jq -n '$ARGS' --arg a 1 --argjson b 2 --args x y # {"positional":["x","y"],"named":{"a":"1","b":2}}
jq -n --arg re "$PATTERN" 'inputs | select(.msg | test($re))' < events.jsonl # regex from the shell, safely
jq -n 'env.HOME, $ENV.PATH' # environment: env is a function, $ENV a variable
jq -n '$ENV | with_entries(select(.key | startswith("MYAPP_")))' # environment subset as an object
MYAPP_PORT=8080 jq -n '{port: ($ENV.MYAPP_PORT | tonumber)}'
jq --slurpfile defaults defaults.json '$defaults[0] * .' # deep-merge a file of defaults under the input
jq --rawfile tmpl template.txt -n '$tmpl | gsub("\\{\\{name\\}\\}"; "my-app")' # text templating with a raw file
jq -L ~/.jq/lib 'import "k8s" as k; k::pod_images' pods.json # module from a search pathNamed functions and modules keep long programs readable. def works inline or in a .jq file; ~/.jq (a file) is loaded automatically, and -L dir adds a directory that import "name" as alias; and include "name"; resolve against.
# ~/.jq: functions available in every interactive invocation
def pod_images: .items[] | .spec.containers[].image;
def bytes_h: if . > 1073741824 then "\(. / 1073741824 * 10 | round / 10)Gi"
elif . > 1048576 then "\(. / 1048576 | round)Mi" else "\(.)B" end;
def kv: to_entries[] | "\(.key)=\(.value)";kubectl get pods -A -o json | jq -r 'pod_images' | sort -u
jq -r '.size | bytes_h' file.json
jq 'def inc(f): f |= . + 1; inc(.a) | inc(.b)' # function with a filter argument; $x makes it a value argumentOneliners#
# Kubernetes: pods with restarts
kubectl get pods -A -o json | jq -r '.items[] | select(any(.status.containerStatuses[]?; .restartCount > 0)) | "\(.metadata.namespace)/\(.metadata.name) \([.status.containerStatuses[].restartCount] | add)"'
# Terraform: IDs of one resource type from state
terraform show -json | jq -r '.values.root_module.resources[] | select(.type == "aws_instance") | .values.id'
# AWS: instance ID, state and Name tag
aws ec2 describe-instances | jq -r '.Reservations[].Instances[] | [.InstanceId, .State.Name, (.Tags // [] | from_entries | .Name // "-")] | @tsv'
# GitHub API: open PRs with author
curl -fsS "https://api.github.com/repos/OWNER/REPO/pulls?state=open" | jq -r '.[] | [.number, .user.login, .title] | @tsv'
# Docker: container to image mapping
docker inspect $(docker ps -q) | jq -r '.[] | [.Name, .Config.Image] | @tsv'
# Flatten to dotted paths with values
jq -r 'paths(type | . != "object" and . != "array") as $p | "\($p | join(".")) = \(getpath($p))"' file.json
# Compare the key structure of two documents
diff <(jq -r 'paths | join(".")' a.json | sort) <(jq -r 'paths | join(".")' b.json | sort)
# Compare two documents ignoring key order
diff <(jq -S . a.json) <(jq -S . b.json)
# Deep merge, right side wins
jq -s '.[0] * .[1]' base.json override.json
# JSON lines to an array
jq -s '.' events.jsonl
# Top 10 by a field
jq -r 'sort_by(-.bytes) | .[:10][] | [.name, .bytes] | @tsv' file.json
# Fail if any element lacks a field
jq -e 'all(.items[]; has("id"))' file.json >/dev/null
# Remove null values recursively
jq 'walk(if type == "object" then with_entries(select(.value != null)) else . end)' file.json
# KEY=value file to a JSON object (skips comments and blank lines, keeps "=" in values)
jq -Rn '[inputs | select(test("^[A-Za-z_][A-Za-z0-9_]*=")) | capture("^(?<key>[^=]+)=(?<value>.*)$")] | from_entries' < .env
# JSON object to KEY=value lines, shell-quoted, ready to source
jq -r 'to_entries[] | "\(.key)=\(.value | tostring | @sh)"' config.json
# Count log levels in a JSON-lines file without loading it all
jq -n 'reduce inputs as $l ({}; .[$l.level // "none"] += 1)' app.jsonl
# Records in the last hour from a JSON-lines log with ISO timestamps
jq -c --arg since "$(date -u -d '1 hour ago' +%FT%TZ)" 'select(.ts >= $since)' app.jsonl
# Histogram of HTTP status codes from structured access logs
jq -r '.status' access.jsonl | sort | uniq -c | sort -rn
# Pretty-print a JSON string embedded in a field
jq '.payload | fromjson' event.json
# Set a nested value only when the path is missing
jq 'if .spec.replicas == null then .spec.replicas = 1 else . end' deploy.json
# Rename a key everywhere in the document
jq 'walk(if type == "object" and has("old") then .new = .old | del(.old) else . end)' file.json
# Kubernetes: images and their pull policy per container
kubectl get deploy -A -o json | jq -r '.items[] | .metadata.name as $d | .spec.template.spec.containers[] | [$d, .image, .imagePullPolicy] | @tsv'
# Kubernetes: decode every key of a Secret
kubectl get secret my-secret -o json | jq -r '.data | to_entries[] | "\(.key)=\(.value | @base64d)"'
# Kubernetes: nodes with allocatable CPU and memory
kubectl get nodes -o json | jq -r '.items[] | [.metadata.name, .status.allocatable.cpu, .status.allocatable.memory] | @tsv'
# Helm: chart versions of every release across namespaces
helm list -A -o json | jq -r '.[] | [.namespace, .name, .chart, .status] | @tsv' | column -t
# Terraform: resources whose plan action is delete or replace
terraform show -json plan.out | jq -r '.resource_changes[] | select(.change.actions | index("delete")) | [.address, (.change.actions | join(","))] | @tsv'
# AWS: security group rules that allow 0.0.0.0/0
aws ec2 describe-security-groups | jq -r '.SecurityGroups[] | .GroupId as $g | .IpPermissions[] | select(any(.IpRanges[]?; .CidrIp == "0.0.0.0/0")) | [$g, (.FromPort // "all" | tostring), .IpProtocol] | @tsv'
# Docker: published ports for every running container
docker inspect $(docker ps -q) | jq -r '.[] | .Name as $n | .NetworkSettings.Ports // {} | to_entries[] | select(.value) | "\($n) \(.key) -> \(.value[0].HostPort)"'
# Turn a CSV with a header row into an array of objects
jq -Rn '(input | split(",")) as $h | [inputs | split(",") | [$h, .] | transpose | map({(.[0]): .[1]}) | add]' data.csv
# Build a JSON payload from shell variables without quoting problems
jq -n --arg name "$NAME" --argjson replicas "$REPLICAS" '{name: $name, spec: {replicas: $replicas}}'
# Validate that a file is JSON, silently
jq -e . file.json >/dev/null 2>&1 && echo valid
# Emit the same value as one compact line per input file, prefixed with the file name
jq -c '{file: input_filename, keys: keys}' *.json
# Sum a field across many files
jq -s 'map(.total) | add' reports/*.json
# Diff two arrays of objects by an identity field: entries only in the second file
jq -n --slurpfile a a.json --slurpfile b b.json '($a[0] | map(.id)) as $ids | $b[0] | map(select(.id | IN($ids[]) | not))'
# Percent-encode a query string value
jq -rn --arg q "$QUERY" '$q | @uri'Scripts#
Watch a JSON health endpoint and exit non-zero when any component is unhealthy, printing the failures as tab-separated rows. Suits a systemd timer or a CI smoke test.
#!/usr/bin/env bash
# usage: health-json.sh https://api.example.com/health
set -euo pipefail
url=${1:?health URL required}
body=$(curl -fsS --max-time 10 "$url") || { echo "fetch failed: $url" >&2; exit 2; }
# Expected shape: {"status":"ok","components":{"db":{"status":"ok"},"cache":{"status":"degraded","error":"..."}}}
if ! jq -e '.components | type == "object"' <<< "$body" >/dev/null; then
echo 'unexpected response shape' >&2
jq -c 'paths(scalars) | join(".")' <<< "$body" | head -20 >&2 || true
exit 2
fi
failing=$(jq -r '.components | to_entries[] | select(.value.status != "ok") | [.key, .value.status, (.value.error // "-")] | @tsv' <<< "$body")
if [[ -n $failing ]]; then
printf 'component\tstatus\terror\n%s\n' "$failing"
exit 1
fi
printf 'all %d components ok\n' "$(jq '.components | length' <<< "$body")"Edit a JSON configuration file in place by path and value, keeping a backup and refusing to write if the result is not valid JSON. Values are parsed as JSON when they look like JSON and treated as strings otherwise.
#!/usr/bin/env bash
# usage: json-set.sh FILE PATH VALUE e.g. json-set.sh app.json .server.port 8080
set -euo pipefail
file=${1:?file} path=${2:?path} value=${3:?value}
[[ -f $file ]] || { echo "no such file: $file" >&2; exit 2; }
[[ $path == .* ]] || { echo "path must start with . (got $path)" >&2; exit 2; }
# Try the value as JSON first (numbers, true/false/null, arrays, objects); fall back to a string.
if json=$(jq -cn --argjson v "$value" '$v' 2>/dev/null); then :; else json=$(jq -n --arg v "$value" '$v'); fi
tmp=$(mktemp "${file}.XXXXXX")
trap 'rm -f -- "$tmp"' EXIT
# The path comes from the caller, so it is the one thing spliced into the program; the value goes in via --argjson.
jq --argjson v "$json" "${path} = \$v" "$file" > "$tmp"
jq -e . "$tmp" >/dev/null # belt and braces: refuse to replace the file with garbage
cp -p -- "$file" "${file}.bak"
mv -f -- "$tmp" "$file"
trap - EXIT
printf '%s: set %s = %s\n' "$file" "$path" "$json"Summarise a JSON-lines log: total records, count per level, top error messages and the time span, in a single pass with reduce so multi-gigabyte files fit in memory.
#!/usr/bin/env bash
# usage: jsonl-summary.sh app.jsonl [level-field] [message-field] [timestamp-field]
set -euo pipefail
f=${1:?log file} lvl=${2:-level} msg=${3:-msg} ts=${4:-ts}
[[ -r $f ]] || { echo "cannot read $f" >&2; exit 2; }
jq -n --arg lvl "$lvl" --arg msg "$msg" --arg ts "$ts" '
reduce inputs as $r (
{n: 0, levels: {}, errors: {}, first: null, last: null};
.n += 1
| .levels[$r[$lvl] // "none"] += 1
| if ($r[$lvl] // "") | ascii_downcase | IN("error", "fatal") then .errors[$r[$msg] // "?"] += 1 else . end
| .first = ([.first, $r[$ts]] | map(select(. != null)) | min)
| .last = ([.last, $r[$ts]] | map(select(. != null)) | max)
)
| "records: \(.n)",
"span: \(.first) .. \(.last)",
"levels: \(.levels | to_entries | sort_by(-.value) | map("\(.key)=\(.value)") | join(" "))",
"top errors:",
(.errors | to_entries | sort_by(-.value) | .[:10][] | " \(.value)\t\(.key)")
' -r "$f"Troubleshooting#
| Symptom | Cause | Fix |
|---|---|---|
Cannot index array with string "name" | Input is an array; the filter expects an object | .[].name or map(.name) |
Cannot iterate over null | Path is missing on some elements | .items[]? or (.items // [])[] |
Cannot index string with string "x" | Value is JSON encoded as a string | .field | fromjson | .x |
| Output has quotes in a shell variable | Missing -r | jq -r |
parse error: Invalid numeric literal | Input is not JSON (HTML error page, YAML, log prefix) | Inspect with head -c 200; curl -f to fail on HTTP errors |
jq: error: x/0 is not defined or syntax error | Shell expanded $x or quotes inside the filter | Single-quote the program; pass values with --arg |
| Numbers compare as unequal | --arg gave a string | --argjson, or tonumber |
| Each result printed separately, not as one array | Filter is a stream | Wrap in [...] or use map |
| Large integer IDs change | Pre-1.7 jq converts all numbers to IEEE 754 doubles | jq 1.7+ keeps the literal when a number passes through unchanged; arithmetic still converts |
Empty file after jq ... f > f | Shell truncated f before jq read it | Write to a temporary file, then mv |
| Out of memory on a large file | Whole document parsed into memory | --stream (see large inputs) |
Invalid path expression with result ... | Left side of |=, del or path() computes a value instead of addressing one | Rewrite so the left side is a pure path: del(.a[] | select(.x)) not del(.a | map(select(.x))) |
--arg regex never matches | Backslashes were consumed by the shell, or the pattern needs flags | Single-quote the shell argument; pass flags as test($re; "i") |
Cannot use null (null) as object key | Grouping or indexing on a field some records lack | .[$r.level // "none"], or select(.level != null) first |
Date "..." does not match format "%Y-%m-%dT%H:%M:%SZ" | fromdate accepts only that exact form | Use strptime with the real format, or sub("\\.[0-9]+"; "") to drop fractional seconds |
| Program reads stdin and hangs | No -n and no file given, so jq waits for input | Add -n when the program does not use . |
jq: error: Could not open file for a module | -L not set or module not on the search path | jq -L dir 'import "name" as n; ...'; the file is dir/name.jq |
try swallowed every element after the first error | The error stopped the whole generator inside try | Move try inside: .items[] | try f |
@tsv output shows \t or \n literally in a value | @tsv escapes control characters by design | Expected; use @text if the consumer handles raw values |
Interactive jq shows colours but a script gets escape codes | -C was forced, or JQ_COLORS set in a wrapper | Use -M in scripts; colour is off by default when stdout is not a terminal |
gojq is a Go implementation with the same language and clearer error messages. For YAML, the Go yq (mikefarah) uses a jq-like but not identical syntax; the Python yq converts YAML to JSON and runs real jq.
Further reading#
- jq manual: every builtin, operator and command-line flag, with the version selector for 1.6 and 1.7 differences.
- jq wiki: jq language description: the evaluation model behind generators, paths and
reduce. - jq wiki: FAQ: number handling, streaming and portability answers.
- RFC 7464: JSON Text Sequences: the format behind
--seq. - Oniguruma regular expression syntax: what
test,match,captureandsubaccept.