jq · Software Engineering Wiki

Cheatsheet #

Task	Filter
Pretty print	`jq .`
Top-level keys	`jq 'keys'`
Shape of a document	`jq 'to_entries \| map({key, type: (.value\|type)})'`
One field	`jq -r '.name'`
Nested, tolerant of missing	`jq -r '.a?.b? // "none"'`
Every array element	`jq '.items[]'`
Filter	`jq '.items[] \| select(.status=="ready")'`
Map to new objects	`jq '[.items[] \| {name, ip: .addr}]'`
Count	`jq '[.items[]] \| length'`
Sum	`jq '[.items[].bytes] \| add'`
Group	`jq 'group_by(.zone) \| map({zone: .[0].zone, n: length})'`
Sort descending	`jq 'sort_by(-.size)'`
Raw strings, no quotes	`jq -r`
Tab-separated line	`jq -r '[.a,.b] \| @tsv'`
Each object on one line	`jq -c '.[]'`
Read shell variable	`jq --arg v "$x" 'select(.name==$v)'`
Exit non-zero when empty	`jq -e '.items \| length > 0'`

Look at the shape first #

Most jq frustration is guessing at structure. Print the top level, then descend.

jq 'keys' file.json
jq '.items | length' file.json
jq '.items[0]' file.json                      # one representative element
jq -r 'paths(scalars) | join(".")' file.json | sort -u | head -40   # every leaf path
jq 'map(keys) | add | unique' file.json       # union of keys across an array

paths(scalars) is the fastest way to learn an unfamiliar API response: it prints every addressable leaf.

Selecting #

jq '.name'                 # field; null if absent
jq '.a.b.c'                # nested
jq '.["odd-key"]'          # keys that are not identifiers
jq '.items[]'              # stream array elements
jq '.items[2]'             # index
jq '.items[-1]'            # last
jq '.items[1:3]'           # slice
jq '..|.id? // empty'      # every id at any depth
jq '.a? // "default"'      # fall back when null or missing

? suppresses the error when the input is the wrong type; // supplies a value when the left side is null or false. They solve different problems and are often needed together.

Filtering #

jq '.items[] | select(.status == "ready")'
jq '.items[] | select(.count > 10 and .zone != "a")'
jq '.items[] | select(.name | startswith("prod"))'
jq '.items[] | select(.name | test("^api-[0-9]+$"))'      # regex
jq '.items[] | select(.tags | index("urgent"))'           # array contains
jq '.items[] | select(has("error"))'                      # key present
jq '.items | map(select(.active)) | length'               # count matches
jq 'del(.items[] | select(.deleted))'                     # remove matching elements

select passes the whole input through when the condition is true and emits nothing when false, which is why it composes with map and with [] streams alike.

Reshaping #

jq '{name, ip: .addr}'                                # shorthand keeps the key name
jq '{id: .metadata.uid, ns: .metadata.namespace}'
jq '.items | map({name: .metadata.name, images: [.spec.containers[].image]})'
jq 'to_entries | map({k: .key, v: .value})'           # object to array of pairs
jq 'from_entries'                                     # and back
jq 'with_entries(.value |= ascii_downcase)'           # transform every value
jq '.metadata.labels |= (. + {env: "prod"})'          # merge into a nested object
jq 'del(.metadata.managedFields)'                     # drop noise
jq '.a as $x | .b | {x: $x, y: .}'                    # bind a value for later
jq -s 'add'                                           # slurp several documents into one array
jq '.[] | flatten'                                    # collapse nested arrays

|= updates in place with a filter, = assigns a literal, and += adds. Assignment paths must exist unless you build them explicitly.

Aggregating #

jq '[.items[].bytes] | add'
jq '[.items[].bytes] | add / length'                  # mean
jq '[.items[].latency] | sort | .[length/2|floor]'    # median
jq 'group_by(.zone) | map({zone: .[0].zone, count: length, total: (map(.bytes)|add)})'
jq 'map(.status) | group_by(.) | map({status: .[0], n: length})'
jq 'max_by(.age)' ; jq 'min_by(.age)'
jq 'unique_by(.host)'
jq 'sort_by(.name) | reverse'
jq 'reduce .items[] as $i (0; . + $i.count)'          # explicit fold

group_by requires sorted input and does it for you; on large arrays sort once and reuse rather than grouping repeatedly.

Output for the shell #

jq -r '.name'                       # raw: no quotes, no escapes
jq -c '.items[]'                    # compact: one JSON object per line
jq -r '.items[] | [.name, .ip] | @tsv'
jq -r '.items[] | "\(.name)=\(.ip)"'          # string interpolation
jq -r '@base64d'                              # decode base64 (and @base64 to encode)
jq -r '.cmd | @sh'                            # shell-quote a value before eval
jq -r 'to_entries[] | "export \(.key)=\(.value|@sh)"'
jq --tab .                                    # tabs instead of spaces
jq -r '.[] | @csv'

@tsv with -r is the joint between JSON and awk, cut or while read. @sh is the only safe way to interpolate untrusted JSON into a shell command.

Arguments and exit codes #

jq --arg name "$NAME" '.items[] | select(.name == $name)'
jq --argjson limit 5 '.items[] | select(.count > $limit)'
jq --slurpfile extra other.json '. + $extra[0]'
jq --rawfile body payload.txt '{body: $body}'
jq -e '.items | length > 0' >/dev/null || echo 'nothing matched'
jq -n '{ts: now | todate, host: env.HOSTNAME}'        # build JSON from nothing

--arg always produces a string; --argjson parses its value, which is how you pass numbers and booleans. -e sets exit status 1 when the output is false or null, making jq usable directly in if.

Large inputs #

jq -c '.[]' huge.json | while IFS= read -r line; do process "$line"; done
jq --stream 'select(length == 2)' huge.json | head            # events, not a parse tree
jq -n --stream 'fromstream(1|truncate_stream(inputs))' huge.json   # element at a time
curl -sN https://example.com/stream | jq -c --unbuffered '.event'

Normal jq builds the whole document in memory. --stream emits [path, value] events as it parses, which is the difference between working and being OOM killed on a multi-gigabyte file. --unbuffered matters for live streams.

Oneliners #

# Kubernetes: pods with restarts
kubectl get pods -A -o json | jq -r '.items[] | select(any(.status.containerStatuses[]?; .restartCount > 0)) | "\(.metadata.namespace)/\(.metadata.name) \(.status.containerStatuses[0].restartCount)"'

# Terraform: resources of one type from state
terraform show -json | jq -r '.values.root_module.resources[] | select(.type=="aws_instance") | .values.id'

# AWS: instances with name tags
aws ec2 describe-instances | jq -r '.Reservations[].Instances[] | [.InstanceId, .State.Name, (.Tags[]?|select(.Key=="Name")|.Value)] | @tsv'

# GitHub API: open PRs by author
curl -s "https://api.github.com/repos/o/r/pulls?state=open" | jq -r '.[] | [.number, .user.login, .title] | @tsv'

# Docker: container to image mapping
docker inspect $(docker ps -q) | jq -r '.[] | [.Name, .Config.Image] | @tsv'

# Flatten nested JSON to dotted paths
jq -r 'paths(scalars) as $p | "\($p | join(".")) = \(getpath($p))"' file.json

# Diff two JSON documents by key set
diff <(jq -S 'paths(scalars)|join(".")' a.json) <(jq -S 'paths(scalars)|join(".")' b.json)

# Merge two documents, right wins
jq -s '.[0] * .[1]' base.json override.json

# Convert JSON lines to an array
jq -s '.' events.jsonl

# Top 10 by a field
jq -r 'sort_by(-.bytes) | .[:10][] | [.name, .bytes] | @tsv' file.json

# Validate: exit non-zero if any element lacks a field
jq -e 'all(.items[]; has("id"))' file.json >/dev/null

# Strip nulls recursively
jq 'walk(if type == "object" then with_entries(select(.value != null)) else . end)' file.json

# Turn an env-style file into JSON
jq -Rn '[inputs | split("=") | {(.[0]): .[1]}] | add' < .env