# Elasticsearch

> Check cluster health, fix unassigned shards, write mappings, queries and aggregations, and tune slow search or indexing in Elasticsearch.

Canonical: https://www.wiki.jodisand.me/elasticsearch/
Reviewed: 2026-09-24
Related: [jq](https://www.wiki.jodisand.me/jq/index.md), [HTTP and curl](https://www.wiki.jodisand.me/http/index.md), [OpenTelemetry](https://www.wiki.jodisand.me/opentelemetry/index.md), [Kubernetes](https://www.wiki.jodisand.me/kubernetes/index.md)


## Cheatsheet

Requests are shown in Kibana Dev Tools form. The `curl` equivalents below use the `es` helper from [Connecting with curl](#connecting-with-curl).

| Task | Request |
| --- | --- |
| Cluster health | `GET _cluster/health` |
| Why a shard is unassigned | `GET _cluster/allocation/explain` |
| Indices by size | `GET _cat/indices?v&s=store.size:desc` |
| Shard placement and state | `GET _cat/shards?v&s=state&h=index,shard,prirep,state,node,unassigned.reason` |
| Node heap, CPU, disk | `GET _cat/nodes?v&h=name,node.role,heap.percent,cpu,load_1m,disk.used_percent` |
| Disk per node | `GET _cat/allocation?v` |
| Pending cluster tasks | `GET _cat/pending_tasks?v` |
| Running searches | `GET _tasks?actions=*search*&detailed` |
| Mapping of an index | `GET my-index/_mapping` |
| Count matching documents | `GET my-index/_count` with a `query` body |
| Why a document scored as it did | `GET my-index/_explain/<id>` with a `query` body |
| Validate a query | `GET my-index/_validate/query?explain` |
| Show analyser tokens | `GET my-index/_analyze` |
| Make recent writes searchable now | `POST my-index/_refresh` |
| Effective cluster settings | `GET _cluster/settings?include_defaults&flat_settings` |

## Connecting with curl

Since 8.0, a new self-managed cluster enables security by default: HTTPS on port 9200, authentication required, and a generated CA at `config/certs/http_ca.crt`. Plain `curl localhost:9200` returns an empty reply or a certificate error.

```sh
export ES_URL=https://localhost:9200
export ES_API_KEY='<encoded-api-key>'      # from Kibana or POST _security/api_key
es() { curl -sS --cacert http_ca.crt -H "Authorization: ApiKey $ES_API_KEY" -H 'Content-Type: application/json' "$@"; }

es "$ES_URL/"                               # cluster name and version
es "$ES_URL/_cluster/health?pretty"
```

Use an API key scoped to the task rather than the `elastic` superuser password. See [HTTP](https://www.wiki.jodisand.me/http/) for curl flags and [jq](https://www.wiki.jodisand.me/jq/) for the output.

## Health and shards

Green: every primary and replica shard is assigned. Yellow: all primaries are assigned but some replicas are not; searches work but a node loss may lose data. Red: at least one primary is unassigned, so part of the data cannot be searched or written. A single-node cluster with `number_of_replicas: 1` is permanently yellow, because a replica never goes on the same node as its primary.

```sh
es "$ES_URL/_cluster/health?pretty"
es "$ES_URL/_cat/indices?v&health=red&s=index"
es "$ES_URL/_cat/shards?v&s=state&h=index,shard,prirep,state,node,unassigned.reason" | grep -v STARTED
es "$ES_URL/_cluster/allocation/explain?pretty"         # no body: explains the first unassigned shard
es "$ES_URL/_cluster/allocation/explain?pretty" -d '{"index": "logs-2026.09.01", "shard": 0, "primary": true}'
```

The allocation explain API states the reason directly, including which allocation decider said no (disk watermark, allocation filter, same-shard rule, attribute awareness) on each node. Read `allocate_explanation` and the `deciders` list before changing anything.

`unassigned.reason` records why the shard became unassigned; the explain API says why it is still unassigned.

| `unassigned.reason` | Meaning |
| --- | --- |
| `INDEX_CREATED` | Normal for a moment after index creation |
| `NODE_LEFT` / `NODE_RESTARTING` | The node holding it left; replicas are promoted or recovery waits for the node |
| `ALLOCATION_FAILED` | Allocation failed `index.allocation.max_retries` times (default 5); fix the cause, then `POST _cluster/reroute?retry_failed=true` |
| `CLUSTER_RECOVERED` | Full cluster restart; normal while recovering |
| `REPLICA_ADDED` | `number_of_replicas` was raised |
| `PRIMARY_FAILED` | The primary failed while this replica was initialising |

If explain reports no valid shard copy for a primary, the data on disk is gone from every node that held it. Recovery is a snapshot restore. The reroute commands `allocate_stale_primary` and `allocate_empty_primary` accept data loss explicitly and should be a last resort.

### Disk watermarks

| Setting | Default | Effect |
| --- | --- | --- |
| `cluster.routing.allocation.disk.watermark.low` | 85% (max headroom 200GB) | No new shards allocated to the node |
| `cluster.routing.allocation.disk.watermark.high` | 90% (max headroom 150GB) | Shards relocated away from the node |
| `cluster.routing.allocation.disk.watermark.flood_stage` | 95% (max headroom 100GB) | `index.blocks.read_only_allow_delete` set on every index with a shard on the node |

Max headroom applies when the percentage is left at its default, so on large disks the thresholds trigger at a fixed free-space amount instead. At flood stage, writes fail with `cluster_block_exception ... read-only / allow delete` while health can still be green. The block is removed automatically once disk use falls below the high watermark; free space (delete old indices, add nodes) rather than raising the threshold. See the [allocation settings reference](https://www.elastic.co/docs/reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings).

```sh
es "$ES_URL/_cat/allocation?v&h=node,shards,disk.percent,disk.used,disk.avail"

# Temporary relief while you free space; remove afterwards by setting it to null
es -X PUT "$ES_URL/_cluster/settings" -d '{"persistent": {"cluster.routing.allocation.disk.watermark.flood_stage": "97%"}}'
```

## Indexing and mapping

A mapping decides how each field is stored and analysed, and therefore how it can be searched. `text` fields are analysed into terms for full-text search. `keyword` fields are stored whole for exact matches, sorting and aggregations. Most "my `term` query returns nothing" problems are a `term` query against a `text` field, where the stored terms are lowercased tokens.

```json
PUT my-index
{
  "settings": { "number_of_shards": 1, "number_of_replicas": 1, "refresh_interval": "5s" },
  "mappings": {
    "dynamic": "strict",
    "properties": {
      "service":     { "type": "keyword" },
      "message":     { "type": "text" },
      "level":       { "type": "keyword" },
      "duration_ms": { "type": "float" },
      "@timestamp":  { "type": "date" },
      "host": {
        "properties": { "name": { "type": "keyword" }, "ip": { "type": "ip" } }
      }
    }
  }
}
```

You cannot change the type of an existing field. Changing it means a new index with the new mapping and a `_reindex` into it. `dynamic: strict` rejects documents with unmapped fields instead of guessing a type and fixing it for the life of the index. Dynamic mapping maps a new string field as `text` with a `.keyword` sub-field, which doubles the storage for fields that only need one of the two.

```sh
es -X POST "$ES_URL/my-index/_doc" -d '{"service":"api","message":"timeout"}'           # auto ID
es -X PUT  "$ES_URL/my-index/_doc/1" -d '{"service":"api"}'                            # create or replace ID 1
es -X POST "$ES_URL/my-index/_update/1" -d '{"doc":{"level":"error"}}'                 # partial update
es -X POST "$ES_URL/_bulk" -H 'Content-Type: application/x-ndjson' --data-binary @bulk.ndjson
```

```json
{"index":{"_index":"my-index","_id":"1"}}
{"service":"api","message":"timeout","@timestamp":"2026-09-01T00:00:00Z"}
```

Use `_bulk` for any volume: one request per document pays a round trip and indexing overhead each time. The body is newline-delimited JSON and must end with a newline, and `--data-binary` preserves the newlines that `-d` would strip. A bulk response returns HTTP 200 even when items fail; check `"errors": true` and each item's `error`.

```sh
es -X POST "$ES_URL/_bulk" -H 'Content-Type: application/x-ndjson' --data-binary @bulk.ndjson | jq '{errors, failed: [.items[] | .[] | select(.error) | .error.reason] | .[:5]}'
```

### Mapping parameters

A few parameters decide most of the storage and query behaviour of a field.

```json
PUT my-index
{
  "mappings": {
    "properties": {
      "title":    { "type": "text", "fields": { "raw": { "type": "keyword", "ignore_above": 256 } } },
      "trace_id": { "type": "keyword", "doc_values": false },
      "payload":  { "type": "object", "enabled": false },
      "tags":     { "type": "keyword", "null_value": "none" },
      "price":    { "type": "scaled_float", "scaling_factor": 100 },
      "labels":   { "type": "flattened" },
      "items":    { "type": "nested", "properties": { "sku": { "type": "keyword" }, "qty": { "type": "integer" } } }
    },
    "runtime": {
      "duration_s": { "type": "double", "script": { "source": "emit(doc['duration_ms'].value / 1000.0)" } }
    }
  }
}
```

| Parameter | Effect |
| --- | --- |
| `fields` (multi-field) | Index the same value twice, typically `text` for search and `keyword` for sort and aggregate; query the sub-field as `title.raw` |
| `ignore_above` | Do not index `keyword` values longer than N characters; stops one huge value blowing the term dictionary |
| `doc_values: false` | Saves disk for fields never sorted or aggregated; you cannot turn it back on without a reindex |
| `enabled: false` | Store the object in `_source` but do not index any of it |
| `index: false` | Stored, aggregatable, not searchable |
| `flattened` | One field for an object with arbitrary keys; every leaf is a keyword, so no field explosion |
| `nested` | Array of objects queried as independent documents; each nested object is a hidden Lucene document, so keep arrays short |
| `runtime` | Computed at query time from `_source` or doc values; slow to query, free to add, useful to test a field before reindexing |

Put shared mappings in an index template so every matching new index gets them. Component templates hold reusable pieces; the index template composes them and sets the priority that decides which template wins when patterns overlap.

```sh
es -X PUT "$ES_URL/_component_template/logs-mappings" -d '{"template": {"mappings": {"dynamic": "strict", "properties": {"@timestamp": {"type": "date"}, "service": {"type": "keyword"}, "message": {"type": "text"}}}}}'
es -X PUT "$ES_URL/_index_template/logs-my-app" -d '{"index_patterns": ["logs-my-app-*"], "data_stream": {}, "priority": 200, "composed_of": ["logs-mappings"], "template": {"settings": {"index.lifecycle.name": "logs"}}}'
es -X POST "$ES_URL/_index_template/_simulate_index/logs-my-app-default?pretty"   # what a new index with this name would get, and which template won
```

`_simulate_index` is the quickest answer to "why did this index get the wrong mapping": it prints the resolved settings and mappings and lists the templates it overlapped with.

## Analysers

An analyser turns text into the terms stored in the inverted index: character filters clean the input, a tokenizer splits it, token filters change each token. The `standard` analyser splits on Unicode word boundaries and lowercases, so `Connection-Timeout` is stored as `connection` and `timeout`, and a `match` query goes through the same analyser before looking up terms. `keyword` fields skip analysis entirely; a `normalizer` (lowercase, asciifolding) is the only processing they allow.

```json
PUT my-index
{
  "settings": {
    "analysis": {
      "char_filter": { "strip_html": { "type": "html_strip" } },
      "filter": {
        "english_stop": { "type": "stop", "stopwords": "_english_" },
        "edge_3_20":    { "type": "edge_ngram", "min_gram": 3, "max_gram": 20 }
      },
      "analyzer": {
        "message_index":  { "type": "custom", "char_filter": ["strip_html"], "tokenizer": "standard", "filter": ["lowercase", "asciifolding", "english_stop"] },
        "prefix_index":   { "type": "custom", "tokenizer": "standard", "filter": ["lowercase", "edge_3_20"] },
        "prefix_search":  { "type": "custom", "tokenizer": "standard", "filter": ["lowercase"] }
      },
      "normalizer": { "lc": { "type": "custom", "filter": ["lowercase", "asciifolding"] } }
    }
  },
  "mappings": {
    "properties": {
      "message":  { "type": "text", "analyzer": "message_index" },
      "hostname": { "type": "text", "analyzer": "prefix_index", "search_analyzer": "prefix_search" },
      "service":  { "type": "keyword", "normalizer": "lc" }
    }
  }
}
```

The `hostname` field shows the usual autocomplete pattern: edge n-grams at index time, plain tokens at search time, so `web-0` matches `web-01.example.com` without generating n-grams of the query too. Analysis settings are static; changing an analyser means closing the index (`POST my-index/_close`) or reindexing, and changing one used by an existing field always means reindexing, because the stored terms were produced by the old chain.

```sh
# Run text through a named analyser, a field's analyser, or an ad-hoc chain; explain shows each stage's output
es "$ES_URL/my-index/_analyze" -d '{"analyzer":"message_index","text":"<p>Connection timed out to Zürich</p>"}' | jq -r '.tokens[].token'
es "$ES_URL/my-index/_analyze" -d '{"field":"hostname","text":"web-01.example.com"}' | jq -r '.tokens[].token'
es "$ES_URL/_analyze" -d '{"tokenizer":"whitespace","filter":["lowercase",{"type":"stop","stopwords":["to"]}],"text":"Connection timed out to Zürich","explain":true}'
```

Built-in language analysers (`english`, `german`, ...) add stemming, so `timeouts` and `timeout` match. That is what you want for prose and what you do not want for identifiers, log levels and hostnames, which belong in `keyword`.

## Searching

```json
GET my-index/_search
{
  "size": 20,
  "query": {
    "bool": {
      "must":     [{ "match": { "message": "connection timeout" } }],
      "filter":   [
        { "term":  { "service": "api" } },
        { "range": { "@timestamp": { "gte": "now-1h" } } }
      ],
      "must_not": [{ "term": { "level": "debug" } }],
      "should":   [{ "match_phrase": { "message": "connection refused" } }]
    }
  },
  "sort": [{ "@timestamp": "desc" }],
  "_source": ["service", "message", "@timestamp"]
}
```

| Clause | Effect |
| --- | --- |
| `must` | Must match; contributes to the score |
| `filter` | Must match; no scoring, results cacheable. Use for structured conditions |
| `should` | Raises the score when it matches. Required to match only if there is no `must` or `filter`, or `minimum_should_match` is set |
| `must_not` | Excludes; no scoring |
| `match` | Analyses the query text, then matches terms |
| `term` | Exact, unanalysed value; against a `text` field it usually matches nothing |
| `match_phrase` | Terms adjacent and in order |
| `wildcard` / `regexp` | Expensive; a leading wildcard scans every term in the field |

Put dates, IDs and enumerations in `filter`, not `must`. Filters skip scoring and their results can be cached per segment.

By default `hits.total` counts accurately only up to 10,000 (`"relation": "gte"` beyond that). `"track_total_hits": true` counts all matches at extra cost. `from + size` is limited to 10,000 results (`index.max_result_window`). To page deeper or export, use `search_after` with a point in time (PIT); the scroll API is no longer recommended for deep paging.

```json
POST my-index/_pit?keep_alive=2m
```

```json
GET _search
{
  "size": 1000,
  "pit": { "id": "<pit-id>", "keep_alive": "2m" },
  "sort": [{ "@timestamp": "asc" }],
  "search_after": [1788220800000]
}
```

Pass the `sort` values of the last hit as `search_after` in the next request, and delete the PIT with `DELETE _pit` when done.

### Query DSL examples

```json
GET my-index/_search
{
  "query": {
    "multi_match": {
      "query": "connection timeout",
      "fields": ["message^3", "title"],
      "type": "best_fields",
      "operator": "and"
    }
  },
  "highlight": { "fields": { "message": {} } }
}
```

`multi_match` runs one text query across several fields; `^3` boosts `message`. `best_fields` takes the score of the single best field, `most_fields` sums them (for the same text analysed several ways), `cross_fields` treats the fields as one (first name and surname split across two fields). `operator: and` requires every term instead of any.

```json
GET my-index/_search
{
  "query": {
    "bool": {
      "filter": [
        { "terms":  { "service": ["api", "worker"] } },
        { "exists": { "field": "host.ip" } },
        { "prefix": { "host.name": "web-" } },
        { "range":  { "duration_ms": { "gte": 500, "lt": 2000 } } }
      ],
      "should": [
        { "match_phrase": { "message": { "query": "connection refused", "slop": 1 } } },
        { "match": { "message": { "query": "reset", "boost": 0.5 } } }
      ],
      "minimum_should_match": 1
    }
  }
}
```

`terms` is an exact match against any listed value; `exists` finds documents where the field has a non-null value; `prefix` on a `keyword` is cheap where `wildcard` is not; `slop` lets phrase terms be one position apart. `minimum_should_match: 1` turns `should` into "at least one of these", which otherwise is optional once a `filter` is present.

```json
GET my-index/_search
{
  "query": {
    "nested": {
      "path": "items",
      "query": { "bool": { "filter": [{ "term": { "items.sku": "ABC-1" } }, { "range": { "items.qty": { "gte": 10 } } }] } },
      "inner_hits": {}
    }
  }
}
```

A `nested` query is the only way to require two conditions on the same array element; on a plain `object` array the fields are flattened and `sku: ABC-1` in one element combines with `qty: 10` from another. `inner_hits` returns which nested objects matched.

```json
GET my-index/_search
{
  "query": { "query_string": { "query": "service:api AND level:(error OR warn) AND NOT message:\"health check\"", "default_field": "message" } }
}
```

`query_string` parses Lucene syntax and is what Kibana's KQL is compiled from. It throws on a malformed query, so use `simple_query_string` for anything typed by users; it never errors, and drops invalid parts instead. Neither belongs in an API that accepts untrusted input against sensitive fields, because the field name is part of the syntax.

### Aggregations

```json
GET my-index/_search
{
  "size": 0,
  "aggs": {
    "by_service": {
      "terms": { "field": "service", "size": 10 },
      "aggs": {
        "p95":       { "percentiles": { "field": "duration_ms", "percents": [95] } },
        "over_time": { "date_histogram": { "field": "@timestamp", "fixed_interval": "5m" } }
      }
    }
  }
}
```

`"size": 0` skips returning documents when only aggregations are needed. Aggregations and sorting need `keyword`, numeric or date fields (doc values); a `text` field fails unless `fielddata` is enabled, which loads it onto the heap and should be avoided. `terms` counts on a sharded index are approximate. To walk every bucket of a high-cardinality field, use a `composite` aggregation and page with its `after_key`.

```json
GET my-index/_search
{
  "size": 0,
  "query": { "range": { "@timestamp": { "gte": "now-24h" } } },
  "aggs": {
    "hosts":        { "cardinality": { "field": "host.name", "precision_threshold": 3000 } },
    "error_rate":   { "filters": { "filters": { "errors": { "term": { "level": "error" } }, "all": { "match_all": {} } } } },
    "latency":      { "range": { "field": "duration_ms", "ranges": [{ "to": 100 }, { "from": 100, "to": 1000 }, { "from": 1000 }] } },
    "slowest":      { "top_hits": { "size": 3, "sort": [{ "duration_ms": "desc" }], "_source": ["service", "message", "duration_ms"] } },
    "per_service":  {
      "terms": { "field": "service", "size": 50, "order": { "p99": "desc" } },
      "aggs": {
        "p99":  { "percentiles": { "field": "duration_ms", "percents": [99] } },
        "top3": { "bucket_sort": { "size": 3 } }
      }
    },
    "walk": {
      "composite": {
        "size": 1000,
        "sources": [{ "svc": { "terms": { "field": "service" } } }, { "host": { "terms": { "field": "host.name" } } }]
      }
    }
  }
}
```

`cardinality` is a HyperLogLog++ estimate, exact below `precision_threshold` (default 3000, max 40000). `filters` with a named `match_all` bucket gives you a numerator and denominator in one request. `bucket_sort` trims buckets after the parent aggregation has computed them, so it is a presentation step, not a cheaper query. `terms` with `size: 50` fetches `shard_size` (default `size * 1.5 + 10`) candidates per shard, which is where the approximate counts come from; raise `shard_size` if `doc_count_error_upper_bound` in the response is not 0 and the numbers matter. `composite` returns `after_key`; pass it back as `"after": {...}` to get the next page, and it is the only aggregation that pages.

## Reindex, aliases and lifecycle

An alias lets you replace the index behind a name without changing clients. Reindex into a new index, then swap the alias in one atomic `_aliases` call.

```sh
es -X POST "$ES_URL/_reindex?wait_for_completion=false" -d '{"source": {"index": "my-index-v1"}, "dest": {"index": "my-index-v2"}}'
es "$ES_URL/_tasks/<task-id>?pretty"        # progress of the reindex task

es -X POST "$ES_URL/_aliases" -d '{
  "actions": [
    {"remove": {"index": "my-index-v1", "alias": "my-index"}},
    {"add":    {"index": "my-index-v2", "alias": "my-index", "is_write_index": true}}
  ]
}'
```

### Reindex options

`_reindex` is a scroll-and-bulk loop run by the cluster. It copies `_source`, so fields with `_source` disabled or `enabled: false` objects cannot be rebuilt from it, and it does not copy settings, mappings or aliases: create the destination first.

```sh
# Subset of documents, only some fields, into an index that already has the right mapping
es -X POST "$ES_URL/_reindex?wait_for_completion=false&slices=auto" -d '{
  "conflicts": "proceed",
  "source": { "index": "logs-2026.09.*", "query": { "range": { "@timestamp": { "gte": "2026-09-01" } } }, "_source": ["@timestamp", "service", "message"] },
  "dest":   { "index": "logs-archive", "op_type": "create" }
}'

# Transform on the way through: rename a field and drop another
es -X POST "$ES_URL/_reindex" -d '{
  "source": { "index": "my-index-v1" },
  "dest":   { "index": "my-index-v2" },
  "script": { "lang": "painless", "source": "ctx._source.service = ctx._source.remove(\"svc\"); ctx._source.remove(\"debug\")" }
}'

# From another cluster; the remote host must be listed in reindex.remote.whitelist on this cluster
es -X POST "$ES_URL/_reindex" -d '{"source": {"remote": {"host": "https://old.example.com:9200", "username": "reindex"}, "index": "my-index"}, "dest": {"index": "my-index"}}'

es -X POST "$ES_URL/_reindex/<task-id>/_rethrottle?requests_per_second=500"   # slow a running reindex down; -1 removes the limit
es "$ES_URL/_tasks?actions=*reindex&detailed" | jq '.nodes[].tasks[] | {description, status: .status | {total, created, updated, version_conflicts}}'
```

`op_type: create` skips documents that already exist in the destination, which with `conflicts: proceed` makes a reindex resumable. `slices=auto` runs one slice per shard in parallel; do not use it against a remote source. For a live index, reindex once, then a second time with a `range` on a modified-timestamp field to catch what changed while the first ran, and swap the alias between the two.

### ILM policies

Index lifecycle management moves an index through phases (`hot`, `warm`, `cold`, `frozen`, `delete`) based on age since rollover. Rollover is the mechanism that makes it work: the policy watches the write index behind an alias or data stream and, when a threshold is hit, creates the next index and points writes at it. `min_age` of every later phase counts from rollover, not from index creation.

```sh
es -X PUT "$ES_URL/_ilm/policy/logs" -d '{
  "policy": {
    "phases": {
      "hot":    { "actions": { "rollover": { "max_primary_shard_size": "50gb", "max_age": "1d" }, "set_priority": { "priority": 100 } } },
      "warm":   { "min_age": "2d", "actions": { "shrink": { "number_of_shards": 1 }, "forcemerge": { "max_num_segments": 1 }, "allocate": { "number_of_replicas": 1, "require": { "data": "warm" } }, "set_priority": { "priority": 50 } } },
      "cold":   { "min_age": "14d", "actions": { "allocate": { "number_of_replicas": 0, "require": { "data": "cold" } }, "set_priority": { "priority": 0 } } },
      "delete": { "min_age": "30d", "actions": { "delete": {} } }
    }
  }
}'
```

For a data stream the template only needs `index.lifecycle.name`; for a classic alias-based setup it also needs `index.lifecycle.rollover_alias`, the first index must be created with a name ending in `-000001`, and the alias must be added with `is_write_index: true`. Forgetting any of those three puts the index into the ILM `ERROR` step at rollover time.

```sh
es -X PUT "$ES_URL/my-index-000001" -d '{"aliases": {"my-index": {"is_write_index": true}}}'
es -X POST "$ES_URL/my-index/_rollover?dry_run=true" -d '{"conditions": {"max_age": "1d", "max_primary_shard_size": "50gb"}}'   # would it roll now, and why
es -X PUT "$ES_URL/_cluster/settings" -d '{"persistent": {"indices.lifecycle.poll_interval": "1m"}}'                        # default 10m; ILM only checks conditions this often
es "$ES_URL/my-index-*/_ilm/explain?pretty&only_errors=true"
es -X POST "$ES_URL/my-index-000003/_ilm/retry"                                                                              # after fixing the cause of an ERROR step
es -X POST "$ES_URL/_ilm/move/my-index-000002" -d '{"current_step": {"phase": "hot", "action": "complete", "name": "complete"}, "next_step": {"phase": "warm"}}'   # force a phase change
```

`shrink` needs every primary on one node and enough disk there for a full copy; it is the usual cause of an index parked in `warm` for hours. `forcemerge` down to one segment is only worth doing on an index that will never be written again. The data stream lifecycle (`PUT _data_stream/<name>/_lifecycle` with `data_retention`) covers the retention-only case with far less to misconfigure, and since 8.x is the default for new Elastic integrations. Do not delete dated indices from cron.

```sh
es "$ES_URL/_ilm/policy/logs?pretty"
es "$ES_URL/my-index/_ilm/explain?pretty"        # current phase, step and any ERROR step
es "$ES_URL/_data_stream/logs-my-app-default?pretty"
```

## cat APIs

The `_cat` endpoints return columns for people and scripts where the JSON APIs return trees. Every one takes `?v` (headers), `?h=` (choose columns), `?s=` (sort, `:desc` per column), `?format=json|yaml`, `?bytes=b|kb|mb|gb` and `?time=ms|s` for fixed units, and `?help` to list every available column.

```sh
es "$ES_URL/_cat/health?v&ts=false"
es "$ES_URL/_cat/nodes?v&h=name,node.role,master,heap.percent,ram.percent,cpu,load_1m,disk.used_percent,version&s=name"
es "$ES_URL/_cat/indices?v&h=index,health,status,pri,rep,docs.count,store.size,pri.store.size&s=pri.store.size:desc&bytes=gb"
es "$ES_URL/_cat/shards/logs-*?v&h=index,shard,prirep,state,docs,store,node&s=store:desc&bytes=mb"
es "$ES_URL/_cat/segments/my-index?v&h=index,shard,segment,docs.count,size&s=size:desc"   # many tiny segments means forcemerge or a busier merge scheduler
es "$ES_URL/_cat/recovery?v&active_only=true&h=index,shard,type,stage,source_node,target_node,bytes_percent,time"
es "$ES_URL/_cat/thread_pool?v&h=node_name,name,active,queue,rejected,completed&s=rejected:desc"
es "$ES_URL/_cat/fielddata?v&s=size:desc"        # heap held by fielddata; anything large here is a text field being aggregated
es "$ES_URL/_cat/templates?v&s=order"
es "$ES_URL/_cat/aliases?v&s=alias"
es "$ES_URL/_cat/tasks?v&detailed&s=running_time:desc"
es "$ES_URL/_cat/master?v"
es "$ES_URL/_cat/count/my-index?v"
es "$ES_URL/_cat/indices?h=index&s=index" | wc -l   # column-only output is script-friendly without jq
```

Node roles in `_cat/nodes` are single letters: `c` cold, `d` data, `f` frozen, `h` hot, `i` ingest, `m` master-eligible, `r` remote cluster client, `s` content, `t` transform, `v` voting-only, `w` warm, `l` machine learning, and `-` for a coordinating-only node. The `master` column shows `*` on the elected master. The cat APIs are for humans and shell scripts; they are not stable enough to build applications on, and Elastic says so in the docs.

## Performance

```sh
es "$ES_URL/_nodes/stats/jvm,indices?pretty" | jq '.nodes[] | {name, heap: .jvm.mem.heap_used_percent, old_gc: .jvm.gc.collectors.old.collection_count, query_ms: .indices.search.query_time_in_millis}'
es "$ES_URL/_cat/thread_pool/search,write?v&h=node_name,name,active,queue,rejected"
es "$ES_URL/_nodes/hot_threads"
```

| Symptom | Usual cause |
| --- | --- |
| `rejected` rising in a thread pool | Queue full: too many concurrent requests, or many tiny shards per request |
| Frequent old-generation GC, high heap | Large aggregations, `fielddata` on `text`, too many shards or fields, heap above about 31 GB (loses compressed pointers) |
| Slow search, indexing fine | Too many small shards, leading wildcards, scripts, scoring where a filter would do |
| Slow indexing | `refresh_interval` too short, replicas during a bulk load, small bulk batches |
| New documents not in results | Not refreshed yet: default refresh is 1s; if `refresh_interval` is not set explicitly, shards with no searches for 30s skip scheduled refreshes until searched again |

For a one-off bulk load, set `"refresh_interval": "-1"` and `"number_of_replicas": 0`, then restore both afterwards; without replicas, a node loss during the load loses data.

Shard sizing, per the [Elastic guidance](https://www.elastic.co/docs/deploy-manage/production-guidance/optimize-performance/size-shards): aim for 10 to 50 GB per shard and under 200 million documents per shard. Over-sharding (many small shards) is the most common self-inflicted performance problem. The older "20 shards per GB of heap" rule is no longer in the guidance.

```sh
# Search and indexing slow logs for one index
es -X PUT "$ES_URL/my-index/_settings" -d '{
  "index.search.slowlog.threshold.query.warn": "2s",
  "index.indexing.slowlog.threshold.index.warn": "1s"
}'
```

## Snapshots

```sh
# Register an S3 repository (credentials go in the Elasticsearch keystore, not the request)
es -X PUT "$ES_URL/_snapshot/backups" -d '{"type": "s3", "settings": {"bucket": "my-es-backups"}}'
es -X POST "$ES_URL/_snapshot/backups/_verify?pretty"

es -X PUT "$ES_URL/_snapshot/backups/snap-$(date +%F)?wait_for_completion=false"
es "$ES_URL/_cat/snapshots/backups?v"

# Restore one index under a new name so it does not collide with the live index
es -X POST "$ES_URL/_snapshot/backups/snap-2026-09-01/_restore" -d '{"indices": "my-index", "rename_pattern": "(.+)", "rename_replacement": "restored-$1"}'
```

Schedule snapshots with snapshot lifecycle management (`PUT _slm/policy/<name>`) rather than cron. Snapshots are incremental at the segment level, so frequent snapshots are cheap.

## Troubleshooting

| Symptom | Likely cause | Check |
| --- | --- | --- |
| `curl: (52) Empty reply` or certificate error | HTTPS with security enabled; plain HTTP or unknown CA used | `https://` and `--cacert` |
| `401` / `security_exception` | Missing or wrong credentials, or API key lacks privileges | `GET _security/_authenticate` |
| Cluster yellow | Replicas unassigned: single node, disk low watermark, allocation filters | `_cluster/allocation/explain` |
| Cluster red | A primary is unassigned: node lost with no replica, corruption | Explain API; restore from snapshot |
| Writes fail, `read-only / allow delete` | Flood-stage disk watermark | `_cat/allocation`; free space |
| `circuit_breaking_exception` | Request would exceed a heap limit | Aggregation size, `fielddata`, heap usage |
| `es_rejected_execution_exception` | Thread pool queue full | `_cat/thread_pool`; client concurrency |
| `mapper_parsing_exception` | Value does not match the field type, or `dynamic: strict` | `GET my-index/_mapping/field/<name>` |
| `illegal_argument_exception: Text fields are not optimised...` | Aggregating or sorting on `text` | Use the `keyword` field |
| `term` query finds nothing | Query on a `text` field | `_analyze` to see stored tokens |
| `Result window is too large` | `from + size` over 10,000 | PIT with `search_after` |
| ILM stuck in `ERROR` | Rollover alias, shard allocation or permissions | `_ilm/explain`; `POST my-index/_ilm/retry` after fixing |
| `illegal_argument_exception: index.lifecycle.rollover_alias does not point to index` | Alias missing, or `is_write_index` not set on the current index | `GET _cat/aliases/my-index?v`; add the alias with `is_write_index: true` |
| Index sits in `warm` for hours | `shrink` waiting for all primaries on one node with enough disk | `_ilm/explain` shows `step: shrink` or `wait-for-shard-history-leases`; `_cat/allocation` |
| `too_many_clauses` or `maxClauseCount` | `terms`, `wildcard` or `query_string` expanded past `indices.query.bool.max_clause_count` (default 4096, dynamic since 8.x) | Rewrite as `terms` lookup or filter; shorten the wildcard prefix |
| `search_phase_execution_exception: Fielddata is disabled on [message]` | Sorting or aggregating on `text` | Aggregate on `message.keyword` or add a multi-field and reindex |
| `version_conflict_engine_exception` | Two writers updated the same document; `_update_by_query` or `_reindex` raced a live write | `conflicts=proceed`, or retry with `if_seq_no`/`if_primary_term` from the failed response |
| `index_not_found_exception` on a template-managed name | Data stream not created because the template's `index_patterns` or `priority` lost to another template | `POST _index_template/_simulate_index/<name>` shows the winner |
| New index gets a wrong mapping | Dynamic mapping guessed from the first document, or a lower-priority template matched | `_simulate_index`; set `dynamic: strict` in the template |
| `cluster_block_exception ... index read-only` without disk pressure | `index.blocks.write` or `index.blocks.read_only` set by a shrink, a restore or a person | `GET my-index/_settings?flat_settings&filter_path=**.blocks*`; set the block to `null` |
| `master_not_discovered_exception` | Fewer than a quorum of master-eligible nodes reachable, or `cluster.initial_master_nodes` wrong on a new cluster | `_cat/master`; node logs for `discovery`; check `discovery.seed_hosts` |
| Snapshot `PARTIAL` | A shard was unassigned or a node dropped out during the snapshot | `GET _snapshot/backups/<name>?pretty` lists failed shards; retake after the cluster is green |
| Bulk indexing slower than expected, no rejections | Small batches, one client thread, `refresh_interval` 1s, replicas rebuilt per request | 5 to 15 MB per bulk request; several clients; `refresh_interval: 30s` during the load |

## Oneliners

```sh
# Cluster status on one line
es "$ES_URL/_cluster/health" | jq -r '[.status, .number_of_nodes, .active_shards, .unassigned_shards] | @tsv'

# Indices sorted by size in bytes
es "$ES_URL/_cat/indices?h=index,store.size,docs.count&bytes=b" | sort -k2 -rn | head

# Shards that are not STARTED, with reasons
es "$ES_URL/_cat/shards?h=index,shard,prirep,state,unassigned.reason" | awk '$4!="STARTED"'

# Retry allocations that hit the retry limit (after fixing the cause)
es -X POST "$ES_URL/_cluster/reroute?retry_failed=true" -o /dev/null -w '%{http_code}\n'

# Mapping of one field across all indices
es "$ES_URL/_mapping/field/service?pretty"

# Tokens the analyser produces for a string
es "$ES_URL/my-index/_analyze" -d '{"field":"message","text":"Connection timed out"}' | jq -r '.tokens[].token'

# Document count per value of a keyword field
es "$ES_URL/my-index/_search" -d '{"size":0,"aggs":{"s":{"terms":{"field":"service","size":20}}}}' | jq -r '.aggregations.s.buckets[] | [.key, .doc_count] | @tsv'

# Cancel searches running longer than 30 seconds
es "$ES_URL/_tasks?actions=*search*&detailed" | jq -r '.nodes[].tasks | to_entries[] | select(.value.running_time_in_nanos > 30e9) | .key' | xargs -r -I{} curl -sS --cacert http_ca.crt -H "Authorization: ApiKey $ES_API_KEY" -X POST "$ES_URL/_tasks/{}/_cancel"

# Count documents from the last hour
es "$ES_URL/my-index/_count" -d '{"query":{"range":{"@timestamp":{"gte":"now-1h"}}}}' | jq .count

# Which node is the elected master, and every master-eligible node
es "$ES_URL/_cat/master?h=node"; es "$ES_URL/_cat/nodes?h=name,node.role" | awk '$2 ~ /m/'

# Nodes above 75% heap
es "$ES_URL/_cat/nodes?h=name,heap.percent" | awk '$2 > 75'

# Indices with more than one primary shard per node's worth of data (candidates for shrink)
es "$ES_URL/_cat/indices?h=index,pri,pri.store.size&bytes=gb" | awk '$2 > 1 && $3/$2 < 10 {print $1, $2, $3"gb"}'

# Total shard count against the default per-node limit of 1000
es "$ES_URL/_cluster/health" | jq '{active_shards, nodes: .number_of_data_nodes, per_node: (.active_shards / .number_of_data_nodes | floor)}'

# Every non-default persistent and transient cluster setting
es "$ES_URL/_cluster/settings?flat_settings" | jq '.persistent + .transient'

# Every index setting that differs from the defaults, one index
es "$ES_URL/my-index/_settings?flat_settings" | jq '.[].settings'

# Which index template a new index name would use
es -X POST "$ES_URL/_index_template/_simulate_index/logs-my-app-2026.10.01" | jq '{overlapping, template: .template.settings}'

# ILM phase of every index, grouped
es "$ES_URL/logs-*/_ilm/explain" | jq -r '.indices[] | [.index, .phase, .action, .step] | @tsv' | sort -k2

# Indices in an ILM ERROR step with the failure reason
es "$ES_URL/*/_ilm/explain?only_errors=true" | jq -r '.indices[] | "\(.index)\t\(.step_info.reason // .step_info.type)"'

# Data streams and their backing index count
es "$ES_URL/_data_stream" | jq -r '.data_streams[] | [.name, (.indices | length), .ilm_policy // .lifecycle.data_retention] | @tsv'

# Force a data stream to roll over now (for example after changing the template mapping)
es -X POST "$ES_URL/logs-my-app-default/_rollover" | jq '{old_index, new_index, rolled_over}'

# Segments per shard for an index; hundreds means merges are behind
es "$ES_URL/_cat/segments/my-index?h=shard,segment" | awk '{c[$1]++} END {for (s in c) print s, c[s]}'

# Fields in an index, flattened, to spot field explosion (limit is index.mapping.total_fields.limit, default 1000)
es "$ES_URL/my-index/_mapping" | jq '[.. | objects | select(has("type")) ] | length'

# Recovery progress for shards currently moving or rebuilding
es "$ES_URL/_cat/recovery?active_only=true&h=index,shard,stage,source_node,target_node,bytes_percent,translog_ops_percent"

# Move a shard off a node explicitly (allocation may move it straight back if deciders disagree)
es -X POST "$ES_URL/_cluster/reroute" -d '{"commands":[{"move":{"index":"my-index","shard":0,"from_node":"es-data-1","to_node":"es-data-2"}}]}' | jq .acknowledged

# Drain a node before maintenance: exclude it, then watch its shard count fall to 0
es -X PUT "$ES_URL/_cluster/settings" -d '{"transient":{"cluster.routing.allocation.exclude._name":"es-data-3"}}'; watch -n5 "curl -sS --cacert http_ca.crt -H \"Authorization: ApiKey $ES_API_KEY\" '$ES_URL/_cat/allocation?h=node,shards' | grep es-data-3"

# Undo the drain
es -X PUT "$ES_URL/_cluster/settings" -d '{"transient":{"cluster.routing.allocation.exclude._name":null}}'

# Disable replica allocation for a rolling restart (primaries still allocate), then re-enable
es -X PUT "$ES_URL/_cluster/settings" -d '{"persistent":{"cluster.routing.allocation.enable":"primaries"}}'
es -X PUT "$ES_URL/_cluster/settings" -d '{"persistent":{"cluster.routing.allocation.enable":null}}'

# Flush before stopping a node so recovery has less translog to replay
es -X POST "$ES_URL/_flush"

# Slow searches from the slow log, grouped by index
journalctl -u elasticsearch --since -1h | grep 'index.search.slowlog' | grep -o '\[[^]]*\]\[[0-9]*\]' | sort | uniq -c | sort -rn | head

# Long-running tasks of any kind on any node
es "$ES_URL/_tasks?detailed" | jq -r '.nodes[].tasks[] | select(.running_time_in_nanos > 60e9) | [(.running_time_in_nanos/1e9|floor), .action, .description[:80]] | @tsv' | sort -rn

# Estimate how many hits a query has without fetching any
es "$ES_URL/my-index/_search?size=0&track_total_hits=true" -d '{"query":{"match":{"message":"timeout"}}}' | jq .hits.total.value

# Profile a slow query: per-shard time by query component
es "$ES_URL/my-index/_search" -d '{"profile":true,"size":0,"query":{"match":{"message":"timeout"}}}' | jq '[.profile.shards[].searches[].query[] | {type, time_in_nanos, description: .description[:60]}] | sort_by(-.time_in_nanos) | .[:5]'

# Create an API key limited to reading one index pattern, valid for 30 days
es -X POST "$ES_URL/_security/api_key" -d '{"name":"readonly-logs","expiration":"30d","role_descriptors":{"ro":{"indices":[{"names":["logs-*"],"privileges":["read","view_index_metadata"]}]}}}' | jq -r .encoded

# Who am I and what roles do I have
es "$ES_URL/_security/_authenticate" | jq '{username, roles}'
```

> [!WARNING] Delete by query is permanent
> Run the same query through `_count` first. There is no undo short of a snapshot restore.

```sh
es -X POST "$ES_URL/my-index/_delete_by_query?conflicts=proceed" -d '{"query":{"term":{"level":"debug"}}}' | jq '{deleted, failures}'
```

## Scripts

Cluster health report: status, node pressure, unassigned shards with their explain reason and the indices closest to the per-shard size guidance, in one screen. Read-only.

```sh
#!/usr/bin/env bash
# usage: ES_URL=https://es.example.com:9200 ES_API_KEY=... es-report.sh [ca.crt]
set -euo pipefail
: "${ES_URL:?}" "${ES_API_KEY:?}"
ca=${1:-http_ca.crt}
es() { curl -sS --max-time 20 --cacert "$ca" -H "Authorization: ApiKey $ES_API_KEY" -H 'Content-Type: application/json' "$@"; }

es "$ES_URL/_cluster/health" | jq -r '"cluster \(.cluster_name): \(.status)  nodes=\(.number_of_nodes) shards=\(.active_shards) unassigned=\(.unassigned_shards) pending_tasks=\(.number_of_pending_tasks)"'

printf '\n-- nodes over 75%% heap or 85%% disk --\n'
es "$ES_URL/_cat/nodes?h=name,heap.percent,disk.used_percent,cpu,load_1m" | awk '$2 > 75 || $3 > 85' || true

printf '\n-- unassigned shards --\n'
es "$ES_URL/_cat/shards?h=index,shard,prirep,state,unassigned.reason" | awk '$4 == "UNASSIGNED"' | head -20 || true
if es "$ES_URL/_cat/shards?h=state" | grep -q UNASSIGNED; then
  es "$ES_URL/_cluster/allocation/explain" | jq -r '"first unassigned: \(.index)[\(.shard)] \(.allocate_explanation // .unassigned_info.reason)"'
fi

printf '\n-- indices with primary shards over 50gb or under 1gb (and more than 1 primary) --\n'
es "$ES_URL/_cat/indices?h=index,pri,pri.store.size&bytes=gb&s=pri.store.size:desc" \
  | awk '$3 > 0 { per = $3 / $2; if (per > 50 || ($2 > 1 && per < 1)) printf "%-50s pri=%s avg=%.1fgb\n", $1, $2, per }' || true

printf '\n-- thread pool rejections --\n'
es "$ES_URL/_cat/thread_pool/search,write,get?h=node_name,name,rejected" | awk '$3 > 0' || true

printf '\n-- ILM errors --\n'
es "$ES_URL/*/_ilm/explain?only_errors=true" | jq -r '.indices[] | "\(.index): \(.step_info.reason // .step_info.type // "unknown")"' || true
```

Retention enforcement for indices that are not under ILM: deletes indices matching a pattern whose newest document is older than a threshold. Prints the plan and exits unless `--apply` is given. Deletes indices when run with `--apply`.

```sh
#!/usr/bin/env bash
# usage: es-prune.sh [--apply] PATTERN DAYS   e.g. es-prune.sh 'logs-legacy-*' 90
set -euo pipefail
apply=0; [[ ${1:-} == --apply ]] && { apply=1; shift; }
pattern=${1:?index pattern required}; days=${2:?days required}
[[ $days =~ ^[0-9]+$ ]] || { echo 'DAYS must be an integer' >&2; exit 2; }
es() { curl -sS --max-time 30 --cacert "${ES_CA:-http_ca.crt}" -H "Authorization: ApiKey $ES_API_KEY" -H 'Content-Type: application/json' "$@"; }
cutoff=$(date -u -d "-${days} days" +%s)

mapfile -t indices < <(es "$ES_URL/_cat/indices/$pattern?h=index&s=index")
(( ${#indices[@]} )) || { echo "no indices match $pattern" >&2; exit 0; }

to_delete=()
for idx in "${indices[@]}"; do
  # Skip anything that is a write target or under ILM; those are someone else's job
  if es "$ES_URL/$idx/_ilm/explain" | jq -e '.indices[].managed' >/dev/null 2>&1; then printf 'skip %s: ILM managed\n' "$idx"; continue; fi
  if es "$ES_URL/_alias/$idx" 2>/dev/null | jq -e '.[].aliases[] | select(.is_write_index == true)' >/dev/null; then printf 'skip %s: write index\n' "$idx"; continue; fi
  newest=$(es "$ES_URL/$idx/_search?size=0" -d '{"aggs":{"m":{"max":{"field":"@timestamp"}}}}' | jq -r '.aggregations.m.value // empty')
  [[ -n $newest ]] || { printf 'skip %s: no @timestamp\n' "$idx"; continue; }
  newest_s=$(( ${newest%.*} / 1000 ))
  if (( newest_s < cutoff )); then
    printf 'delete %s (newest doc %s)\n' "$idx" "$(date -u -d "@$newest_s" +%F)"
    to_delete+=("$idx")
  fi
done

(( ${#to_delete[@]} )) || { echo 'nothing to delete'; exit 0; }
(( apply )) || { printf '\n%d indices would be deleted; rerun with --apply\n' "${#to_delete[@]}"; exit 0; }
for idx in "${to_delete[@]}"; do
  es -X DELETE "$ES_URL/$idx" | jq -c --arg i "$idx" '{index: $i, acknowledged}'
done
```

Export every document matching a query to NDJSON with a point in time and `search_after`, which works past the 10,000-hit window and does not hold a scroll context open on the cluster longer than `keep_alive`.

```python
#!/usr/bin/env python3
"""usage: ES_URL=... ES_API_KEY=... es-export.py INDEX 'QUERY_JSON' > out.ndjson"""
import json, os, sys, urllib.request, ssl

url, key = os.environ["ES_URL"], os.environ["ES_API_KEY"]
index, query = sys.argv[1], json.loads(sys.argv[2] if len(sys.argv) > 2 else '{"match_all": {}}')
ctx = ssl.create_default_context(cafile=os.environ.get("ES_CA", "http_ca.crt"))

def call(method, path, body=None):
    req = urllib.request.Request(url + path, method=method, data=json.dumps(body).encode() if body else None,
                                 headers={"Authorization": f"ApiKey {key}", "Content-Type": "application/json"})
    with urllib.request.urlopen(req, context=ctx, timeout=60) as r:
        return json.load(r)

pit = call("POST", f"/{index}/_pit?keep_alive=2m")["id"]
search_after, total = None, 0
try:
    while True:
        body = {"size": 1000, "query": query, "pit": {"id": pit, "keep_alive": "2m"},
                "sort": [{"@timestamp": "asc"}, {"_shard_doc": "asc"}]}   # _shard_doc breaks ties so no hit is skipped
        if search_after:
            body["search_after"] = search_after
        res = call("GET", "/_search", body)
        hits = res["hits"]["hits"]
        if not hits:
            break
        for h in hits:
            sys.stdout.write(json.dumps(h["_source"]) + "\n")
        total += len(hits)
        search_after = hits[-1]["sort"]
        pit = res["pit_id"]                                                # the PIT id can change between pages
finally:
    call("DELETE", "/_pit", {"id": pit})
print(f"exported {total} documents", file=sys.stderr)
```

## Further reading

- [Elasticsearch REST API reference](https://www.elastic.co/docs/api/doc/elasticsearch/): every endpoint with request and response schemas.
- [Query DSL](https://www.elastic.co/docs/reference/query-languages/querydsl): full-text, term-level, compound and joining queries.
- [Mapping and field types](https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/field-data-types): every type and mapping parameter.
- [Text analysis](https://www.elastic.co/docs/manage-data/data-store/text-analysis): analysers, tokenizers, token and character filters.
- [Index lifecycle management](https://www.elastic.co/docs/manage-data/lifecycle/index-lifecycle-management): phases, actions and rollover.
- [Size your shards](https://www.elastic.co/docs/deploy-manage/production-guidance/optimize-performance/size-shards): current sizing guidance and how to fix an oversharded cluster.