# Grafana and Loki

> Provision Grafana datasources, dashboards and alerts as code, drive it through the HTTP API, and ship, query and tune logs in Loki with LogQL and logcli.

Canonical: https://www.wiki.jodisand.me/grafana/
Reviewed: 2026-09-24
Related: [Prometheus](https://www.wiki.jodisand.me/prometheus/index.md), [OpenTelemetry](https://www.wiki.jodisand.me/opentelemetry/index.md), [Kubernetes](https://www.wiki.jodisand.me/kubernetes/index.md), [Docker Compose](https://www.wiki.jodisand.me/docker-compose/index.md)


## Cheatsheet

| Task | Command |
| --- | --- |
| Is Grafana up | `curl -fsS http://grafana.example.com:3000/api/health` |
| Find a dashboard | `curl -fsS -H "Authorization: Bearer $GRAFANA_TOKEN" "http://grafana.example.com:3000/api/search?query=my-app&type=dash-db"` |
| Export a dashboard | `curl -fsS -H "Authorization: Bearer $GRAFANA_TOKEN" http://grafana.example.com:3000/api/dashboards/uid/my-app \| jq .dashboard > my-app.json` |
| Import a dashboard | `jq '{dashboard: (. + {id: null}), overwrite: true, folderUid: "ops"}' my-app.json \| curl -fsS -X POST -H "Authorization: Bearer $GRAFANA_TOKEN" -H 'Content-Type: application/json' -d @- http://grafana.example.com:3000/api/dashboards/db` |
| List datasources | `curl -fsS -H "Authorization: Bearer $GRAFANA_TOKEN" http://grafana.example.com:3000/api/datasources \| jq '.[] \| {name, type, uid}'` |
| Datasource health | `curl -fsS -H "Authorization: Bearer $GRAFANA_TOKEN" http://grafana.example.com:3000/api/datasources/uid/prometheus/health` |
| Reload provisioned dashboards | `curl -fsS -X POST -H "Authorization: Bearer $GRAFANA_TOKEN" http://grafana.example.com:3000/api/admin/provisioning/dashboards/reload` |
| Export alert rules as provisioning YAML | `curl -fsS -H "Authorization: Bearer $GRAFANA_TOKEN" "http://grafana.example.com:3000/api/v1/provisioning/alert-rules/export?format=yaml"` |
| Firing alerts | `curl -fsS -H "Authorization: Bearer $GRAFANA_TOKEN" http://grafana.example.com:3000/api/alertmanager/grafana/api/v2/alerts \| jq '.[].labels.alertname'` |
| Is Loki ready | `curl -fsS http://loki.example.com:3100/ready` |
| Label names in Loki | `logcli labels` |
| Values of one label | `logcli labels app` |
| Query the last hour | `logcli query --since 1h --limit 200 '{app="my-app"} \|= "error"'` |
| Follow logs | `logcli query --tail '{app="my-app"}'` |
| Raw lines only | `logcli query -o raw --since 15m '{app="my-app"} \| json \| level="error"'` |
| Metric query | `logcli instant-query 'sum by (app) (rate({namespace="prod"} \|= "error" [5m]))'` |
| Stream cardinality | `logcli series '{namespace="prod"}' --analyze-labels` |
| Loki config as running | `curl -fsS http://loki.example.com:3100/config` |
| Push a test line | `curl -fsS -X POST -H 'Content-Type: application/json' http://loki.example.com:3100/loki/api/v1/push -d "{\"streams\":[{\"stream\":{\"app\":\"test\"},\"values\":[[\"$(date +%s%N)\",\"hello\"]]}]}"` |
| Alloy config check | `alloy fmt --verify /etc/alloy/config.alloy` |

Behaviour below is Grafana 12.x and Loki 3.x with the TSDB index (schema v13). Promtail is deprecated and replaced by Alloy; the Promtail examples remain because existing deployments run it. References: [Grafana documentation](https://grafana.com/docs/grafana/latest/) and [Loki documentation](https://grafana.com/docs/loki/latest/).

## Provisioning as code

Grafana reads `provisioning/` (`/etc/grafana/provisioning` in the package and image, or `[paths] provisioning` in `grafana.ini`) at startup: `datasources/`, `dashboards/`, `alerting/` and `plugins/`. Provisioned objects are marked read-only in the UI unless the file allows edits, and a UI edit to a provisioned dashboard is overwritten on the next reload. That is the point: the repository is the source of truth, the same way [Argo CD](https://www.wiki.jodisand.me/argocd/) owns manifests.

```yaml
# provisioning/datasources/datasources.yaml
apiVersion: 1
datasources:
  - name: Prometheus
    uid: prometheus                    # fixed uid so dashboards and alert rules can reference it without an id lookup
    type: prometheus
    access: proxy                      # Grafana's backend makes the request; the browser never reaches the datasource
    url: http://prometheus.example.com:9090
    isDefault: true
    editable: false
    jsonData:
      httpMethod: POST
      timeInterval: 15s                # scrape interval, drives $__rate_interval
      exemplarTraceIdDestinations:
        - name: trace_id
          datasourceUid: tempo
  - name: Loki
    uid: loki
    type: loki
    access: proxy
    url: http://loki.example.com:3100
    jsonData:
      maxLines: 1000
      derivedFields:                   # turn a trace id in a log line into a link
        - name: TraceID
          matcherRegex: 'trace_id=(\w+)'
          url: '$${__value.raw}'       # $$ escapes Grafana's own interpolation in provisioning files
          datasourceUid: tempo
    secureJsonData:
      httpHeaderValue1: "$LOKI_TENANT" # environment variables expand; keep real values out of the file
    jsonData:
      httpHeaderName1: X-Scope-OrgID
deleteDatasources:
  - name: Old Prometheus
    orgId: 1
```

Datasource files are re-read only at startup or through `POST /api/admin/provisioning/datasources/reload`. Dashboards are different: a provider watches a directory and reloads changed files every `updateIntervalSeconds`.

```yaml
# provisioning/dashboards/default.yaml
apiVersion: 1
providers:
  - name: repo
    orgId: 1
    type: file
    updateIntervalSeconds: 30
    allowUiUpdates: false              # true lets people save from the UI, and the file still wins at the next change
    disableDeletion: false             # false: removing the file removes the dashboard
    options:
      path: /var/lib/grafana/dashboards
      foldersFromFilesStructure: true  # subdirectory name becomes the folder
```

Two dashboards with the same `uid` in the tree make the provisioner log an error and skip one; a dashboard without a `uid` gets a random one on every import, breaking links. Give every dashboard file a fixed `uid` and a `title` unique within its folder.

Alerting is provisioned from `provisioning/alerting/*.yaml` with top-level keys `contactPoints`, `policies`, `muteTimes`, `templates` and `groups` (rule groups), plus `deleteContactPoints`, `deleteRules` and so on for removal. The `data` block of a rule is the same JSON the UI produces, so the workflow that works is: build the rule in the UI, export it with `GET /api/v1/provisioning/alert-rules/<uid>/export?format=yaml` (or the Export button), commit, and restart or reload. Hand-writing the `model` blocks is error-prone.

```yaml
# provisioning/alerting/contact-points.yaml
apiVersion: 1
contactPoints:
  - orgId: 1
    name: oncall-slack
    receivers:
      - uid: oncall-slack
        type: slack
        settings:
          url: "$SLACK_WEBHOOK_URL"    # expanded from the environment at load
          recipient: "#oncall"
          title: '{{ template "default.title" . }}'
        disableResolveMessage: false
policies:
  - orgId: 1
    receiver: oncall-slack             # root policy: everything not matched below
    group_by: [alertname, namespace]
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 4h
    routes:
      - receiver: oncall-slack
        object_matchers:
          - [severity, "=", critical]
        repeat_interval: 1h
        continue: false
```

Provisioning applies contact points first, then policies, then rules, then deletes; there is no rollback, so a broken rule file leaves earlier changes applied. Check `journalctl -u grafana-server` (or the container log) for `provisioning.alerting` errors after every change.

## Dashboard JSON essentials

A dashboard is one JSON document. The fields that matter when writing or reviewing one:

```json
{
  "uid": "my-app",
  "title": "my-app",
  "tags": ["team-platform"],
  "editable": false,
  "schemaVersion": 41,
  "time": { "from": "now-6h", "to": "now" },
  "refresh": "1m",
  "templating": { "list": [] },
  "panels": [
    {
      "type": "timeseries",
      "title": "Requests per second",
      "gridPos": { "x": 0, "y": 0, "w": 12, "h": 8 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "targets": [
        {
          "refId": "A",
          "expr": "sum by (status) (rate(http_requests_total{job=\"my-app\", namespace=\"$namespace\"}[$__rate_interval]))",
          "legendFormat": "{{status}}"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "unit": "reqps",
          "min": 0,
          "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 500 } ] }
        },
        "overrides": [
          { "matcher": { "id": "byRegexp", "options": "5.." }, "properties": [ { "id": "color", "value": { "mode": "fixed", "fixedColor": "red" } } ] }
        ]
      },
      "options": { "legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max", "lastNotNull"] }, "tooltip": { "mode": "multi", "sort": "desc" } }
    }
  ]
}
```

`gridPos` is a 24-column grid; `y` positions are relative to the row and Grafana packs panels upward. `id` is per-instance and must be `null` (or absent) when importing through the API. A dashboard exported with "Export for sharing externally" carries an `__inputs` block and `${DS_PROMETHEUS}` datasource placeholders; those are for the import UI only and fail under file provisioning, so export without that option and reference datasources by fixed `uid`.

Panel types worth knowing: `timeseries` (default graph), `stat` (single value with sparkline; set the target to `instant: true` and reduce to `lastNotNull`), `gauge` and `bargauge` (thresholds as fill), `table` (needs `format: table` on Prometheus targets and a Transform to organise columns), `logs` (Loki streams), `heatmap` (Prometheus histograms with `format: heatmap`), `text` (Markdown), `row` (collapsible group). Removed panel types are migrated automatically on load, but the migrated JSON is what you should commit.

## Variables and templating

Variables live in `templating.list` and are referenced as `$var` or `${var}`, with a format modifier for multi-value: `${var:regex}` produces `(a|b)`, `${var:csv}` `a,b`, `${var:pipe}` `a|b`, `${var:queryparam}` for links.

| Type | Use | Example |
| --- | --- | --- |
| `query` | Values from a datasource | `label_values(up{job="my-app"}, namespace)` |
| `query` chained | Depends on another variable; refreshes when it changes | `label_values(kube_pod_info{namespace="$namespace"}, pod)` |
| `custom` | Fixed list | `prod,staging,dev` |
| `interval` | Step size selector, sets `$interval` | `1m,5m,15m,1h` with `auto` |
| `datasource` | Pick a datasource of a type | Multi-cluster dashboards, one Prometheus per cluster |
| `textbox` | Free text | Trace ID, user ID |
| `constant` | Hidden value used in queries | Cluster name in a copied dashboard |
| `adhoc` | Key/value filters the viewer adds; applied to every query on that datasource | Datasource-wide filtering |

```json
{
  "name": "namespace",
  "type": "query",
  "datasource": { "type": "prometheus", "uid": "prometheus" },
  "query": { "query": "label_values(kube_namespace_status_phase, namespace)", "refId": "ns" },
  "refresh": 2,
  "regex": "/^(?!kube-).*/",
  "sort": 1,
  "multi": true,
  "includeAll": true,
  "allValue": ".+",
  "current": { "text": "All", "value": "$__all" }
}
```

`refresh: 1` re-queries on dashboard load, `2` on time range change. Multi-value and `All` variables must be used with `=~`, and `allValue: ".+"` is faster than the default, which expands `All` to every value joined with `|`. Built-in variables: `$__rate_interval` (at least four scrape intervals; use it in every `rate()`), `$__interval` (the current step), `$__range` (the whole time range, for `increase(...[$__range])` in a stat), `$__from` and `$__to` (epoch milliseconds, `${__from:date:iso}` for ISO), `$__dashboard`, `$__org` and `$__user.login`. Loki queries use `$__auto` as the range selector, which is the Loki equivalent of `$__rate_interval`.

## Alert rules

A Grafana-managed rule is a chain of queries and expressions: a datasource query (`A`), a `reduce` expression (`B`, last or mean over the range) and a `threshold` or `math` expression (`C`) that is the condition. The rule fires when the condition is non-zero for `for`, evaluated every `interval` of its group. Rules in one group are evaluated sequentially, so keep a group to rules that share an interval and are cheap.

```yaml
apiVersion: 1
groups:
  - orgId: 1
    name: my-app
    folder: platform
    interval: 1m
    rules:
      - uid: my-app-error-ratio
        title: my-app 5xx ratio above 5%
        condition: C
        for: 10m
        noDataState: NoData          # a query returning nothing is not an outage; alert on absent() separately
        execErrState: Error
        labels: { severity: critical, team: platform }
        annotations:
          summary: "{{ $labels.namespace }}/my-app 5xx ratio is {{ $values.B | printf \"%.1f\" }}%"
          runbook_url: https://wiki.example.com/runbooks/my-app-5xx
        data:
          - refId: A
            relativeTimeRange: { from: 600, to: 0 }
            datasourceUid: prometheus
            model:
              expr: 100 * sum by (namespace) (rate(http_requests_total{job="my-app",status=~"5.."}[5m])) / sum by (namespace) (rate(http_requests_total{job="my-app"}[5m]))
              instant: true
              refId: A
          - refId: B
            datasourceUid: __expr__
            model: { type: reduce, expression: A, reducer: last, refId: B }
          - refId: C
            datasourceUid: __expr__
            model: { type: threshold, expression: B, refId: C, conditions: [ { evaluator: { type: gt, params: [5] } } ] }
```

Each distinct label set from query A becomes its own alert instance, so a `sum by (namespace)` produces one alert per namespace and a query without aggregation produces one per series, which is usually noise. Labels on the rule plus the series labels are what notification policies match on; annotations are for humans and can template `$labels` and `$values`. Rules that need the same Prometheus query as a dashboard panel should be Prometheus [alerting rules](https://www.wiki.jodisand.me/prometheus/#alerting-rules) evaluated by Prometheus, with Grafana only as the viewer, when the metrics side already runs Alertmanager; Grafana-managed rules are the choice when the source is Loki, SQL or a mix of datasources.

Silence and inspect from the API: `GET /api/alertmanager/grafana/api/v2/alerts?active=true`, `POST /api/alertmanager/grafana/api/v2/silences` with the Alertmanager silence body, and `GET /api/prometheus/grafana/api/v1/rules` for rule state including the last evaluation error.

## Useful panel patterns

```promql
# Error ratio as a percentage, safe when the denominator is zero
100 * sum(rate(http_requests_total{job="my-app",status=~"5.."}[$__rate_interval])) / clamp_min(sum(rate(http_requests_total{job="my-app"}[$__rate_interval])), 1)

# p99 latency from a histogram; a heatmap panel takes the same query without histogram_quantile and format=heatmap
histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket{job="my-app"}[$__rate_interval])))

# Stat panel: total over the dashboard time range, one instant value
sum(increase(jobs_completed_total{job="my-app"}[$__range]))

# Table: current value per pod with restart count, format=table, instant=true, then Transform > Organize fields
kube_pod_container_status_restarts_total{namespace="$namespace"}

# Legend with variables and a fixed unit: legendFormat "{{pod}} ({{container}})", unit "bytes"
container_memory_working_set_bytes{namespace="$namespace", container!=""}

# Deployment markers: annotation query on a counter that increments on rollout
changes(kube_deployment_status_observed_generation{namespace="$namespace"}[1m]) > 0
```

Other patterns that pay off: a `Reduce` transform with `Series to rows` to get a top-N table from a time series; `Value mappings` to turn `0`/`1` into `down`/`up` in a stat; an override matching `byFrameRefID` to put a second query on the right axis; a `Join by field (time)` transform to divide two datasources' results; and the panel `interval` set to the scrape interval on counters so `rate()` never sees a range shorter than two samples. Set `maxDataPoints` low (a few hundred) on dashboards used over long ranges; it caps the step Grafana asks Prometheus for and is the difference between a two-second and a twenty-second load.

## The HTTP API with curl

Authenticate with a service account token (Administration > Service accounts, or the API), never a user password. Tokens carry the account's role; create a `Viewer` account for read-only automation.

```sh
G=http://grafana.example.com:3000
H="Authorization: Bearer $GRAFANA_TOKEN"
curl -fsS "$G/api/health"                                          # no auth: {"database":"ok","version":...}
curl -fsS -H "$H" "$G/api/org"                                     # confirms the token works and which org it is in
curl -fsS -H "$H" "$G/api/search?type=dash-db&tag=team-platform" | jq -r '.[] | "\(.uid)\t\(.folderTitle)/\(.title)"'
curl -fsS -H "$H" "$G/api/dashboards/uid/my-app" | jq '.dashboard' > my-app.json     # .meta has folder, version, provisioned flag
curl -fsS -H "$H" "$G/api/dashboards/uid/my-app/versions" | jq '.versions[] | {version, created, createdBy, message}'
curl -fsS -H "$H" "$G/api/folders" | jq '.[] | {uid, title}'
curl -fsS -X POST -H "$H" -H 'Content-Type: application/json' "$G/api/folders" -d '{"uid":"ops","title":"Ops"}'
# Import: id must be null, overwrite replaces a dashboard with the same uid; provisioned dashboards return 400
jq '{dashboard: (. + {id: null}), folderUid: "ops", overwrite: true, message: "from CI"}' my-app.json \
  | curl -fsS -X POST -H "$H" -H 'Content-Type: application/json' "$G/api/dashboards/db" -d @-
curl -fsS -X DELETE -H "$H" "$G/api/dashboards/uid/old-dashboard"   # permanent; export first
curl -fsS -H "$H" "$G/api/datasources" | jq '.[] | {name, type, uid, url}'
curl -fsS -H "$H" "$G/api/datasources/uid/prometheus/health"
# Run a query through Grafana (uses the datasource's credentials, useful when the datasource is not reachable directly)
curl -fsS -X POST -H "$H" -H 'Content-Type: application/json' "$G/api/ds/query" -d '{"queries":[{"refId":"A","datasource":{"uid":"prometheus"},"expr":"up","instant":true}],"from":"now-5m","to":"now"}' | jq '.results.A.frames[0].data'
# Annotation on a dashboard, for deploy markers
curl -fsS -X POST -H "$H" -H 'Content-Type: application/json' "$G/api/annotations" -d "{\"dashboardUID\":\"my-app\",\"time\":$(date +%s000),\"tags\":[\"deploy\"],\"text\":\"my-app 1.4.2\"}"
# Alerting provisioning API: rules, contact points and policies as the same YAML the files use
curl -fsS -H "$H" "$G/api/v1/provisioning/alert-rules" | jq '.[] | {uid, title, folderUID}'
curl -fsS -H "$H" "$G/api/v1/provisioning/contact-points/export?format=yaml"
curl -fsS -H "$H" "$G/api/v1/provisioning/policies/export?format=yaml"
# Service account and token (admin)
curl -fsS -X POST -H "$H" -H 'Content-Type: application/json' "$G/api/serviceaccounts" -d '{"name":"ci","role":"Editor"}'
curl -fsS -X POST -H "$H" -H 'Content-Type: application/json' "$G/api/serviceaccounts/12/tokens" -d '{"name":"ci-token"}'   # prints the key once
```

Objects changed through the API but defined in provisioning files revert at the next reload; use `X-Disable-Provenance: true` on `/api/v1/provisioning/*` requests when a resource should be editable in the UI afterwards.

## Loki architecture

Loki indexes labels, not log content. Every unique combination of label values is a stream; each stream's lines are batched into chunks, compressed and written to object storage (S3, GCS, Azure, or the filesystem for a single node), and the TSDB index records which chunks exist for which stream and time range. A query selects streams by label, fetches their chunks for the time range, and then filters and parses lines in the queriers. Cost therefore scales with the number of streams touched and bytes scanned, not with the number of lines that match, which is the opposite of [Elasticsearch](https://www.wiki.jodisand.me/elasticsearch/).

| Component | Role |
| --- | --- |
| Distributor | Receives pushes, validates labels and rate limits per tenant, hashes streams to ingesters |
| Ingester | Buffers lines per stream in memory (with a WAL), cuts chunks, flushes to object storage |
| Querier | Executes queries against ingesters (recent data) and storage |
| Query frontend and scheduler | Split long range queries by `split_queries_by_interval`, cache results, queue work fairly across tenants |
| Compactor | Merges index tables, applies retention, processes delete requests; exactly one instance |
| Ruler | Evaluates LogQL alerting and recording rules and sends to Alertmanager |
| Index gateway | Serves the index to queriers so they do not each download it |

Deployment modes are monolithic (`-target=all`, one process, fine to tens of GB per day), simple scalable (`read`, `write` and `backend` targets, each scaled separately, the Helm chart's default) and microservices. Configuration is the same file for all three; the mode only decides which components a process runs.

```yaml
# /etc/loki/config.yaml, single node with filesystem storage
auth_enabled: false                 # single tenant; multi-tenant deployments require X-Scope-OrgID on every request
server:
  http_listen_port: 3100
  grpc_listen_port: 9096
common:
  path_prefix: /var/lib/loki
  replication_factor: 1
  ring:
    kvstore: { store: inmemory }
  storage:
    filesystem:
      chunks_directory: /var/lib/loki/chunks
      rules_directory: /var/lib/loki/rules
schema_config:
  configs:
    - from: "2024-01-01"
      store: tsdb
      object_store: filesystem
      schema: v13
      index: { prefix: index_, period: 24h }
limits_config:
  retention_period: 744h            # 31 days; needs the compactor block below
  ingestion_rate_mb: 8
  ingestion_burst_size_mb: 16
  per_stream_rate_limit: 5MB
  per_stream_rate_limit_burst: 20MB
  max_query_series: 1000
  max_query_length: 0h              # 0 removes the 30-day default cap
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  allow_structured_metadata: true
  volume_enabled: true
compactor:
  working_directory: /var/lib/loki/compactor
  retention_enabled: true
  delete_request_store: filesystem
ruler:
  alertmanager_url: http://alertmanager.example.com:9093
```

Labels should be few and bounded: `namespace`, `app`, `pod`, `container`, `level`, `host`, `job`. A label with an unbounded value set (user ID, request ID, path) creates a stream per value, each with its own chunk, which is how a Loki gets to millions of tiny chunks and 429s. Put such values in the line, or in structured metadata (Loki 3.0+), which is stored with the chunk, not the index, and is queryable with the same label-filter syntax.

## Shipping logs with Promtail and Alloy

Promtail tails files, attaches labels through relabelling, runs pipeline stages, and pushes batches. Alloy is Grafana's successor: the same concepts with `loki.*` components in its own configuration language, plus the Prometheus and OpenTelemetry pipelines in one binary. Promtail reached end of life in March 2026; convert with `alloy convert --source-format=promtail --output=config.alloy promtail.yaml`.

```yaml
# /etc/promtail/config.yaml
server: { http_listen_port: 9080 }
positions: { filename: /var/lib/promtail/positions.yaml }   # where each file was left off; losing it re-reads everything
clients:
  - url: http://loki.example.com:3100/loki/api/v1/push
    external_labels: { host: web-1 }
scrape_configs:
  - job_name: journal
    journal:
      max_age: 12h
      labels: { job: systemd-journal }
    relabel_configs:
      - source_labels: ['__journal__systemd_unit']
        target_label: unit
  - job_name: my-app
    static_configs:
      - targets: [localhost]
        labels: { job: my-app, __path__: /var/log/my-app/*.log }
    pipeline_stages:
      - json:
          expressions: { level: level, ts: timestamp }
      - timestamp: { source: ts, format: RFC3339Nano }
      - labels: { level: "" }      # promote level to a label; bounded set, safe
```

```alloy
// /etc/alloy/config.alloy
loki.source.journal "journal" {
  max_age       = "12h"
  labels        = { job = "systemd-journal" }
  relabel_rules = loki.relabel.journal.rules
  forward_to    = [loki.write.default.receiver]
}

loki.relabel "journal" {
  forward_to = []
  rule {
    source_labels = ["__journal__systemd_unit"]
    target_label  = "unit"
  }
}

local.file_match "my_app" {
  path_targets = [{ __path__ = "/var/log/my-app/*.log", job = "my-app" }]
}

loki.source.file "my_app" {
  targets    = local.file_match.my_app.targets
  forward_to = [loki.process.my_app.receiver]
}

loki.process "my_app" {
  stage.json {
    expressions = { level = "level", ts = "timestamp" }
  }
  stage.timestamp {
    source = "ts"
    format = "RFC3339Nano"
  }
  stage.labels {
    values = { level = "" }
  }
  stage.structured_metadata {
    values = { trace_id = "" }        // high cardinality: metadata, not a label
  }
  forward_to = [loki.write.default.receiver]
}

loki.write "default" {
  endpoint {
    url = "http://loki.example.com:3100/loki/api/v1/push"
  }
  external_labels = { host = constants.hostname }
}
```

```sh
alloy fmt --verify /etc/alloy/config.alloy       # syntax and formatting; exit 1 on a diff
alloy run --server.http.listen-addr=127.0.0.1:12345 /etc/alloy/config.alloy   # UI at /  shows every component, its health and discovered targets
curl -fsS http://127.0.0.1:12345/-/ready
curl -fsS http://127.0.0.1:12345/metrics | grep -E '^loki_(source_file_read_bytes_total|write_(sent|dropped)_entries_total)'
```

On Kubernetes the pattern is `discovery.kubernetes "pods"` feeding `discovery.relabel` (to map `__meta_kubernetes_namespace` and friends to `namespace`, `pod`, `container`) into `loki.source.kubernetes`, which reads through the API rather than mounting `/var/log/pods`. The Grafana `k8s-monitoring` Helm chart ships that wiring; see [Kubernetes](https://www.wiki.jodisand.me/kubernetes/) for the cluster side.

## LogQL

A query starts with a stream selector, which must contain at least one non-empty matcher, and continues through a pipeline of stages. Order matters for cost: line filters run before parsers on raw bytes and are cheap; parsers run per line and are expensive; label filters after a parser run on the parsed fields.

```logql
{namespace="prod", app="my-app"}                                   # selector: =, !=, =~, !~ on labels
{namespace="prod", app=~"my-app|my-worker"} |= "timeout"             # line contains
{app="my-app"} |= "timeout" != "healthz"                             # contains, and does not contain
{app="my-app"} |~ "status=5[0-9]{2}"                                 # regexp (RE2, no lookaround)
{app="my-app"} |= ip("192.0.2.0/24")                                 # IP match on the line
{app="my-app"} | json                                                # every JSON key becomes a label; nested keys joined with _
{app="my-app"} | json method="request.method", path="request.path"  # only these, with JMESPath-style paths
{app="my-app"} | logfmt                                              # key=value pairs
{app="nginx"} | regexp `(?P<method>\w+) (?P<path>\S+) HTTP/\S+" (?P<status>\d{3})`   # named groups become labels
{app="nginx"} | pattern `<ip> - - <_> "<method> <path> <_>" <status> <size>`         # cheaper than regexp for fixed layouts
{app="my-app"} | json | level="error" | duration > 500ms             # label filters: string, number, duration, bytes
{app="my-app"} | json | status >= 500 and path != "/healthz"
{app="my-app"} | json | __error__=""                                 # drop lines the parser could not handle
{app="my-app"} | json | line_format "{{.method}} {{.path}} {{.duration}}"   # rewrite the line; Go templates
{app="my-app"} | json | label_format route=`{{ .method }} {{ .path }}` | keep route, status
{app="my-app"} | decolorize | logfmt | drop __error__, __error_details__
{app="my-app"} | trace_id="4bf92f3577b34da6"                         # structured metadata is filtered like a label
```

A parser that fails sets `__error__` (`JSONParserErr`, `LogfmtParserErr`) instead of dropping the line, so filter on `__error__=""` when mixed formats share a stream. Backticks avoid escaping backslashes in regexps.

Metric queries wrap a log query in a range function and return series that Grafana graphs like PromQL:

```logql
rate({app="my-app"} |= "error" [5m])                                  # lines per second, per stream
sum by (app) (rate({namespace="prod"} |= "error" [5m]))               # aggregated
count_over_time({app="my-app"} | json | level="error" [1h])          # lines in the window
sum by (level) (count_over_time({app="my-app"} | logfmt [$__auto]))   # in Grafana: $__auto matches the panel step
bytes_rate({namespace="prod"}[5m])                                    # bytes per second, for finding noisy apps
topk(5, sum by (app) (bytes_over_time({namespace="prod"}[1h])))
# Unwrap turns a parsed numeric label into the sample value
quantile_over_time(0.99, {app="my-app"} | json | unwrap duration_ms [5m]) by (path)
sum by (path) (rate({app="my-app"} | json | unwrap bytes(response_size) [5m]))   # bytes() and duration() convert units
avg_over_time({app="my-app"} | logfmt | unwrap duration(latency) | __error__="" [5m])
absent_over_time({app="my-app"}[10m])                                  # 1 when no lines arrived: a dead-log alert
sum(rate({app="my-app"} |= "error" [5m])) / sum(rate({app="my-app"}[5m])) > 0.05   # error ratio
```

`rate` and `count_over_time` count lines; `bytes_rate` and `bytes_over_time` count bytes; the `_over_time` unwrapped functions (`sum`, `avg`, `min`, `max`, `stddev`, `quantile`, `first`, `last`, `rate` with `unwrap`, `absent`) need a numeric label. Use `offset 1h` after the range to compare with the past. Ruler rules use the same expressions in Prometheus rule-file format under `ruler.storage`.

## logcli

```sh
export LOKI_ADDR=http://loki.example.com:3100
export LOKI_ORG_ID=my-tenant                       # only when auth_enabled: true
logcli labels                                       # label names in the default last hour
logcli labels app --since 24h
logcli series '{namespace="prod"}' --since 1h       # every stream
logcli series '{namespace="prod"}' --analyze-labels # per-label value counts: the high ones are your cardinality problem
logcli query --since 1h --limit 500 '{app="my-app"} |= "error"'                   # default: timestamp, labels, line
logcli query --since 1h -o raw '{app="my-app"} | json | line_format "{{.msg}}"'  # lines only
logcli query --since 1h -o jsonl '{app="my-app"}' | jq -r '.line'
logcli query --from="2026-09-23T22:00:00Z" --to="2026-09-24T02:00:00Z" --timezone=UTC --limit 0 --forward '{app="my-app"}' > incident.log   # --limit 0 removes the cap
logcli query --tail --delay-for 5 '{app="my-app"} |= "error"'      # follow, buffering 5 s for late lines
logcli query --stats --since 1h '{app="my-app"} |~ "timeout"' 2>&1 >/dev/null | grep -E 'Summary|TotalBytesProcessed|ExecTime'   # cost of a query
logcli instant-query 'sum by (app) (count_over_time({namespace="prod"} |= "error" [1h]))'
logcli query --since 6h --step 5m 'sum(rate({app="my-app"} |= "error" [5m]))' -o jsonl   # range metric query
logcli volume '{namespace="prod"}' --since 24h                                           # bytes per stream (volume_enabled: true)
logcli query --parallel-duration=15m --parallel-max-workers=4 --part-path-prefix=/var/tmp/export/my-app --merge-parts --from="2026-09-23T00:00:00Z" --to="2026-09-24T00:00:00Z" '{app="my-app"}'   # large exports in parallel parts
```

The same operations over HTTP: `GET /loki/api/v1/labels`, `GET /loki/api/v1/label/app/values`, `GET /loki/api/v1/series?match[]={app="x"}`, `GET /loki/api/v1/query_range?query=...&start=<ns>&end=<ns>&limit=100&direction=backward`, `GET /loki/api/v1/query?query=<metric query>` for instant, `GET /loki/api/v1/index/volume?query=...`. Times are Unix nanoseconds or RFC3339. `/ready`, `/metrics`, `/config` and `/services` on each component report health.

## Retention and deletion

Retention is applied by the compactor from the `limits_config` values, globally with `retention_period` and per stream with `retention_stream`. Both require `compactor.retention_enabled: true` and `delete_request_store`; without them `retention_period` is silently ignored and storage grows forever.

```yaml
limits_config:
  retention_period: 744h
  retention_stream:
    - selector: '{namespace="dev"}'
      priority: 1
      period: 72h
    - selector: '{app="audit"}'
      priority: 2
      period: 8760h
compactor:
  retention_enabled: true
  delete_request_store: s3
  compaction_interval: 10m
  retention_delete_delay: 2h          # marked chunks are deleted this long after being marked
  retention_delete_worker_count: 150
```

Per-tenant overrides go in the `overrides` file referenced by `runtime_config.file` and hot-reload without a restart. Retention works on chunks, so a chunk containing both 30-day-old and 32-day-old lines from a slow stream is kept until its newest line ages out. Object storage lifecycle rules are not a substitute: deleting chunks under Loki leaves index entries pointing at nothing and queries return errors.

Targeted deletion (a leaked secret in a log line, a GDPR request) uses the delete API, enabled by `limits_config.deletion_mode: filter-and-delete`. The request is processed by the compactor at its next run, and until then the lines still return from queries.

```sh
curl -fsS -X POST -G "$LOKI_ADDR/loki/api/v1/delete" --data-urlencode 'query={app="my-app"} |= "password="' --data-urlencode "start=$(date -d '7 days ago' +%s)" --data-urlencode "end=$(date +%s)"   # deletes matching lines; irreversible once applied
curl -fsS "$LOKI_ADDR/loki/api/v1/delete" | jq .                                                                          # pending requests and status
curl -fsS -X DELETE "$LOKI_ADDR/loki/api/v1/delete?request_id=<id>"                                                       # cancel while still pending
```

## Oneliners

```sh
# Every dashboard as a JSON file, one per uid, into ./dashboards
curl -fsS -H "Authorization: Bearer $GRAFANA_TOKEN" "$G/api/search?type=dash-db" | jq -r '.[].uid' | while read -r u; do curl -fsS -H "Authorization: Bearer $GRAFANA_TOKEN" "$G/api/dashboards/uid/$u" | jq '.dashboard' > "dashboards/$u.json"; done

# Dashboards nobody has opened in 90 days (needs [analytics] enabled; sort by usage in the UI otherwise)
curl -fsS -H "Authorization: Bearer $GRAFANA_TOKEN" "$G/api/search?type=dash-db&sort=views-asc&limit=20" | jq -r '.[] | .title'

# Datasources that fail their health check
curl -fsS -H "Authorization: Bearer $GRAFANA_TOKEN" "$G/api/datasources" | jq -r '.[].uid' | while read -r u; do s=$(curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $GRAFANA_TOKEN" "$G/api/datasources/uid/$u/health"); [[ $s == 200 ]] || echo "$u $s"; done

# Which dashboards use a given datasource uid
grep -l '"uid": "old-prometheus"' dashboards/*.json

# Which dashboards reference a metric
grep -l 'http_requests_total' dashboards/*.json

# Alert rules and their state, with the last error if any
curl -fsS -H "Authorization: Bearer $GRAFANA_TOKEN" "$G/api/prometheus/grafana/api/v1/rules" | jq -r '.data.groups[].rules[] | "\(.state)\t\(.name)\t\(.lastError // "")"'

# Silence one alert for two hours
curl -fsS -X POST -H "Authorization: Bearer $GRAFANA_TOKEN" -H 'Content-Type: application/json' "$G/api/alertmanager/grafana/api/v2/silences" -d "{\"matchers\":[{\"name\":\"alertname\",\"value\":\"my-app 5xx ratio above 5%\",\"isEqual\":true}],\"startsAt\":\"$(date -u +%FT%TZ)\",\"endsAt\":\"$(date -u -d '+2 hours' +%FT%TZ)\",\"createdBy\":\"ops\",\"comment\":\"deploy\"}"

# Grafana version and database status
curl -fsS "$G/api/health" | jq .

# Rows in the Grafana SQLite database that hold dashboards (on the server)
sqlite3 /var/lib/grafana/grafana.db 'select uid, title, updated from dashboard where is_folder = 0 order by updated desc limit 10'

# Loki: streams per app, highest first
logcli series '{namespace="prod"}' --since 1h -q | sed -E 's/.*app="([^"]*)".*/\1/' | sort | uniq -c | sort -rn | head

# Loki: bytes ingested per app over the last day
logcli instant-query 'topk(10, sum by (app) (bytes_over_time({namespace="prod"}[24h])))'

# Loki: apps that stopped logging in the last 30 minutes but logged in the hour before
logcli instant-query 'sum by (app) (count_over_time({namespace="prod"}[1h] offset 30m)) unless sum by (app) (count_over_time({namespace="prod"}[30m]))'

# Loki: error lines per minute for one app, as CSV
logcli query --since 6h --step 1m -o jsonl 'sum(count_over_time({app="my-app"} | json | level="error" [1m]))' | jq -r '.[] | .values[] | @csv' 2>/dev/null || true

# Loki: ingestion rate limits being hit, per tenant (Prometheus query against Loki's metrics)
curl -fsS "$LOKI_ADDR/metrics" | grep -E '^loki_discarded_samples_total' | grep -v ' 0$'

# Loki: ring health (every ingester should be ACTIVE)
curl -fsS "$LOKI_ADDR/ring" | grep -oE '(ACTIVE|LEAVING|PENDING|UNHEALTHY)' | sort | uniq -c

# Loki: flush in-memory chunks before a planned ingester shutdown
curl -fsS -X POST "$LOKI_ADDR/flush"

# Loki: compactor status and retention progress
curl -fsS "$LOKI_ADDR/compactor/ring"; curl -fsS "$LOKI_ADDR/metrics" | grep -E '^loki_compactor_(apply_retention_last_successful_run_timestamp_seconds|deleted_chunks_total)'

# Alloy: components in an unhealthy state
curl -fsS http://127.0.0.1:12345/api/v0/web/components | jq -r '.[] | select(.health.state != "healthy") | "\(.localID)\t\(.health.message)"'

# Alloy: lines dropped by the Loki writer (429s and 400s)
curl -fsS http://127.0.0.1:12345/metrics | grep -E '^loki_write_dropped_entries_total'
```

## Scripts

Back up every Grafana dashboard into a directory tree by folder, suitable for committing.

```sh
#!/usr/bin/env bash
# usage: GRAFANA_TOKEN=... grafana-export.sh http://grafana.example.com:3000 ./dashboards
set -euo pipefail
G=${1:?grafana url} out=${2:-dashboards}
H="Authorization: Bearer ${GRAFANA_TOKEN:?}"
mkdir -p "$out"
curl -fsS -H "$H" "$G/api/search?type=dash-db&limit=5000" \
  | jq -r '.[] | [.uid, (.folderTitle // "General")] | @tsv' \
  | while IFS=$'\t' read -r uid folder; do
      dir="$out/${folder// /-}"
      mkdir -p "$dir"
      curl -fsS -H "$H" "$G/api/dashboards/uid/$uid" | jq '.dashboard | .id = null' > "$dir/$uid.json"
      printf '%s/%s.json\n' "$dir" "$uid"
    done
```

Push every dashboard JSON under a directory to Grafana, creating folders from directory names.

```sh
#!/usr/bin/env bash
# usage: GRAFANA_TOKEN=... grafana-import.sh http://grafana.example.com:3000 ./dashboards
set -euo pipefail
G=${1:?grafana url} src=${2:-dashboards}
H="Authorization: Bearer ${GRAFANA_TOKEN:?}"
J='Content-Type: application/json'
find "$src" -mindepth 2 -name '*.json' | while read -r f; do
  folder=$(basename "$(dirname "$f")")
  fuid=$(curl -fsS -H "$H" "$G/api/folders" | jq -r --arg t "$folder" '.[] | select(.title == $t) | .uid')
  if [[ -z $fuid ]]; then
    fuid=$(curl -fsS -X POST -H "$H" -H "$J" "$G/api/folders" -d "$(jq -n --arg t "$folder" '{title: $t}')" | jq -r .uid)
  fi
  jq --arg fu "$fuid" '{dashboard: (. + {id: null}), folderUid: $fu, overwrite: true, message: "import script"}' "$f" \
    | curl -fsS -X POST -H "$H" -H "$J" "$G/api/dashboards/db" -d @- | jq -r '"\(.status)\t\(.uid)\t\(.url)"'
done
```

Report Loki label cardinality for a selector and flag labels with more than a threshold of values.

```sh
#!/usr/bin/env bash
# usage: LOKI_ADDR=... loki-cardinality.sh '{namespace="prod"}' 100
set -euo pipefail
sel=${1:-'{}'} threshold=${2:-100}
logcli series "$sel" --since 1h --analyze-labels -q \
  | awk -v t="$threshold" 'NR>1 && $2 ~ /^[0-9]+$/ { flag = ($2 > t) ? "HIGH" : ""; printf "%-40s %8s values %6s streams %s\n", $1, $2, $3, flag }' \
  | sort -k2 -nr
```

Check the whole logging path: Alloy healthy, Loki ready, and lines for a job seen in the last five minutes.

```sh
#!/usr/bin/env bash
set -euo pipefail
job=${1:?job label to check}
alloy=${ALLOY_ADDR:-http://127.0.0.1:12345}
: "${LOKI_ADDR:?}"
rc=0
curl -fsS --max-time 5 "$alloy/-/ready" >/dev/null || { echo "alloy not ready"; rc=1; }
curl -fsS --max-time 5 "$LOKI_ADDR/ready" >/dev/null || { echo "loki not ready"; rc=1; }
n=$(logcli instant-query -q "sum(count_over_time({job=\"$job\"}[5m]))" 2>/dev/null | jq -r '.[0].value[1] // "0"')
if [[ ${n%.*} -eq 0 ]]; then echo "no lines for job=$job in 5m"; rc=1; else echo "job=$job: $n lines in 5m"; fi
exit "$rc"
```

## Troubleshooting

| Symptom | Cause | Fix |
| --- | --- | --- |
| Grafana panel shows `No data` | Wrong datasource uid, variable resolving to nothing, or time range | Panel > Inspect > Query shows the exact request and response; check `$namespace` value in the URL |
| Provisioned dashboard missing | Duplicate `uid`, invalid JSON, or wrong `path` | `journalctl -u grafana-server \| grep provisioning`; `jq . file.json` |
| Dashboard edits disappear | File provisioning overwrote them | Edit the file; or `allowUiUpdates: true` and remove the file |
| `Datasource provisioning error: datasource.yaml config is invalid` | Bad YAML or a `jsonData` key for the wrong type | Compare with `GET /api/datasources/uid/<uid>` from a working instance |
| Alert stays `Pending` | `for` not yet elapsed, or evaluation slower than the group interval | Rule state in `/api/prometheus/grafana/api/v1/rules`; shorten the query or move to its own group |
| Alert `Error` state | Query failed | `lastError` in the rules API; usually a datasource timeout or a removed metric |
| Loki: `Data source connected, but no labels were received` | Nothing ingested in the lookback, or the wrong tenant | `logcli labels --since 24h`; check `X-Scope-OrgID` when `auth_enabled: true` |
| Loki: no logs for a new host | Shipper cannot reach Loki, or lines rejected | Alloy UI component health; `loki_write_dropped_entries_total`; Loki `loki_discarded_samples_total{reason=...}` |
| `entry too far behind` / `timestamp too old` | Line older than `reject_old_samples_max_age` (default 7 days), or clock skew | Fix the clock; raise the limit for backfills |
| `per stream rate limit exceeded` (429) | One stream over `per_stream_rate_limit` (3 MB/s default) | Split the stream with another bounded label, or raise the limit |
| `Ingestion rate limit exceeded` (429) | Tenant over `ingestion_rate_mb` | Raise `ingestion_rate_mb` and `ingestion_burst_size_mb`; find the noisy app with `bytes_rate` |
| `Maximum active stream limit exceeded` | Too many streams: a high-cardinality label | `logcli series --analyze-labels`; move the label into the line or structured metadata |
| `maximum of series (500) reached for a single query` | Metric query grouped by a high-cardinality label | Aggregate with `sum by (...)` on fewer labels; `max_query_series` raises the cap at querier memory cost |
| Query slow or times out | Parser before the line filter, wide selector, long range | Put `\|=` before `\| json`; add labels to the selector; check `--stats` bytes processed; raise `split_queries_by_interval` parallelism and querier count |
| `too many outstanding requests` | Query scheduler queue full | Fewer or narrower concurrent queries; more queriers; `max_outstanding_requests_per_tenant` |
| Storage never shrinks | Retention not enabled | `compactor.retention_enabled: true` and `delete_request_store`; confirm with `loki_compactor_apply_retention_last_successful_run_timestamp_seconds` |
| Logs appear twice | Two shippers on the same files, or Alloy restarted without a persisted positions file | One shipper per host; persist `/var/lib/alloy/data` |

## Further reading

- [Grafana provisioning](https://grafana.com/docs/grafana/latest/administration/provisioning/)
- [Grafana HTTP API](https://grafana.com/docs/grafana/latest/developers/http_api/)
- [Grafana alerting file provisioning](https://grafana.com/docs/grafana/latest/alerting/set-up/provision-alerting-resources/file-provisioning/)
- [LogQL reference](https://grafana.com/docs/loki/latest/query/)
- [Loki configuration reference](https://grafana.com/docs/loki/latest/configure/)
- [Loki retention](https://grafana.com/docs/loki/latest/operations/storage/retention/)
- [Alloy Loki components](https://grafana.com/docs/alloy/latest/reference/components/loki/)