# Traefik

> Configure Traefik v3 entrypoints, routers, middlewares and services from Docker, Kubernetes or files, and debug a route that does not match.

Canonical: https://www.wiki.jodisand.me/traefik/
Reviewed: 2026-09-24
Related: [Gateway API](https://www.wiki.jodisand.me/gateway-api/index.md), [TLS and certificates](https://www.wiki.jodisand.me/tls/index.md), [HTTP and curl](https://www.wiki.jodisand.me/http/index.md), [Docker Compose](https://www.wiki.jodisand.me/docker-compose/index.md), [Kubernetes](https://www.wiki.jodisand.me/kubernetes/index.md)


## Cheatsheet

The API commands assume the API is reachable on `localhost:8080`, which is the `traefik` entrypoint that `api.insecure=true` creates. On a secured setup, use the dashboard hostname and credentials instead.

| Task | Command or setting |
| --- | --- |
| What routers exist | `curl -s localhost:8080/api/http/routers \| jq -r '.[] \| [.name, .rule, .status] \| @tsv'` |
| Routers that failed, with the error | `curl -s localhost:8080/api/http/routers \| jq '.[] \| select(.status!="enabled")'` |
| Services and their servers | `curl -s localhost:8080/api/http/services \| jq` |
| Providers and object counts | `curl -s localhost:8080/api/overview \| jq` |
| Traefik version | `curl -s localhost:8080/api/version` |
| Debug logs | `--log.level=DEBUG` |
| JSON access logs | `--accesslog=true --accesslog.format=json` |
| Route a container | label ``traefik.http.routers.app.rule=Host(`app.example.com`)`` |
| Host plus path prefix | ``Host(`app.example.com`) && PathPrefix(`/api`)`` |
| Pick the container port | `traefik.http.services.app.loadbalancer.server.port=8080` |
| Strip a prefix | `traefik.http.middlewares.strip.stripprefix.prefixes=/api` |
| Redirect HTTP to HTTPS | entrypoint `http.redirections.entryPoint.to=websecure` |
| Test a route without DNS | `curl -vk --resolve app.example.com:443:127.0.0.1 https://app.example.com/` |

## How Traefik routes a request

A request arrives at an **entrypoint** (a listening port), is matched by a **router** (a rule on host, path, headers and so on), passes through the router's **middlewares** in order, and reaches a **service**, which load balances across servers.

Configuration has two halves:

| Kind | Contains | When it is read |
| --- | --- | --- |
| Install (static) configuration | Entrypoints, providers, certificate resolvers, API, logs, metrics | Once at start-up, from file, CLI flags or environment. Changes need a restart |
| Routing (dynamic) configuration | Routers, middlewares, services, TLS options | Continuously from **providers** (Docker labels, Kubernetes CRDs, Ingress, Gateway API, files) and applied without restart |

Objects are named `<name>@<provider>`, for example `app@docker` or `secure-headers@file`. Referencing an object from another provider needs the full name.

A route that "does not work" is almost always one of four things, and the API shows each:

1. The provider never produced the router (container not enabled, labels wrong, CRD in an unwatched namespace).
2. The router exists but its rule does not match the request, or a higher-priority router matches first.
3. A middleware rejected or rewrote the request.
4. The service has no healthy servers.

```sh
curl -s localhost:8080/api/overview | jq                     # counts, warnings, errors per type and enabled providers
curl -s localhost:8080/api/http/routers | jq -r '.[] | [.name, .rule, .status] | @tsv'
curl -s localhost:8080/api/http/services | jq -r '.[] | [.name, .status, ((.serverStatus // {}) | tostring)] | @tsv'
curl -s localhost:8080/api/http/middlewares | jq -r '.[] | [.name, .status] | @tsv'
curl -s localhost:8080/api/rawdata | jq                      # the full dynamic config Traefik is running
```

## Static configuration

```yaml
# traefik.yaml, read once at start-up
entryPoints:
  web:
    address: ":80"
    http:
      redirections:
        entryPoint: { to: websecure, scheme: https, permanent: true }
  websecure:
    address: ":443"
    http:
      tls:
        certResolver: letsencrypt
    transport:
      respondingTimeouts: { readTimeout: 60s, writeTimeout: 0s, idleTimeout: 180s }   # v3 defaults
    forwardedHeaders:
      trustedIPs: ["198.51.100.0/24"]   # only when a load balancer in front sets X-Forwarded-For

providers:
  docker:
    exposedByDefault: false          # containers need traefik.enable=true
  kubernetesCRD:
    allowCrossNamespace: false
  file:
    directory: /etc/traefik/dynamic
    watch: true

certificatesResolvers:
  letsencrypt:
    acme:
      email: ops@example.com
      storage: /data/acme.json       # must persist across restarts; keep mode 600
      tlsChallenge: {}

api:
  dashboard: true
ping: {}
accessLog:
  format: json
  filters: { statusCodes: ["400-599"], retryAttempts: true, minDuration: 1s }
log:
  level: INFO
```

`exposedByDefault` defaults to `true` on the Docker provider, which makes every container with a reachable port routable. Set it to `false` and opt in per container with `traefik.enable=true`.

`readTimeout` covers reading the whole request, body included. It defaults to 60 s since v2.11.2 (before that there was no limit), so long uploads over a slow link are cut off. Raise it on the entrypoint that receives them.

ACME challenge types:

| Challenge | Config | Requirement |
| --- | --- | --- |
| TLS-ALPN-01 | `tlsChallenge: {}` | Port 443 reachable from the internet on this Traefik |
| HTTP-01 | `httpChallenge: { entryPoint: web }` | Port 80 reachable from the internet |
| DNS-01 | `dnsChallenge: { provider: <name> }` plus provider credentials in env | Needed for wildcards and internal-only hosts |

Traefik renews certificates 30 days before expiry. Point `caServer` at the Let's Encrypt staging directory while testing to avoid production rate limits. See [TLS](https://www.wiki.jodisand.me/tls/) for inspecting the result.

## Routers and rules

```yaml
http:
  routers:
    api:
      rule: "Host(`api.example.com`) && PathPrefix(`/v1`)"
      entryPoints: [websecure]
      middlewares: [ratelimit, secure-headers]
      service: api
      priority: 100
      tls: { certResolver: letsencrypt }
```

| Matcher | Example |
| --- | --- |
| `Host` | ``Host(`api.example.com`)`` |
| `HostRegexp` | ``HostRegexp(`^.+\.example\.com$`)`` |
| `Path` | ``Path(`/healthz`)`` |
| `PathPrefix` | ``PathPrefix(`/api`)`` |
| `PathRegexp` | ``PathRegexp(`^/v[0-9]+/`)`` |
| `Header` / `HeaderRegexp` | ``Header(`X-Env`, `prod`)`` |
| `Query` / `QueryRegexp` | ``Query(`debug`, `true`)``, ``Query(`debug`)`` |
| `Method` | ``Method(`POST`)`` |
| `ClientIP` | ``ClientIP(`192.0.2.0/24`)`` |

Combine with `&&`, `||`, `!` and parentheses. Regex matchers use Go regexp syntax.

v3 changed the rule syntax. Each matcher takes one value, so v2's ``Host(`a.example.com`, `b.example.com`)`` becomes two `Host` calls joined with `||`; `Headers` became `Header`; `HostHeader` is gone; and `HostRegexp` takes a real regular expression instead of `{name:pattern}` placeholders. Setting `core.defaultRuleSyntax: v2` or a per-router `ruleSyntax: v2` keeps old rules working during a migration. See the [v2 to v3 migration guide](https://doc.traefik.io/traefik/migrate/v2-to-v3/).

Priority defaults to the length of the rule string, so longer rules win. Set `priority` explicitly whenever two routers can match the same request: relying on length is how a catch-all silently takes traffic from a specific route.

`ClientIP` matches the address of the TCP peer and never reads `X-Forwarded-For`. Behind a load balancer it sees the balancer's address.

## Services

```yaml
http:
  services:
    api:
      loadBalancer:
        servers:
          - url: "http://192.0.2.5:8080"
          - url: "http://192.0.2.6:8080"
        healthCheck: { path: /healthz, interval: 10s, timeout: 3s }
        sticky:
          cookie: { name: srv, httpOnly: true, secure: true, sameSite: lax }
    api-split:
      weighted:
        services:
          - { name: api-v1, weight: 90 }
          - { name: api-v2, weight: 10 }
    api-mirror:
      mirroring:
        service: api-v1
        mirrors: [{ name: api-v2, percent: 10 }]
```

A failing health check removes that server from rotation until it passes again; a service whose servers all fail returns `503`. `weighted` splits traffic by ratio, which is the canary mechanism. `mirroring` sends a copy of requests to the mirror and discards its responses, so a new version can take real traffic with no user impact. Weighted and mirroring services cannot be defined with Docker labels; use the file provider or Kubernetes CRDs.

## Middlewares

Middlewares run in the order listed on the router. Put authentication before rate limiting to protect the backend from unauthenticated floods, or rate limiting first to protect the auth service itself.

```yaml
http:
  middlewares:
    secure-headers:
      headers:
        stsSeconds: 31536000
        frameDeny: true
        contentTypeNosniff: true
        referrerPolicy: no-referrer
    ratelimit:
      rateLimit: { average: 100, burst: 200, period: 1s }
    basic-auth:
      basicAuth: { usersFile: /etc/traefik/users }        # htpasswd format
    forward-auth:
      forwardAuth:
        address: http://auth:4180/verify
        authResponseHeaders: [X-Auth-User, X-Auth-Groups]
    strip:
      stripPrefix: { prefixes: ["/api"] }
    retry:
      retry: { attempts: 3, initialInterval: 100ms }
    circuit:
      circuitBreaker: { expression: "NetworkErrorRatio() > 0.30 || ResponseCodeRatio(500, 600, 0, 600) > 0.25" }
    compress:
      compress: {}
    allowlist:
      ipAllowList:
        sourceRange: ["192.0.2.0/24", "198.51.100.0/24"]
        ipStrategy: { depth: 1 }     # only when behind one trusted proxy; see below
```

`ipAllowList` (the renamed `ipWhiteList`, whose old name v3 removed) matches `sourceRange` against the TCP peer address unless `ipStrategy` is set. Behind a load balancer every request comes from the balancer, so either set `ipStrategy.depth` to pick the client address from `X-Forwarded-For` (counting from the right) or list the proxies in `ipStrategy.excludedIPs`. Only do this when the header is set by a proxy you control; otherwise clients can forge it.

`retry` resends a request when the connection to a server fails; it does not retry on HTTP error status codes.

### Middleware catalogue

| Middleware | Purpose | Key options |
| --- | --- | --- |
| `addPrefix` | Prepend a path before forwarding | `prefix` |
| `stripPrefix` / `stripPrefixRegex` | Remove a path prefix; the original is passed in `X-Forwarded-Prefix` | `prefixes`, `regex` |
| `replacePath` / `replacePathRegex` | Rewrite the whole path; original kept in `X-Replaced-Path` | `path`, `regex`, `replacement` |
| `redirectScheme` | Redirect to another scheme or port | `scheme`, `port`, `permanent` |
| `redirectRegex` | Redirect by regex on the full URL | `regex`, `replacement`, `permanent` |
| `headers` | Set, add or remove request and response headers; security headers; CORS | `customRequestHeaders`, `customResponseHeaders`, `accessControlAllow*`, `stsSeconds`, `frameDeny` |
| `basicAuth` / `digestAuth` | Static credential check | `users`, `usersFile`, `realm`, `headerField`, `removeHeader` |
| `forwardAuth` | Delegate the decision to an HTTP service; 2xx allows, anything else is returned to the client | `address`, `authResponseHeaders`, `authRequestHeaders`, `trustForwardHeader`, `forwardBody` |
| `ipAllowList` | Allow by source range | `sourceRange`, `ipStrategy` |
| `rateLimit` | Token bucket per source | `average`, `burst`, `period`, `sourceCriterion` (IP strategy, request header or host) |
| `inFlightReq` | Cap concurrent requests per source | `amount`, `sourceCriterion` |
| `circuitBreaker` | Trip on error or latency ratios, then recover gradually | `expression`, `checkPeriod`, `fallbackDuration`, `recoveryDuration` |
| `retry` | Retry on connection failure | `attempts`, `initialInterval` |
| `buffering` | Buffer bodies, cap request size, retry on body errors | `maxRequestBodyBytes`, `memRequestBodyBytes`, `retryExpression` |
| `compress` | gzip, br and zstd responses | `excludedContentTypes`, `minResponseBodyBytes`, `encodings` |
| `errors` | Serve custom pages for status ranges from another service | `status`, `service`, `query` |
| `chain` | Reuse an ordered list of middlewares under one name | `middlewares` |
| `contentType` | Control auto-detection of `Content-Type` | `autoDetect` |
| `passTLSClientCert` | Forward client certificate details in `X-Forwarded-Tls-Client-Cert` | `pem`, `info` |
| `grpcWeb` | Translate gRPC-Web to gRPC | `allowOrigins` |
| `encodedCharacters` | Allow or reject percent-encoded characters in request paths | per-character allow list; see the reference for the exact keys |

`rateLimit` counts per `sourceCriterion`, which defaults to the client IP with the same `ipStrategy` caveat as `ipAllowList`; behind a load balancer without `depth` every client shares one bucket. `buffering.maxRequestBodyBytes` is the upload limit and returns 413 when exceeded. A `chain` is the cleanest way to give every router the same security stack:

```yaml
http:
  middlewares:
    public:
      chain:
        middlewares: [secure-headers, ratelimit, compress]
    upload-limit:
      buffering: { maxRequestBodyBytes: 104857600 }   # 100 MiB, 413 above
    cors:
      headers:
        accessControlAllowOriginList: ["https://app.example.com"]
        accessControlAllowMethods: [GET, POST, OPTIONS]
        accessControlAllowHeaders: [Authorization, Content-Type]
        accessControlMaxAge: 600
        addVaryHeader: true
    inflight:
      inFlightReq: { amount: 50 }
    error-pages:
      errors:
        status: ["500-599"]
        service: error-pages
        query: "/{status}.html"
```

## TLS options

`tls.options` is dynamic configuration (file provider or the `TLSOption` CRD) that sets protocol versions, ciphers and client authentication per router. The option named `default` applies to every TLS router that does not name one, and to the default certificate.

```yaml
tls:
  options:
    default:
      minVersion: VersionTLS12
      sniStrict: true                          # refuse connections whose SNI matches no certificate instead of serving the default
      cipherSuites:                            # TLS 1.2 only; TLS 1.3 suites are not configurable in Go
        - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
        - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
        - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
      curvePreferences: [X25519, CurveP256]
      alpnProtocols: [h2, http/1.1]
    mtls:
      minVersion: VersionTLS13
      clientAuth:
        caFiles: [/etc/traefik/certs/client-ca.pem]
        clientAuthType: RequireAndVerifyClientCert   # NoClientCert, RequestClientCert, RequireAnyClientCert, VerifyClientCertIfGiven
  certificates:                                # static certificates, matched to SNI automatically
    - certFile: /etc/traefik/certs/example.com.crt
      keyFile: /etc/traefik/certs/example.com.key
  stores:
    default:
      defaultCertificate:                      # served when no certificate matches and sniStrict is off
        certFile: /etc/traefik/certs/default.crt
        keyFile: /etc/traefik/certs/default.key
http:
  routers:
    admin:
      rule: "Host(`admin.example.com`)"
      service: admin
      tls: { options: mtls }                   # mtls@file from another provider
      middlewares: [client-cert-info]
  middlewares:
    client-cert-info:
      passTLSClientCert: { info: { subject: { commonName: true, organization: true } } }
```

Two routers on the same host with different `options` is a conflict: TLS options are selected by SNI before any HTTP routing, so Traefik logs an error and falls back to `default`. The same rule means options cannot differ by path. Traefik-to-backend TLS is a separate object, `serversTransport`, referenced from the service:

```yaml
http:
  serversTransports:
    internal-ca:
      serverName: my-app.internal              # SNI and name verified on the backend certificate
      rootCAs: [/etc/traefik/certs/internal-ca.pem]
      certificates: [{ certFile: /etc/traefik/certs/traefik-client.crt, keyFile: /etc/traefik/certs/traefik-client.key }]
      insecureSkipVerify: false
      maxIdleConnsPerHost: 200
      forwardingTimeouts: { dialTimeout: 5s, responseHeaderTimeout: 30s, idleConnTimeout: 90s }
  services:
    my-app:
      loadBalancer:
        serversTransport: internal-ca
        servers: [{ url: "https://192.0.2.5:8443" }]
```

Verify what a router serves with `openssl s_client -connect 127.0.0.1:443 -servername admin.example.com -tls1_2 </dev/null`; a handshake failure with `-tls1_2` and success without it confirms `minVersion: VersionTLS13` took effect. See [TLS](https://www.wiki.jodisand.me/tls/) for the rest of the inspection toolkit.

## Dynamic configuration in files

The file provider is the one that works everywhere: no labels, no CRDs, and every object in one place for review. With `watch: true` Traefik reloads on change; a file with a syntax error is rejected as a whole and the previous configuration stays in force, so check `traefik_config_last_reload_success` after editing.

```yaml
# /etc/traefik/dynamic/apps.yaml
http:
  routers:
    app:
      rule: "Host(`app.example.com`)"
      entryPoints: [websecure]
      middlewares: [public@file]
      service: app
      tls: { certResolver: letsencrypt, domains: [{ main: example.com, sans: ["*.example.com"] }] }   # wildcard via DNS-01
    app-canary:
      rule: "Host(`app.example.com`) && Header(`X-Canary`, `always`)"
      entryPoints: [websecure]
      priority: 200
      service: app-v2
      tls: {}
    legacy:
      rule: "Host(`old.example.com`)"
      entryPoints: [websecure]
      middlewares: [to-new]
      service: noop@internal                   # a redirect needs no backend
      tls: {}
  middlewares:
    to-new:
      redirectRegex:
        regex: "^https://old\\.example\\.com/(.*)"
        replacement: "https://app.example.com/${1}"
        permanent: true
  services:
    app:
      weighted:
        services: [{ name: app-v1, weight: 95 }, { name: app-v2, weight: 5 }]
    app-v1:
      loadBalancer:
        servers: [{ url: "http://192.0.2.5:8080" }, { url: "http://192.0.2.6:8080" }]
        healthCheck: { path: /healthz, interval: 10s, timeout: 3s, followRedirects: false }
        passHostHeader: true
    app-v2:
      loadBalancer:
        servers: [{ url: "http://192.0.2.7:8080" }]
tcp:
  routers:
    db:
      rule: "HostSNI(`db.example.com`)"
      entryPoints: [postgres]                   # a TCP entrypoint declared in static config, address ":5432"
      service: db
      tls: { passthrough: true }                # SNI routing without terminating
  services:
    db:
      loadBalancer:
        servers: [{ address: "192.0.2.20:5432" }]
```

Objects defined in files can be referenced from Docker labels or CRDs by their full name (`public@file`), which is how one shared middleware chain serves containers on a Compose host. Environment variables in the file are expanded with `{{ env "VAR" }}` Go templating when the file is loaded, and a `directory` of files is merged, so one file per application keeps diffs small. `noop@internal` is the built-in service for routers that only redirect; `api@internal`, `dashboard@internal` and `ping@internal` are the others.

`HostSNI(`*`)` is the catch-all TCP rule and the only rule possible for non-TLS TCP, since without TLS there is no SNI to inspect. UDP routers have no rules at all: one router per entrypoint. For a TCP router with `tls: { passthrough: true }`, health checks against the backend must be TCP-level too, which the `tcp` load balancer does not do; put the backend behind something that removes dead members itself.

## Kubernetes

```yaml
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata: { name: api, namespace: my-namespace }
spec:
  entryPoints: [websecure]
  routes:
    - match: Host(`api.example.com`) && PathPrefix(`/v1`)
      kind: Rule
      priority: 100
      middlewares:
        - { name: secure-headers }
      services:
        - { name: api, port: 80 }
  tls: { certResolver: letsencrypt }
```

```yaml
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata: { name: secure-headers, namespace: my-namespace }
spec:
  headers: { stsSeconds: 31536000, frameDeny: true }
```

v3 serves CRDs only under the `traefik.io` API group; the old `traefik.containo.us` group was removed, so upgrade the CRDs and manifests together. A CRD object referencing a middleware or service in another namespace needs `namespace:` on the reference and `allowCrossNamespace: true` on the provider. From a plain `Ingress`, attach a CRD middleware with the annotation `traefik.ingress.kubernetes.io/router.middlewares: my-namespace-secure-headers@kubernetescrd`.

Traefik also implements Gateway API through the `kubernetesGateway` provider. Use it when manifests should stay portable between controllers; see [Gateway API](https://www.wiki.jodisand.me/gateway-api/).

### The CRD set

| Kind | Purpose |
| --- | --- |
| `IngressRoute` | HTTP router: rules, middlewares, services, TLS |
| `IngressRouteTCP` / `IngressRouteUDP` | TCP by SNI (or `HostSNI(`*`)`) and UDP by entrypoint |
| `Middleware` / `MiddlewareTCP` | Any middleware from the catalogue, namespaced |
| `TraefikService` | Weighted round robin or mirroring across Services or other TraefikServices |
| `TLSOption` | TLS versions, ciphers, client auth; `default` in the Traefik namespace applies globally |
| `TLSStore` | Default certificate from a Secret |
| `ServersTransport` / `ServersTransportTCP` | Traefik-to-backend TLS and timeouts |

```yaml
apiVersion: traefik.io/v1alpha1
kind: TraefikService
metadata: { name: app-split, namespace: my-namespace }
spec:
  weighted:
    services:
      - { name: app-v1, port: 80, weight: 95 }
      - { name: app-v2, port: 80, weight: 5 }
    sticky: { cookie: { name: app, secure: true, httpOnly: true } }
---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata: { name: app, namespace: my-namespace }
spec:
  entryPoints: [websecure]
  routes:
    - match: Host(`app.example.com`) && Header(`X-Canary`, `always`)
      kind: Rule
      priority: 200
      services: [{ name: app-v2, port: 80 }]
    - match: Host(`app.example.com`)
      kind: Rule
      middlewares: [{ name: public, namespace: traefik }]     # cross-namespace needs allowCrossNamespace
      services:
        - { name: app-split, kind: TraefikService }             # kind defaults to Service
  tls:
    secretName: app-example-com-tls                             # cert-manager Secret; omit for certResolver
    options: { name: strict, namespace: traefik }
---
apiVersion: traefik.io/v1alpha1
kind: TLSOption
metadata: { name: strict, namespace: traefik }
spec:
  minVersion: VersionTLS13
  sniStrict: true
---
apiVersion: traefik.io/v1alpha1
kind: ServersTransport
metadata: { name: internal-ca, namespace: my-namespace }
spec:
  serverName: my-app.my-namespace.svc
  rootCAsSecrets: [internal-ca]                                 # Secret with ca.crt
---
apiVersion: traefik.io/v1alpha1
kind: IngressRouteTCP
metadata: { name: db, namespace: my-namespace }
spec:
  entryPoints: [postgres]
  routes:
    - match: HostSNI(`db.example.com`)
      services: [{ name: my-db, port: 5432 }]
  tls: { passthrough: true }
```

Services referenced from an `IngressRoute` are Kubernetes Services in the same namespace by default; Traefik load balances across the endpoints directly rather than through the ClusterIP unless `nativeLB: true` is set on the service reference, so `externalTrafficPolicy` and kube-proxy do not apply. A `Middleware` referenced from an `IngressRoute` in the same namespace needs only `name`; from a Docker label or file it is `my-namespace-public@kubernetescrd`. Traefik 3 also reads Gateway API `HTTPRoute`, `GRPCRoute`, `TCPRoute` and `TLSRoute` when the `kubernetesGateway` provider is on, and the Helm chart (`traefik/traefik`) installs the CRDs and both providers; `helm show values traefik/traefik | grep -A5 providers` shows the switches.

```sh
kubectl get ingressroute,ingressroutetcp,middleware,traefikservice,tlsoption -A
kubectl get crd | grep traefik.io                                  # v3 group; traefik.containo.us means old CRDs
kubectl logs -n traefik deploy/traefik | grep -iE 'error|warn' | tail -20
kubectl port-forward -n traefik deploy/traefik 8080:8080           # API on localhost:8080 when api.insecure is off but the port exists
```

## Docker labels

```yaml
services:
  app:
    image: my-app:1.0
    labels:
      - traefik.enable=true
      - traefik.http.routers.app.rule=Host(`app.example.com`)
      - traefik.http.routers.app.entrypoints=websecure
      - traefik.http.routers.app.tls.certresolver=letsencrypt
      - traefik.http.routers.app.middlewares=app-strip
      - traefik.http.middlewares.app-strip.stripprefix.prefixes=/api
      - traefik.http.services.app.loadbalancer.server.port=8080
      - traefik.docker.network=edge
    networks: [edge]
```

Traefik connects to the container's IP on a network they share. Set `traefik.docker.network` when the container is on more than one network, or Traefik may pick one it cannot reach. `loadbalancer.server.port` is required when the image exposes several ports or none. In Compose files, a `$` in label values (for example a bcrypt hash in `basicauth.users`) must be written as `$$`. See [Docker Compose](https://www.wiki.jodisand.me/docker-compose/).

## Dashboard and API

`api.insecure=true` publishes the API and dashboard without authentication on the `traefik` entrypoint (port 8080). Use it only on a host where that port is firewalled. In production, route to the internal service with authentication:

```yaml
http:
  routers:
    dashboard:
      rule: "Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
      entryPoints: [websecure]
      service: api@internal
      middlewares: [basic-auth]
      tls: { certResolver: letsencrypt }
```

The dashboard lives at `/dashboard/` and the trailing slash is required.

## Observability

```yaml
metrics:
  prometheus:
    buckets: [0.1, 0.3, 1.2, 5.0]      # the default
    addEntryPointsLabels: true         # default true
    addRoutersLabels: true             # default false
    addServicesLabels: true            # default true
tracing:
  otlp:
    http: { endpoint: http://otel-collector:4318/v1/traces }
```

Useful series: `traefik_service_requests_total`, `traefik_service_request_duration_seconds_bucket`, `traefik_service_server_up`, `traefik_open_connections`, `traefik_config_last_reload_success` and `traefik_tls_certs_not_after`. v3 replaced v2's `traefik_entrypoint_open_connections` and `traefik_service_open_connections` with the single `traefik_open_connections`. See [Prometheus](https://www.wiki.jodisand.me/prometheus/) and [OpenTelemetry](https://www.wiki.jodisand.me/opentelemetry/).

```promql
# 5xx ratio per service
sum by (service) (rate(traefik_service_requests_total{code=~"5.."}[5m]))
  / sum by (service) (rate(traefik_service_requests_total[5m]))
```

```promql
# Days until each certificate expires
(traefik_tls_certs_not_after - time()) / 86400
```

Metrics can also go to an OTLP endpoint (`metrics.otlp.http.endpoint`) or StatsD/Datadog/InfluxDB, and the same `add*Labels` switches apply. `addRoutersLabels` multiplies series by router and is off by default for that reason; turn it on only when routers are few. Series worth alerting on:

| Series | Meaning |
| --- | --- |
| `traefik_config_last_reload_success` | 0 after a rejected dynamic config; the previous config stays active |
| `traefik_config_last_reload_failure` | Timestamp of the last failed reload |
| `traefik_service_server_up{service,url}` | 0 when a health check removed the server |
| `traefik_entrypoint_requests_total{code,method,protocol}` | Volume per entrypoint before routing |
| `traefik_router_requests_total` | Per-router volume (needs `addRoutersLabels`) |
| `traefik_service_requests_total{code}` and `_duration_seconds_bucket` | Per-service status and latency |
| `traefik_service_retries_total` | Retry middleware activity; rising means backends refusing connections |
| `traefik_open_connections{entrypoint,protocol}` | Current connections |
| `traefik_tls_certs_not_after{cn,sans,serial}` | Certificate expiry as a Unix timestamp |
| `traefik_entrypoint_requests_bytes_total`, `_responses_bytes_total` | Throughput |

```promql
# Backends removed by health checks right now
traefik_service_server_up == 0

# p95 latency per service
histogram_quantile(0.95, sum by (le, service) (rate(traefik_service_request_duration_seconds_bucket[5m])))

# Certificates expiring within 14 days
(traefik_tls_certs_not_after - time()) / 86400 < 14

# Requests that matched no router (404 at the entrypoint, not from a service)
sum(rate(traefik_entrypoint_requests_total{code="404"}[5m])) - sum(rate(traefik_service_requests_total{code="404"}[5m]))
```

```yaml
# Alerting rules for the essentials
groups:
  - name: traefik
    rules:
      - alert: TraefikConfigReloadFailed
        expr: traefik_config_last_reload_success == 0
        for: 5m
        labels: { severity: page }
      - alert: TraefikBackendDown
        expr: traefik_service_server_up == 0
        for: 2m
        labels: { severity: ticket }
      - alert: TraefikCertificateExpiring
        expr: (traefik_tls_certs_not_after - time()) / 86400 < 14
        labels: { severity: ticket }
      - alert: TraefikHighErrorRatio
        expr: sum by (service) (rate(traefik_service_requests_total{code=~"5.."}[5m])) / sum by (service) (rate(traefik_service_requests_total[5m])) > 0.05
        for: 10m
        labels: { severity: page }
```

## Running more than one instance

Traefik Proxy stores ACME state in a local `acme.json` and has no shared store, and an ACME challenge can land on an instance that did not request it. The documentation states that several instances cannot run Let's Encrypt together. For more than one replica, issue certificates with cert-manager into Kubernetes Secrets (or terminate TLS upstream) and let every Traefik replica read them.

## Troubleshooting

| Symptom | Cause | Check |
| --- | --- | --- |
| `404 page not found` (plain text) | No router matched: wrong host, path, entrypoint, or router never created | `/api/http/routers`; is the router listed and `enabled`? |
| Router missing entirely | Provider did not produce it: `traefik.enable` absent, label typo, wrong namespace or ingress class | `/api/overview` provider list; DEBUG logs name the rejected object |
| Router status `disabled` with an error | Rule syntax error (often v2 syntax on v3), unknown middleware or service reference | `.error` field in `/api/http/routers` |
| `502 Bad Gateway` | Traefik reached no server: wrong port, container on a network Traefik is not on, server refused | `/api/http/services` server URLs; `loadbalancer.server.port`; `traefik.docker.network` |
| `503 Service Unavailable` | Every server failed its health check, or the service has no servers | `serverStatus` in the service API |
| `504 Gateway Timeout` | Backend slower than the forwarding timeouts | `serversTransport` `forwardingTimeouts` |
| Default "TRAEFIK DEFAULT CERT" served | ACME failed, or the router has no `tls` / `certResolver` | Logs at DEBUG for `acme`; challenge port reachable from the internet |
| Wrong router handles the request | Higher priority (longer rule) router matched first | Compare `priority` values in the router API |
| `ipAllowList` blocks everyone behind a load balancer | Middleware sees the balancer IP | Set `ipStrategy.depth` or `excludedIPs` |
| Redirect loop to HTTPS | TLS terminated upstream, request arrives on `web` again | Remove the entrypoint redirect and let the upstream balancer redirect, or send upstream traffic to `websecure` |
| Long uploads cut at 60 s | Default `readTimeout: 60s` (since v2.11.2) | Raise `transport.respondingTimeouts.readTimeout` |
| Dynamic file edited, nothing changed | File rejected as a whole; previous config kept | `traefik_config_last_reload_success`, logs for `Error while parsing`; `watch: true` on the provider |
| `413 Request Entity Too Large` | `buffering.maxRequestBodyBytes` on a middleware in the chain | Router's middleware list; raise or remove the limit |
| TLS options ignored, `default` used | Two routers on the same host name different options, or option name lacks `@provider` | Logs for `conflicting TLS options`; use `name@file` / `name@kubernetescrd` |
| Handshake fails only for some clients | `minVersion` or `cipherSuites` too strict, or `sniStrict` with a client that sends no SNI | `openssl s_client -tls1_2`, without `-servername` |
| mTLS router accepts any certificate | `clientAuthType` is `RequestClientCert` or `VerifyClientCertIfGiven` | Set `RequireAndVerifyClientCert`; check `caFiles` |
| Backend TLS error `x509: certificate signed by unknown authority` | Traefik verifies the backend against system CAs | `serversTransport.rootCAs`, or `serverName` for SNI mismatch |
| Rate limit hits everyone at once | All requests share one bucket: balancer IP as the source | `sourceCriterion.ipStrategy.depth`, or `requestHeaderName` |
| `IngressRoute` accepted, `404` | Wrong `entryPoints` name, or Service port name/number mismatch | `/api/http/routers` shows `entryPoints`; compare `port` with the Service |
| Middleware from another namespace not found | `allowCrossNamespace` is false | Set it on the provider, or move the middleware |
| Every request logged as `-` router with 404 | Router matched but the entrypoint is wrong, or the request arrived on `web` after a redirect loop | Access log `RouterName` and `entryPointName` fields |
| Traefik pod ready but Gateway API routes ignored | `kubernetesGateway` provider off, or GatewayClass `controllerName` mismatch | `providers.kubernetesGateway` in values; `kubectl get gatewayclass` |

```sh
docker logs traefik 2>&1 | grep -iE 'error|level=error' | tail -20     # provider and ACME errors
curl -vk --resolve app.example.com:443:127.0.0.1 https://app.example.com/v1/healthz   # exact SNI and Host, bypassing DNS
curl -s localhost:8080/ping                                               # prints OK when Traefik is healthy (needs ping enabled)
```

## Oneliners

```sh
# Routers that are not enabled, with their error
curl -s localhost:8080/api/http/routers | jq -r '.[] | select(.status!="enabled") | [.name, (.error // [] | join("; "))] | @tsv'

# Services with zero servers
curl -s localhost:8080/api/http/services | jq -r '.[] | select((.loadBalancer.servers // []) | length == 0) | .name'

# Routers whose rule mentions a host
curl -s localhost:8080/api/http/routers | jq -r '.[] | select(.rule | test("api.example.com")) | [.name, .priority, .rule] | @tsv'

# Certificate domains stored by the "letsencrypt" resolver
jq -r '.letsencrypt.Certificates[].domain.main' /data/acme.json

# Expiry of the first stored certificate
jq -r '.letsencrypt.Certificates[0].certificate' /data/acme.json | base64 -d | openssl x509 -noout -enddate

# Tail access logs for 5xx only
tail -f /var/log/traefik/access.log | jq -r 'select(.DownstreamStatus >= 500) | [.StartUTC, .RouterName, .DownstreamStatus, .RequestPath] | @tsv'

# Slowest requests in a log file (Duration is in nanoseconds)
jq -r '[.Duration/1000000, .RouterName, .RequestPath] | @tsv' access.log | sort -rn | head

# Requests per router
jq -r '.RouterName' access.log | sort | uniq -c | sort -rn | head

# Traefik labels on a container
docker inspect my-app | jq '.[0].Config.Labels | with_entries(select(.key | startswith("traefik")))'

# Routers by priority, highest first (who wins a conflict)
curl -s localhost:8080/api/http/routers | jq -r '.[] | [.priority // 0, .name, .rule] | @tsv' | sort -rn

# Routers with no TLS on the websecure entrypoint
curl -s localhost:8080/api/http/routers | jq -r '.[] | select((.entryPoints // []) | index("websecure")) | select(.tls == null) | .name'

# Every middleware in use and the routers using it
curl -s localhost:8080/api/http/routers | jq -r '.[] | .name as $r | (.middlewares // [])[] | "\(.)\t\($r)"' | sort

# Middlewares defined but referenced by no router
comm -23 <(curl -s localhost:8080/api/http/middlewares | jq -r '.[].name' | sort) <(curl -s localhost:8080/api/http/routers | jq -r '.[].middlewares[]?' | sort -u)

# Servers currently marked down by health checks
curl -s localhost:8080/api/http/services | jq -r '.[] | .name as $s | (.serverStatus // {}) | to_entries[] | select(.value != "UP") | "\($s)\t\(.key)\t\(.value)"'

# Per-provider object counts and warnings
curl -s localhost:8080/api/overview | jq '{http: .http, tcp: .tcp, providers}'

# Entry points and their addresses
curl -s localhost:8080/api/entrypoints | jq -r '.[] | [.name, .address] | @tsv'

# TLS options and the routers that use each
curl -s localhost:8080/api/http/routers | jq -r '.[] | [(.tls.options // "default"), .name] | @tsv' | sort

# TCP and UDP routers
curl -s localhost:8080/api/tcp/routers | jq -r '.[] | [.name, .rule, .status] | @tsv'; curl -s localhost:8080/api/udp/routers | jq -r '.[] | [.name, .status] | @tsv'

# Full dynamic config, saved for a diff after a change
curl -s localhost:8080/api/rawdata | jq -S . > rawdata-before.json

# Does a request match a router: send it and read the router from the access log
curl -s -o /dev/null -H 'Host: app.example.com' http://127.0.0.1/api/health; tail -1 /var/log/traefik/access.log | jq -r '[.RouterName, .ServiceName, .DownstreamStatus] | @tsv'

# Status code distribution in the last 10,000 log lines
tail -10000 access.log | jq -r .DownstreamStatus | sort | uniq -c | sort -rn

# Top client IPs (ClientHost) in a log file
jq -r .ClientHost access.log | sort | uniq -c | sort -rn | head

# Requests slower than 2 s, by service
jq -r 'select(.Duration > 2000000000) | [.ServiceName, .RequestPath, (.Duration/1e9|tostring)] | @tsv' access.log | sort | uniq -c | sort -rn | head

# Retries per service from the access log (RetryAttempts field)
jq -r 'select(.RetryAttempts > 0) | .ServiceName' access.log | sort | uniq -c

# Certificates stored per resolver with expiry, without printing keys
jq -r '.[] | .Certificates[]? | .domain.main as $d | .certificate | @base64d' /data/acme.json 2>/dev/null | openssl x509 -noout -subject -enddate 2>/dev/null

# Certificate served for a host, straight from Traefik
openssl s_client -connect 127.0.0.1:443 -servername app.example.com </dev/null 2>/dev/null | openssl x509 -noout -subject -issuer -enddate

# Which TLS versions a router accepts
for v in tls1_2 tls1_3; do printf '%s: ' "$v"; openssl s_client -connect 127.0.0.1:443 -servername app.example.com -"$v" </dev/null >/dev/null 2>&1 && echo yes || echo no; done

# Validate a static config file without starting a listener (parse errors print, then Ctrl-C)
traefik --configfile=traefik.yaml --entrypoints.web.address=:0 --log.level=DEBUG 2>&1 | head -20

# Traefik version and Go runtime
traefik version

# Reload dynamic config by touching the directory (file provider with watch: true)
touch /etc/traefik/dynamic/apps.yaml

# Kubernetes: IngressRoutes referencing a Service that does not exist
kubectl get ingressroute -A -o json | jq -r '.items[] | .metadata.namespace as $ns | .metadata.name as $n | .spec.routes[].services[]? | select(.kind == null or .kind == "Service") | "\($ns) \($n) \(.namespace // $ns) \(.name)"' | while read -r ns n sns svc; do kubectl get svc -n "$sns" "$svc" >/dev/null 2>&1 || echo "$ns/$n -> missing $sns/$svc"; done

# Kubernetes: middlewares referenced by IngressRoutes that do not exist
kubectl get ingressroute -A -o json | jq -r '.items[] | .metadata.namespace as $ns | .spec.routes[].middlewares[]? | "\(.namespace // $ns) \(.name)"' | sort -u | while read -r ns m; do kubectl get middleware -n "$ns" "$m" >/dev/null 2>&1 || echo "missing $ns/$m"; done

# Kubernetes: Helm values in force
helm get values traefik -n traefik
```

## Scripts

Report the health of a Traefik instance from its API: disabled routers with errors, services with down servers, and certificates near expiry, exiting non-zero for a monitoring hook.

```sh
#!/usr/bin/env bash
# traefik-health.sh [api-url] [days]: routers in error, servers down, certificates expiring within N days
set -euo pipefail
api=${1:-http://localhost:8080}; days=${2:-14}; rc=0
curl -sf --max-time 5 "$api/ping" >/dev/null || { echo "CRIT: ping failed at $api"; exit 2; }

bad=$(curl -sf "$api/api/http/routers" | jq -r '.[] | select(.status!="enabled") | "  \(.name)\t\(.error // [] | join("; "))"')
[ -z "$bad" ] || { echo "WARN: routers not enabled:"; echo "$bad"; rc=1; }

down=$(curl -sf "$api/api/http/services" | jq -r '.[] | .name as $s | (.serverStatus // {}) | to_entries[] | select(.value!="UP") | "  \($s)\t\(.key)"')
[ -z "$down" ] || { echo "WARN: servers down:"; echo "$down"; rc=1; }

empty=$(curl -sf "$api/api/http/services" | jq -r '.[] | select(.loadBalancer != null and ((.loadBalancer.servers // []) | length == 0)) | "  \(.name)"')
[ -z "$empty" ] || { echo "WARN: services with no servers:"; echo "$empty"; rc=1; }

if [ -r "${ACME_JSON:-/data/acme.json}" ]; then
  now=$(date +%s)
  while IFS=$'\t' read -r domain cert; do
    exp=$(printf '%s' "$cert" | base64 -d | openssl x509 -noout -enddate | cut -d= -f2)
    left=$(( ($(date -d "$exp" +%s) - now) / 86400 ))
    [ "$left" -ge "$days" ] || { echo "WARN: $domain expires in ${left}d"; rc=1; }
  done < <(jq -r '.[] | .Certificates[]? | [.domain.main, .certificate] | @tsv' "${ACME_JSON:-/data/acme.json}")
fi

warnings=$(curl -sf "$api/api/overview" | jq '[.http.routers.warnings, .http.services.warnings, .http.middlewares.warnings] | add // 0')
[ "$warnings" -eq 0 ] || { echo "WARN: $warnings configuration warnings in /api/overview"; rc=1; }
[ $rc -eq 0 ] && echo "OK: $(curl -sf "$api/api/http/routers" | jq length) routers, $(curl -sf "$api/api/http/services" | jq length) services"
exit $rc
```

Summarise a JSON access log into a per-service table of request count, error ratio and p95 latency, for a quick look before opening Grafana.

```python
#!/usr/bin/env python3
"""Per-service summary of a Traefik JSON access log.

Usage: traefik-log-summary.py access.log [minutes]
Only lines from the last N minutes are counted when given.
"""
import json
import sys
from collections import defaultdict
from datetime import datetime, timedelta, timezone

path = sys.argv[1]
cutoff = datetime.now(timezone.utc) - timedelta(minutes=int(sys.argv[2])) if len(sys.argv) > 2 else None
stats = defaultdict(lambda: {"n": 0, "5xx": 0, "4xx": 0, "durations": []})
with open(path) as f:
    for line in f:
        try:
            e = json.loads(line)
        except json.JSONDecodeError:
            continue
        if cutoff and datetime.fromisoformat(e["StartUTC"].replace("Z", "+00:00")) < cutoff:
            continue
        s = stats[e.get("ServiceName") or e.get("RouterName") or "-"]
        s["n"] += 1
        code = int(e.get("DownstreamStatus", 0))
        s["5xx"] += code >= 500
        s["4xx"] += 400 <= code < 500
        s["durations"].append(e.get("Duration", 0) / 1e6)   # ns -> ms

def pct(xs, p):
    xs = sorted(xs)
    return xs[min(len(xs) - 1, int(p * len(xs)))] if xs else 0

print(f"{'service':40} {'reqs':>8} {'5xx%':>6} {'4xx%':>6} {'p50ms':>8} {'p95ms':>8} {'p99ms':>8}")
for name, s in sorted(stats.items(), key=lambda kv: -kv[1]["n"]):
    d = s["durations"]
    print(f"{name[:40]:40} {s['n']:8d} {100 * s['5xx'] / s['n']:6.2f} {100 * s['4xx'] / s['n']:6.2f} {pct(d, .5):8.1f} {pct(d, .95):8.1f} {pct(d, .99):8.1f}")
```

Rotate a basic-auth users file safely: generate a bcrypt entry with `htpasswd`, write the file atomically with mode 600, and confirm Traefik reloaded it.

```sh
#!/usr/bin/env bash
# traefik-user.sh USERS_FILE USERNAME: add or replace one user (password prompted), atomic write, reload check
set -euo pipefail
file=$1; user=$2
[ -f "$file" ] || : > "$file"
tmp=$(mktemp "${file}.XXXXXX"); trap 'rm -f "$tmp"' EXIT
grep -v "^$user:" "$file" > "$tmp" || true                 # drop any existing entry for the user
htpasswd -nB "$user" >> "$tmp"                             # bcrypt; prompts twice, never on the command line
chmod 600 "$tmp"; mv "$tmp" "$file"; trap - EXIT
echo "wrote $(wc -l < "$file") users to $file"
# the basicAuth middleware reads usersFile at each request, so no reload is needed; prove it
if curl -sf -o /dev/null -w '%{http_code}\n' -u "$user:" "${VERIFY_URL:-http://localhost/}" | grep -q '^401$'; then
  echo "middleware answering 401 for an empty password: file in use"
fi
```

## Further reading

- [Traefik documentation](https://doc.traefik.io/traefik/) and [v2 to v3 migration](https://doc.traefik.io/traefik/migrate/v2-to-v3/)
- [HTTP middlewares reference](https://doc.traefik.io/traefik/reference/routing-configuration/http/middlewares/overview/)
- [TLS options](https://doc.traefik.io/traefik/reference/routing-configuration/http/tls/tls-options/) and [ACME](https://doc.traefik.io/traefik/reference/install-configuration/tls/certificate-resolvers/acme/)
- [Kubernetes CRD provider](https://doc.traefik.io/traefik/reference/routing-configuration/kubernetes/crd/http/ingressroute/)
- [Metrics](https://doc.traefik.io/traefik/reference/install-configuration/observability/metrics/) and [access logs](https://doc.traefik.io/traefik/reference/install-configuration/observability/logs-and-accesslogs/)

> [!CAUTION]
> `acme.json` contains the ACME account key and every certificate's private key. Do not paste its contents into tickets or chat; the commands above print only domains and dates.