# Vault

> Operate HashiCorp Vault: seal state, tokens, auth methods, policies, KV, dynamic credentials, PKI and transit, and why a request is denied.

Canonical: https://www.wiki.jodisand.me/vault/
Reviewed: 2026-09-24
Related: [TLS and certificates](https://www.wiki.jodisand.me/tls/index.md), [Kubernetes](https://www.wiki.jodisand.me/kubernetes/index.md), [PostgreSQL](https://www.wiki.jodisand.me/postgresql/index.md), [Terraform](https://www.wiki.jodisand.me/terraform/index.md), [AWS](https://www.wiki.jodisand.me/aws/index.md)


## Cheatsheet

| Task | Command |
| --- | --- |
| Server state (exit 0 unsealed, 2 sealed, 1 error) | `vault status` |
| Who is this token | `vault token lookup` |
| What this token may do at a path | `vault token capabilities secret/data/app/db` |
| Log in with OIDC | `vault login -method=oidc` |
| Read a KV v2 secret | `vault kv get -mount=secret app/db` |
| One field only | `vault kv get -mount=secret -field=password app/db` |
| Write a value from stdin | `vault kv put -mount=secret app/db password=-` |
| Read an older version | `vault kv get -mount=secret -version=3 app/db` |
| Restore a soft-deleted version | `vault kv undelete -mount=secret -versions=3 app/db` |
| List secrets engines | `vault secrets list -detailed` |
| List auth methods | `vault auth list` |
| Read a policy | `vault policy read my-app` |
| Get a dynamic database credential | `vault read database/creds/readonly` |
| Renew a lease | `vault lease renew <lease_id>` |
| Revoke all leases under a prefix | `vault lease revoke -prefix database/creds/readonly` |

The CLI reads `VAULT_ADDR` (default `https://127.0.0.1:8200`), `VAULT_TOKEN`, `VAULT_NAMESPACE` (Enterprise and HCP) and `VAULT_CACERT`. Most "works for me, fails for them" reports come from one of these differing.

## A request that is denied

```sh
vault status                                             # reachable, unsealed, which version
vault token lookup -format=json | jq '.data | {display_name, policies, identity_policies, ttl, renewable}'
vault token capabilities secret/data/app/db              # e.g. "read" or "deny"
vault read sys/internal/ui/mounts/secret/app/db          # which mount serves the path, and its type and version
vault policy read my-app                                 # the rules actually attached
```

`capabilities` answers the question directly: it returns the combined capabilities of every policy on the token (including identity group policies) for that exact API path. If it says `deny` but the policy looks right, the path is usually wrong. For KV v2 the API path includes `data/` or `metadata/` (see [Policies](#policies)). The audit log records every denied request with the path and the token's policies; use it when the client cannot tell you which call failed.

## Seal, unseal and tokens

Vault encrypts all stored data with a key that is itself protected by a root key. At start-up Vault is sealed: it holds the encrypted data but not the root key. With Shamir seals, the root key is split into key shares, and a threshold of them must be entered to unseal. With auto-unseal, a cloud KMS or HSM decrypts the root key at start-up and the operator holds recovery keys instead.

```sh
vault status                        # Sealed, Seal Type, HA Mode, Version, Storage Type
vault operator unseal               # prompts for one key share; repeat until the threshold is met
vault token lookup                  # policies, TTL, renewable, entity, parent
vault token revoke -self
```

> [!WARNING] `vault operator seal` stops all service
> Sealing discards the in-memory root key; every client fails until enough key holders (or the KMS) unseal it again. Use it only as an emergency stop, for example after a suspected compromise.

Every request carries a token. Each token has policies, a TTL and usually a parent; revoking a parent revokes all its children. Clients must renew renewable tokens and leases before they expire, up to their max TTL, after which they must log in again. Recovery keys for auto-unseal need the same care as unseal keys: they authorise root token generation and rekeying.

## Auth methods

People log in through OIDC or LDAP. Workloads log in with an identity the platform already proves (Kubernetes service account, cloud IAM role, AppRole). Avoid long-lived static tokens.

```sh
vault auth enable oidc
vault auth enable kubernetes
vault auth enable approle
```

### Kubernetes auth

The pod sends its service account JWT; Vault checks it with the Kubernetes TokenReview API and returns a Vault token with the role's policies. Nothing secret sits in the manifest or the image. When Vault runs inside the cluster, the CA certificate and reviewer JWT default to Vault's own pod service account.

```sh
vault write auth/kubernetes/config kubernetes_host=https://kubernetes.default.svc

vault write auth/kubernetes/role/my-app \
  bound_service_account_names=my-app \
  bound_service_account_namespaces=my-namespace \
  token_policies=my-app \
  token_ttl=1h
```

`policies` and `ttl` are deprecated aliases of `token_policies` and `token_ttl`. The optional `audience` parameter makes Vault verify the JWT `aud` claim; set it and request a projected token with the same audience. See the [Kubernetes auth API](https://developer.hashicorp.com/vault/api-docs/auth/kubernetes) and [Kubernetes](https://www.wiki.jodisand.me/kubernetes/).

When Vault runs outside the cluster it needs the cluster CA and a way to call TokenReview. Give it a dedicated service account bound to `system:auth-delegator` and a long-lived token, or leave `token_reviewer_jwt` unset so Vault uses the client's own JWT for the review (the default since 1.9, which requires the client service account to have TokenReview permission).

```sh
vault write auth/kubernetes/config \
  kubernetes_host="https://k8s.example.com:6443" \
  kubernetes_ca_cert=@ca.crt \
  token_reviewer_jwt=@reviewer.jwt

# From a pod, by hand, to prove the role works
JWT=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
vault write -field=token auth/kubernetes/login role=my-app jwt="$JWT"
```

Three ways to get the secret into the pod: the Vault Agent injector (a sidecar writes files, no application change), the Vault Secrets Operator (a CRD syncs Vault data into a Kubernetes Secret, so `envFrom` and existing tooling keep working) and the CSI provider (mounted as a volume). The operator is the current recommendation for new deployments; see [Vault Agent](#vault-agent).

```yaml
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: my-app-db
  namespace: my-namespace
spec:
  vaultAuthRef: default            # a VaultAuth in the same namespace pointing at auth/kubernetes role my-app
  mount: secret
  type: kv-v2
  path: app/db
  refreshAfter: 60s
  destination:
    name: my-app-db                # the Kubernetes Secret written
    create: true
  rolloutRestartTargets:           # restart on change
    - kind: Deployment
      name: my-app
```

### OIDC

For people. Vault is an OIDC relying party: the CLI opens a browser, the provider (Keycloak, Entra ID, Okta, Google) authenticates, and Vault maps claims to a role. Group membership arrives as a claim and is matched to external identity groups, which is where policies for teams are attached. See [IdM](https://www.wiki.jodisand.me/idm/) for the Keycloak side.

```sh
vault write auth/oidc/config \
  oidc_discovery_url="https://sso.example.com/realms/example" \
  oidc_client_id="vault" \
  oidc_client_secret="$OIDC_CLIENT_SECRET" \
  default_role="default"

vault write auth/oidc/role/default \
  bound_audiences="vault" \
  allowed_redirect_uris="https://vault.example.com/ui/vault/auth/oidc/oidc/callback" \
  allowed_redirect_uris="http://localhost:8250/oidc/callback" \
  user_claim="preferred_username" \
  groups_claim="groups" \
  oidc_scopes="openid,profile,groups" \
  token_policies="default" \
  token_ttl=8h

# Map a provider group to a Vault group that carries a policy
vault write -format=json identity/group name=platform-admins type=external policies=platform-admin | jq -r .data.id > group.id
vault read -field=accessor sys/auth/oidc > accessor.id
vault write identity/group-alias name="platform-admins" mount_accessor="$(cat accessor.id)" canonical_id="$(cat group.id)"

vault login -method=oidc role=default
vault login -method=oidc -path=oidc role=default -no-print   # in a headless shell: prints a URL to open elsewhere
```

The second `allowed_redirect_uris` is the CLI's local listener on port 8250. `name` on the group alias must equal the value of the group claim exactly (Keycloak sends `/platform-admins` when the path mapper is used; strip it with `Full group path` off). Policies attached to an external group show up under `identity_policies` in `vault token lookup`, not `policies`, and `vault token capabilities` accounts for both.

### Identity

Every login creates or reuses an entity with an alias per auth method, so the same person via OIDC and LDAP is one entity with one set of identity policies and one count against the client licence. Entities also power templated policies: a rule can reference `{{identity.entity.name}}` or a metadata field, so one policy grants each user their own path.

```sh
vault read identity/entity/name/jane
vault write identity/entity/name/jane metadata=team=platform
vault list identity/group/name
vault write identity/lookup/entity alias_name=jane alias_mount_accessor="$(vault read -field=accessor sys/auth/oidc)"
```

### Userpass and LDAP

`userpass` is for break-glass accounts and local testing; `ldap` for sites without an OIDC provider.

```sh
vault auth enable userpass
vault write auth/userpass/users/breakglass password=- token_policies=admin token_ttl=1h token_max_ttl=4h   # password from stdin
vault login -method=userpass username=breakglass

vault auth enable ldap
vault write auth/ldap/config url="ldaps://idm.example.com" binddn="uid=vault,cn=sysaccounts,cn=etc,dc=example,dc=com" bindpass=- \
  userdn="cn=users,cn=accounts,dc=example,dc=com" userattr=uid \
  groupdn="cn=groups,cn=accounts,dc=example,dc=com" groupattr=cn groupfilter='(member={{.UserDN}})' \
  certificate=@ipa-ca.pem
vault write auth/ldap/groups/platform-admins policies=platform-admin
```

### AppRole

For workloads outside Kubernetes and without a cloud identity. The role ID is like a user name; the secret ID is the credential and should be short-lived and delivered by a trusted system (CI, config management).

```sh
vault write auth/approle/role/ci token_policies=ci token_ttl=20m secret_id_ttl=10m secret_id_num_uses=1
vault read -field=role_id auth/approle/role/ci/role-id
vault write -f -field=secret_id auth/approle/role/ci/secret-id    # generates a new secret ID
vault write auth/approle/login role_id="$ROLE_ID" secret_id="$SECRET_ID"
```

## Policies

Policies are deny-by-default rules on API paths. Capabilities are `create`, `read`, `update`, `patch`, `delete`, `list`, `sudo`, `deny`, plus `subscribe` (events) and `recover` (snapshot recovery). `deny` on a matched path overrides every other capability, including `sudo`.

```hcl
# my-app.hcl
path "secret/data/app/*" {
  capabilities = ["read"]
}

path "secret/metadata/app/*" {
  capabilities = ["list"]
}

path "database/creds/readonly" {
  capabilities = ["read"]
}

path "secret/data/app/admin" {
  capabilities = ["deny"]
}
```

```sh
vault policy write my-app my-app.hcl
vault policy read my-app
vault token capabilities "$APP_TOKEN" secret/data/app/db    # check another token without logging in as it
```

When several rules match a path, Vault uses the most specific one; it does not merge a glob rule with an exact rule. Exact paths beat globs, and a later wildcard beats an earlier one. The same path in several policies on one token has its capabilities unioned. `*` is a glob only at the end of a path; `+` matches one path segment.

KV v2 separates the logical path from the API path. The secret at `app/db` on mount `secret` is read at `secret/data/app/db`, listed at `secret/metadata/app/`, and deleted or destroyed through `secret/delete/`, `secret/undelete/` and `secret/destroy/`. A policy on `secret/app/*` grants nothing on a KV v2 mount. Use `-mount=` with `vault kv` so the CLI builds these paths for you. See the [policy docs](https://developer.hashicorp.com/vault/docs/concepts/policies).

### Templated policies and parameter constraints

A policy can interpolate identity data, so one rule serves every user or team, and can constrain the parameters of a write, not only the path.

```hcl
# Each user gets a private KV area named after their entity
path "secret/data/users/{{identity.entity.name}}/*" {
  capabilities = ["create", "read", "update", "delete"]
}
path "secret/metadata/users/{{identity.entity.name}}/*" {
  capabilities = ["list", "read"]
}

# Team path from entity metadata set by an administrator
path "secret/data/teams/{{identity.entity.metadata.team}}/*" {
  capabilities = ["read"]
}

# May issue certificates from this role, but only for hosts under one subdomain and never a CA
path "pki_int/issue/internal" {
  capabilities = ["update"]
  allowed_parameters = {
    "common_name" = []                       # any value allowed for this key
    "ttl"         = ["24h", "72h"]
  }
  denied_parameters = {
    "ip_sans" = []
  }
  required_parameters = ["common_name"]
}

# Read-only mounts view for auditors
path "sys/mounts" {
  capabilities = ["read"]
}
```

Group-based templating uses `{{identity.groups.names.<name>.id}}` or a group's metadata. Templated policies cannot be checked with `vault token capabilities` for another token; log in as a test user from the same group. Parameter constraints apply to `create` and `update` only, and an empty list means "any value". Together with `sudo`-protected paths (listed in the [policy docs](https://developer.hashicorp.com/vault/docs/concepts/policies#root-protected-api-endpoints)), these are the whole authorisation model; there are no roles above policies apart from the identity groups that attach them.

## KV secrets

```sh
vault secrets enable -path=secret -version=2 kv
vault kv put -mount=secret app/db username=my-app password=-              # password read from stdin
vault kv get -mount=secret -format=json app/db | jq -r '.data.data.password'
vault kv patch -mount=secret app/db password=-                           # change one field, keep the rest
vault kv metadata get -mount=secret app/db                               # versions, deletion times
vault kv delete -mount=secret app/db                                     # soft delete latest version; undelete works
vault kv undelete -mount=secret -versions=3 app/db
```

`kv patch` needs the `patch` capability (or falls back to read then write, which needs `read` and `update`).

> [!WARNING] Irreversible deletes
> `vault kv destroy -mount=secret -versions=3 app/db` removes the data of those versions permanently. `vault kv metadata delete -mount=secret app/db` removes every version and the metadata. Neither can be undone.

A secret passed as `key=value` on the command line lands in shell history and is visible in the process list. Use `key=-` for stdin or `key=@file` to read from a file.

### Versions, check-and-set and retention

Every `put` creates a new version and keeps the previous ones up to `max_versions` (default 10). `patch` and `put` differ in what they do to fields you do not mention: `put` replaces the whole secret, so a `put` with one key deletes every other key. Check-and-set stops two writers clobbering each other.

```sh
vault kv put -mount=secret -cas=0 app/new key=value          # only succeeds if the secret does not exist yet
vault kv put -mount=secret -cas=4 app/db password=-           # only succeeds if the current version is 4
vault kv metadata put -mount=secret -cas-required=true app/db  # every write must now carry -cas
vault kv metadata put -mount=secret -max-versions=5 -delete-version-after=2160h app/db   # keep 5 versions; auto soft-delete after 90 days
vault kv metadata put -mount=secret -custom-metadata=owner=platform -custom-metadata=rotation=quarterly app/db
vault kv rollback -mount=secret -version=3 app/db             # writes version 3's data as a new version
vault kv get -mount=secret -format=json app/db | jq '.data.metadata'   # created_time, version, custom_metadata, destroyed
vault kv destroy -mount=secret -versions=1,2 app/db           # irreversible: removes the data of those versions
```

`delete-version-after` runs from each version's creation time, so a secret nobody rewrites disappears on schedule; do not set it on a mount that holds long-lived configuration. Custom metadata is readable with `read` on the metadata path only, which makes it the right place for owner and rotation tags an auditor should see without the secret value.

`vault kv` also takes `-mount` on KV v1, where there is no versioning, no metadata and no `patch`; `vault secrets list -detailed` shows the version under `options`. Upgrading a v1 mount in place (`vault kv enable-versioning secret/`) works but blocks the mount briefly and changes every API path clients use, so it is a change to policies as much as to data.

## Dynamic secrets

Vault creates a credential on request and revokes it when its lease expires. A leaked credential has a bounded lifetime, and the audit log ties it to the requester.

```sh
vault secrets enable database

vault write database/config/prod \
  plugin_name=postgresql-database-plugin \
  allowed_roles=readonly \
  connection_url='postgresql://{{username}}:{{password}}@db.internal.example:5432/app?sslmode=verify-full' \
  username=vault-admin \
  password="$DB_ADMIN_PASSWORD"

vault write database/roles/readonly \
  db_name=prod \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl=1h max_ttl=24h

vault read database/creds/readonly     # username, password, lease_id, lease_duration
vault lease renew <lease_id>           # extend, up to max_ttl
vault lease revoke <lease_id>          # drop the database role now
```

After configuring, rotate the admin password so only Vault knows it: `vault write -f database/rotate-root/prod`. Keep a break-glass path, because nobody can log in with the old password afterwards. The same lease pattern applies to AWS (`aws/creds/<role>`), other clouds and SSH. See [PostgreSQL](https://www.wiki.jodisand.me/postgresql/) for the database side.

### Static roles and root rotation

Not every application can pick up a new username. A static role manages the password of one existing database account and rotates it on a schedule; clients read the current password rather than a lease. Since Vault 1.15 the root credential itself can rotate on a schedule too.

```sh
vault write database/static-roles/reporting \
  db_name=prod username=reporting \
  rotation_period=24h \
  rotation_statements="ALTER ROLE \"{{name}}\" WITH PASSWORD '{{password}}';"
vault read database/static-creds/reporting              # current password, last_vault_rotation, ttl until next
vault write -f database/rotate-role/reporting            # rotate now

vault write database/config/prod rotation_schedule="0 3 * * 6" rotation_window=1h   # rotate the root on Saturdays at 03:00
vault write database/roles/readonly revocation_statements="REASSIGN OWNED BY \"{{name}}\" TO vault-admin; DROP ROLE IF EXISTS \"{{name}}\";"
```

Default `revocation_statements` for PostgreSQL revoke privileges and `DROP ROLE`, which fails if the role still owns objects; set them explicitly for roles that create tables. `vault lease revoke -force -prefix database/creds/readonly` removes leases whose revocation keeps failing, leaving the database roles behind for you to clean up.

```sh
# AWS: an IAM role assumed with STS, no long-lived keys anywhere
vault secrets enable aws
vault write aws/config/root region=ap-southeast-2                    # credentials from the Vault server's own IAM role
vault write aws/roles/s3-reader credential_type=assumed_role role_arns=arn:aws:iam::123456789012:role/s3-reader default_sts_ttl=1h
vault read aws/creds/s3-reader                                       # access_key, secret_key, security_token
```

Dynamic credentials have leases; `sys/leases` is where to look when the database has thousands of `v-kubernetes-my-app-readonly-*` roles. `vault lease lookup <id>` shows expiry and `vault list sys/leases/lookup/database/creds/readonly` lists them (needs `sudo` on that path). Lease counts grow fast when clients request credentials per request instead of per process; the metric `vault.expire.num_leases` is the one to graph.

## PKI

```sh
vault secrets enable pki
vault secrets tune -max-lease-ttl=87600h pki
vault write -field=certificate pki/root/generate/internal common_name="Example Internal Root" ttl=87600h > root.pem
vault write pki/roles/internal allowed_domains=example.com allow_subdomains=true max_ttl=720h
vault write pki/issue/internal common_name=api.example.com ttl=72h
```

`pki/issue` returns the private key in the response and Vault does not store it; `pki/sign` signs a CSR you generated locally instead. In production, keep the root offline or in its own mount and issue from an intermediate. Short-lived certificates issued on demand turn renewal into routine automation. See [TLS](https://www.wiki.jodisand.me/tls/) for reading and verifying the issued chain, and the [PKI docs](https://developer.hashicorp.com/vault/docs/secrets/pki).

### Root and intermediate

The root lives in one mount with a long TTL and issues nothing but the intermediate; the intermediate lives in a second mount, has the URLs clients need, and does all the work. Issued certificates then carry `ca_chain` with the intermediate, and clients trust only the root.

```sh
vault secrets enable -path=pki_root pki
vault secrets tune -max-lease-ttl=87600h pki_root
vault write -field=certificate pki_root/root/generate/internal common_name="Example Internal Root" issuer_name=root-2026 key_type=ec key_bits=384 ttl=87600h > root.pem

vault secrets enable -path=pki_int pki
vault secrets tune -max-lease-ttl=43800h pki_int
vault write -format=json pki_int/intermediate/generate/internal common_name="Example Internal Intermediate" key_type=ec key_bits=256 | jq -r .data.csr > int.csr
vault write -format=json pki_root/root/sign-intermediate csr=@int.csr format=pem_bundle ttl=43800h | jq -r .data.certificate > int.pem
vault write pki_int/intermediate/set-signed certificate=@int.pem

# AIA and CRL URLs embedded in every issued certificate; clients fetch these, so they must be reachable
vault write pki_int/config/urls \
  issuing_certificates="https://vault.example.com:8200/v1/pki_int/ca" \
  crl_distribution_points="https://vault.example.com:8200/v1/pki_int/crl" \
  ocsp_servers="https://vault.example.com:8200/v1/pki_int/ocsp"
vault write pki_int/config/crl expiry=72h auto_rebuild=true auto_rebuild_grace_period=12h

vault write pki_int/roles/internal \
  allowed_domains=example.com allow_subdomains=true allow_bare_domains=false allow_ip_sans=true \
  key_type=ec key_bits=256 max_ttl=720h ttl=72h \
  server_flag=true client_flag=false require_cn=false \
  allowed_uri_sans="spiffe://example.com/*"
vault write -format=json pki_int/issue/internal common_name=api.example.com alt_names=api.internal.example.com ip_sans=192.0.2.10 ttl=72h > issued.json
jq -r .data.certificate issued.json > cert.pem; jq -r .data.private_key issued.json > key.pem; jq -r '.data.ca_chain[]' issued.json > chain.pem

vault list pki_int/certs                                 # serials of every issued certificate still stored
vault write pki_int/revoke serial_number="$(openssl x509 -in cert.pem -noout -serial | cut -d= -f2 | sed 's/../&:/g; s/:$//')"
vault write pki_int/tidy tidy_cert_store=true tidy_revoked_certs=true safety_buffer=72h   # remove expired certificates from storage
vault list pki_int/issuers; vault read pki_int/issuer/default   # several issuers per mount since 1.11, for CA rotation without a new mount
```

Every issued certificate is stored until `tidy` removes it, so an issuer handing out 24-hour certificates to a fleet fills storage unless `auto_tidy` is configured (`vault write pki_int/config/auto-tidy enabled=true tidy_cert_store=true interval_duration=12h`). Roles default to `key_type=rsa`, `key_bits=2048` and generating the key in Vault; for `pki/sign` use `pki_int/sign/internal csr=@req.csr` and `pki_int/sign-verbatim` only where you trust the requester with every extension.

### ACME

Since 1.14 the PKI engine speaks ACME, so certbot, cert-manager, Caddy and Traefik can obtain certificates from your internal CA with no Vault token at all: domain control is proven the same way as with a public CA.

```sh
vault secrets tune -passthrough-request-headers=If-Modified-Since -allowed-response-headers=Last-Modified \
  -allowed-response-headers=Location -allowed-response-headers=Replay-Nonce -allowed-response-headers=Link pki_int
vault write pki_int/config/cluster path=https://vault.example.com:8200/v1/pki_int aia_path=https://vault.example.com:8200/v1/pki_int
vault write pki_int/config/acme enabled=true allowed_roles=internal default_directory_policy=role:internal
vault read pki_int/config/acme

certbot certonly --standalone --server https://vault.example.com:8200/v1/pki_int/acme/directory -d api.internal.example.com --register-unsafely-without-email --agree-tos
```

Vault's ACME server validates `http-01`, `dns-01` and `tls-alpn-01` challenges from the Vault servers, so they need to reach the requesting host and resolve its names. Names still have to satisfy the role's `allowed_domains`.

## Transit

Encryption as a service: the key never leaves Vault, so an application encrypts and decrypts without holding key material. Plaintext must be base64-encoded.

```sh
vault secrets enable transit
vault write -f transit/keys/my-app
vault write -field=ciphertext transit/encrypt/my-app plaintext="$(printf '%s' 'card number' | base64)"
vault write -field=plaintext transit/decrypt/my-app ciphertext='vault:v1:...' | base64 -d
vault write -f transit/keys/my-app/rotate                 # new key version; old ciphertext still decrypts
vault write transit/rewrap/my-app ciphertext='vault:v1:...'   # re-encrypt with the latest version, no plaintext exposed
```

Set `min_decryption_version` on the key once all data is rewrapped, so old key versions can no longer decrypt.

## Audit devices

```sh
vault audit enable file file_path=/var/log/vault/audit.log
vault audit list -detailed
```

Enable at least one audit device in production, and preferably two. If audit devices are enabled and none of them can write a request, Vault refuses the request: no audit, no access. A full disk under the audit log therefore presents as a Vault outage. Sensitive values in the log are HMAC-hashed; `vault write sys/audit-hash/file input=<value>` produces the hash to search for.

```sh
# Denied requests in the last hour, by path and requesting entity
jq -r 'select(.type == "response" and (.error // "" | test("permission denied"))) | [.time, .request.path, .auth.display_name // .auth.entity_id] | @tsv' /var/log/vault/audit.log | tail -50

# Who read a secret
jq -r 'select(.request.path == "secret/data/app/db" and .request.operation == "read") | [.time, .auth.display_name, .request.remote_address] | @tsv' /var/log/vault/audit.log
```

## Vault Agent

Vault Agent is a client-side daemon that logs in with an auth method, keeps the token renewed, and renders secrets into files from templates, re-rendering when leases roll over and optionally running a command afterwards. The application reads a file; it never speaks to Vault or handles a token. The same binary in `proxy` mode fronts the Vault API on localhost with a cached token for applications that do use the API.

```hcl
# /etc/vault-agent/agent.hcl
vault {
  address = "https://vault.example.com:8200"
  retry { num_retries = 5 }
}

auto_auth {
  method "approle" {
    mount_path = "auth/approle"
    config = {
      role_id_file_path                   = "/etc/vault-agent/role-id"
      secret_id_file_path                 = "/etc/vault-agent/secret-id"
      remove_secret_id_file_after_reading = true
    }
  }
  sink "file" {
    config = { path = "/run/vault-agent/token", mode = 0640 }
  }
}

template_config {
  exit_on_retry_failure         = true      # fail loudly instead of running with stale files
  static_secret_render_interval = "5m"      # how often KV (non-leased) secrets are re-read
}

template {
  destination = "/etc/my-app/db.env"
  perms       = "0640"
  command     = "systemctl reload my-app"
  error_on_missing_key = true
  contents = <<-EOT
    {{- with secret "database/creds/readonly" }}
    DB_USER={{ .Data.username }}
    DB_PASSWORD={{ .Data.password }}
    {{- end }}
    {{- with secret "secret/data/app/db" }}
    DB_HOST={{ .Data.data.host }}
    {{- end }}
  EOT
}

template {
  source      = "/etc/vault-agent/tls.ctmpl"
  destination = "/etc/my-app/tls.pem"
  perms       = "0600"
  command     = "systemctl reload my-app"
}
```

```text
{{/* tls.ctmpl: certificate, key and chain from PKI, renewed at two thirds of its TTL */}}
{{- with pkiCert "pki_int/issue/internal" "common_name=api.example.com" "ttl=72h" }}
{{ .Cert }}
{{ .Key }}
{{ .CA }}
{{- end }}
```

```sh
vault agent -config=/etc/vault-agent/agent.hcl
vault agent -config=/etc/vault-agent/agent.hcl -log-level=debug   # shows each template render and lease renewal
VAULT_TOKEN=$(cat /run/vault-agent/token) vault token lookup     # the agent's token, if the sink is readable
```

KV v2 data sits under `.Data.data` in templates because the API wraps it in metadata; database credentials sit directly under `.Data`. `pkiCert` (Vault 1.11+) renews before expiry and, unlike `secret "pki_int/issue/..."`, does not request a new certificate on every agent restart. The templates are Consul Template syntax, so `{{ .Data.data | toJSON }}` and `{{ range $k, $v := .Data.data }}` work.

In Kubernetes, the agent injector adds the same agent as an init container and sidecar when a pod carries annotations; the pod's own service account JWT is the credential.

```yaml
metadata:
  annotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/role: "my-app"
    vault.hashicorp.com/agent-inject-secret-db.env: "secret/data/app/db"
    vault.hashicorp.com/agent-inject-template-db.env: |
      {{- with secret "secret/data/app/db" -}}
      DB_PASSWORD={{ .Data.data.password }}
      {{- end }}
    vault.hashicorp.com/agent-inject-command-db.env: "kill -HUP $(pidof my-app)"
    vault.hashicorp.com/agent-pre-populate-only: "false"    # true: init container only, no sidecar, no renewal
```

Files land under `/vault/secrets/` in the application container. `kubectl logs my-app-xxx -c vault-agent` is where login and template errors go; `-c vault-agent-init` for the first render.

## Raft storage operations

Integrated storage (Raft) is the supported backend for self-managed Vault. Snapshots are the backup; storage is encrypted, so a snapshot is useless without the unseal or recovery keys.

```sh
vault operator raft list-peers                                    # node id, address, state (leader/follower), voter
vault operator raft autopilot state                               # health per server, failure tolerance, last contact
vault operator raft snapshot save "backup-$(date +%FT%H%M).snap"   # from any node; forwarded to the leader
vault operator raft snapshot restore -force backup.snap           # restores onto a cluster with a different unseal key when -force is given; overwrites everything
vault operator raft remove-peer vault-3                           # after a node is gone for good; do not remove a live voter you still need
vault operator raft join https://vault-1.example.com:8200         # from a new node
vault operator step-down                                          # leader hands over; useful before maintenance
vault operator rekey -init -key-shares=5 -key-threshold=3         # new unseal keys (Shamir), multi-step
vault operator generate-root -init                                # new root token using a quorum of key shares; revoke it when done
vault operator members                                            # every node's version, hostname and active status
```

Losing more than half the voters loses quorum: the cluster stays up for reads on the leader until it steps down, then stops. Recovery is `peers.json` in the Raft data directory of the surviving node with the remaining peers listed, followed by a restart; it is documented in the [Raft recovery guide](https://developer.hashicorp.com/vault/tutorials/raft/raft-lost-quorum). Automated snapshots (`sys/storage/raft/snapshot-auto/config`) are Enterprise only; on Community edition schedule `snapshot save` with a timer, keep several days, and test a restore into a scratch cluster.

Rolling upgrades: upgrade standbys first, then step down the leader and upgrade it. Vault's version skew rules allow one minor version between nodes during the upgrade only. `vault status` on each node shows `Version` and `Active Node Address`.

## Troubleshooting

| Symptom | Likely cause | Check |
| --- | --- | --- |
| `403 permission denied` | Policy path wrong (KV v2 `data/` / `metadata/`), wrong namespace, or a `deny` rule | `vault token capabilities <path>`; audit log |
| `missing client token` | `VAULT_TOKEN` unset, or Vault Agent's sink file not mounted | `vault print token`; agent logs |
| `503 Vault is sealed` | Restarted without auto-unseal, KMS unreachable, or sealed on purpose | `vault status`; server log for seal errors |
| Credentials stop working after an hour | Lease or token expired and nothing renewed it | `vault token lookup` TTL; lease `lease_duration` |
| `connection refused` / `http: server gave HTTP response to HTTPS client` | Wrong `VAULT_ADDR` host, port or scheme | `echo "$VAULT_ADDR"`; listener config |
| `x509: certificate signed by unknown authority` | Client does not trust Vault's CA | `VAULT_CACERT`; [TLS](https://www.wiki.jodisand.me/tls/) |
| Kubernetes login `permission denied` | Service account or namespace not bound, audience mismatch, TokenReview denied | `vault read auth/kubernetes/role/my-app`; Vault log |
| Every request fails after disk fills | Audit device cannot write | `vault audit list`; disk space |
| Standby returns errors or redirects | HA standby forwarding or `api_addr` misconfigured | `vault status` HA fields; `vault operator raft list-peers` |
| `vault kv put` deleted keys I did not mention | `put` replaces the whole secret | `vault kv rollback -version=N`; use `kv patch` for one field |
| `check-and-set parameter required` | Mount or secret has `cas_required` | `vault kv put -cas=<current version>` |
| OIDC login opens the browser then fails with `redirect uri not allowed` | `allowed_redirect_uris` on the role lacks the CLI (`localhost:8250`) or UI callback | `vault read auth/oidc/role/default`; add the exact URI |
| OIDC user gets `default` only, no team policy | Group claim name or value differs from the group alias `name`, or `groups_claim` unset | `vault login -method=oidc -format=json` then `vault read identity/entity/id/<id>`; `vault list identity/group-alias/id` |
| `token_reviewer_jwt` login `permission denied` with `service account name not authorized` | Pod service account or namespace not in the role's bound lists | `vault read auth/kubernetes/role/my-app`; pod's `serviceAccountName` |
| Kubernetes login `invalid audience` | Projected token audience differs from the role's `audience` | `kubectl create token my-app --audience=vault` matches; check `audience` on the role |
| Agent renders nothing and exits | `exit_on_retry_failure` with a path the token cannot read | `vault agent -log-level=debug`; `vault token capabilities` with the agent's token |
| Agent template shows `<no value>` | Wrong data path (`.Data.data` for KV v2, `.Data` for everything else) | `vault read -format=json <path>` to see the shape; set `error_on_missing_key` |
| PKI clients cannot fetch the CA or CRL | `config/urls` points at an unreachable address or not set at all | `vault read pki_int/config/urls`; `curl -sS "$VAULT_ADDR/v1/pki_int/crl/pem" \| openssl crl -noout -text` |
| `pki` mount grows without bound | Every issued certificate is stored until tidy | `vault read pki_int/tidy-status`; enable `config/auto-tidy` |
| `database/creds` returns `pq: role ... already exists` or revocation fails | Username template collided or the role still owns objects | Set `username_template`; explicit `revocation_statements` with `REASSIGN OWNED` |
| Raft node stuck `follower` with `last contact` growing | Network partition, or the node was removed but kept its data | `vault operator raft autopilot state`; rejoin with a clean data directory |
| Lease count climbing, memory too | Clients request credentials per request, or long `max_ttl` | `vault.expire.num_leases` metric; `vault list sys/leases/lookup/database/creds/<role>`; shorten TTLs |

## Oneliners

```sh
# Export one secret field into the environment without writing it to disk
export DB_PASSWORD="$(vault kv get -mount=secret -field=password app/db)"

# Keys directly under a path (not recursive; entries ending in / are folders)
vault kv list -mount=secret -format=json app | jq -r '.[]'

# Compare two policies
diff <(vault policy read app-a) <(vault policy read app-b)

# Render a .env file from a secret (the file contains secrets: restrict permissions)
(umask 077; vault kv get -mount=secret -format=json app/db | jq -r '.data.data | to_entries[] | "\(.key)=\(.value)"' > .env)

# Leases issued by a role (needs sudo on sys/leases/lookup)
vault list sys/leases/lookup/database/creds/readonly

# Confirm a Kubernetes role binds the expected service account and namespace
vault read -format=json auth/kubernetes/role/my-app | jq '.data | {bound_service_account_names, bound_service_account_namespaces, token_policies}'

# Test an AppRole login without replacing your stored token
VAULT_TOKEN= vault write -field=token auth/approle/login role_id="$ROLE_ID" secret_id="$SECRET_ID"

# Seal and HA state across nodes
for h in vault-1 vault-2 vault-3; do printf '%s ' "$h"; VAULT_ADDR="https://$h.example.com:8200" vault status -format=json | jq -r '[.sealed, .ha_enabled, .is_self // "-"] | @tsv'; done

# Health endpoint without a token; 200 active, 429 standby, 472 DR secondary, 473 perf standby, 501 uninitialised, 503 sealed
curl -sS -o /dev/null -w '%{http_code}\n' "$VAULT_ADDR/v1/sys/health"

# Every mount with type and version
vault secrets list -format=json | jq -r 'to_entries[] | [.key, .value.type, (.value.options.version // "-")] | @tsv'

# Recursive listing of a KV v2 mount (folders end in /)
walk() { vault kv list -mount=secret -format=json "$1" 2>/dev/null | jq -r '.[]' | while read -r k; do if [[ $k == */ ]]; then walk "$1$k"; else printf '%s%s\n' "$1" "$k"; fi; done; }; walk ''

# Secrets whose latest version is older than 90 days (candidates for rotation)
walk '' | while read -r p; do t=$(vault kv metadata get -mount=secret -format=json "$p" | jq -r .data.updated_time); (( $(date -d "$t" +%s) < $(date -d '-90 days' +%s) )) && printf '%s\t%s\n' "${t%%T*}" "$p"; done

# Copy one secret to another path, all fields, as a new version
vault kv get -mount=secret -format=json app/db | jq '.data.data' | vault kv put -mount=secret app/db-copy -

# Diff two versions of a secret without printing values (keys and whether changed)
diff <(vault kv get -mount=secret -version=3 -format=json app/db | jq -S '.data.data | map_values(. | @base64 | .[0:8])') <(vault kv get -mount=secret -version=4 -format=json app/db | jq -S '.data.data | map_values(. | @base64 | .[0:8])')

# Which policies grant anything on a path (scans every policy)
for p in $(vault policy list); do vault policy read "$p" | grep -q 'secret/data/app' && echo "$p"; done

# Tokens with the root policy (accessor list; needs sudo on auth/token/accessors)
vault list -format=json auth/token/accessors | jq -r '.[]' | while read -r a; do vault write -format=json auth/token/lookup-accessor accessor="$a" | jq -r 'select(.data.policies | index("root")) | [.data.accessor, .data.display_name, .data.creation_time] | @tsv'; done

# Revoke a token by accessor when you do not hold the token itself
vault token revoke -accessor "$ACCESSOR"

# Create a short-lived token for a script with only one policy and one use
vault token create -policy=my-app -ttl=15m -use-limit=3 -field=token

# Wrap a secret for handover: the recipient unwraps once, and Vault records who did (response-wrapping)
vault kv get -mount=secret -wrap-ttl=15m app/db | grep -E 'wrapping_token:' ; vault unwrap "$WRAP_TOKEN"

# Entities and which auth methods they have aliases on
vault list -format=json identity/entity/id | jq -r '.[]' | while read -r id; do vault read -format=json "identity/entity/id/$id" | jq -r '[.data.name, ([.data.aliases[].mount_type] | join(","))] | @tsv'; done

# Issue a certificate and split the response into files in one go
vault write -format=json pki_int/issue/internal common_name=api.example.com ttl=72h | jq -r '.data | .certificate, .private_key, (.ca_chain | join("\n"))' | awk '/BEGIN CERT/{f="cert.pem"; if (n++) f="chain.pem"} /BEGIN .*PRIVATE/{f="key.pem"} {print > f}'

# CRL from the PKI mount, expiry and revoked count
curl -sS "$VAULT_ADDR/v1/pki_int/crl/pem" | openssl crl -noout -nextupdate; curl -sS "$VAULT_ADDR/v1/pki_int/crl/pem" | openssl crl -noout -text | grep -c 'Serial Number'

# Certificates issued by the PKI mount expiring within 7 days (walks the cert store; slow on large mounts)
for s in $(vault list -format=json pki_int/certs | jq -r '.[]'); do vault read -field=certificate "pki_int/cert/$s" | openssl x509 -noout -checkend 604800 >/dev/null || echo "$s"; done

# Renew the current token if it is renewable and under an hour left
vault token lookup -format=json | jq -e '.data.renewable and .data.ttl < 3600' >/dev/null && vault token renew >/dev/null

# Rotate a KV secret and a database static role together, then confirm the version bumped
openssl rand -base64 32 | vault kv put -mount=secret app/api api_key=- >/dev/null; vault kv metadata get -mount=secret -format=json app/api | jq .data.current_version

# Leases about to expire in the next 10 minutes for one role
vault list -format=json sys/leases/lookup/database/creds/readonly | jq -r '.[]' | while read -r l; do vault lease lookup -format=json "database/creds/readonly/$l" | jq -r 'select(.data.ttl < 600) | [.data.id, .data.ttl] | @tsv'; done

# Vault server log filtered to seal and storage errors (systemd install)
journalctl -u vault --since -1h | grep -Ei 'seal|raft|storage|error'

# Take a Raft snapshot and verify it is a valid gzip archive with a SHA256SUMS entry
vault operator raft snapshot save snap.snap && tar -tzf snap.snap | head

# Enable a secrets engine at a path with a description and a default lease TTL
vault secrets enable -path=secret-team-a -version=2 -description='Team A KV' -default-lease-ttl=24h kv

# Move a mount without losing data (blocks the mount during the move; policies must be updated)
vault secrets move secret-old secret-team-a

# Namespaces (Enterprise/HCP): run a command against one namespace
VAULT_NAMESPACE=admin/team-a vault kv list -mount=secret /
```

## Scripts

Renew a certificate from the PKI engine only when it is close to expiry, write the files atomically and reload the service. For hosts that cannot run Vault Agent, from a systemd timer.

```sh
#!/usr/bin/env bash
# usage: pki-renew.sh HOSTNAME OUTDIR [SERVICE]   e.g. pki-renew.sh api.example.com /etc/pki/my-app my-app
# needs VAULT_ADDR and a token (VAULT_TOKEN or ~/.vault-token) with update on pki_int/issue/internal
set -euo pipefail
cn=${1:?common name}; out=${2:?output dir}; svc=${3:-}
threshold=${THRESHOLD_SECONDS:-$(( 24 * 3600 ))}; ttl=${TTL:-72h}

if [[ -f $out/cert.pem ]] && openssl x509 -in "$out/cert.pem" -noout -checkend "$threshold" >/dev/null; then
  exit 0                                         # still valid for longer than the threshold
fi

umask 077; mkdir -p "$out"
tmp=$(mktemp -d "$out/.new.XXXXXX"); trap 'rm -rf -- "$tmp"' EXIT
vault write -format=json pki_int/issue/internal common_name="$cn" ttl="$ttl" > "$tmp/issue.json"
jq -r .data.certificate  "$tmp/issue.json" > "$tmp/cert.pem"
jq -r .data.private_key  "$tmp/issue.json" > "$tmp/key.pem"
jq -r '.data.ca_chain[]' "$tmp/issue.json" > "$tmp/chain.pem"
cat "$tmp/cert.pem" "$tmp/chain.pem" > "$tmp/fullchain.pem"
rm -f "$tmp/issue.json"

openssl verify -CAfile "$tmp/chain.pem" "$tmp/cert.pem" >/dev/null
diff <(openssl pkey -in "$tmp/key.pem" -pubout) <(openssl x509 -in "$tmp/cert.pem" -noout -pubkey) >/dev/null
for f in cert.pem key.pem chain.pem fullchain.pem; do mv -f -- "$tmp/$f" "$out/$f"; done
chmod 644 "$out/cert.pem" "$out/chain.pem" "$out/fullchain.pem"
[[ -n $svc ]] && systemctl reload "$svc"
openssl x509 -in "$out/cert.pem" -noout -enddate
```

Raft snapshot with rotation and a restore-ability check. Keeps `KEEP` snapshots, fails if the snapshot is empty or not a gzip archive, and prints the size so a monitoring job can graph it. Deletes old snapshot files in the target directory.

```sh
#!/usr/bin/env bash
# usage: vault-snapshot.sh /var/backups/vault [KEEP]
set -euo pipefail
dir=${1:?snapshot directory}; keep=${2:-14}
: "${VAULT_ADDR:?}"
umask 077; mkdir -p "$dir"
name="$dir/vault-$(date -u +%Y%m%dT%H%M%SZ).snap"

status=$(curl -sS -o /dev/null -w '%{http_code}' --max-time 10 "$VAULT_ADDR/v1/sys/health")
[[ $status == 200 || $status == 429 ]] || { echo "vault health $status, not snapshotting" >&2; exit 1; }

timeout 300 vault operator raft snapshot save "$name"
[[ -s $name ]] || { echo 'empty snapshot' >&2; rm -f -- "$name"; exit 1; }
gzip -t "$name" || { echo 'snapshot is not a valid gzip archive' >&2; rm -f -- "$name"; exit 1; }
tar -tzf "$name" | grep -q '^SHA256SUMS$' || { echo 'snapshot missing SHA256SUMS' >&2; exit 1; }
printf 'saved %s (%s bytes)\n' "$name" "$(stat -c %s "$name")"

# Rotation: newest first, remove everything past $keep
mapfile -t old < <(ls -1t "$dir"/vault-*.snap | tail -n +"$(( keep + 1 ))")
for f in "${old[@]}"; do rm -f -- "$f"; printf 'removed %s\n' "$f"; done
```

Audit report from the file audit device: denied requests, root token use and secrets read by humans in the last day, for a daily review. Read-only.

```sh
#!/usr/bin/env bash
# usage: vault-audit-report.sh /var/log/vault/audit.log [HOURS]
set -euo pipefail
log=${1:?audit log}; hours=${2:-24}
since=$(date -u -d "-${hours} hours" +%FT%TZ)
recent() { jq -c --arg s "$since" 'select(.time >= $s)' "$log"; }

printf '== denied requests (top 20 by path) ==\n'
recent | jq -r 'select(.type == "response" and ((.error // "") | test("permission denied"))) | "\(.request.path)\t\(.auth.display_name // .auth.entity_id // "anon")"' | sort | uniq -c | sort -rn | head -20

printf '\n== requests made with a root-policy token ==\n'
recent | jq -r 'select(.type == "request" and (.auth.policies // [] | index("root"))) | [.time, .request.operation, .request.path, .request.remote_address] | @tsv' | head -50

printf '\n== KV reads by human logins (oidc, ldap, userpass) ==\n'
recent | jq -r 'select(.type == "response" and .request.operation == "read" and (.request.path | startswith("secret/data/")) and ((.auth.display_name // "") | test("^(oidc|ldap|userpass)-"))) | [.time, .auth.display_name, .request.path] | @tsv' | sort | uniq -c | sort -rn | head -50

printf '\n== logins by auth method ==\n'
recent | jq -r 'select(.type == "response" and (.request.path | test("^auth/.*/login"))) | .request.path | split("/")[1]' | sort | uniq -c | sort -rn
```

## Further reading

- [Vault documentation](https://developer.hashicorp.com/vault/docs): concepts, auth methods, secrets engines and operations.
- [Vault API reference](https://developer.hashicorp.com/vault/api-docs): every path and parameter the CLI maps to.
- [Policies](https://developer.hashicorp.com/vault/docs/concepts/policies): syntax, templating, parameter constraints and the sudo-protected paths.
- [PKI secrets engine](https://developer.hashicorp.com/vault/docs/secrets/pki) and its [ACME support](https://developer.hashicorp.com/vault/docs/secrets/pki/acme).
- [Vault Agent](https://developer.hashicorp.com/vault/docs/agent-and-proxy/agent) and [Vault Secrets Operator](https://developer.hashicorp.com/vault/docs/platform/k8s/vso).
- [Integrated storage (Raft)](https://developer.hashicorp.com/vault/docs/configuration/storage/raft) and the [production hardening guide](https://developer.hashicorp.com/vault/tutorials/operations/production-hardening).