# Cilium

> Understand Cilium's identity-based policy and eBPF datapath, then trace dropped or misrouted flows with cilium-dbg and Hubble.

Canonical: https://www.wiki.jodisand.me/cilium/
Reviewed: 2026-09-24
Related: [Kubernetes](https://www.wiki.jodisand.me/kubernetes/index.md), [Gateway API](https://www.wiki.jodisand.me/gateway-api/index.md), [iproute2](https://www.wiki.jodisand.me/iproute2/index.md), [DNS](https://www.wiki.jodisand.me/dns/index.md), [Linux performance](https://www.wiki.jodisand.me/linux-performance/index.md)


## Cheatsheet

Written against Cilium 1.20. `cilium` is the cluster-level CLI run from your workstation. `cilium-dbg` runs inside each agent pod and only sees its own node. `hubble` needs Hubble Relay reachable, usually through `cilium hubble port-forward`.

| Task | Command |
| --- | --- |
| Overall health | `cilium status --wait` |
| Agent detail on one node | `kubectl exec -n kube-system <cilium-pod> -c cilium-agent -- cilium-dbg status --verbose` |
| Endpoints on that node | `kubectl exec -n kube-system <cilium-pod> -- cilium-dbg endpoint list` |
| Identity to label mapping | `kubectl exec -n kube-system <cilium-pod> -- cilium-dbg identity list` |
| Watch drops live on that node | `kubectl exec -n kube-system <cilium-pod> -- cilium-dbg monitor --type drop` |
| Flows for a pod, cluster-wide | `hubble observe --pod my-namespace/my-app --last 50` |
| Only dropped flows | `hubble observe --verdict DROPPED --last 100` |
| Why a flow was dropped | `hubble observe --to-pod my-namespace/my-app --verdict DROPPED -o json \| jq -r .flow.drop_reason_desc` |
| Services programmed in eBPF | `kubectl exec -n kube-system <cilium-pod> -- cilium-dbg service list` |
| Is kube-proxy replaced | `kubectl exec -n kube-system <cilium-pod> -- cilium-dbg status \| grep KubeProxyReplacement` |
| End-to-end test (creates pods) | `cilium connectivity test` |
| Policy realised on an endpoint | `kubectl exec -n kube-system <cilium-pod> -- cilium-dbg endpoint get <id> -o json \| jq '.[0].status.policy.realized'` |
| Support bundle | `cilium sysdump` |

`kubectl exec ds/cilium` picks an arbitrary agent. Datapath state is per node, so target the agent on the node that runs the pod you are debugging:

```sh
NODE=$(kubectl get pod my-app-7c9d -n my-namespace -o jsonpath='{.spec.nodeName}')
kubectl get pod -n kube-system -l k8s-app=cilium --field-selector spec.nodeName="$NODE" -o name
```

## How Cilium sees the cluster

Cilium attaches eBPF programs to kernel networking hooks (TC, XDP and sockets) on each node, so packets are handled in the kernel rather than by iptables chains. Every pod becomes an *endpoint* with a numeric *identity* derived from its security-relevant labels. Policy is enforced on identities, not IP addresses.

That indirection is the important part. Pods with the same labels share an identity, a rescheduled pod gets the same identity on its new node, and a policy decision is a hash-map lookup on (identity, port, protocol) rather than a walk through rules. The ipcache map translates remote IPs to identities, so a stale ipcache entry shows up as a wrong verdict.

```sh
cilium status --wait                                                       # from outside the cluster
kubectl exec -n kube-system <cilium-pod> -- cilium-dbg status --verbose    # per-node agent view
kubectl exec -n kube-system <cilium-pod> -- cilium-dbg endpoint list       # endpoint ID, identity, labels, policy enforcement
kubectl exec -n kube-system <cilium-pod> -- cilium-dbg identity list       # identity to label mapping
kubectl exec -n kube-system <cilium-pod> -- cilium-dbg bpf ipcache list    # IP to identity (and tunnel endpoint) mapping
```

Reserved identities appear in verdicts and policy:

| Identity | Meaning |
| --- | --- |
| `reserved:host` | The local node, including host-network pods |
| `reserved:remote-node` | Other nodes in the cluster |
| `reserved:kube-apiserver` | The API server endpoints |
| `reserved:world` | Anything outside the cluster not matched by a CIDR rule |
| `reserved:health` | Cilium health-check endpoints |
| `reserved:init` | An endpoint whose identity is not resolved yet |
| `reserved:unmanaged` | Pods not managed by Cilium, such as those started before it |
| `reserved:ingress` | Cilium's Envoy for Ingress and Gateway API traffic |

## Datapath and IPAM

| Routing mode | How packets cross nodes | When it applies |
| --- | --- | --- |
| Encapsulation (`routingMode=tunnel`, default) | VXLAN (default) or Geneve tunnel between nodes | Works on any underlay; costs 50 bytes (VXLAN) of MTU |
| Native routing (`routingMode=native`) | The underlay routes pod CIDRs | The network knows pod routes, via BGP, cloud routes or `autoDirectNodeRoutes` on one L2 segment |
| ENI, Azure or GKE IPAM | Pods get VPC addresses | Cloud-native addressing, no tunnel overhead |

The default IPAM mode is `cluster-pool`: the operator hands each node a pod CIDR from a cluster-wide pool, and the agent allocates pod IPs from it.

```sh
kubectl exec -n kube-system <cilium-pod> -- cilium-dbg status | grep -E 'Routing|IPAM|Masquerading'
kubectl exec -n kube-system <cilium-pod> -- cilium-dbg status --all-addresses   # every allocated IP and its owner
kubectl get ciliumnode -o custom-columns='NODE:.metadata.name,CIDR:.spec.ipam.podCIDRs'
kubectl -n kube-system get cm cilium-config -o yaml | grep -E 'routing-mode|tunnel-protocol|ipam|masquerade|kube-proxy-replacement'
```

Address exhaustion shows as pods stuck in `ContainerCreating` with a CNI error in `kubectl describe pod`. The IPAM line in `cilium-dbg status` shows allocated versus available for that node.

## kube-proxy replacement

With `kubeProxyReplacement=true`, Cilium implements Services in eBPF. Socket-level load balancing translates a ClusterIP to a backend at `connect()` time, so the packet never carries the ClusterIP. `iptables -t nat -L` shows nothing useful on such a cluster; Service state lives in eBPF maps. The Helm default is `false`, which still load-balances ClusterIP traffic per packet but leaves NodePort and LoadBalancer handling to kube-proxy.

```sh
kubectl exec -n kube-system <cilium-pod> -- cilium-dbg status | grep KubeProxyReplacement
kubectl exec -n kube-system <cilium-pod> -- cilium-dbg service list    # frontends and their backends
kubectl exec -n kube-system <cilium-pod> -- cilium-dbg bpf lb list     # the same, read from the eBPF map
kubectl exec -n kube-system <cilium-pod> -- cilium-dbg bpf ct list | head   # connection tracking entries
```

For node-external traffic (NodePort, LoadBalancer, externalIPs), Cilium uses SNAT by default: the node receiving the request forwards it to a backend on another node and the reply returns through the same node. `loadBalancer.mode=dsr` lets the backend reply directly to the client and preserves the client source IP. DSR needs native routing, or Geneve tunnelling with `loadBalancer.dsrDispatch=geneve`, and on AWS the source/destination check disabled. `loadBalancer.algorithm=maglev` gives consistent backend selection across nodes, so a node failure does not reshuffle existing flows. See [kube-proxy free](https://docs.cilium.io/en/stable/network/kubernetes/kubeproxy-free/) for the kernel requirements.

## Network policy

Kubernetes NetworkPolicy works unchanged. CiliumNetworkPolicy adds L7 rules, DNS-based egress, entity selectors, deny rules and a cluster-wide variant. Policies are allow-lists: once any policy selects an endpoint for a direction, everything else in that direction is denied. Deny rules take precedence over allow rules.

```yaml
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata: { name: my-app, namespace: my-namespace }
spec:
  endpointSelector:
    matchLabels: { app: my-app }
  ingress:
    - fromEndpoints: [{ matchLabels: { app: web } }]
      toPorts:
        - ports: [{ port: "8080", protocol: TCP }]
          rules:
            http:
              - { method: "GET", path: "/v1/.*" }      # L7: enforced by Envoy
  egress:
    - toEndpoints: [{ matchLabels: { "k8s:io.kubernetes.pod.namespace": kube-system, "k8s:k8s-app": kube-dns } }]
      toPorts:
        - ports: [{ port: "53", protocol: ANY }]
          rules:
            dns: [{ matchPattern: "*" }]              # routes DNS through the DNS proxy; toFQDNs needs this
    - toFQDNs: [{ matchName: "api.vendor.example.com" }]
      toPorts: [{ ports: [{ port: "443", protocol: TCP }] }]
    - toEntities: ["kube-apiserver"]
```

L7 rules redirect matching traffic through the node-local Envoy proxy (the `cilium-envoy` DaemonSet by default), which adds latency and changes the datapath. Apply them where that cost is worth it. `toFQDNs` works by observing DNS responses in the agent's DNS proxy and allowing the returned IPs, which is why the DNS rule with `rules.dns` must also be present. A name resolved before the policy existed, or by a resolver that bypasses the proxy, is not allowed.

`CiliumClusterwideNetworkPolicy` uses the same schema without a namespace. Use it for baseline rules such as "everything may reach CoreDNS, nothing may reach the cloud metadata service". Host policies are clusterwide policies with a `nodeSelector` instead of an `endpointSelector`, and only take effect with `hostFirewall.enabled=true`.

> [!WARNING] Host policies can remove your access
> A host policy that selects a node without allowing SSH, kubelet (10250) and API server traffic cuts the node off. Put the host endpoint in audit mode first (`cilium-dbg endpoint config <host-endpoint-id> PolicyAuditMode=Enabled`), confirm with `cilium-dbg monitor -t policy-verdict` that only expected flows show `action audit`, and keep console access to one node while rolling out. See [host firewall](https://docs.cilium.io/en/stable/security/host-firewall/).

```sh
kubectl get cnp,ccnp -A
kubectl exec -n kube-system <cilium-pod> -- cilium-dbg policy get                    # all rules the agent holds
kubectl exec -n kube-system <cilium-pod> -- cilium-dbg endpoint get <id> -o json | jq '.[0].status.policy.realized'
```

### Policy examples by layer

Start every namespace with a default deny that still allows DNS and health checks, then add per-application allow rules. Without the DNS egress, the first `toFQDNs` rule silently blocks everything.

```yaml
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata: { name: default-deny, namespace: my-namespace }
spec:
  endpointSelector: {}                       # every pod in the namespace
  ingress:
    - fromEntities: [health]                 # keep cilium-health probes working
  egress:
    - toEndpoints:
        - matchLabels: { "k8s:io.kubernetes.pod.namespace": kube-system, "k8s:k8s-app": kube-dns }
      toPorts:
        - ports: [{ port: "53", protocol: ANY }]
          rules: { dns: [{ matchPattern: "*" }] }
```

L3 rules select by identity, CIDR or entity. `toCIDR` matches only addresses outside the cluster (pod and node IPs are always resolved to identities first), and `toCIDRSet` with `except` carves holes.

```yaml
  egress:
    - toCIDRSet:
        - cidr: 192.0.2.0/24
          except: [192.0.2.1/32]             # allow the subnet but not the router
    - toEntities: [world]                    # anything not in the cluster; broad, prefer toFQDNs or toCIDR
      toPorts: [{ ports: [{ port: "443", protocol: TCP }] }]
    - icmps:                                 # ICMP is not covered by toPorts
        - fields: [{ type: 8, family: IPv4 }]   # echo request
```

L4 rules narrow by port and protocol; a `toPorts` entry with a `rules` block becomes L7 and is redirected through Envoy. Kafka and generic L7 (`l7proto`) exist alongside `http` and `dns`. An explicit deny wins over every allow and does not need a port to be broad:

```yaml
apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata: { name: block-metadata }
spec:
  endpointSelector: {}
  egressDeny:
    - toCIDR: [169.254.169.254/32]           # cloud metadata endpoint, denied for every pod in the cluster
```

```yaml
  ingress:
    - fromEndpoints: [{ matchLabels: { app: frontend } }]
      toPorts:
        - ports: [{ port: "8080", protocol: TCP }]
          rules:
            http:
              - method: GET
                path: "/api/v1/.*"
                headers: ["X-Request-Source: frontend"]   # header must be present with this value
              - method: POST
                path: "/api/v1/orders"
```

An L7 rule that matches nothing returns HTTP 403 from Envoy, visible in Hubble as an `http-request` flow with verdict `DROPPED` rather than a packet-level `POLICY_DENIED`. Use `hubble observe --type l7 --http-status 403` to find them. Policies can also carry `enableDefaultDeny: { ingress: false }` (1.15+) to add allow rules without switching the endpoint into default-deny for that direction, which is the safe way to introduce policy into a namespace that had none.

## Hubble flow observability

Hubble reads flow events from the same eBPF programs that enforce policy, so a verdict in Hubble is the verdict the datapath applied, not a reconstruction. Relay aggregates flows from every node; each agent keeps a ring buffer (4095 flows by default), so `--last` only reaches back a few seconds on a busy node.

```sh
cilium hubble enable --ui                          # enables Hubble, deploys Relay and the UI
cilium hubble port-forward &                       # exposes Relay on localhost:4245
hubble status
hubble observe --pod my-namespace/my-app --last 50
hubble observe --verdict DROPPED --last 100
hubble observe --to-pod my-namespace/my-app --port 8080 -f
hubble observe --protocol dns --last 20
hubble observe --verdict DROPPED -o json | jq -r '.flow | [.source.pod_name, .destination.pod_name, .drop_reason_desc] | @tsv'
```

`-o json` prints one object per line with the flow under `.flow`.

| Drop reason | Meaning |
| --- | --- |
| `POLICY_DENIED` | No allow rule for this identity pair and port |
| `POLICY_DENY` | An explicit deny rule matched |
| `AUTH_REQUIRED` | The policy requires mutual authentication that has not completed |
| `CT_MAP_INSERTION_FAILED` | Connection tracking table full; raise `bpf.ctTcpMax` / `bpf.ctAnyMax` or `bpf.mapDynamicSizeRatio` |
| `SERVICE_BACKEND_NOT_FOUND` | Service has no backend in the datapath, usually zero ready endpoints |
| `STALE_OR_UNROUTABLE_IP` | Packet addressed to an IP no longer owned by a local endpoint |
| `UNSUPPORTED_L3_PROTOCOL` | Non-IP traffic on a managed interface |
| `INVALID_SOURCE_IP` | Source IP not owned by the sending endpoint (spoofing or misconfigured pod) |

### Hubble CLI filters

Every filter has a `--from-` and `--to-` variant and an undirected form; `--not` negates the filter that follows it. Filters of the same kind are ORed, different kinds are ANDed, and `--print-raw-filters` shows exactly what the CLI sends to Relay.

```sh
hubble observe --since 5m --namespace my-namespace                          # time window instead of --last
hubble observe --from-pod my-namespace/my-app --to-fqdn '*.example.com'     # DNS-resolved destinations
hubble observe --type policy-verdict --verdict DROPPED --since 2m           # policy decisions only
hubble observe --type l7 --http-method POST --http-status 5xx -f            # L7 flows through Envoy
hubble observe --type l7 --protocol dns --since 1m -o json | jq -r 'select(.flow.l7.dns.rcode==3) | .flow.l7.dns.query' | sort | uniq -c   # NXDOMAIN
hubble observe --from-label reserved:world --to-namespace my-namespace      # what is reaching in from outside
hubble observe --identity 16777217 --since 1m                               # by numeric identity (CIDR identities start at 16777216)
hubble observe --node-name ip-10-0-1-23.example.com --verdict DROPPED       # one node's drops
hubble observe --not --to-namespace kube-system --verdict DROPPED           # exclude noise
hubble observe --drop-reason-desc CT_MAP_INSERTION_FAILED --since 10m       # a specific drop reason
hubble observe --to-service my-namespace/my-app --since 1m                  # traffic to a ClusterIP
hubble observe -o compact --since 30s | head                                # shorter than the default output
hubble list nodes                                                            # which agents Relay can reach
hubble list namespaces                                                       # namespaces with flows in the last hour
```

`--type` accepts `drop`, `trace`, `l7`, `policy-verdict`, `capture` and `trace-sock`. `-o jsonpb` prints protobuf JSON identical to the API, and `--cel-expression` (1.16+) accepts a CEL filter for anything the flags cannot express. Hubble metrics (`hubble.metrics.enabled` in Helm, for example `{dns,drop,tcp,flow,port-distribution,icmp,httpV2:exemplars=true;labelsContext=source_ip\,source_namespace\,destination_namespace}`) export the same data to [Prometheus](https://www.wiki.jodisand.me/prometheus/) as `hubble_*` series and are the right tool for sustained drop rates rather than the ring buffer.

## Cluster Mesh

Cluster Mesh joins several clusters into one identity and service domain. Each cluster runs a `clustermesh-apiserver` that exposes its state through a shared etcd; agents in every other cluster connect to it, learn remote identities and endpoints, and program them into their ipcache. Policy then works across clusters by label, and a Service annotated as global load-balances to backends in every cluster.

Prerequisites are strict: every cluster needs a unique `cluster.name` and `cluster.id` (1 to 255, or up to 511 with `maxConnectedClusters=511`), non-overlapping pod CIDRs, node-to-node reachability on the pod network (tunnel or native), and the API server reachable from remote nodes, typically as a LoadBalancer Service.

```sh
# During install: identify each cluster
cilium install --set cluster.name=sydney --set cluster.id=1 --context sydney
cilium install --set cluster.name=melbourne --set cluster.id=2 --context melbourne

cilium clustermesh enable --context sydney --service-type LoadBalancer   # deploys clustermesh-apiserver and certs
cilium clustermesh enable --context melbourne --service-type LoadBalancer
cilium clustermesh connect --context sydney --destination-context melbourne   # exchanges CA and endpoints both ways
cilium clustermesh status --context sydney --wait                          # every remote cluster "connected"
cilium connectivity test --context sydney --multi-cluster melbourne        # cross-cluster suite

kubectl exec -n kube-system <cilium-pod> -- cilium-dbg troubleshoot clustermesh     # per-node: DNS, TCP, TLS, etcd to each remote
kubectl exec -n kube-system <cilium-pod> -- cilium-dbg status --verbose | sed -n '/ClusterMesh/,/^$/p'
```

A global Service exists with the same name and namespace in each cluster and carries annotations that control the spread:

```yaml
apiVersion: v1
kind: Service
metadata:
  name: my-app
  namespace: my-namespace
  annotations:
    service.cilium.io/global: "true"      # merge backends from every cluster
    service.cilium.io/shared: "true"      # default; "false" consumes remote backends without exporting local ones
    service.cilium.io/affinity: local     # prefer local backends, fail over to remote; also "remote" or "none"
spec:
  selector: { app: my-app }
  ports: [{ port: 80, targetPort: 8080 }]
```

`cilium-dbg service list --clustermesh-affinity` shows which backends are local and remote for each frontend. Cross-cluster policy uses the same `fromEndpoints` selectors plus `io.cilium.k8s.policy.cluster: melbourne` to restrict a rule to one cluster. Identities are namespace-scoped labels, so `app: web` in cluster A and `app: web` in cluster B share an identity unless you add the cluster label. Hubble flows carry `--cluster` and `--from-cluster` filters once Relay is configured with `hubble.relay` pointing at remote peers or you run `hubble observe` against each cluster.

## Datapath troubleshooting

Work from the pod outwards. Each step reads the eBPF map the datapath actually consulted, so a mismatch between what Kubernetes says and what the map holds is the finding.

```sh
POD=my-app-7c9d; NS=my-namespace
NODE=$(kubectl get pod "$POD" -n "$NS" -o jsonpath='{.spec.nodeName}')
AGENT=$(kubectl get pod -n kube-system -l k8s-app=cilium --field-selector spec.nodeName="$NODE" -o name)
EP=$(kubectl exec -n kube-system "$AGENT" -- cilium-dbg endpoint list -o json | jq -r --arg p "$POD" '.[] | select(.status."external-identifiers"."k8s-pod-name"==$p) | .id')

kubectl exec -n kube-system "$AGENT" -- cilium-dbg endpoint get "$EP" -o json | jq '.[0].status | {state, identity: .identity.id, policy: .policy.realized."policy-enabled", labels: .labels."security-relevant"}'
kubectl exec -n kube-system "$AGENT" -- cilium-dbg bpf policy get "$EP"            # the (identity, port, proto) allow map for this endpoint
kubectl exec -n kube-system "$AGENT" -- cilium-dbg bpf ipcache get 10.244.3.17     # identity the datapath believes a remote IP has
kubectl exec -n kube-system "$AGENT" -- cilium-dbg bpf endpoint list               # local IP to endpoint mapping
kubectl exec -n kube-system "$AGENT" -- cilium-dbg bpf tunnel list                 # remote pod CIDR to node IP (tunnel mode)
kubectl exec -n kube-system "$AGENT" -- cilium-dbg bpf nat list | grep 10.244.3.17 # SNAT entries for masqueraded egress
kubectl exec -n kube-system "$AGENT" -- cilium-dbg bpf ct list global | grep 10.244.3.17   # conntrack; flags show direction and state
kubectl exec -n kube-system "$AGENT" -- cilium-dbg monitor -v --related-to "$EP"   # every event touching this endpoint, with L4 detail
kubectl exec -n kube-system "$AGENT" -- cilium-dbg bpf metrics list                # datapath counters: drops and forwards by reason
kubectl exec -n kube-system "$AGENT" -- cilium-dbg debuginfo > "$NODE-debuginfo.txt"   # everything above in one file
```

`cilium-dbg monitor -v` prints the policy verdict with the source and destination identities, so a `Policy verdict log: ... action deny` for identity pair (12345, 67890) can be resolved with `cilium-dbg identity get 67890`. When the ipcache entry for a remote pod is missing, the packet is treated as `reserved:world` and dropped by any policy that only allows in-cluster identities; check that the remote node's `CiliumNode` object is present and that the agent on that node is healthy. For encapsulation problems, capture the tunnel on the node with `tcpdump -ni eth0 udp port 8472` (VXLAN) or `6081` (Geneve) and confirm packets leave and arrive, then confirm the underlay MTU with `ip link` (see [iproute2](https://www.wiki.jodisand.me/iproute2/)). `cilium-dbg bpf ct list` entries with `TxClosing`/`RxClosing` piling up on one destination point to a backend that stopped answering rather than a Cilium fault.

For kernel-level packet tracing beyond what monitor shows, `pwru` (packet, where are you) from the Cilium project hooks every kernel function that handles an skb and shows exactly where a packet was dropped; run it on the node with `pwru --filter-dst-ip 10.244.3.17 --output-tuple`.

## Troubleshooting

```sh
cilium status --wait
cilium connectivity test                                           # full suite in namespace cilium-test-1
kubectl exec -n kube-system <cilium-pod> -- cilium-dbg monitor --type drop --type policy-verdict
kubectl exec -n kube-system <cilium-pod> -- cilium-dbg map list --verbose
kubectl logs -n kube-system <cilium-pod> -c cilium-agent --previous | tail -50   # why the agent restarted
cilium sysdump                                                     # zip of logs, maps and policies for a bug report
```

| Symptom | Likely cause | Check |
| --- | --- | --- |
| Pods cannot resolve names | DNS egress not allowed, or CoreDNS has no endpoints | `hubble observe --protocol dns`, then [DNS](https://www.wiki.jodisand.me/dns/#a-name-that-will-not-resolve) |
| Small requests work, large transfers stall between nodes | MTU: tunnel overhead exceeds the underlay MTU | `kubectl exec my-app-7c9d -- cat /sys/class/net/eth0/mtu` against the node's NIC MTU |
| Service IP unreachable | No backends, or kube-proxy replacement not active | `cilium-dbg service list`, then EndpointSlice readiness |
| Policy has no effect | Selector labels do not match, or enforcement disabled on the endpoint | `cilium-dbg endpoint list` labels and `POLICY (ingress) ENFORCEMENT` columns |
| `toFQDNs` rule blocks traffic | DNS rule missing, or the client cached the answer before the policy | `cilium-dbg fqdn cache list`, `hubble observe --protocol dns` |
| Traffic allowed that should be denied | A broader clusterwide policy or an entity rule (`world`, `all`) | `kubectl get ccnp`, `cilium-dbg policy get` |
| Agent crash-loops | Missing kernel features, or a config change the agent rejects | `kubectl logs --previous -c cilium-agent`, `cilium-dbg status --verbose` |
| Pods stuck `ContainerCreating` | IPAM exhausted or agent not ready on the node | `kubectl describe pod`, IPAM line of `cilium-dbg status` |
| L7 policy returns 403 but Hubble shows no `POLICY_DENIED` | Envoy rejected at L7; the packet was allowed at L4 | `hubble observe --type l7 --http-status 403`, `cilium-dbg policy get` for the http rules |
| Latency jumps after adding a policy | `toPorts.rules` turned the port into an L7 redirect through Envoy | `cilium-dbg endpoint get <id> -o json \| jq '.[0].status.policy.realized.l4'` for proxy ports, `cilium-dbg status \| grep Proxy` |
| Remote pod treated as `reserved:world` | ipcache has no entry: remote agent unhealthy or CiliumNode missing | `cilium-dbg bpf ipcache get <ip>`, `kubectl get ciliumnode`, `cilium status` |
| Cluster Mesh shows `connected` but global Service has no remote backends | Service not annotated global in both clusters, or names differ | `cilium-dbg service list --clustermesh-affinity`, annotations on both Services |
| `cilium clustermesh connect` hangs | clustermesh-apiserver LoadBalancer has no address, or port 2379 blocked between clusters | `kubectl get svc -n kube-system clustermesh-apiserver`, `cilium-dbg troubleshoot clustermesh` |
| Source IP lost on LoadBalancer traffic | SNAT mode with backend on another node | `cilium-dbg status \| grep -i 'loadbalancer'`, consider `loadBalancer.mode=dsr` or `externalTrafficPolicy: Local` |
| Drops with `CT_MAP_INSERTION_FAILED` under load | Conntrack map full | `cilium-dbg bpf ct list global \| wc -l` against `cilium-dbg map list` max entries |
| Hubble `--last` returns almost nothing | Ring buffer (4095 flows) overrun on a busy node | Raise `hubble.eventBufferCapacity`, use Hubble metrics for rates |

For link, route and neighbour checks on the node itself, see [iproute2](https://www.wiki.jodisand.me/iproute2/#a-connectivity-problem).

## Oneliners

```sh
# Identity, namespace and pod for every endpoint on one node
kubectl exec -n kube-system <cilium-pod> -- cilium-dbg endpoint list -o json | jq -r '.[] | [.status.identity.id, .status."external-identifiers"."k8s-namespace", .status."external-identifiers"."k8s-pod-name"] | @tsv'

# Agents that are not Running
kubectl get pods -n kube-system -l k8s-app=cilium -o wide | grep -v Running

# Drops per source pod in the recent buffer
hubble observe --verdict DROPPED --last 500 -o json | jq -r '.flow.source.pod_name // "unknown"' | sort | uniq -c | sort -rn

# Top talkers by flow count
hubble observe --last 1000 -o json | jq -r '.flow | [.source.pod_name, .destination.pod_name] | @tsv' | sort | uniq -c | sort -rn | head

# Confirm a Service has backends in the datapath (use your ClusterIP)
kubectl exec -n kube-system <cilium-pod> -- cilium-dbg service list | grep -A2 10.96.0.10

# FQDN to IP mappings learned by the DNS proxy
kubectl exec -n kube-system <cilium-pod> -- cilium-dbg fqdn cache list | head -20

# Compare policy revision across agents (should converge to the same number)
kubectl get pods -n kube-system -l k8s-app=cilium -o name | xargs -I{} sh -c 'printf "%s " {}; kubectl exec -n kube-system {} -c cilium-agent -- cilium-dbg policy get -o json | jq .revision'

# Remove connectivity test namespaces and workloads
cilium connectivity test --cleanup

# Agent pod on the node that runs a given pod
kubectl get pod -n kube-system -l k8s-app=cilium --field-selector spec.nodeName="$(kubectl get pod my-app-7c9d -n my-namespace -o jsonpath='{.spec.nodeName}')" -o name

# Endpoint ID for a pod, then its realised policy
EP=$(kubectl exec -n kube-system <cilium-pod> -- cilium-dbg endpoint list -o json | jq -r '.[] | select(.status."external-identifiers"."k8s-pod-name"=="my-app-7c9d") | .id'); kubectl exec -n kube-system <cilium-pod> -- cilium-dbg bpf policy get "$EP"

# Resolve a numeric identity seen in a verdict to labels
kubectl exec -n kube-system <cilium-pod> -- cilium-dbg identity get 16777217

# Endpoints on this node with policy enforcement disabled in either direction
kubectl exec -n kube-system <cilium-pod> -- cilium-dbg endpoint list -o json | jq -r '.[] | select(.status.policy.realized."policy-enabled" != "both") | "\(.id)\t\(.status.policy.realized."policy-enabled")\t\(.status."external-identifiers"."k8s-namespace")/\(.status."external-identifiers"."k8s-pod-name")"'

# Endpoints not in the ready state on this node
kubectl exec -n kube-system <cilium-pod> -- cilium-dbg endpoint list -o json | jq -r '.[] | select(.status.state != "ready") | "\(.id) \(.status.state)"'

# Namespaces with no CiliumNetworkPolicy at all
comm -23 <(kubectl get ns -o name | cut -d/ -f2 | sort) <(kubectl get cnp -A -o jsonpath='{.items[*].metadata.namespace}' | tr ' ' '\n' | sort -u)

# Policies whose endpointSelector matches nothing (candidates for typos)
kubectl get cnp -A -o json | jq -r '.items[] | .metadata.namespace as $ns | "\($ns) \(.metadata.name) " + ((.spec.endpointSelector.matchLabels // {}) | to_entries | map("\(.key)=\(.value)") | join(","))' | while read -r ns name sel; do [ -z "$sel" ] || [ "$(kubectl get pod -n "$ns" -l "$sel" --no-headers 2>/dev/null | wc -l)" -gt 0 ] || echo "$ns/$name selects no pods"; done

# Drop reasons in the last five minutes, counted
hubble observe --verdict DROPPED --since 5m -o json | jq -r .flow.drop_reason_desc | sort | uniq -c | sort -rn

# Identity pairs being denied, resolved to pod labels
hubble observe --type policy-verdict --verdict DROPPED --since 2m -o json | jq -r '.flow | "\(.source.namespace // "world")/\(.source.pod_name // .source.identity) -> \(.destination.namespace // "world")/\(.destination.pod_name // .destination.identity):\(.l4.TCP.destination_port // .l4.UDP.destination_port)"' | sort | uniq -c | sort -rn

# HTTP 5xx seen by Envoy L7 policy, by destination
hubble observe --type l7 --http-status 5xx --since 5m -o json | jq -r '.flow | "\(.destination.pod_name) \(.l7.http.code) \(.l7.http.method) \(.l7.http.url)"' | sort | uniq -c | sort -rn | head

# DNS queries a pod made in the last minute
hubble observe --from-pod my-namespace/my-app --protocol dns --since 1m -o json | jq -r 'select(.flow.l7.type=="REQUEST") | .flow.l7.dns.query' | sort | uniq -c

# Flows from outside the cluster into a namespace
hubble observe --from-label reserved:world --to-namespace my-namespace --since 5m -o compact

# Datapath drop counters on one node, non-zero only
kubectl exec -n kube-system <cilium-pod> -- cilium-dbg bpf metrics list | awk 'NR==1 || ($3+0 > 0 && $1 ~ /Drop|drop/)'

# Conntrack table usage on one node
kubectl exec -n kube-system <cilium-pod> -- sh -c 'cilium-dbg bpf ct list global | wc -l; cilium-dbg map list | grep -E "ct4_global|ct_any4_global"'

# Identity the datapath assigns to a remote IP
kubectl exec -n kube-system <cilium-pod> -- cilium-dbg bpf ipcache get 10.244.3.17

# Nodes as seen by this agent (tunnel endpoints and health)
kubectl exec -n kube-system <cilium-pod> -- cilium-dbg node list

# Cluster-wide health probe matrix from one agent (node and endpoint reachability)
kubectl exec -n kube-system <cilium-pod> -- cilium-health status --probe

# Every Service frontend without a backend
kubectl exec -n kube-system <cilium-pod> -- cilium-dbg service list -o json | jq -r '.[] | select((.status.realized["backend-addresses"] // []) | length == 0) | .status.realized["frontend-address"] | "\(.ip):\(.port)"'

# FQDN policy: names with no cached IPs (rule will block until resolved through the proxy)
kubectl exec -n kube-system <cilium-pod> -- cilium-dbg fqdn cache list -o json | jq -r '.[] | select((.ips // []) | length == 0) | .fqdn'

# Effective Helm values on the running installation
helm get values cilium -n kube-system

# Cilium version per agent (catches a half-finished upgrade)
kubectl get pods -n kube-system -l k8s-app=cilium -o jsonpath='{range .items[*]}{.spec.nodeName}{"\t"}{.spec.containers[?(@.name=="cilium-agent")].image}{"\n"}{end}' | sort -k2 | uniq -c -f1

# Cluster Mesh: connection state to every remote cluster from one agent
kubectl exec -n kube-system <cilium-pod> -- cilium-dbg status --verbose | sed -n '/ClusterMesh/,/^[A-Z]/p'

# Global Services and their affinity settings
kubectl get svc -A -o json | jq -r '.items[] | select(.metadata.annotations["service.cilium.io/global"]=="true") | "\(.metadata.namespace)/\(.metadata.name)\tshared=\(.metadata.annotations["service.cilium.io/shared"] // "true")\taffinity=\(.metadata.annotations["service.cilium.io/affinity"] // "none")"'

# Put an endpoint into policy audit mode (logs verdicts, enforces nothing) and watch
kubectl exec -n kube-system <cilium-pod> -- cilium-dbg endpoint config "$EP" PolicyAuditMode=Enabled && kubectl exec -n kube-system <cilium-pod> -- cilium-dbg monitor -t policy-verdict --related-to "$EP"
```

## Scripts

Report, for every node, agent readiness, endpoint counts, IPAM headroom and datapath drops, so a single unhealthy node stands out before it pages you.

```sh
#!/usr/bin/env bash
# cilium-node-report.sh: per-node agent health, endpoints, IPAM usage and drop counters
set -euo pipefail
printf '%-40s %-8s %-6s %-14s %-10s %s\n' NODE READY EPS IPAM_USED/MAX DROPS WARNINGS
kubectl get pods -n kube-system -l k8s-app=cilium -o json \
  | jq -r '.items[] | "\(.spec.nodeName) \(.metadata.name) \(.status.containerStatuses[] | select(.name=="cilium-agent") | .ready)"' \
  | while read -r node pod ready; do
    if [ "$ready" != true ]; then printf '%-40s %-8s\n' "$node" NOTREADY; continue; fi
    status=$(kubectl exec -n kube-system "$pod" -c cilium-agent -- cilium-dbg status -o json 2>/dev/null) || { printf '%-40s %-8s\n' "$node" EXEC-FAIL; continue; }
    eps=$(kubectl exec -n kube-system "$pod" -c cilium-agent -- cilium-dbg endpoint list -o json | jq 'length')
    ipam=$(jq -r '.ipam | "\(.allocations | length)/\(((.ipv4 // []) | length) + (.allocations | length))"' <<<"$status")   # used / (used + free)
    drops=$(kubectl exec -n kube-system "$pod" -c cilium-agent -- cilium-dbg bpf metrics list -o json | jq '[.[] | select(.reason | test("Policy|Stale|Unsupported|Invalid")) | .packets] | add // 0')
    warn=$(jq -r '[.controllers[]? | select(.status["consecutive-failure-count"] > 0) | .name] | join(",")' <<<"$status")
    printf '%-40s %-8s %-6s %-14s %-10s %s\n' "$node" ok "$eps" "$ipam" "$drops" "${warn:--}"
  done
```

Audit a namespace before enabling default deny: collect the identity pairs and ports actually in use over a window, and emit a CiliumNetworkPolicy skeleton that allows exactly those flows for review.

```python
#!/usr/bin/env python3
"""Generate a CiliumNetworkPolicy allow-list from observed Hubble flows.

Usage: hubble observe --to-namespace my-namespace --since 30m -o json | ./flows-to-policy.py my-namespace my-app > policy.yaml
Review the output before applying; it reflects what happened, not what should be allowed.
"""
import json
import sys
from collections import defaultdict

import yaml

ns, app = sys.argv[1], sys.argv[2]
pairs: dict[tuple, set] = defaultdict(set)
for line in sys.stdin:
    flow = json.loads(line).get("flow", {})
    dst = flow.get("destination", {})
    if dst.get("namespace") != ns or not any(l == f"k8s:app={app}" for l in dst.get("labels", [])):
        continue
    if flow.get("verdict") not in ("FORWARDED", "DROPPED"):
        continue
    src = flow.get("source", {})
    src_labels = tuple(sorted(l for l in src.get("labels", []) if l.startswith("k8s:app=") or l.startswith("k8s:io.kubernetes.pod.namespace=") or l.startswith("reserved:")))
    l4 = flow.get("l4", {})
    for proto in ("TCP", "UDP"):
        if proto in l4:
            pairs[src_labels].add((str(l4[proto]["destination_port"]), proto))

ingress = []
for labels, ports in sorted(pairs.items()):
    if any(l.startswith("reserved:") for l in labels):
        ingress.append({"fromEntities": [l.split(":", 1)[1] for l in labels], "toPorts": [{"ports": [{"port": p, "protocol": pr} for p, pr in sorted(ports)]}]})
        continue
    match = {l.split("=", 1)[0]: l.split("=", 1)[1] for l in labels}
    ingress.append({"fromEndpoints": [{"matchLabels": match}], "toPorts": [{"ports": [{"port": p, "protocol": pr} for p, pr in sorted(ports)]}]})

policy = {"apiVersion": "cilium.io/v2", "kind": "CiliumNetworkPolicy",
          "metadata": {"name": f"{app}-observed", "namespace": ns},
          "spec": {"endpointSelector": {"matchLabels": {"app": app}}, "ingress": ingress}}
yaml.safe_dump(policy, sys.stdout, sort_keys=False)
```

Upstream reference: [Cilium documentation](https://docs.cilium.io/en/stable/), [troubleshooting guide](https://docs.cilium.io/en/stable/operations/troubleshooting/).