# DNS > Resolve and trace names with dig and resolvectl, read the answer, reason about TTLs and caching, and debug search domains and Kubernetes DNS. Canonical: https://www.wiki.jodisand.me/dns/ Reviewed: 2026-09-24 Related: [HTTP and curl](https://www.wiki.jodisand.me/http/index.md), [TLS and certificates](https://www.wiki.jodisand.me/tls/index.md), [iproute2](https://www.wiki.jodisand.me/iproute2/index.md), [Kubernetes](https://www.wiki.jodisand.me/kubernetes/index.md), [systemd](https://www.wiki.jodisand.me/systemd/index.md) ## Cheatsheet | Task | Command | | --- | --- | | Resolve a name | `dig +short api.example.com` | | Resolve as applications do (hosts file, NSS) | `getent ahosts api.example.com` | | Which resolver answered | `dig api.example.com \| grep SERVER` | | Ask a specific resolver, bypassing the local one | `dig @1.1.1.1 api.example.com` | | Ask the authoritative server, no recursion | `dig +norecurse @ns1.example.com api.example.com` | | Follow the delegation from the root | `dig +trace api.example.com` | | Reverse lookup | `dig -x 203.0.113.10 +short` | | Nameservers for a zone | `dig NS example.com +short` | | Mail records | `dig MX example.com +short` | | TXT (SPF, verification) | `dig TXT example.com +short` | | Zone serial | `dig SOA example.com +short` | | Remaining TTL in a cache | `dig api.example.com +noall +answer` (second column counts down) | | What systemd-resolved does | `resolvectl query api.example.com` | | Flush systemd-resolved cache | `resolvectl flush-caches` | | Name from inside a cluster | `kubectl run -it --rm dnstest --image=busybox --restart=Never -- nslookup api.my-namespace` | `dig` and `delv` come from BIND (`bind-utils` on Fedora and RHEL, `dnsutils` or `bind9-dnsutils` on Debian and Ubuntu). ## A name that will not resolve Answer three questions in order: does the authoritative server have the record, does the recursive resolver return it, and does this host (or container) use that resolver. ```sh getent ahosts api.example.com # what applications on this host get, including /etc/hosts dig +short api.example.com # what the configured resolver returns dig @1.1.1.1 +short api.example.com # what a public resolver returns dig NS example.com +short # who is authoritative dig +norecurse @ns1.example.com api.example.com # straight from the zone, no cache involved dig +trace api.example.com | tail -20 # delegation from the root, done by dig itself resolvectl status | head -20 # which servers and search domains this host uses ``` `dig` talks DNS directly to the server in `/etc/resolv.conf`. It ignores `/etc/hosts`, `nsswitch.conf`, mDNS and per-link routing in systemd-resolved. When `dig` works and the application does not, compare with `getent`. | Result | Meaning | | --- | --- | | `NXDOMAIN` | The name does not exist at all; check for a typo, a missing record or the wrong zone | | `NOERROR` with an empty answer (NODATA) | The name exists but has no record of that type, for example A queried where only AAAA exists | | `SERVFAIL` | The resolver could not get an answer: broken delegation, unreachable or lame authoritative servers, or DNSSEC validation failure | | `REFUSED` | The server will not answer this query from you, often recursion not allowed for your source address | | `connection timed out; no servers could be reached` | Nothing answered on port 53; firewall, wrong server address or resolver down | | Answer differs by resolver | Caching within TTL, split-horizon DNS, or geo-based answers | | Resolves but the application fails | `/etc/hosts` override, a different resolver in the container, or the application cached the address at startup | `dig +short` prints nothing for both `NXDOMAIN` and NODATA. Use plain `dig` when diagnosing, because the status line and flags are the diagnosis. ## Reading dig output ```text ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 23405 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1 ;; ANSWER SECTION: api.example.com. 60 IN CNAME lb.example.net. lb.example.net. 30 IN A 203.0.113.10 ;; SERVER: 192.0.2.53#53(192.0.2.53) (UDP) ;; Query time: 4 msec ``` | Field | Meaning | | --- | --- | | `status` | Response code; `NOERROR` even with zero answers | | `flags: aa` | Authoritative answer, straight from the zone | | `flags: rd` / `ra` | Recursion desired by the client / available at this server | | `flags: ad` | The resolver validated the answer with DNSSEC | | `flags: tc` | Response truncated; the client should retry over TCP | | TTL column | Seconds left in the cache that answered; a full TTL means it was just fetched | | `SERVER` | Which server actually answered. Check this first when results disagree | | `Query time` | A slow first query and a fast repeat means the first one recursed and the second came from cache | ## dig, drill and delv `dig` options starting with `+` control the query and the display; the ones starting with `-` select the transport, the type and the server. Options accumulate left to right, so `+noall +answer` clears every section and then re-enables the answer. ```sh dig -t MX example.com +noall +answer # one section only; -t and a bare type name are equivalent dig example.com +short +identify # answer plus the server and port that supplied it dig SOA example.com +multiline # SOA fields on separate lines with field names dig +nssearch example.com # SOA from every authoritative server, one line each, with query time dig +norecurse +dnssec DNSKEY example.com @ns1.example.com # zone keys with RRSIGs, from the source dig +tcp AXFR example.com @ns1.example.com # zone transfer; refused unless your address is allowed dig +tls @1.1.1.1 example.com # DNS over TLS, port defaults to 853 dig +https @1.1.1.1 example.com # DNS over HTTPS, port defaults to 443 dig +subnet=192.0.2.0/24 example.com @8.8.8.8 # EDNS Client Subnet: what a client in that network would be told dig +bufsize=512 +noedns example.com # force pre-EDNS behaviour to reproduce truncation dig +timeout=2 +tries=1 example.com # fail fast instead of waiting 5 s and retrying 3 times dig -4 example.com; dig -6 example.com # restrict the transport, not the record type dig -p 5353 @127.0.0.1 example.com # a resolver on a non-standard port dig -f queries.txt +short # batch mode: one query per line in the file dig +qr example.com # print the query as sent; shows EDNS options and flags ``` `+nssearch` is the quickest way to find a secondary that has not picked up a change: every serial in the output should match. `+ttlunits` prints TTLs as `1h` rather than `3600` and is easier to read on long answers. `drill` (from ldns, package `ldns-utils` on Fedora, `ldnsutils` on Debian) covers the same ground with different spelling: `drill -T` traces from the root, `drill -D` sets the DO bit, `drill -S name` chases signatures up to a trust anchor, `drill -x 203.0.113.10` reverses, `drill -t` uses TCP and `drill -a` retries a truncated answer with a larger buffer and then TCP. `drill -s DNSKEY example.com` prints the matching DS records next to each key, which is what you paste into the parent zone. `delv` is a validating client. It does its own resolution and validation rather than trusting the `ad` flag from a resolver, so it answers the question "would a validating resolver accept this zone" from any host. ```sh delv example.com # "; fully validated", "; unsigned answer" or an explanation of the failure delv +vtrace example.com # every validation step: which key, which DS, which RRSIG delv +rtrace example.com # every query delv sent while resolving delv @ns1.example.com +cdflag example.com # fetch with checking disabled, still validate locally delv -a /etc/trusted-key.key +root=. example.com # alternate trust anchor file ``` ## Record types | Type | Purpose | Note | | --- | --- | --- | | `A` / `AAAA` | Name to IPv4 / IPv6 | Clients may query both; a slow or broken AAAA path delays connections | | `CNAME` | Alias to another name | Cannot coexist with other records at the same name, so never at a zone apex | | `ALIAS` / `ANAME` | Apex alias | Provider-specific, flattened to A/AAAA on the server | | `HTTPS` / `SVCB` | Service endpoint, ALPN and hints (RFC 9460) | Browsers use `HTTPS` to learn HTTP/3 support and apex aliasing | | `MX` | Mail exchangers with preference | Lower number is tried first | | `TXT` | Arbitrary text | SPF, DKIM, DMARC, domain verification | | `SRV` | Service host and port | Named `_service._proto.name` | | `NS` | Delegates a zone | The set at the parent should match the set in the child zone | | `SOA` | Zone metadata, serial, negative-cache TTL | Serial must increase for secondaries to transfer | | `PTR` | Reverse mapping | Lives under `in-addr.arpa` / `ip6.arpa`, controlled by whoever owns the address block | | `CAA` | CAs allowed to issue certificates for the name | A CA must refuse issuance if not listed; see [TLS](https://www.wiki.jodisand.me/tls/) | Record data as it appears in a zone file or a `dig` answer: ```text example.com. 3600 IN SOA ns1.example.com. hostmaster.example.com. 2026092401 7200 900 1209600 300 example.com. 3600 IN NS ns1.example.com. example.com. 3600 IN MX 10 mail.example.com. example.com. 3600 IN TXT "v=spf1 mx -all" example.com. 3600 IN CAA 0 issue "letsencrypt.org" example.com. 3600 IN HTTPS 1 . alpn="h2,h3" www.example.com. 300 IN CNAME lb.example.net. _sip._tcp.example.com. 3600 IN SRV 10 60 5060 sip.example.com. 10.2.0.192.in-addr.arpa. 3600 IN PTR api.example.com. ``` The SOA fields after the two names are serial, refresh, retry, expire and minimum. Refresh is how often a secondary checks the serial, retry how soon it re-checks after a failure, expire how long it keeps serving the zone without reaching the primary, and minimum is the negative-cache TTL. A serial in `YYYYMMDDnn` form leaves 99 changes a day and stays comparable; secondaries transfer only when the serial is greater in serial-number arithmetic (RFC 1982), so a serial that goes backwards is silently ignored. `TXT` strings are limited to 255 characters per string; longer values such as DKIM keys are split into several quoted strings in one record and concatenated by the reader. `SRV` fields are priority, weight, port and target; a target of `.` means the service is explicitly absent. `HTTPS` with priority `0` is an alias form that behaves like a CNAME at the apex; priority `1` and above carry parameters such as `alpn` and `ipv4hint`. ## Zone files A zone file is the text form of a zone as BIND, NSD, Knot and most providers' import tools read it (RFC 1035 master file format). Names without a trailing dot are relative to `$ORIGIN`; a bare `@` is the origin itself. A line without a name inherits the previous name. `$TTL` sets the default TTL for records without one. ```text $ORIGIN example.com. $TTL 3600 @ IN SOA ns1 hostmaster ( 2026092401 ; serial 7200 ; refresh 900 ; retry 1209600 ; expire 300 ) ; negative-cache TTL IN NS ns1 IN NS ns2 IN MX 10 mail ns1 IN A 192.0.2.53 ns2 IN A 198.51.100.53 mail IN A 192.0.2.25 api 60 IN A 192.0.2.10 www IN CNAME api ``` Check a zone before loading it. `named-checkzone` parses the file as `named` would and reports the serial it read; `named-checkconf -z` does that for every zone in the configuration. ```sh named-checkzone example.com /var/named/example.com.zone # "OK" and the loaded serial, or the line that failed named-checkconf -z /etc/named.conf # configuration syntax and every zone file named-checkconf -p /etc/named.conf | less # the configuration with includes expanded and defaults applied rndc reload example.com # reload one zone after editing its file; rndc reload alone reloads all rndc zonestatus example.com # serial, type, next refresh and whether it is loaded rndc retransfer example.com # on a secondary: force a transfer regardless of serial ``` The common mistakes are a missing trailing dot on a fully qualified name inside the file (`mail.example.com` becomes `mail.example.com.example.com.`), a serial that was not incremented so secondaries never transfer, and a CNAME sharing a name with any other record. `named-checkzone` catches the last; the first two look valid and show up as wrong answers. ## TTL and caching Changes do not propagate. A new record is visible immediately to anyone who asks the authoritative servers. Resolvers that cached the previous answer keep serving it until its TTL runs out. ```sh dig api.example.com +noall +answer # TTL remaining in your resolver's cache dig +norecurse @ns1.example.com api.example.com # the record and full TTL at the source watch -n5 'dig +short @8.8.8.8 api.example.com' # watch one public resolver pick up a change ``` Before a planned change, lower the TTL (for example to 60 seconds) and wait at least one old TTL so every cache holds the short value. Make the change, then raise the TTL again. Negative answers are cached too (RFC 2308). The negative TTL is the lower of the SOA record's own TTL and its MINIMUM field. An `NXDOMAIN` fetched just before a record was created lingers for that long. Some clients cache beyond DNS: the JVM caches lookups according to `networkaddress.cache.ttl`, and many applications resolve once at startup and never again. Restarting the application is sometimes the only fix. ## Search domains and ndots For a name with fewer dots than `ndots`, the resolver tries each search domain appended first and the name as written last. A name with at least `ndots` dots is tried as written first. A trailing dot (`api.example.com.`) marks the name fully qualified and skips the search list. ```text nameserver 10.96.0.10 search my-namespace.svc.cluster.local svc.cluster.local cluster.local options ndots:5 ``` That is a typical Kubernetes pod `/etc/resolv.conf`. `api.example.com` has two dots, fewer than five, so the resolver tries `api.example.com.my-namespace.svc.cluster.local`, then the other two cluster suffixes and any search domains inherited from the node, and only then `api.example.com`. Each attempt is usually sent for both A and AAAA. In hot paths, use a trailing dot for external names or lower `ndots` per pod with `dnsConfig.options`. The glibc and musl (Alpine) resolvers handle search lists and parallel A/AAAA queries differently, so the same image logic can behave differently after a base image change. ```sh cat /etc/resolv.conf resolvectl query api.example.com # shows the link and whether the answer came from cache resolvectl statistics # cache hits, misses and DNSSEC counters resolvectl flush-caches # needs root or polkit authorisation ``` On hosts with systemd-resolved, `/etc/resolv.conf` usually points at the stub `127.0.0.53`, and the real upstream servers are listed per link in `resolvectl status`. See [systemd](https://www.wiki.jodisand.me/systemd/#a-failing-service) if `systemd-resolved` itself is failing. ## systemd-resolved systemd-resolved routes each query to a link based on the link's search and routing domains, then falls back to whichever links have the default route. A routing domain is written with a leading `~` and steers queries without being appended to short names, which is what a VPN link needs: `~corp.example.com` sends only that domain to the VPN's resolver, and `~.` makes a link the default for everything else. When two links both claim the default route, resolved sends the query to both and takes the first answer, which is the usual cause of "DNS works only sometimes after connecting the VPN". ```sh resolvectl status eth0 # servers, search domains, DNSSEC and DNS-over-TLS state for one link resolvectl query --type=MX example.com # explicit type; disables search-domain logic resolvectl query --cache=no api.example.com # bypass the cache for this lookup resolvectl query --validate=no api.example.com # skip DNSSEC validation for this lookup resolvectl service _sip._tcp example.com # SRV lookup with the targets resolved resolvectl dns eth0 192.0.2.53 2001:db8::53 # set servers for a link until the network manager changes them resolvectl domain eth0 '~corp.example.com' # routing domain: send only this suffix to eth0's servers resolvectl default-route eth0 false # stop this link from receiving unrelated queries resolvectl revert eth0 # drop the per-link overrides made with the commands above resolvectl show-cache # dump cached records (needs root) resolvectl log-level debug # verbose logs in journalctl -u systemd-resolved until reset to info ``` Settings made with `resolvectl dns` and `resolvectl domain` are runtime only. Persistent configuration belongs in `/etc/systemd/resolved.conf.d/*.conf` for global values or in the network manager's per-connection settings (`nmcli connection modify my-vpn ipv4.dns-search '~corp.example.com'`, or `[Network] Domains=~corp.example.com` in a `.network` file). ```ini # /etc/systemd/resolved.conf.d/upstream.conf [Resolve] DNS=192.0.2.53#dns.example.com 2001:db8::53 # server with the name to verify for DNS-over-TLS FallbackDNS= DNSOverTLS=opportunistic # yes enforces it and fails closed DNSSEC=allow-downgrade # yes fails closed on unsigned-looking answers Domains=~. Cache=yes ``` `/etc/resolv.conf` can be one of three things: a symlink to `/run/systemd/resolve/stub-resolv.conf` (the stub at `127.0.0.53`, search domains included), to `/run/systemd/resolve/resolv.conf` (the upstream servers directly, which bypasses per-link routing and the cache), or a static file managed by something else. `ls -l /etc/resolv.conf` tells you which, and a static file that another tool keeps rewriting is a fight between NetworkManager, resolved and a VPN client. ## Kubernetes DNS CoreDNS serves the cluster domain (usually `cluster.local`) and forwards everything else to the upstream servers in its `forward` plugin, often the node's `/etc/resolv.conf`. A Service name resolves to its ClusterIP; a headless Service resolves to the ready pod IPs. | Name | Resolves to | | --- | --- | | `api` | Service `api` in the pod's own namespace, via the search list | | `api.my-namespace` | Service `api` in `my-namespace` | | `api.my-namespace.svc.cluster.local` | Fully qualified, no search expansion | | `10-1-2-3.my-namespace.pod.cluster.local` | Pod with IP 10.1.2.3 (CoreDNS `pods` option must be enabled) | | `_grpc._tcp.api.my-namespace.svc.cluster.local` | SRV record for the Service port named `grpc` | ```sh kubectl run -it --rm dnstest --image=nicolaka/netshoot --restart=Never -- dig api.my-namespace.svc.cluster.local kubectl -n kube-system logs -l k8s-app=kube-dns --tail 50 kubectl -n kube-system get configmap coredns -o yaml kubectl -n kube-system get endpointslice -l kubernetes.io/service-name=kube-dns ``` Intermittent resolution failures in a cluster usually come from unhealthy or overloaded CoreDNS replicas, a NetworkPolicy that blocks UDP and TCP 53 to `kube-system`, or conntrack races on UDP. The records themselves are rarely the problem. See [Kubernetes](https://www.wiki.jodisand.me/kubernetes/#start-with-a-failing-workload) and [Cilium](https://www.wiki.jodisand.me/cilium/) for policy checks. ## DNSSEC DNSSEC signs records so a validating resolver can prove an answer came from the zone owner unmodified. It does not encrypt queries; DNS over TLS or HTTPS does that. ```sh dig +dnssec api.example.com | grep -E 'RRSIG|flags:' # ad in flags = validated by the resolver dig +cdflag api.example.com # checking disabled: does it resolve without validation? delv api.example.com # validates locally and explains failures dig DS example.com +short # delegation signer record at the parent ``` A `SERVFAIL` that turns into `NOERROR` with `+cdflag` is a validation failure: expired signatures, or a DS record at the parent that no longer matches the zone's keys after a key rollover or DNS provider move. The chain of trust runs from the root key, through a `DS` record in each parent that hashes the child's key-signing key (`DNSKEY` with flags `257`), to the zone-signing key (flags `256`) that signs the records. Every link is checkable by hand. ```sh dig DNSKEY example.com +short | awk '$1 == 257' # the KSK(s), what the parent's DS must hash dig DS example.com @a.gtld-servers.net +norecurse +short # DS as published at the parent dig +dnssec +multiline SOA example.com | grep -A1 RRSIG # signature validity window: inception and expiration dnssec-dsfromkey -2 Kexample.com.+013+12345.key # SHA-256 DS record to give the registrar, from the KSK file dnssec-verify -o example.com /var/named/example.com.zone.signed # verify a signed zone file offline before publishing delv +vtrace example.com 2>&1 | grep -iE 'validat|bogus|insecure' # where a broken chain breaks ``` `RRSIG` expiration is the silent failure: a zone signed once by hand validates until the signatures expire, typically after 30 days, and then every validating resolver returns `SERVFAIL` at once. Use automatic signing (`dnssec-policy` in BIND, or the provider's managed signing) and alert on the earliest expiration in the zone. When moving a signed zone between providers, remove the DS at the registrar first, wait the parent's DS TTL, move the zone, then publish the new DS. A DS pointing at keys the new provider does not hold makes the zone bogus for every validating client, which is most of them. ## Split horizon Split horizon serves different answers for the same name depending on who asks: internal clients get private addresses, the internet gets public ones. It is usually the explanation when `dig` on the VPN and `dig @1.1.1.1` disagree and nothing is cached. In BIND, `view` blocks with `match-clients` select a complete configuration per client set; the first matching view wins and each view carries its own zones, so the same zone name loads from two different files: ```text view "internal" { match-clients { 192.0.2.0/24; 2001:db8::/32; localhost; }; recursion yes; zone "example.com" { type primary; file "internal/example.com.zone"; }; }; view "external" { match-clients { any; }; recursion no; zone "example.com" { type primary; file "external/example.com.zone"; }; }; ``` In unbound, `local-zone` and `local-data` override a name for every client, and `forward-zone` sends a suffix to an internal authoritative server instead of the internet. `transparent` answers from local data where it exists and recurses for everything else under the name; `static` refuses to recurse, so an unlisted name under it is `NXDOMAIN`; `redirect` answers the whole subtree with the same data. ```text server: local-zone: "corp.example.com." transparent local-data: "git.corp.example.com. IN A 192.0.2.40" local-data-ptr: "192.0.2.40 git.corp.example.com" private-domain: "corp.example.com." # allow RFC 1918 answers from this domain despite rebinding protection forward-zone: name: "example.internal." forward-addr: 192.0.2.53 ``` Clients on a split-horizon network need their queries to reach the internal resolver; a laptop with a hard-coded public resolver bypasses the whole arrangement, and a resolver with `private-address` rebinding protection strips internal addresses returned by external names unless the domain is listed in `private-domain`. ## Running a resolver A local recursive resolver removes the dependency on an upstream and gives you a cache you can inspect and flush. unbound is the smaller choice for a recursive-only role; BIND does recursion and authoritative service in one daemon. Minimal unbound as a caching resolver for one network, validating DNSSEC with the root anchor that `unbound-anchor` maintains: ```text # /etc/unbound/unbound.conf server: interface: 0.0.0.0 interface: ::0 access-control: 192.0.2.0/24 allow access-control: 2001:db8::/32 allow # everything not listed is refused auto-trust-anchor-file: "/var/lib/unbound/root.key" prefetch: yes # refresh popular records before they expire cache-min-ttl: 0 # honour published TTLs; raising this serves stale data harden-dnssec-stripped: yes verbosity: 1 log-queries: no # yes for a short diagnosis only; one line per query remote-control: control-enable: yes control-interface: 127.0.0.1 ``` ```sh unbound-checkconf # parse the config; run before every reload sudo systemctl reload unbound # or unbound-control reload, which also flushes the cache unbound-control status # version, threads, uptime; exit code 3 if not running unbound-control stats_noreset | grep -E 'total.num.(queries|cachehits|cachemiss)|num.answer.rcode' unbound-control lookup api.example.com # which nameservers unbound would ask, with their RTT unbound-control flush_zone example.com # drop everything at and below the name unbound-control flush_negative # drop cached NXDOMAIN, NODATA and SERVFAIL answers unbound-control dump_cache > cache.txt # full cache in text; load_cache reads the same format unbound-control local_data 'test.example.com. IN A 192.0.2.99' # runtime override, lost on restart unbound-host -C /etc/unbound/unbound.conf -v api.example.com # resolve with that config and print the security status ``` Minimal BIND as a recursive resolver plus one authoritative zone: ```text # /etc/named.conf options { directory "/var/named"; listen-on { 192.0.2.53; 127.0.0.1; }; allow-query { 192.0.2.0/24; localhost; }; allow-recursion { 192.0.2.0/24; localhost; }; # separate from allow-query; an open recursor is an amplifier recursion yes; dnssec-validation auto; # built-in root trust anchor, RFC 5011 tracked }; zone "example.com" { type primary; file "example.com.zone"; allow-transfer { 198.51.100.53; }; # secondaries only; set it explicitly rather than relying on the default notify yes; }; ``` ```sh named-checkconf -z && rndc reconfig # reconfig loads new zones without reloading existing ones rndc status # zones loaded, recursive clients, query logging state rndc dumpdb -cache && less /var/named/data/cache_dump.db # cache contents with remaining TTLs rndc flushname api.example.com # one name; rndc flushtree drops the subtree, rndc flush drops everything rndc querylog # toggle per-query logging to the configured channel rndc dnssec -status example.com # key states under dnssec-policy rndc sync -clean example.com # write dynamic-update journal to the zone file ``` `rndc freeze example.com` stops dynamic updates so the zone file can be edited by hand; `rndc thaw example.com` reloads it and resumes. Editing the file while it is not frozen loses the edit at the next journal write. ## Troubleshooting | Symptom | Likely cause | Check | | --- | --- | --- | | `NXDOMAIN` for a record you just created | Negative answer cached before creation | SOA negative TTL; query the authoritative server with `+norecurse` | | Old address after a change | Positive cache within the old TTL, or the application cached it | `dig` against several resolvers; restart the client | | `SERVFAIL` from one resolver only | That resolver validates DNSSEC and the zone fails validation | `dig +cdflag`; `delv` | | `SERVFAIL` everywhere | Lame delegation: parent NS points at servers that do not serve the zone | `dig +trace`; query each NS with `+norecurse` | | `dig` works, application does not | `/etc/hosts`, NSS order, or a different resolver in the container | `getent ahosts`; `cat /etc/resolv.conf` inside the container | | Slow first request, fast repeats | Search list expansion or a slow upstream | Count queries with `resolvectl monitor` or CoreDNS logs; fully qualify the name | | 5 second delays on lookups | UDP reply lost (conntrack race or firewall), resolver waits for its timeout | Packet capture on port 53; `options single-request-reopen` (glibc) | | Large responses fail | UDP truncated and TCP 53 blocked | `dig +tcp`; allow TCP 53 | | Reverse lookup returns nothing | No PTR record, or the address owner has not delegated the reverse zone | `dig -x` against the provider's servers | | Name resolves in the cluster only intermittently | CoreDNS replica unhealthy or NetworkPolicy drop | CoreDNS logs and EndpointSlice; policy on port 53 | | Secondaries serve the old zone | Serial not incremented, or transfer refused | `dig +nssearch example.com`; `rndc zonestatus`; `allow-transfer` on the primary | | Every validating resolver fails at once | RRSIGs expired | `dig +dnssec +multiline SOA example.com`, compare the RRSIG expiration with today | | Zone bogus after a provider move | DS at the registrar hashes keys the new provider does not have | `dig DS example.com +short` against `dnssec-dsfromkey` of the live KSK; `delv +vtrace` | | Internal names fail after connecting to a VPN | Routing domain missing or a second link owns the default route | `resolvectl status`; `resolvectl domain vpn0 '~corp.example.com'` | | Private address stripped from an answer | Resolver rebinding protection (`private-address`) | `unbound-control list_local_zones`; add `private-domain` for that suffix | | `REFUSED` from your own resolver | Client address outside `access-control` or `allow-recursion` | `unbound-checkconf`; `named-checkconf -p \| grep -A3 allow-recursion` | | Name resolves for `dig` but `nslookup` or the app returns a wrong address | `/etc/hosts` entry, or `myhostname`/`mdns` in `nsswitch.conf` | `getent ahosts`; `grep hosts /etc/nsswitch.conf` | | `rndc: connect failed: connection refused` | `named` not running, or `controls` not listening on `127.0.0.1#953` | `systemctl status named`; `ss -ltnp \| grep 953`; `rndc-confgen` for the key | | Changes to a zone file disappear | Dynamic updates rewrote the file from the journal | `rndc freeze` before editing, `rndc thaw` after | For connectivity to the resolver itself, see [iproute2](https://www.wiki.jodisand.me/iproute2/#a-connectivity-problem). ## Oneliners ```sh # Compare answers across public resolvers for r in 1.1.1.1 8.8.8.8 9.9.9.9; do printf '%s: %s\n' "$r" "$(dig +short @"$r" api.example.com | tr '\n' ' ')"; done # Common record types for a name for t in A AAAA CNAME HTTPS MX TXT NS SOA CAA; do printf '%-6s %s\n' "$t" "$(dig +short "$t" example.com | tr '\n' ' ')"; done # Serial on every authoritative server, to check zone transfer and replication for ns in $(dig +short NS example.com); do printf '%s %s\n' "$ns" "$(dig +short SOA @"$ns" example.com | awk '{print $3}')"; done # Reverse lookups for part of a subnet for i in $(seq 1 20); do dig +short -x "192.0.2.$i" | sed "s/^/192.0.2.$i /"; done # Resolve through one interface's resolver (systemd-resolved) resolvectl query --interface=eth0 api.example.com # Watch queries as they happen (systemd-resolved, needs root) resolvectl monitor # SPF and DMARC records dig +short TXT example.com | grep -i spf; dig +short TXT _dmarc.example.com # Which CAs may issue for this name dig +short CAA example.com # Time a lookup from inside a pod kubectl run -it --rm t --image=nicolaka/netshoot --restart=Never -- sh -c 'time dig +short api.example.com' # Watch a record change during a cutover while :; do printf '%s %s\n' "$(date +%T)" "$(dig +short @1.1.1.1 api.example.com)"; sleep 5; done # Serial and response time from every authoritative server in one command dig +nssearch example.com # Full delegation chain, only the last hop's answer and the servers consulted dig +trace +nodnssec api.example.com | grep -E '^(api\.example\.com|;; Received)' # Does the parent's NS set match the child's diff <(dig +short NS example.com @a.gtld-servers.net +norecurse | sort) <(dig +short NS example.com @ns1.example.com +norecurse | sort) # Does every glue record resolve to the address the parent publishes for ns in $(dig +short NS example.com); do printf '%-24s %s\n' "$ns" "$(dig +short A "$ns" | tr '\n' ' ')"; done # TTLs in human units for an answer with many records dig +noall +answer +ttlunits example.com # All records at the apex, whether ANY is honoured or not (RFC 8482 servers return a synthetic HINFO) for t in SOA NS A AAAA MX TXT CAA HTTPS DNSKEY; do dig +noall +answer "$t" example.com; done # RRSIG expiration dates for a name, oldest first dig +dnssec +noall +answer example.com SOA | awk '$4 == "RRSIG" {print $9}' | sort # Validate a zone from any host, without trusting the local resolver delv +short example.com && echo validated # Resolve over DNS-over-TLS and confirm the transport in the footer dig +tls @1.1.1.1 example.com | grep -E 'SERVER|TLS' # What a client in another network would be told (EDNS Client Subnet, geo-routed zones) dig +short +subnet=198.51.100.0/24 @8.8.8.8 api.example.com # Reproduce a truncation problem: small buffer, no EDNS, then confirm TCP works dig +bufsize=512 +ignore DNSKEY example.com | grep flags; dig +tcp DNSKEY example.com | grep status # Query a resolver on a non-standard port, IPv4 only, fail after 2 seconds dig -4 -p 5353 +timeout=2 +tries=1 @127.0.0.1 api.example.com # Batch resolve a file of names, one line each: name, then addresses dig -f names.txt +short +identify # Zone transfer, when your address is allowed: full zone as records dig +tcp AXFR example.com @ns1.example.com | grep -v '^;' # DS record to hand to the registrar, from the live KSK dig +short DNSKEY example.com | awk '$1 == 257' | sed 's/^/example.com. IN DNSKEY /' | dnssec-dsfromkey -2 -f /dev/stdin example.com # drill: chase signatures to the root and print the trust path drill -S -k /var/lib/unbound/root.key example.com # systemd-resolved: cache hit rate since the last reset resolvectl statistics | grep -E 'Current Cache Size|Cache Hits|Cache Misses' # systemd-resolved: which link answers a given domain resolvectl status | awk '/^Link/ {link=$0} /DNS Domain/ {print link ": " $0}' # systemd-resolved: JSON answer for scripting resolvectl query --json=short api.example.com | jq -r '.answer[].rr.a.address // empty' # What resolves this host, ordered as applications use it grep -E '^hosts:' /etc/nsswitch.conf; ls -l /etc/resolv.conf; grep -E '^(nameserver|search|options)' /etc/resolv.conf # unbound: top cache misses by name, from a short query-log sample unbound-control stats_noreset | grep -E 'num\.(queries|cachehits|cachemiss)' # unbound: which servers it would ask for a name, with RTTs unbound-control lookup api.example.com # BIND: dump the cache and list what it holds for one zone rndc dumpdb -cache && grep -A2 'example.com' /var/named/data/cache_dump.db | head # Reverse zone name for a prefix, for delegation and PTR work arpaname 192.0.2.10 2001:db8::10 # Names in a CoreDNS log that hit search-domain expansion (queries ending in the cluster suffix for external names) kubectl -n kube-system logs -l k8s-app=kube-dns --tail 2000 | grep -oE '[a-z0-9.-]+\.com\.[a-z-]+\.svc\.cluster\.local\.' | sort | uniq -c | sort -rn | head # Measure lookup latency through the local resolver, 20 samples for i in $(seq 20); do dig api.example.com | awk '/Query time/ {print $4}'; done | sort -n | awk '{a[NR]=$1} END {print "min", a[1], "median", a[int(NR/2)+1], "max", a[NR]}' # Which pods override cluster DNS with their own dnsPolicy or dnsConfig kubectl get pods -A -o json | jq -r '.items[] | select(.spec.dnsPolicy != "ClusterFirst" or .spec.dnsConfig != null) | "\(.metadata.namespace)/\(.metadata.name) \(.spec.dnsPolicy)"' ``` ## Scripts Check that every authoritative server for a zone answers, agrees on the serial and is not a lame delegation; exit non-zero if any disagree, so it can run from cron before and after a zone change. ```sh #!/usr/bin/env bash # Usage: zone-consistency.sh example.com set -euo pipefail zone=${1:?zone required} mapfile -t parent_ns < <(dig +short NS "$zone" | sort) (( ${#parent_ns[@]} > 0 )) || { printf 'no NS records for %s\n' "$zone" >&2; exit 1; } declare -A serial rc=0 for ns in "${parent_ns[@]}"; do out=$(dig +norecurse +time=3 +tries=1 SOA "$zone" @"$ns" 2>&1) || { printf '%-28s unreachable\n' "$ns"; rc=1; continue; } if ! grep -q 'flags:.* aa' <<<"$out"; then printf '%-28s not authoritative (lame)\n' "$ns"; rc=1; continue; fi s=$(awk -v z="$zone." '$1 == z && $4 == "SOA" {print $7}' <<<"$out") serial[$ns]=$s printf '%-28s serial %s\n' "$ns" "$s" done distinct=$(printf '%s\n' "${serial[@]}" | sort -u | wc -l) if (( distinct > 1 )); then printf 'serials disagree across %d servers\n' "${#serial[@]}" >&2; rc=1; fi child_ns=$(dig +short +norecurse NS "$zone" @"${parent_ns[0]}" | sort) if [[ "$child_ns" != "$(printf '%s\n' "${parent_ns[@]}")" ]]; then printf 'NS set at parent differs from NS set in zone\n' >&2; rc=1; fi exit "$rc" ``` Warn when DNSSEC signatures on a set of zones are within a given number of days of expiring, or when the DS at the parent no longer matches a published key-signing key. ```sh #!/usr/bin/env bash # Usage: dnssec-expiry.sh 7 example.com example.net set -euo pipefail days=${1:?days required}; shift now=$(date +%s); rc=0 for zone in "$@"; do exp=$(dig +dnssec +noall +answer SOA "$zone" | awk '$4 == "RRSIG" {print $9}' | sort | head -1) if [[ -z "$exp" ]]; then printf '%-24s unsigned or RRSIG not returned\n' "$zone"; continue; fi exp_epoch=$(date -u -d "${exp:0:8} ${exp:8:2}:${exp:10:2}:${exp:12:2}" +%s) # RRSIG time is YYYYMMDDHHmmss UTC left=$(( (exp_epoch - now) / 86400 )) status=ok; (( left < days )) && { status=EXPIRING; rc=1; } ds_parent=$(dig +short DS "$zone" | awk '{print $1}' | sort -u | tr '\n' ' ') ds_zone=$(dig +short DNSKEY "$zone" | awk '$1 == 257' | sed "s/^/$zone. IN DNSKEY /" | dnssec-dsfromkey -2 -f /dev/stdin "$zone" 2>/dev/null | awk '{print $4}' | sort -u | tr '\n' ' ') match=match for tag in $ds_parent; do [[ " $ds_zone " == *" $tag "* ]] || { match=MISMATCH; rc=1; }; done printf '%-24s %s signatures expire in %3d days; DS %s (parent tags: %s)\n' "$zone" "$status" "$left" "$match" "${ds_parent:-none}" done exit "$rc" ``` Report how resolution behaves for one name across every layer a host uses: NSS, the stub resolver, each upstream server, and a public resolver, so a "works here, not there" report can be pinned to a layer in one run. ```python #!/usr/bin/env python3 """Usage: resolve-layers.py api.example.com""" import socket, subprocess, sys name = sys.argv[1] def dig(*args): out = subprocess.run(["dig", "+short", "+time=2", "+tries=1", *args, name], capture_output=True, text=True, timeout=10) return " ".join(out.stdout.split()) or f"(none, rc={out.returncode})" try: nss = sorted({r[4][0] for r in socket.getaddrinfo(name, None)}) except socket.gaierror as e: nss = [f"gaierror: {e}"] print(f"{'getaddrinfo (NSS)':<28} {' '.join(nss)}") print(f"{'stub resolver (dig)':<28} {dig()}") with open("/etc/resolv.conf") as f: servers = [l.split()[1] for l in f if l.startswith("nameserver")] if "127.0.0.53" in servers: r = subprocess.run(["resolvectl", "status"], capture_output=True, text=True, timeout=10) servers = [w for line in r.stdout.splitlines() if "DNS Servers:" in line for w in line.split(":", 1)[1].split()] for s in servers: print(f"{'upstream ' + s:<28} {dig('@' + s)}") for s in ("1.1.1.1", "8.8.8.8"): print(f"{'public ' + s:<28} {dig('@' + s)}") print(f"{'authoritative':<28} {dig('+norecurse', '@' + (dig('NS').split() or ['.'])[0])}") ``` ## Further reading - `man dig`, `man delv`, `man drill`, `man resolvectl`, `man resolv.conf`, `man resolved.conf`, `man unbound.conf`, `man named.conf` - [RFC 1034](https://www.rfc-editor.org/rfc/rfc1034) for the model, [RFC 1035](https://www.rfc-editor.org/rfc/rfc1035) for the zone file format, [RFC 2308](https://www.rfc-editor.org/rfc/rfc2308) for negative caching, [RFC 9460](https://www.rfc-editor.org/rfc/rfc9460) for SVCB and HTTPS records - [BIND 9 Administrator Reference Manual](https://bind9.readthedocs.io/en/latest/) for `named.conf`, views, `rndc` and the `dnssec-*` tools - [Unbound documentation](https://unbound.docs.nlnetlabs.nl/en/latest/) for `unbound.conf` and `unbound-control` - [systemd-resolved.service](https://www.freedesktop.org/software/systemd/man/latest/systemd-resolved.service.html) for link routing and the stub resolver - [Kubernetes DNS for Services and Pods](https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/)