Software Engineering WikiSE Wiki

DNS

Resolve and trace names with dig and resolvectl, read the answer, reason about TTLs and caching, and debug search domains and Kubernetes DNS.

Reviewed MarkdownEdit

On this page

Cheatsheet#

TaskCommand
Resolve a namedig +short api.example.com
Resolve as applications do (hosts file, NSS)getent ahosts api.example.com
Which resolver answereddig api.example.com | grep SERVER
Ask a specific resolver, bypassing the local onedig @1.1.1.1 api.example.com
Ask the authoritative server, no recursiondig +norecurse @ns1.example.com api.example.com
Follow the delegation from the rootdig +trace api.example.com
Reverse lookupdig -x 203.0.113.10 +short
Nameservers for a zonedig NS example.com +short
Mail recordsdig MX example.com +short
TXT (SPF, verification)dig TXT example.com +short
Zone serialdig SOA example.com +short
Remaining TTL in a cachedig api.example.com +noall +answer (second column counts down)
What systemd-resolved doesresolvectl query api.example.com
Flush systemd-resolved cacheresolvectl flush-caches
Name from inside a clusterkubectl run -it --rm dnstest --image=busybox --restart=Never -- nslookup api.my-namespace

dig and delv come from BIND (bind-utils on Fedora and RHEL, dnsutils or bind9-dnsutils on Debian and Ubuntu).

A name that will not resolve#

Answer three questions in order: does the authoritative server have the record, does the recursive resolver return it, and does this host (or container) use that resolver.

getent ahosts api.example.com                    # what applications on this host get, including /etc/hosts
dig +short api.example.com                       # what the configured resolver returns
dig @1.1.1.1 +short api.example.com              # what a public resolver returns
dig NS example.com +short                        # who is authoritative
dig +norecurse @ns1.example.com api.example.com  # straight from the zone, no cache involved
dig +trace api.example.com | tail -20            # delegation from the root, done by dig itself
resolvectl status | head -20                     # which servers and search domains this host uses

dig talks DNS directly to the server in /etc/resolv.conf. It ignores /etc/hosts, nsswitch.conf, mDNS and per-link routing in systemd-resolved. When dig works and the application does not, compare with getent.

ResultMeaning
NXDOMAINThe name does not exist at all; check for a typo, a missing record or the wrong zone
NOERROR with an empty answer (NODATA)The name exists but has no record of that type, for example A queried where only AAAA exists
SERVFAILThe resolver could not get an answer: broken delegation, unreachable or lame authoritative servers, or DNSSEC validation failure
REFUSEDThe server will not answer this query from you, often recursion not allowed for your source address
connection timed out; no servers could be reachedNothing answered on port 53; firewall, wrong server address or resolver down
Answer differs by resolverCaching within TTL, split-horizon DNS, or geo-based answers
Resolves but the application fails/etc/hosts override, a different resolver in the container, or the application cached the address at startup

dig +short prints nothing for both NXDOMAIN and NODATA. Use plain dig when diagnosing, because the status line and flags are the diagnosis.

Reading dig output#

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 23405
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; ANSWER SECTION:
api.example.com.   60   IN   CNAME   lb.example.net.
lb.example.net.    30   IN   A       203.0.113.10

;; SERVER: 192.0.2.53#53(192.0.2.53) (UDP)
;; Query time: 4 msec
FieldMeaning
statusResponse code; NOERROR even with zero answers
flags: aaAuthoritative answer, straight from the zone
flags: rd / raRecursion desired by the client / available at this server
flags: adThe resolver validated the answer with DNSSEC
flags: tcResponse truncated; the client should retry over TCP
TTL columnSeconds left in the cache that answered; a full TTL means it was just fetched
SERVERWhich server actually answered. Check this first when results disagree
Query timeA slow first query and a fast repeat means the first one recursed and the second came from cache

dig, drill and delv#

dig options starting with + control the query and the display; the ones starting with - select the transport, the type and the server. Options accumulate left to right, so +noall +answer clears every section and then re-enables the answer.

dig -t MX example.com +noall +answer          # one section only; -t and a bare type name are equivalent
dig example.com +short +identify              # answer plus the server and port that supplied it
dig SOA example.com +multiline                # SOA fields on separate lines with field names
dig +nssearch example.com                     # SOA from every authoritative server, one line each, with query time
dig +norecurse +dnssec DNSKEY example.com @ns1.example.com   # zone keys with RRSIGs, from the source
dig +tcp AXFR example.com @ns1.example.com    # zone transfer; refused unless your address is allowed
dig +tls @1.1.1.1 example.com                 # DNS over TLS, port defaults to 853
dig +https @1.1.1.1 example.com               # DNS over HTTPS, port defaults to 443
dig +subnet=192.0.2.0/24 example.com @8.8.8.8 # EDNS Client Subnet: what a client in that network would be told
dig +bufsize=512 +noedns example.com          # force pre-EDNS behaviour to reproduce truncation
dig +timeout=2 +tries=1 example.com           # fail fast instead of waiting 5 s and retrying 3 times
dig -4 example.com; dig -6 example.com        # restrict the transport, not the record type
dig -p 5353 @127.0.0.1 example.com            # a resolver on a non-standard port
dig -f queries.txt +short                     # batch mode: one query per line in the file
dig +qr example.com                           # print the query as sent; shows EDNS options and flags

+nssearch is the quickest way to find a secondary that has not picked up a change: every serial in the output should match. +ttlunits prints TTLs as 1h rather than 3600 and is easier to read on long answers.

drill (from ldns, package ldns-utils on Fedora, ldnsutils on Debian) covers the same ground with different spelling: drill -T traces from the root, drill -D sets the DO bit, drill -S name chases signatures up to a trust anchor, drill -x 203.0.113.10 reverses, drill -t uses TCP and drill -a retries a truncated answer with a larger buffer and then TCP. drill -s DNSKEY example.com prints the matching DS records next to each key, which is what you paste into the parent zone.

delv is a validating client. It does its own resolution and validation rather than trusting the ad flag from a resolver, so it answers the question “would a validating resolver accept this zone” from any host.

delv example.com                      # "; fully validated", "; unsigned answer" or an explanation of the failure
delv +vtrace example.com              # every validation step: which key, which DS, which RRSIG
delv +rtrace example.com              # every query delv sent while resolving
delv @ns1.example.com +cdflag example.com   # fetch with checking disabled, still validate locally
delv -a /etc/trusted-key.key +root=. example.com   # alternate trust anchor file

Record types#

TypePurposeNote
A / AAAAName to IPv4 / IPv6Clients may query both; a slow or broken AAAA path delays connections
CNAMEAlias to another nameCannot coexist with other records at the same name, so never at a zone apex
ALIAS / ANAMEApex aliasProvider-specific, flattened to A/AAAA on the server
HTTPS / SVCBService endpoint, ALPN and hints (RFC 9460)Browsers use HTTPS to learn HTTP/3 support and apex aliasing
MXMail exchangers with preferenceLower number is tried first
TXTArbitrary textSPF, DKIM, DMARC, domain verification
SRVService host and portNamed _service._proto.name
NSDelegates a zoneThe set at the parent should match the set in the child zone
SOAZone metadata, serial, negative-cache TTLSerial must increase for secondaries to transfer
PTRReverse mappingLives under in-addr.arpa / ip6.arpa, controlled by whoever owns the address block
CAACAs allowed to issue certificates for the nameA CA must refuse issuance if not listed; see TLS

Record data as it appears in a zone file or a dig answer:

example.com.        3600 IN SOA   ns1.example.com. hostmaster.example.com. 2026092401 7200 900 1209600 300
example.com.        3600 IN NS    ns1.example.com.
example.com.        3600 IN MX    10 mail.example.com.
example.com.        3600 IN TXT   "v=spf1 mx -all"
example.com.        3600 IN CAA   0 issue "letsencrypt.org"
example.com.        3600 IN HTTPS 1 . alpn="h2,h3"
www.example.com.     300 IN CNAME lb.example.net.
_sip._tcp.example.com. 3600 IN SRV 10 60 5060 sip.example.com.
10.2.0.192.in-addr.arpa. 3600 IN PTR api.example.com.

The SOA fields after the two names are serial, refresh, retry, expire and minimum. Refresh is how often a secondary checks the serial, retry how soon it re-checks after a failure, expire how long it keeps serving the zone without reaching the primary, and minimum is the negative-cache TTL. A serial in YYYYMMDDnn form leaves 99 changes a day and stays comparable; secondaries transfer only when the serial is greater in serial-number arithmetic (RFC 1982), so a serial that goes backwards is silently ignored.

TXT strings are limited to 255 characters per string; longer values such as DKIM keys are split into several quoted strings in one record and concatenated by the reader. SRV fields are priority, weight, port and target; a target of . means the service is explicitly absent. HTTPS with priority 0 is an alias form that behaves like a CNAME at the apex; priority 1 and above carry parameters such as alpn and ipv4hint.

Zone files#

A zone file is the text form of a zone as BIND, NSD, Knot and most providers’ import tools read it (RFC 1035 master file format). Names without a trailing dot are relative to $ORIGIN; a bare @ is the origin itself. A line without a name inherits the previous name. $TTL sets the default TTL for records without one.

$ORIGIN example.com.
$TTL 3600
@       IN SOA  ns1 hostmaster (
                2026092401 ; serial
                7200       ; refresh
                900        ; retry
                1209600    ; expire
                300 )      ; negative-cache TTL
        IN NS   ns1
        IN NS   ns2
        IN MX   10 mail
ns1     IN A    192.0.2.53
ns2     IN A    198.51.100.53
mail    IN A    192.0.2.25
api  60 IN A    192.0.2.10
www     IN CNAME api

Check a zone before loading it. named-checkzone parses the file as named would and reports the serial it read; named-checkconf -z does that for every zone in the configuration.

named-checkzone example.com /var/named/example.com.zone     # "OK" and the loaded serial, or the line that failed
named-checkconf -z /etc/named.conf                          # configuration syntax and every zone file
named-checkconf -p /etc/named.conf | less                   # the configuration with includes expanded and defaults applied
rndc reload example.com                                     # reload one zone after editing its file; rndc reload alone reloads all
rndc zonestatus example.com                                 # serial, type, next refresh and whether it is loaded
rndc retransfer example.com                                 # on a secondary: force a transfer regardless of serial

The common mistakes are a missing trailing dot on a fully qualified name inside the file (mail.example.com becomes mail.example.com.example.com.), a serial that was not incremented so secondaries never transfer, and a CNAME sharing a name with any other record. named-checkzone catches the last; the first two look valid and show up as wrong answers.

TTL and caching#

Changes do not propagate. A new record is visible immediately to anyone who asks the authoritative servers. Resolvers that cached the previous answer keep serving it until its TTL runs out.

dig api.example.com +noall +answer                  # TTL remaining in your resolver's cache
dig +norecurse @ns1.example.com api.example.com     # the record and full TTL at the source
watch -n5 'dig +short @8.8.8.8 api.example.com'     # watch one public resolver pick up a change

Before a planned change, lower the TTL (for example to 60 seconds) and wait at least one old TTL so every cache holds the short value. Make the change, then raise the TTL again.

Negative answers are cached too (RFC 2308). The negative TTL is the lower of the SOA record’s own TTL and its MINIMUM field. An NXDOMAIN fetched just before a record was created lingers for that long.

Some clients cache beyond DNS: the JVM caches lookups according to networkaddress.cache.ttl, and many applications resolve once at startup and never again. Restarting the application is sometimes the only fix.

Search domains and ndots#

For a name with fewer dots than ndots, the resolver tries each search domain appended first and the name as written last. A name with at least ndots dots is tried as written first. A trailing dot (api.example.com.) marks the name fully qualified and skips the search list.

nameserver 10.96.0.10
search my-namespace.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

That is a typical Kubernetes pod /etc/resolv.conf. api.example.com has two dots, fewer than five, so the resolver tries api.example.com.my-namespace.svc.cluster.local, then the other two cluster suffixes and any search domains inherited from the node, and only then api.example.com. Each attempt is usually sent for both A and AAAA. In hot paths, use a trailing dot for external names or lower ndots per pod with dnsConfig.options.

The glibc and musl (Alpine) resolvers handle search lists and parallel A/AAAA queries differently, so the same image logic can behave differently after a base image change.

cat /etc/resolv.conf
resolvectl query api.example.com          # shows the link and whether the answer came from cache
resolvectl statistics                     # cache hits, misses and DNSSEC counters
resolvectl flush-caches                   # needs root or polkit authorisation

On hosts with systemd-resolved, /etc/resolv.conf usually points at the stub 127.0.0.53, and the real upstream servers are listed per link in resolvectl status. See systemd if systemd-resolved itself is failing.

systemd-resolved#

systemd-resolved routes each query to a link based on the link’s search and routing domains, then falls back to whichever links have the default route. A routing domain is written with a leading ~ and steers queries without being appended to short names, which is what a VPN link needs: ~corp.example.com sends only that domain to the VPN’s resolver, and ~. makes a link the default for everything else. When two links both claim the default route, resolved sends the query to both and takes the first answer, which is the usual cause of “DNS works only sometimes after connecting the VPN”.

resolvectl status eth0                          # servers, search domains, DNSSEC and DNS-over-TLS state for one link
resolvectl query --type=MX example.com          # explicit type; disables search-domain logic
resolvectl query --cache=no api.example.com     # bypass the cache for this lookup
resolvectl query --validate=no api.example.com  # skip DNSSEC validation for this lookup
resolvectl service _sip._tcp example.com        # SRV lookup with the targets resolved
resolvectl dns eth0 192.0.2.53 2001:db8::53     # set servers for a link until the network manager changes them
resolvectl domain eth0 '~corp.example.com'      # routing domain: send only this suffix to eth0's servers
resolvectl default-route eth0 false             # stop this link from receiving unrelated queries
resolvectl revert eth0                          # drop the per-link overrides made with the commands above
resolvectl show-cache                           # dump cached records (needs root)
resolvectl log-level debug                      # verbose logs in journalctl -u systemd-resolved until reset to info

Settings made with resolvectl dns and resolvectl domain are runtime only. Persistent configuration belongs in /etc/systemd/resolved.conf.d/*.conf for global values or in the network manager’s per-connection settings (nmcli connection modify my-vpn ipv4.dns-search '~corp.example.com', or [Network] Domains=~corp.example.com in a .network file).

# /etc/systemd/resolved.conf.d/upstream.conf
[Resolve]
DNS=192.0.2.53#dns.example.com 2001:db8::53   # server with the name to verify for DNS-over-TLS
FallbackDNS=
DNSOverTLS=opportunistic                        # yes enforces it and fails closed
DNSSEC=allow-downgrade                          # yes fails closed on unsigned-looking answers
Domains=~.
Cache=yes

/etc/resolv.conf can be one of three things: a symlink to /run/systemd/resolve/stub-resolv.conf (the stub at 127.0.0.53, search domains included), to /run/systemd/resolve/resolv.conf (the upstream servers directly, which bypasses per-link routing and the cache), or a static file managed by something else. ls -l /etc/resolv.conf tells you which, and a static file that another tool keeps rewriting is a fight between NetworkManager, resolved and a VPN client.

Kubernetes DNS#

CoreDNS serves the cluster domain (usually cluster.local) and forwards everything else to the upstream servers in its forward plugin, often the node’s /etc/resolv.conf. A Service name resolves to its ClusterIP; a headless Service resolves to the ready pod IPs.

NameResolves to
apiService api in the pod’s own namespace, via the search list
api.my-namespaceService api in my-namespace
api.my-namespace.svc.cluster.localFully qualified, no search expansion
10-1-2-3.my-namespace.pod.cluster.localPod with IP 10.1.2.3 (CoreDNS pods option must be enabled)
_grpc._tcp.api.my-namespace.svc.cluster.localSRV record for the Service port named grpc
kubectl run -it --rm dnstest --image=nicolaka/netshoot --restart=Never -- dig api.my-namespace.svc.cluster.local
kubectl -n kube-system logs -l k8s-app=kube-dns --tail 50
kubectl -n kube-system get configmap coredns -o yaml
kubectl -n kube-system get endpointslice -l kubernetes.io/service-name=kube-dns

Intermittent resolution failures in a cluster usually come from unhealthy or overloaded CoreDNS replicas, a NetworkPolicy that blocks UDP and TCP 53 to kube-system, or conntrack races on UDP. The records themselves are rarely the problem. See Kubernetes and Cilium for policy checks.

DNSSEC#

DNSSEC signs records so a validating resolver can prove an answer came from the zone owner unmodified. It does not encrypt queries; DNS over TLS or HTTPS does that.

dig +dnssec api.example.com | grep -E 'RRSIG|flags:'   # ad in flags = validated by the resolver
dig +cdflag api.example.com                           # checking disabled: does it resolve without validation?
delv api.example.com                                  # validates locally and explains failures
dig DS example.com +short                             # delegation signer record at the parent

A SERVFAIL that turns into NOERROR with +cdflag is a validation failure: expired signatures, or a DS record at the parent that no longer matches the zone’s keys after a key rollover or DNS provider move.

The chain of trust runs from the root key, through a DS record in each parent that hashes the child’s key-signing key (DNSKEY with flags 257), to the zone-signing key (flags 256) that signs the records. Every link is checkable by hand.

dig DNSKEY example.com +short | awk '$1 == 257'                    # the KSK(s), what the parent's DS must hash
dig DS example.com @a.gtld-servers.net +norecurse +short             # DS as published at the parent
dig +dnssec +multiline SOA example.com | grep -A1 RRSIG              # signature validity window: inception and expiration
dnssec-dsfromkey -2 Kexample.com.+013+12345.key                     # SHA-256 DS record to give the registrar, from the KSK file
dnssec-verify -o example.com /var/named/example.com.zone.signed      # verify a signed zone file offline before publishing
delv +vtrace example.com 2>&1 | grep -iE 'validat|bogus|insecure'    # where a broken chain breaks

RRSIG expiration is the silent failure: a zone signed once by hand validates until the signatures expire, typically after 30 days, and then every validating resolver returns SERVFAIL at once. Use automatic signing (dnssec-policy in BIND, or the provider’s managed signing) and alert on the earliest expiration in the zone.

When moving a signed zone between providers, remove the DS at the registrar first, wait the parent’s DS TTL, move the zone, then publish the new DS. A DS pointing at keys the new provider does not hold makes the zone bogus for every validating client, which is most of them.

Split horizon#

Split horizon serves different answers for the same name depending on who asks: internal clients get private addresses, the internet gets public ones. It is usually the explanation when dig on the VPN and dig @1.1.1.1 disagree and nothing is cached.

In BIND, view blocks with match-clients select a complete configuration per client set; the first matching view wins and each view carries its own zones, so the same zone name loads from two different files:

view "internal" {
    match-clients { 192.0.2.0/24; 2001:db8::/32; localhost; };
    recursion yes;
    zone "example.com" { type primary; file "internal/example.com.zone"; };
};
view "external" {
    match-clients { any; };
    recursion no;
    zone "example.com" { type primary; file "external/example.com.zone"; };
};

In unbound, local-zone and local-data override a name for every client, and forward-zone sends a suffix to an internal authoritative server instead of the internet. transparent answers from local data where it exists and recurses for everything else under the name; static refuses to recurse, so an unlisted name under it is NXDOMAIN; redirect answers the whole subtree with the same data.

server:
    local-zone: "corp.example.com." transparent
    local-data: "git.corp.example.com. IN A 192.0.2.40"
    local-data-ptr: "192.0.2.40 git.corp.example.com"
    private-domain: "corp.example.com."     # allow RFC 1918 answers from this domain despite rebinding protection
forward-zone:
    name: "example.internal."
    forward-addr: 192.0.2.53

Clients on a split-horizon network need their queries to reach the internal resolver; a laptop with a hard-coded public resolver bypasses the whole arrangement, and a resolver with private-address rebinding protection strips internal addresses returned by external names unless the domain is listed in private-domain.

Running a resolver#

A local recursive resolver removes the dependency on an upstream and gives you a cache you can inspect and flush. unbound is the smaller choice for a recursive-only role; BIND does recursion and authoritative service in one daemon.

Minimal unbound as a caching resolver for one network, validating DNSSEC with the root anchor that unbound-anchor maintains:

# /etc/unbound/unbound.conf
server:
    interface: 0.0.0.0
    interface: ::0
    access-control: 192.0.2.0/24 allow
    access-control: 2001:db8::/32 allow      # everything not listed is refused
    auto-trust-anchor-file: "/var/lib/unbound/root.key"
    prefetch: yes                            # refresh popular records before they expire
    cache-min-ttl: 0                         # honour published TTLs; raising this serves stale data
    harden-dnssec-stripped: yes
    verbosity: 1
    log-queries: no                          # yes for a short diagnosis only; one line per query
remote-control:
    control-enable: yes
    control-interface: 127.0.0.1
unbound-checkconf                       # parse the config; run before every reload
sudo systemctl reload unbound           # or unbound-control reload, which also flushes the cache
unbound-control status                  # version, threads, uptime; exit code 3 if not running
unbound-control stats_noreset | grep -E 'total.num.(queries|cachehits|cachemiss)|num.answer.rcode'
unbound-control lookup api.example.com  # which nameservers unbound would ask, with their RTT
unbound-control flush_zone example.com  # drop everything at and below the name
unbound-control flush_negative          # drop cached NXDOMAIN, NODATA and SERVFAIL answers
unbound-control dump_cache > cache.txt  # full cache in text; load_cache reads the same format
unbound-control local_data 'test.example.com. IN A 192.0.2.99'   # runtime override, lost on restart
unbound-host -C /etc/unbound/unbound.conf -v api.example.com     # resolve with that config and print the security status

Minimal BIND as a recursive resolver plus one authoritative zone:

# /etc/named.conf
options {
    directory "/var/named";
    listen-on { 192.0.2.53; 127.0.0.1; };
    allow-query { 192.0.2.0/24; localhost; };
    allow-recursion { 192.0.2.0/24; localhost; };   # separate from allow-query; an open recursor is an amplifier
    recursion yes;
    dnssec-validation auto;                          # built-in root trust anchor, RFC 5011 tracked
};
zone "example.com" {
    type primary;
    file "example.com.zone";
    allow-transfer { 198.51.100.53; };               # secondaries only; set it explicitly rather than relying on the default
    notify yes;
};
named-checkconf -z && rndc reconfig      # reconfig loads new zones without reloading existing ones
rndc status                              # zones loaded, recursive clients, query logging state
rndc dumpdb -cache && less /var/named/data/cache_dump.db   # cache contents with remaining TTLs
rndc flushname api.example.com           # one name; rndc flushtree drops the subtree, rndc flush drops everything
rndc querylog                            # toggle per-query logging to the configured channel
rndc dnssec -status example.com          # key states under dnssec-policy
rndc sync -clean example.com             # write dynamic-update journal to the zone file

rndc freeze example.com stops dynamic updates so the zone file can be edited by hand; rndc thaw example.com reloads it and resumes. Editing the file while it is not frozen loses the edit at the next journal write.

Troubleshooting#

SymptomLikely causeCheck
NXDOMAIN for a record you just createdNegative answer cached before creationSOA negative TTL; query the authoritative server with +norecurse
Old address after a changePositive cache within the old TTL, or the application cached itdig against several resolvers; restart the client
SERVFAIL from one resolver onlyThat resolver validates DNSSEC and the zone fails validationdig +cdflag; delv
SERVFAIL everywhereLame delegation: parent NS points at servers that do not serve the zonedig +trace; query each NS with +norecurse
dig works, application does not/etc/hosts, NSS order, or a different resolver in the containergetent ahosts; cat /etc/resolv.conf inside the container
Slow first request, fast repeatsSearch list expansion or a slow upstreamCount queries with resolvectl monitor or CoreDNS logs; fully qualify the name
5 second delays on lookupsUDP reply lost (conntrack race or firewall), resolver waits for its timeoutPacket capture on port 53; options single-request-reopen (glibc)
Large responses failUDP truncated and TCP 53 blockeddig +tcp; allow TCP 53
Reverse lookup returns nothingNo PTR record, or the address owner has not delegated the reverse zonedig -x against the provider’s servers
Name resolves in the cluster only intermittentlyCoreDNS replica unhealthy or NetworkPolicy dropCoreDNS logs and EndpointSlice; policy on port 53
Secondaries serve the old zoneSerial not incremented, or transfer refuseddig +nssearch example.com; rndc zonestatus; allow-transfer on the primary
Every validating resolver fails at onceRRSIGs expireddig +dnssec +multiline SOA example.com, compare the RRSIG expiration with today
Zone bogus after a provider moveDS at the registrar hashes keys the new provider does not havedig DS example.com +short against dnssec-dsfromkey of the live KSK; delv +vtrace
Internal names fail after connecting to a VPNRouting domain missing or a second link owns the default routeresolvectl status; resolvectl domain vpn0 '~corp.example.com'
Private address stripped from an answerResolver rebinding protection (private-address)unbound-control list_local_zones; add private-domain for that suffix
REFUSED from your own resolverClient address outside access-control or allow-recursionunbound-checkconf; named-checkconf -p | grep -A3 allow-recursion
Name resolves for dig but nslookup or the app returns a wrong address/etc/hosts entry, or myhostname/mdns in nsswitch.confgetent ahosts; grep hosts /etc/nsswitch.conf
rndc: connect failed: connection refusednamed not running, or controls not listening on 127.0.0.1#953systemctl status named; ss -ltnp | grep 953; rndc-confgen for the key
Changes to a zone file disappearDynamic updates rewrote the file from the journalrndc freeze before editing, rndc thaw after

For connectivity to the resolver itself, see iproute2.

Oneliners#

# Compare answers across public resolvers
for r in 1.1.1.1 8.8.8.8 9.9.9.9; do printf '%s: %s\n' "$r" "$(dig +short @"$r" api.example.com | tr '\n' ' ')"; done

# Common record types for a name
for t in A AAAA CNAME HTTPS MX TXT NS SOA CAA; do printf '%-6s %s\n' "$t" "$(dig +short "$t" example.com | tr '\n' ' ')"; done

# Serial on every authoritative server, to check zone transfer and replication
for ns in $(dig +short NS example.com); do printf '%s %s\n' "$ns" "$(dig +short SOA @"$ns" example.com | awk '{print $3}')"; done

# Reverse lookups for part of a subnet
for i in $(seq 1 20); do dig +short -x "192.0.2.$i" | sed "s/^/192.0.2.$i /"; done

# Resolve through one interface's resolver (systemd-resolved)
resolvectl query --interface=eth0 api.example.com

# Watch queries as they happen (systemd-resolved, needs root)
resolvectl monitor

# SPF and DMARC records
dig +short TXT example.com | grep -i spf; dig +short TXT _dmarc.example.com

# Which CAs may issue for this name
dig +short CAA example.com

# Time a lookup from inside a pod
kubectl run -it --rm t --image=nicolaka/netshoot --restart=Never -- sh -c 'time dig +short api.example.com'

# Watch a record change during a cutover
while :; do printf '%s %s\n' "$(date +%T)" "$(dig +short @1.1.1.1 api.example.com)"; sleep 5; done

# Serial and response time from every authoritative server in one command
dig +nssearch example.com

# Full delegation chain, only the last hop's answer and the servers consulted
dig +trace +nodnssec api.example.com | grep -E '^(api\.example\.com|;; Received)'

# Does the parent's NS set match the child's
diff <(dig +short NS example.com @a.gtld-servers.net +norecurse | sort) <(dig +short NS example.com @ns1.example.com +norecurse | sort)

# Does every glue record resolve to the address the parent publishes
for ns in $(dig +short NS example.com); do printf '%-24s %s\n' "$ns" "$(dig +short A "$ns" | tr '\n' ' ')"; done

# TTLs in human units for an answer with many records
dig +noall +answer +ttlunits example.com

# All records at the apex, whether ANY is honoured or not (RFC 8482 servers return a synthetic HINFO)
for t in SOA NS A AAAA MX TXT CAA HTTPS DNSKEY; do dig +noall +answer "$t" example.com; done

# RRSIG expiration dates for a name, oldest first
dig +dnssec +noall +answer example.com SOA | awk '$4 == "RRSIG" {print $9}' | sort

# Validate a zone from any host, without trusting the local resolver
delv +short example.com && echo validated

# Resolve over DNS-over-TLS and confirm the transport in the footer
dig +tls @1.1.1.1 example.com | grep -E 'SERVER|TLS'

# What a client in another network would be told (EDNS Client Subnet, geo-routed zones)
dig +short +subnet=198.51.100.0/24 @8.8.8.8 api.example.com

# Reproduce a truncation problem: small buffer, no EDNS, then confirm TCP works
dig +bufsize=512 +ignore DNSKEY example.com | grep flags; dig +tcp DNSKEY example.com | grep status

# Query a resolver on a non-standard port, IPv4 only, fail after 2 seconds
dig -4 -p 5353 +timeout=2 +tries=1 @127.0.0.1 api.example.com

# Batch resolve a file of names, one line each: name, then addresses
dig -f names.txt +short +identify

# Zone transfer, when your address is allowed: full zone as records
dig +tcp AXFR example.com @ns1.example.com | grep -v '^;'

# DS record to hand to the registrar, from the live KSK
dig +short DNSKEY example.com | awk '$1 == 257' | sed 's/^/example.com. IN DNSKEY /' | dnssec-dsfromkey -2 -f /dev/stdin example.com

# drill: chase signatures to the root and print the trust path
drill -S -k /var/lib/unbound/root.key example.com

# systemd-resolved: cache hit rate since the last reset
resolvectl statistics | grep -E 'Current Cache Size|Cache Hits|Cache Misses'

# systemd-resolved: which link answers a given domain
resolvectl status | awk '/^Link/ {link=$0} /DNS Domain/ {print link ": " $0}'

# systemd-resolved: JSON answer for scripting
resolvectl query --json=short api.example.com | jq -r '.answer[].rr.a.address // empty'

# What resolves this host, ordered as applications use it
grep -E '^hosts:' /etc/nsswitch.conf; ls -l /etc/resolv.conf; grep -E '^(nameserver|search|options)' /etc/resolv.conf

# unbound: top cache misses by name, from a short query-log sample
unbound-control stats_noreset | grep -E 'num\.(queries|cachehits|cachemiss)'

# unbound: which servers it would ask for a name, with RTTs
unbound-control lookup api.example.com

# BIND: dump the cache and list what it holds for one zone
rndc dumpdb -cache && grep -A2 'example.com' /var/named/data/cache_dump.db | head

# Reverse zone name for a prefix, for delegation and PTR work
arpaname 192.0.2.10 2001:db8::10

# Names in a CoreDNS log that hit search-domain expansion (queries ending in the cluster suffix for external names)
kubectl -n kube-system logs -l k8s-app=kube-dns --tail 2000 | grep -oE '[a-z0-9.-]+\.com\.[a-z-]+\.svc\.cluster\.local\.' | sort | uniq -c | sort -rn | head

# Measure lookup latency through the local resolver, 20 samples
for i in $(seq 20); do dig api.example.com | awk '/Query time/ {print $4}'; done | sort -n | awk '{a[NR]=$1} END {print "min", a[1], "median", a[int(NR/2)+1], "max", a[NR]}'

# Which pods override cluster DNS with their own dnsPolicy or dnsConfig
kubectl get pods -A -o json | jq -r '.items[] | select(.spec.dnsPolicy != "ClusterFirst" or .spec.dnsConfig != null) | "\(.metadata.namespace)/\(.metadata.name) \(.spec.dnsPolicy)"'

Scripts#

Check that every authoritative server for a zone answers, agrees on the serial and is not a lame delegation; exit non-zero if any disagree, so it can run from cron before and after a zone change.

#!/usr/bin/env bash
# Usage: zone-consistency.sh example.com
set -euo pipefail
zone=${1:?zone required}
mapfile -t parent_ns < <(dig +short NS "$zone" | sort)
(( ${#parent_ns[@]} > 0 )) || { printf 'no NS records for %s\n' "$zone" >&2; exit 1; }
declare -A serial
rc=0
for ns in "${parent_ns[@]}"; do
  out=$(dig +norecurse +time=3 +tries=1 SOA "$zone" @"$ns" 2>&1) || { printf '%-28s unreachable\n' "$ns"; rc=1; continue; }
  if ! grep -q 'flags:.* aa' <<<"$out"; then printf '%-28s not authoritative (lame)\n' "$ns"; rc=1; continue; fi
  s=$(awk -v z="$zone." '$1 == z && $4 == "SOA" {print $7}' <<<"$out")
  serial[$ns]=$s
  printf '%-28s serial %s\n' "$ns" "$s"
done
distinct=$(printf '%s\n' "${serial[@]}" | sort -u | wc -l)
if (( distinct > 1 )); then printf 'serials disagree across %d servers\n' "${#serial[@]}" >&2; rc=1; fi
child_ns=$(dig +short +norecurse NS "$zone" @"${parent_ns[0]}" | sort)
if [[ "$child_ns" != "$(printf '%s\n' "${parent_ns[@]}")" ]]; then printf 'NS set at parent differs from NS set in zone\n' >&2; rc=1; fi
exit "$rc"

Warn when DNSSEC signatures on a set of zones are within a given number of days of expiring, or when the DS at the parent no longer matches a published key-signing key.

#!/usr/bin/env bash
# Usage: dnssec-expiry.sh 7 example.com example.net
set -euo pipefail
days=${1:?days required}; shift
now=$(date +%s); rc=0
for zone in "$@"; do
  exp=$(dig +dnssec +noall +answer SOA "$zone" | awk '$4 == "RRSIG" {print $9}' | sort | head -1)
  if [[ -z "$exp" ]]; then printf '%-24s unsigned or RRSIG not returned\n' "$zone"; continue; fi
  exp_epoch=$(date -u -d "${exp:0:8} ${exp:8:2}:${exp:10:2}:${exp:12:2}" +%s)   # RRSIG time is YYYYMMDDHHmmss UTC
  left=$(( (exp_epoch - now) / 86400 ))
  status=ok; (( left < days )) && { status=EXPIRING; rc=1; }
  ds_parent=$(dig +short DS "$zone" | awk '{print $1}' | sort -u | tr '\n' ' ')
  ds_zone=$(dig +short DNSKEY "$zone" | awk '$1 == 257' | sed "s/^/$zone. IN DNSKEY /" | dnssec-dsfromkey -2 -f /dev/stdin "$zone" 2>/dev/null | awk '{print $4}' | sort -u | tr '\n' ' ')
  match=match
  for tag in $ds_parent; do [[ " $ds_zone " == *" $tag "* ]] || { match=MISMATCH; rc=1; }; done
  printf '%-24s %s signatures expire in %3d days; DS %s (parent tags: %s)\n' "$zone" "$status" "$left" "$match" "${ds_parent:-none}"
done
exit "$rc"

Report how resolution behaves for one name across every layer a host uses: NSS, the stub resolver, each upstream server, and a public resolver, so a “works here, not there” report can be pinned to a layer in one run.

#!/usr/bin/env python3
"""Usage: resolve-layers.py api.example.com"""
import socket, subprocess, sys

name = sys.argv[1]

def dig(*args):
    out = subprocess.run(["dig", "+short", "+time=2", "+tries=1", *args, name], capture_output=True, text=True, timeout=10)
    return " ".join(out.stdout.split()) or f"(none, rc={out.returncode})"

try:
    nss = sorted({r[4][0] for r in socket.getaddrinfo(name, None)})
except socket.gaierror as e:
    nss = [f"gaierror: {e}"]
print(f"{'getaddrinfo (NSS)':<28} {' '.join(nss)}")
print(f"{'stub resolver (dig)':<28} {dig()}")
with open("/etc/resolv.conf") as f:
    servers = [l.split()[1] for l in f if l.startswith("nameserver")]
if "127.0.0.53" in servers:
    r = subprocess.run(["resolvectl", "status"], capture_output=True, text=True, timeout=10)
    servers = [w for line in r.stdout.splitlines() if "DNS Servers:" in line for w in line.split(":", 1)[1].split()]
for s in servers:
    print(f"{'upstream ' + s:<28} {dig('@' + s)}")
for s in ("1.1.1.1", "8.8.8.8"):
    print(f"{'public ' + s:<28} {dig('@' + s)}")
print(f"{'authoritative':<28} {dig('+norecurse', '@' + (dig('NS').split() or ['.'])[0])}")

Further reading#