Tools

Tools: Ultimate Guide: Why Lowering ndots Breaks Alpine Pods (But Not Debian) — A Deep Dive into glibc vs musl Resolvers

2026-05-02 0 views admin
Tools: Ultimate Guide: Why Lowering ndots Breaks Alpine Pods (But Not Debian) — A Deep Dive into glibc vs musl Resolvers

The starting point: 5 queries for one domain

The three config files that matter

How a query travels (and where the retry loop lives)

glibc vs musl: same input, different behavior

glibc — falls back gracefully

musl — stops on the first miss

Why doesn't musl just add the fallback?

Reproducing it with kind

Test: resolve kubernetes.default.svc (2 dots)

Result

CoreDNS logs confirm it

Resolver behavior, side by side

What to do about it

Further reading If you're running Alpine-based pods in Kubernetes and someone tells you to lower ndots for better DNS performance — don't. Or at least, read this first. We had 5 DNS queries firing for every external domain lookup. The fix seemed obvious: drop ndots:5 to ndots:2. An AI reviewer warned me it might break internal service resolution. The reasoning didn't hold up when I read the resolver code, so I went ahead — and broke things in a way I didn't expect. The AI was right about the symptom but wrong about the cause. The breakage is real, but it lives in libc, not in the search algorithm. Every external DNS lookup in our cluster was producing 4-5 queries. This is well-known behavior — it's caused by the default ndots:5 combined with Kubernetes' three-entry search list. The path of least resistance is to lower ndots. Most external domains have dots ≥ 2 (google.com, api.example.com), so ndots:2 skips the search-list traversal for them entirely. Before shipping the change, I asked an AI assistant to review it. It warned that internal service resolution might break, with this reasoning: The "then what" was the question. I read the resolver source and concluded the AI was wrong: after the original query fails, the resolver should fall back to search-list traversal. The lookup should still succeed. I was reading the wrong source. Before going deeper, the surface area of "Kubernetes DNS" lives in three files. Knowing which one controls what saves a lot of pain. A Pod's DNS settings come from spec.dnsPolicy, default ClusterFirst, which inherits the pod's resolv.conf: That's the file libc reads. And libc is the part that decides whether to walk the search list or skip it. The key thing to notice in the flow: when a Pod's resolver gets NXDOMAIN, it retries with the next FQDN from the search list. That retry loop is where query amplification comes from. Lowering ndots is appealing because it skips this loop for high-dot names. CoreDNS itself doesn't care about ndots. It just answers whatever FQDN arrives. The retry decision happens entirely on the client side, inside libc. Here's the part the AI got right (in spirit) and I missed: the resolver isn't part of CoreDNS, Kubernetes, or even your app. It's the libc shipped in your container image. Different libcs implement search/ndots differently. Distros: Debian, Ubuntu, CentOS, RHEL, Amazon Linux. When dots ≥ ndots, glibc tries the original first. If that returns NXDOMAIN, it walks the search list anyway as a fallback. One or two extra queries, but resolution succeeds. The fallback logic lives in __res_context_search(). The relevant part: The critical detail: the as-is attempt and the search loop are sequential, not exclusive. Failure of the first does not prevent the second. musl is intentionally minimal. When dots ≥ ndots, it sets *search = 0 and never enters the search loop. From name_from_dns_search(): Setting *search = 0 isn't a bug. It's deliberate. The next question is why. This has come up on the musl mailing list more than once. Maintainer Rich Felker rejects it consistently. The clearest example is from Andrey Arapov's 2019 thread: If you're on musl and want to avoid this entirely: set ndots:1 and don't depend on short names. This is a values disagreement, not a bug. Both libcs are doing what they intended. The mismatch only becomes a Kubernetes problem because Kubernetes hands every Pod a search list and assumes the resolver will use it. Four pods, two libcs, two ndots values. Patch CoreDNS to log every query: The four test pods (full manifest in the original Korean post). Key bits: This name has 2 dots. Under ndots:5, dots < ndots → search first. Under ndots:2, dots ≥ ndots → original first. The libc difference only surfaces in the ndots:2 case. Same query. Same cluster. Same ndots:2. The only thing that changed is the libc. alpine-ndots2 — only the original name arrives at CoreDNS. No search-expanded queries: debian-ndots2 — original first, then the entire search list, then success: This is exactly what the source code predicted. musl exits the search loop on the first iteration; glibc walks every entry. Under the default ndots:5, most names have fewer than 5 dots, so both libcs try search first and the difference doesn't surface. The moment you lower ndots, more names cross into dots ≥ ndots territory — and that's where musl's missing fallback turns into a real outage. If you want to lower ndots and you have any musl-based workloads: The thing I keep coming back to: the abstraction you're tuning (Kubernetes' ndots) and the layer where the behavior actually lives (libc resolver) can be miles apart. The Kubernetes docs talk about ndots. The Pod spec exposes ndots. CoreDNS configures things adjacent to ndots. And none of them are the layer that decides what happens when dots ≥ ndots. The AI reviewer wasn't wrong to flag the risk. It just couldn't see one layer down. Neither could I, until the test pods told me. When something in a layered system behaves unexpectedly, "why" usually doesn't have a clean answer at the layer you're operating in. Tracing the call all the way down to the C source is, surprisingly often, faster than reading another blog post. Templates let you quickly answer FAQs or store snippets for re-use. Hide child comments as well For further actions, you may consider blocking this person and/or reporting abuse

Code Block

Copy

spec: dnsConfig: options: - name: ndots value: "2" spec: dnsConfig: options: - name: ndots value: "2" spec: dnsConfig: options: - name: ndots value: "2" With ndots:5 and a query for "my-svc.default" (1 dot): dots 1 < ndots 5 → search first my-svc.default.<ns>.svc.cluster.local → NXDOMAIN my-svc.default.svc.cluster.local → NOERROR ✓ With ndots:1: dots 1 ≥ ndots 1 → original first my-svc.default → NXDOMAIN (doesn't exist externally) ... then what? With ndots:5 and a query for "my-svc.default" (1 dot): dots 1 < ndots 5 → search first my-svc.default.<ns>.svc.cluster.local → NXDOMAIN my-svc.default.svc.cluster.local → NOERROR ✓ With ndots:1: dots 1 ≥ ndots 1 → original first my-svc.default → NXDOMAIN (doesn't exist externally) ... then what? With ndots:5 and a query for "my-svc.default" (1 dot): dots 1 < ndots 5 → search first my-svc.default.<ns>.svc.cluster.local → NXDOMAIN my-svc.default.svc.cluster.local → NOERROR ✓ With ndots:1: dots 1 ≥ ndots 1 → original first my-svc.default → NXDOMAIN (doesn't exist externally) ... then what? nameserver 10.96.0.10 search <namespace>.svc.cluster.local svc.cluster.local cluster.local options ndots:5 nameserver 10.96.0.10 search <namespace>.svc.cluster.local svc.cluster.local cluster.local options ndots:5 nameserver 10.96.0.10 search <namespace>.svc.cluster.local svc.cluster.local cluster.local options ndots:5 my-svc.default → NXDOMAIN ↓ search fallback my-svc.default.<ns>.svc.cluster.local → NXDOMAIN my-svc.default.svc.cluster.local → NOERROR ✓ my-svc.default → NXDOMAIN ↓ search fallback my-svc.default.<ns>.svc.cluster.local → NXDOMAIN my-svc.default.svc.cluster.local → NOERROR ✓ my-svc.default → NXDOMAIN ↓ search fallback my-svc.default.<ns>.svc.cluster.local → NXDOMAIN my-svc.default.svc.cluster.local → NOERROR ✓ // dots ≥ ndots OR trailing dot → try as-is first if (dots >= statp->ndots || trailing_dot) { ret = __res_context_querydomain (ctx, name, NULL, class, type, ...); if (ret > 0 || trailing_dot ...) return (ret); saved_herrno = h_errno; tried_as_is++; // ... falls through to search loop below } // Run search list when at least one entry might apply if ((!dots && (statp->options & RES_DEFNAMES) != 0) || (dots && !trailing_dot && (statp->options & RES_DNSRCH) != 0)) { for (size_t domain_index = 0; !done; ++domain_index) { const char *dname = __resolv_context_search_list (ctx, domain_index); if (dname == NULL) break; ret = __res_context_querydomain (ctx, name, dname, class, type, ...); // ... } } // dots ≥ ndots OR trailing dot → try as-is first if (dots >= statp->ndots || trailing_dot) { ret = __res_context_querydomain (ctx, name, NULL, class, type, ...); if (ret > 0 || trailing_dot ...) return (ret); saved_herrno = h_errno; tried_as_is++; // ... falls through to search loop below } // Run search list when at least one entry might apply if ((!dots && (statp->options & RES_DEFNAMES) != 0) || (dots && !trailing_dot && (statp->options & RES_DNSRCH) != 0)) { for (size_t domain_index = 0; !done; ++domain_index) { const char *dname = __resolv_context_search_list (ctx, domain_index); if (dname == NULL) break; ret = __res_context_querydomain (ctx, name, dname, class, type, ...); // ... } } // dots ≥ ndots OR trailing dot → try as-is first if (dots >= statp->ndots || trailing_dot) { ret = __res_context_querydomain (ctx, name, NULL, class, type, ...); if (ret > 0 || trailing_dot ...) return (ret); saved_herrno = h_errno; tried_as_is++; // ... falls through to search loop below } // Run search list when at least one entry might apply if ((!dots && (statp->options & RES_DEFNAMES) != 0) || (dots && !trailing_dot && (statp->options & RES_DNSRCH) != 0)) { for (size_t domain_index = 0; !done; ++domain_index) { const char *dname = __resolv_context_search_list (ctx, domain_index); if (dname == NULL) break; ret = __res_context_querydomain (ctx, name, dname, class, type, ...); // ... } } my-svc.default → NXDOMAIN ✗ End. No search attempted. my-svc.default → NXDOMAIN ✗ End. No search attempted. my-svc.default → NXDOMAIN ✗ End. No search attempted. // Count dots. If dots ≥ ndots OR trailing dot → zero out the search list. for (dots=l=0; name[l]; l++) if (name[l]=='.') dots++; if (dots >= conf.ndots || name[l-1]=='.') *search = 0; // ... // Walk the search list, splitting on whitespace. // When *search = 0 above: *p == 0, z == p, break on the first iteration. // → search is attempted ZERO times. for (p=search; *p; p=z) { for (; isspace(*p); p++); for (z=p; *z && !isspace(*z); z++); if (z==p) break; // ... query combined FQDN } // Final fallback: query the original as-is, exactly once. return name_from_dns(buf, canon, name, family, &conf); // Count dots. If dots ≥ ndots OR trailing dot → zero out the search list. for (dots=l=0; name[l]; l++) if (name[l]=='.') dots++; if (dots >= conf.ndots || name[l-1]=='.') *search = 0; // ... // Walk the search list, splitting on whitespace. // When *search = 0 above: *p == 0, z == p, break on the first iteration. // → search is attempted ZERO times. for (p=search; *p; p=z) { for (; isspace(*p); p++); for (z=p; *z && !isspace(*z); z++); if (z==p) break; // ... query combined FQDN } // Final fallback: query the original as-is, exactly once. return name_from_dns(buf, canon, name, family, &conf); // Count dots. If dots ≥ ndots OR trailing dot → zero out the search list. for (dots=l=0; name[l]; l++) if (name[l]=='.') dots++; if (dots >= conf.ndots || name[l-1]=='.') *search = 0; // ... // Walk the search list, splitting on whitespace. // When *search = 0 above: *p == 0, z == p, break on the first iteration. // → search is attempted ZERO times. for (p=search; *p; p=z) { for (; isspace(*p); p++); for (z=p; *z && !isspace(*z); z++); if (z==p) break; // ... query combined FQDN } // Final fallback: query the original as-is, exactly once. return name_from_dns(buf, canon, name, family, &conf); # kind-config.yaml kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 name: dns-poc nodes: - role: control-plane # kind-config.yaml kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 name: dns-poc nodes: - role: control-plane # kind-config.yaml kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 name: dns-poc nodes: - role: control-plane kind create cluster --config kind-config.yaml kind create cluster --config kind-config.yaml kind create cluster --config kind-config.yaml # coredns-log-patch.yaml apiVersion: v1 kind: ConfigMap metadata: name: coredns namespace: kube-system data: Corefile: | .:53 { log errors health { lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa ttl 30 } prometheus :9153 forward . /etc/resolv.conf { max_concurrent 1000 } cache 30 loop reload loadbalance } # coredns-log-patch.yaml apiVersion: v1 kind: ConfigMap metadata: name: coredns namespace: kube-system data: Corefile: | .:53 { log errors health { lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa ttl 30 } prometheus :9153 forward . /etc/resolv.conf { max_concurrent 1000 } cache 30 loop reload loadbalance } # coredns-log-patch.yaml apiVersion: v1 kind: ConfigMap metadata: name: coredns namespace: kube-system data: Corefile: | .:53 { log errors health { lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa ttl 30 } prometheus :9153 forward . /etc/resolv.conf { max_concurrent 1000 } cache 30 loop reload loadbalance } kubectl apply -f coredns-log-patch.yaml kubectl rollout restart deployment/coredns -n kube-system kubectl apply -f coredns-log-patch.yaml kubectl rollout restart deployment/coredns -n kube-system kubectl apply -f coredns-log-patch.yaml kubectl rollout restart deployment/coredns -n kube-system # alpine-ndots2: musl + ndots:2 spec: dnsConfig: options: - name: ndots value: "2" containers: - name: shell image: alpine:3.20 # ... # alpine-ndots2: musl + ndots:2 spec: dnsConfig: options: - name: ndots value: "2" containers: - name: shell image: alpine:3.20 # ... # alpine-ndots2: musl + ndots:2 spec: dnsConfig: options: - name: ndots value: "2" containers: - name: shell image: alpine:3.20 # ... NAME=kubernetes.default.svc for p in alpine-ndots5 alpine-ndots2 debian-ndots5 debian-ndots2; do printf "===== [%s] =====\n" "$p" kubectl exec "$p" -- sh -c "getent hosts $NAME; echo exit=\$?" done NAME=kubernetes.default.svc for p in alpine-ndots5 alpine-ndots2 debian-ndots5 debian-ndots2; do printf "===== [%s] =====\n" "$p" kubectl exec "$p" -- sh -c "getent hosts $NAME; echo exit=\$?" done NAME=kubernetes.default.svc for p in alpine-ndots5 alpine-ndots2 debian-ndots5 debian-ndots2; do printf "===== [%s] =====\n" "$p" kubectl exec "$p" -- sh -c "getent hosts $NAME; echo exit=\$?" done ===== [alpine-ndots5] ===== 10.96.0.1 kubernetes.default.svc.cluster.local exit=0 ===== [alpine-ndots2] ===== exit=2 # ← musl, no search fallback → fails ===== [debian-ndots5] ===== 10.96.0.1 kubernetes.default.svc.cluster.local exit=0 ===== [debian-ndots2] ===== 10.96.0.1 kubernetes.default.svc.cluster.local exit=0 # ← glibc, NXDOMAIN then search fallback → succeeds ===== [alpine-ndots5] ===== 10.96.0.1 kubernetes.default.svc.cluster.local exit=0 ===== [alpine-ndots2] ===== exit=2 # ← musl, no search fallback → fails ===== [debian-ndots5] ===== 10.96.0.1 kubernetes.default.svc.cluster.local exit=0 ===== [debian-ndots2] ===== 10.96.0.1 kubernetes.default.svc.cluster.local exit=0 # ← glibc, NXDOMAIN then search fallback → succeeds ===== [alpine-ndots5] ===== 10.96.0.1 kubernetes.default.svc.cluster.local exit=0 ===== [alpine-ndots2] ===== exit=2 # ← musl, no search fallback → fails ===== [debian-ndots5] ===== 10.96.0.1 kubernetes.default.svc.cluster.local exit=0 ===== [debian-ndots2] ===== 10.96.0.1 kubernetes.default.svc.cluster.local exit=0 # ← glibc, NXDOMAIN then search fallback → succeeds kubectl logs -n kube-system -l k8s-app=kube-dns -f --tail=20 --prefix kubectl logs -n kube-system -l k8s-app=kube-dns -f --tail=20 --prefix kubectl logs -n kube-system -l k8s-app=kube-dns -f --tail=20 --prefix ... kubernetes.default.svc. AAAA NXDOMAIN ... kubernetes.default.svc. A NXDOMAIN ... kubernetes.default.svc. AAAA NXDOMAIN ... kubernetes.default.svc. A NXDOMAIN ... kubernetes.default.svc. AAAA NXDOMAIN ... kubernetes.default.svc. A NXDOMAIN ... kubernetes.default.svc. A NXDOMAIN ... kubernetes.default.svc.default.svc.cluster.local. A NXDOMAIN ... kubernetes.default.svc.svc.cluster.local. A NXDOMAIN ... kubernetes.default.svc.cluster.local. A NOERROR 10.96.0.1 ... kubernetes.default.svc. A NXDOMAIN ... kubernetes.default.svc.default.svc.cluster.local. A NXDOMAIN ... kubernetes.default.svc.svc.cluster.local. A NXDOMAIN ... kubernetes.default.svc.cluster.local. A NOERROR 10.96.0.1 ... kubernetes.default.svc. A NXDOMAIN ... kubernetes.default.svc.default.svc.cluster.local. A NXDOMAIN ... kubernetes.default.svc.svc.cluster.local. A NXDOMAIN ... kubernetes.default.svc.cluster.local. A NOERROR 10.96.0.1 # cleanup kind delete cluster --name dns-poc # cleanup kind delete cluster --name dns-poc # cleanup kind delete cluster --name dns-poc - Lowering ndots reduces DNS query amplification, but breaks internal service resolution on Alpine pods. - The cause isn't CoreDNS or Kubernetes — it's that musl libc skips the search list when dots ≥ ndots, while glibc falls back gracefully. - If you're on Alpine: switch base images, use FQDNs with a trailing dot, or roll out per-workload via dnsConfig. - The ask: A small DNS misconfiguration causes musl's resolver to stop on a single SERVFAIL. It doesn't even try the FQDN. Is this intentional? - Felker's answer: "If a lookup ends in SERVFAIL, the result is indeterminate. That should be reported to the caller as an error, not silently fallen back from. Otherwise the lookup result depends on transient nameserver failures." The principle is determinism. The moment fallback is allowed, the same query can return different answers between runs. An attacker can induce transient failures to manipulate which answer wins. From day one, musl's stance has been: "we don't reproduce the dangerous behavior of other implementations." - Reconsider your base image. alpine → debian-slim or distroless. The biggest hammer, but it solves the class of problem, not just this one. - Use FQDNs at the application level. my-svc.default.svc.cluster.local. (with the trailing dot) skips the search list regardless of libc. - Roll out per-workload. Apply dnsConfig to specific deployments first, not the whole cluster. - Run NodeLocal DNSCache in parallel. Independent of ndots, a cache layer dramatically cuts CoreDNS load and softens the cost of the search loop on glibc workloads.

The bigger lesson - DNS for Services and Pods (Kubernetes docs)

- Kubernetes DNS debugging guide — includes warnings for Alpine 3.17 and earlier- musl wiki — Functional differences from glibc- Pracucci — ndots:5 and application performance- NodeLocal DNSCacheThe fuller archaeology — every relevant GitHub issue across CoreDNS, kubernetes/dns, cert-manager, openwhisk, kind, and others — is in the original Korean post's appendix.

🏷️ Tags

toolsutilitiessecurity toolsultimateguideloweringndotsbreaksalpinedebian

More from Tools

Tools: Linux Security Essentials: Mastering Cron, Sudoers, and UFW for Cybersecurity (2026)

Tools: Linux Security Essentials: Mastering Cron, Sudoers, and UFW for Cybersecurity (2026)

2026-05-02 0
Tools: to Deploy Llama 3.2 with Ollama + Nginx Load Balancing on a $10/Month DigitalOcean Droplet: High-Availability Inference at 1/50th API Cost How

Tools: to Deploy Llama 3.2 with Ollama + Nginx Load Balancing on a $10/Month DigitalOcean Droplet: High-Availability Inference at 1/50th API Cost How

2026-05-02 0
Tools: Step-by-Step Tutorial for Fine-Tuning CodeLlama 70B with LoRA and 4x A100s A

Tools: Step-by-Step Tutorial for Fine-Tuning CodeLlama 70B with LoRA and 4x A100s A

2026-05-02 0
Tools: Breaking: TestSprite — Localized Dev Review: Pengalaman Nyata di Proyek E-Commerce Indonesia

Tools: Breaking: TestSprite — Localized Dev Review: Pengalaman Nyata di Proyek E-Commerce Indonesia

2026-05-02 0

Trending

1

CVE-2025-61481: Critical Remote Code Execution Vulnerability in MikroTik RouterOS & SwitchOS

2025-10-27 • 189 views
2

CVE-2025-43939: Dell Unity OS Command Injection (High)

2025-10-30 • 148 views
3

Google disputes false claims of massive Gmail data breach

2025-10-30 • 130 views
4

Microsoft: DNS outage impacts Azure and Microsoft 365 services

2025-10-30 • 88 views
5

3.5B Accounts, 1 Critical Flaw: Meta Closes WhatsApp Data-Harvesting

2025-11-25 • 81 views