Set up NodeLocal DNSCache

This document describes how to use NodeLocal DNSCache to reduce DNS lookup latency and improve your application's performance in your Google Kubernetes Engine (GKE) cluster.

NodeLocal DNSCache is a GKE add-on that improves DNS performance by running a DNS cache directly on each cluster node as a DaemonSet. When your Pods make a DNS request, the request first goes to the local cache on the same node. Handling requests locally significantly reduces average DNS lookup times and decreases the load on your cluster's central DNS provider, such as kube-dns or Cloud DNS for GKE. For a detailed explanation of DNS architecture and the benefits of NodeLocal DNSCache, see About service discovery.

In GKE Autopilot clusters, NodeLocal DNSCache is enabled by default, and you can't disable it. In GKE Standard clusters running version 1.33.1 and later, NodeLocal DNSCache is enabled by default, but you can disable it.

This document is for GKE users, including Developers and Admins and architects. To learn more about common roles and example tasks in Google Cloud, see Common GKE Enterprise user roles and tasks.

This document assumes you are familiar with the following:

• Kubernetes Service.
• Multi-cluster Services.
• About service discovery.
• About Cloud DNS.

Architecture

NodeLocal DNSCache is a GKE add-on that you can run in addition to kube-dns.

GKE implements NodeLocal DNSCache as a DaemonSet that runs a DNS cache on each node in your cluster.

When a Pod makes a DNS request, the request goes to the DNS cache that runs on the same node as the Pod. If the cache can't resolve the DNS request, the cache forwards the request to one of the following places, based on the query destination:

• kube-dns: all queries for the cluster DNS domain (cluster.local) are forwarded to kube-dns. The node-local-dns Pods use the kube-dns-upstream Service to access kube-dns Pods.
• Custom stub domains or upstream name servers: queries are forwarded directly from NodeLocal DNSCache Pods.
• Cloud DNS: all other queries are forwarded to the local metadata server that runs on the same node where the query originated. The local metadata server accesses Cloud DNS.

When you enable NodeLocal DNSCache on an existing cluster, GKE re-creates all cluster nodes that run GKE version 1.15 and later according to the node upgrade process.

After GKE re-creates the nodes, GKE automatically adds the addon.gke.io/node-local-dns-ds-ready=true label to the nodes. You must not add this label to the cluster nodes manually.

Benefits of NodeLocal DNSCache

NodeLocal DNSCache provides the following benefits:

• Reduced average DNS lookup time
• Connections from Pods to their local cache don't create conntrack table entries. This behavior prevents dropped and rejected connections that are caused by conntrack table exhaustion and race conditions.
• You can use NodeLocal DNSCache with Cloud DNS for GKE.
• DNS queries for external URLs (URLs that don't refer to cluster resources) are forwarded directly to the local metadata server and bypass kube-dns.
• The local DNS caches automatically pick up stub domains and upstream name servers that are specified in the Adding custom resolvers for stub domains section.

Requirements and limitations

• NodeLocal DNSCache consumes compute resources on each node of your cluster.
• NodeLocal DNSCache is not supported with Windows Server node pools.
• NodeLocal DNSCache requires GKE version 1.15 or later.
• NodeLocal DNSCache accesses kube-dns Pods by using TCP.
• NodeLocal DNSCache accesses upstreamServers and stubDomains by using TCP and UDP on GKE versions 1.18 or later. The DNS server must be reachable by using TCP and UDP.
• DNS records are cached for the following periods:
  • The record's time to live (TTL), or 30 seconds if the TTL is more than 30 seconds.
  • 5 seconds if the DNS response is NXDOMAIN.
• NodeLocal DNSCache Pods listen on port 53, 9253, 9353, and 8080 on the nodes. If you run any other hostNetwork Pod or configure a hostPorts with those ports, NodeLocal DNSCache fails and DNS errors occur. NodeLocal DNSCache Pods don't use hostNetwork mode when they use GKE Dataplane V2 and don't use Cloud DNS for GKE.
• The local DNS cache runs only on node pools that run GKE versions 1.15 and later. If you enable NodeLocal DNSCache in a cluster with nodes that run earlier versions, Pods on those nodes use kube-dns.

Before you begin

Before you start, make sure that you have performed the following tasks:

• Enable the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• If you want to use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running the gcloud components update command. Earlier gcloud CLI versions might not support running the commands in this document.

• Ensure that you have an existing Autopilot or Standard cluster. If you need one, create an Autopilot cluster. For Autopilot clusters, NodeLocal DNSCache is enabled by default and can't be overridden.

Enable NodeLocal DNSCache

For Standard clusters, you can enable NodeLocal DNSCache by using the Google Cloud CLI or the Google Cloud console.

gcloud container clusters update CLUSTER_NAME \  --location=COMPUTE_LOCATION \  --update-addons=NodeLocalDNS=ENABLED 

This change requires recreating the nodes, which can cause disruption to your running workloads. For details about this specific change, find the corresponding row in the manual changes that recreate the nodes using a node upgrade strategy and respecting maintenance policies table. To learn more about node updates, see Planning for node update disruptions.

Verify that NodeLocal DNSCache is enabled

You can verify that NodeLocal DNSCache is running by listing the node-local-dns Pods:

kubectl get pods -n kube-system -o wide | grep node-local-dns 

The output is similar to the following:

node-local-dns-869mt 1/1 Running 0 6m24s 10.128.0.35 gke-test-pool-69efb6b8-5d7m <none> <none> node-local-dns-htx4w 1/1 Running 0 6m24s 10.128.0.36 gke-test-pool-69efb6b8-wssk <none> <none> node-local-dns-v5njk 1/1 Running 0 6m24s 10.128.0.33 gke-test-pool-69efb6b8-bhz3 <none> <none> 

The output shows a node-local-dns Pod for each node that runs GKE version 1.15 or later.

Disable NodeLocal DNSCache

You can disable NodeLocal DNSCache by using the following command:

gcloud container clusters update CLUSTER_NAME \  --location=COMPUTE_LOCATION \  --update-addons=NodeLocalDNS=DISABLED 

Replace the following:

• CLUSTER_NAME: the name of the cluster to disable.
• COMPUTE_LOCATION: the Compute Engine location for the cluster.

This change requires recreating the nodes, which can cause disruption to your running workloads. For details about this specific change, find the corresponding row in the manual changes that recreate the nodes using a node upgrade strategy and respecting maintenance policies table. To learn more about node updates, see Planning for node update disruptions.

Troubleshoot NodeLocal DNSCache

For general information about diagnosing Kubernetes DNS issues, see Debugging DNS Resolution.

NodeLocal DNSCache is not enabled immediately

When you enable NodeLocal DNSCache on an existing cluster, GKE might not update the nodes immediately if the cluster has a configured maintenance window or exclusion. For more information, see caveats for node re-creation and maintenance windows.

If you prefer not to wait, you can manually apply the changes to the nodes by calling the gcloud container clusters upgrade command and passing the --cluster-version flag with the same GKE version that the node pool is already running. You must use the Google Cloud CLI for this workaround.

NodeLocal DNSCache with Cloud DNS

If you use NodeLocal DNSCache with Cloud DNS, the cluster uses the name server IP address 169.254.20.10, as shown in the following diagram:

As a result, the IP address of the kube-dns Service might be different from the name server IP address that your Pods use. This difference in IP addresses is expected, because the 169.254.20.10 name server IP address is required for Cloud DNS to work correctly.

To check the IP addresses, run the following commands:

1. View the IP address of the kube-dns Service:

   kubectl get svc -n kube-system kube-dns -o jsonpath="{.spec.clusterIP}" 

   The output is the IP address of kube-dns, such as 10.0.0.10

2. Open a shell session in your Pod:

   kubectl exec -it POD_NAME -- /bin/bash 

3. In the Pod shell session, read the contents of the /etc/resolv.conf file:

   cat /etc/resolv.conf 

   The output is 169.254.20.10

Network policy with NodeLocal DNSCache

If you use network policy with NodeLocal DNSCache and you are not using Cloud DNS or GKE Dataplane V2, you must configure rules to permit your workloads and the node-local-dns Pods to send DNS queries.

Use an ipBlock rule in your manifest to allow communication between your Pods and kube-dns.

The following manifest describes a network policy that uses an ipBlock rule:

spec:  egress:  - ports:  - port: 53  protocol: TCP  - port: 53  protocol: UDP  to:  - ipBlock:  cidr: KUBE_DNS_SVC_CLUSTER_IP/32  podSelector: {}  policyTypes:  - Egress 

Replace KUBE_DNS_SVC_CLUSTER_IP with the IP address of the kube-dns service. You can get the IP address of the kube-dns service by using the following command:

kubectl get svc -n kube-system kube-dns -o jsonpath="{.spec.clusterIP}" 

Known issues

This section lists known issues with NodeLocal DNSCache.

DNS timeout in ClusterFirstWithHostNet dnsPolicy when using NodeLocal DNSCache and GKE Dataplane V2

On clusters that use GKE Dataplane V2 and NodeLocal DNSCache, Pods with the hostNetwork field set to true and the dnsPolicy field set to ClusterFirstWithHostNet cannot reach cluster DNS backends. DNS logs might contain entries similar to the following:

dnslookup: write to 'a.b.c.d': Operation not permitted ;; connection timed out; no servers could be reached 

The output indicates that the DNS requests cannot reach the backend servers.

A workaround is to set the dnsPolicy and dnsConfig fields for hostNetwork Pods:

spec:  dnsPolicy: "None"  dnsConfig:  nameservers:  - KUBE_DNS_UPSTREAM  searches:  - NAMESPACE.svc.cluster.local  - svc.cluster.local  - cluster.local  - c.PROJECT_ID.internal  - google.internal  options:  - name: ndots  value: "5" 

Replace the following:

• NAMESPACE: the namespace of the hostNetwork pod.
• PROJECT_ID: the ID of your Google Cloud project.

• KUBE_DNS_UPSTREAM: the ClusterIP of the upstream kube-dns service. You can get this value by using the following command:

  kubectl get svc -n kube-system kube-dns-upstream -o jsonpath="{.spec.clusterIP}" 

DNS requests from the Pod can now reach kube-dns and bypass NodeLocal DNSCache.

NodeLocal DNSCache timeout errors

On clusters that have NodeLocal DNSCache enabled, the logs might contain entries similar to the following:

[ERROR] plugin/errors: 2 <hostname> A: read tcp <node IP: port>-><kubedns IP>:53: i/o timeout 

The output includes the IP address of the kube-dns-upstream Cluster IP Service. In this example, the response to a DNS request was not received from kube-dns in two seconds. This issue could be due to one of the following reasons:

• An underlying network connectivity problem.
• Significantly increased DNS queries from the workload or from node pool upscaling.

As a result, the existing kube-dns Pods can't handle all requests in time. The workaround is to increase the number of kube-dns replicas by scaling up kube-dns.

Scaling up kube-dns

You can use a lower value for the nodesPerReplica field to help ensure that more kube-dns Pods are created while cluster nodes scale up. We highly recommend setting an explicit max value to help ensure that the GKE control plane virtual machine (VM) is not overwhelmed by the large number of kube-dns Pods watching the Kubernetes API.

You can set the max field to the number of nodes in the cluster. If the cluster has more than 500 nodes, set the max field to 500.

For Standard clusters, you can modify the number of kube-dns replicas by editing the kube-dns-autoscaler ConfigMap. This configuration is not supported in Autopilot clusters.

kubectl edit configmap kube-dns-autoscaler --namespace=kube-system 

The output is similar to the following:

linear: '{"coresPerReplica":256, "nodesPerReplica":16,"preventSinglePointFailure":true}' 

The number of kube-dns replicas is calculated using the following formula:

replicas = max(ceil(cores * 1/coresPerReplica), ceil(nodes * 1/nodesPerReplica))

If you define the min and max fields in the ConfigMap, the replicas are bounded by these values. To scale up, change the value of the nodesPerReplica field to a smaller value and include a value for the max field:

linear: '{"coresPerReplica":256, "nodesPerReplica":8,"max": 15,"preventSinglePointFailure":true}' 

The configuration creates one kube-dns Pod for every eight nodes in the cluster. A 24-node cluster has three replicas, and a 40-node cluster has five replicas. If the cluster grows beyond 120 nodes, the number of kube-dns replicas does not grow beyond 15, which is the max value.

To help ensure a baseline level of DNS availability in your cluster, set a minimum replica count for kube-dns.

The output for the kube-dns-autoscaler ConfigMap that has a defined min field would be similar to the following:

linear: '{"coresPerReplica":256, "nodesPerReplica":8,"max": 15,"min": 5,"preventSinglePointFailure":true}' 

What's next

• Read an overview of how GKE provides managed DNS.
• Read DNS for Services and Pods for a general overview of how DNS is used in Kubernetes clusters.
• Learn how to use Cloud DNS for GKE.