3

Problem:

Running Ubuntu 17.10

I have been trying to resolv (hehe) this issue for about a week now and despite countless Google searches and about 20 different attempts, I can not stop dnsmasq from periodically causing my CPU to spike for about a minute with the following offenders:

  • systemd-resolved
  • systemd-journald
  • dnsmasq

Monitoring journalctl -f I see this every time it happens:

maximum number of concurrent dns queries reached (150)

Accompanied/preceded by a crazy loop of requests to some domain (usually ubuntu connection check) like the following:

query[A] connectivity-check.ubuntu.com from 127.0.0.1 forwarded connectivity-check.ubuntu.com to 127.0.1.1 forwarded connectivity-check.ubuntu.com to 127.0.0.53 query[A] connectivity-check.ubuntu.com from 127.0.0.1 forwarded connectivity-check.ubuntu.com to 127.0.0.53 query[AAAA] connectivity-check.ubuntu.com from 127.0.0.1 forwarded connectivity-check.ubuntu.com to 127.0.0.53 query[AAAA] connectivity-check.ubuntu.com from 127.0.0.1 forwarded connectivity-check.ubuntu.com to 127.0.0.53 query[A] connectivity-check.ubuntu.com from 127.0.0.1 forwarded connectivity-check.ubuntu.com to 127.0.0.53 query[AAAA] connectivity-check.ubuntu.com from 127.0.0.1 forwarded connectivity-check.ubuntu.com to 127.0.0.53

I've found that changing my /etc/resolv.conf to use nameserver 127.0.0.53 causes the spike to dissipate almost instantaneously.

However, as that file is updated regularly by Network Manager, I have to do this about once an hour.

Configuration:

/etc/resolv.conf

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8) # DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN # 127.0.0.53 is the systemd-resolved stub resolver. # run "systemd-resolve --status" to see details about the actual nameservers. nameserver 127.0.0.1 search fios-router.home

/etc/NetworkManager/NetworkManager.conf

[main] plugins=ifupdown,keyfile [ifupdown] managed=false [device] wifi.scan-rand-mac-address=no

/etc/dnsmasq.conf

// All default except this at the very end for my wildcard DNS address=/asmar.d/127.0.0.1

/run/dnsmasq/resolv.conf

nameserver 127.0.0.53

/run/resolvconf/interfaces:

lo.dnsmasq:

nameserver 127.0.0.1

systemd-resolved:

nameserver 127.0.0.53

/etc/resolvconf/interface-order:

# interface-order(5) lo.inet6 lo.inet lo.@(dnsmasq|pdnsd) lo.!(pdns|pdns-recursor) lo tun* tap* hso* em+([0-9])?(_+([0-9]))* p+([0-9])p+([0-9])?(_+([0-9]))* @(br|eth)*([^.]).inet6 @(br|eth)*([^.]).ip6.@(dhclient|dhcpcd|pump|udhcpc) @(br|eth)*([^.]).inet @(br|eth)*([^.]).@(dhclient|dhcpcd|pump|udhcpc) @(br|eth)* @(ath|wifi|wlan)*([^.]).inet6 @(ath|wifi|wlan)*([^.]).ip6.@(dhclient|dhcpcd|pump|udhcpc) @(ath|wifi|wlan)*([^.]).inet @(ath|wifi|wlan)*([^.]).@(dhclient|dhcpcd|pump|udhcpc) @(ath|wifi|wlan)* ppp* *

systemd-resolve --status:

Global DNS Servers: 127.0.0.1 DNSSEC NTA: 10.in-addr.arpa 16.172.in-addr.arpa 168.192.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa corp d.f.ip6.arpa home internal intranet lan local private test Link 5 (br-b1f5461ac410) Current Scopes: none LLMNR setting: yes MulticastDNS setting: no DNSSEC setting: no DNSSEC supported: no Link 4 (docker0) Current Scopes: none LLMNR setting: yes MulticastDNS setting: no DNSSEC setting: no DNSSEC supported: no Link 3 (wlp62s0) Current Scopes: none LLMNR setting: yes MulticastDNS setting: no DNSSEC setting: no DNSSEC supported: no Link 2 (enp61s0) Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6 LLMNR setting: yes MulticastDNS setting: no DNSSEC setting: no DNSSEC supported: no DNS Servers: 8.8.8.8 8.8.4.4 ::1

Questions:

How can I resolve this issue while still using my wildcard domain name?

Optional: How can I achieve this while using Google DNS?

Please do not recommend upping the concurrent dns queries. That is not a solution.

SOLVED!

See telcoM's DNS crash course (the accepted answer) that led me to the solution

See my follow-up & final solution as I experimented with the knowledge gained from that answer

Improve this question
2
  • do you have any evidence to suggest that it's caused by the existence of a wildcard? does the same thing happen if you comment out your wildcard host? or change it to a non-wildcard? Commented Jan 17, 2018 at 3:34
  • Not necessarily -- I do have evidence that it's caused by dnsmasq (this doesn't happen if I remove it), however. Which is the only way I know of (though I'm sure there are more) to implement a wildcard domain name in Linux. Commented Jan 17, 2018 at 6:57

4 Answers 4

Reset to default
5
+50

It looks like you may have dnsmasq process in 127.0.0.1 and systemd-resolved process in 127.0.0.53 passing queries back and forth between each other, causing a loop. Even dnsmasq alone might be capable of looping, as by default it looks into /etc/resolv.conf to find the real DNS servers to use for the names it does not have information for.

Your DNS configuration probably has quite many layers:

  • first, there is the DNS server information you get from your ISP by DHCP or similar.
  • then, there is NetworkManager, which could be configured to override the information and use dnsmasq instead, but isn't currently configured that way.
  • instead, NetworkManager is configured to use the resolvconf tool to update the real /etc/resolv.conf. And dnsmasq may include a drop-in configuration for resolvconf to override any DNS services received by DHCP and use 127.0.0.1 instead while dnsmasq is running.
  • systemd-resolved may also include a drop-in configuration for resolvconf, but is apparently getting overridden by dnsmasq.

What I don't yet understand is where the 127.0.1.1 and 127.0.0.53 come from. Are they perhaps mentioned in dnsmasq default configuration in Ubuntu?

As it says in the comment of /etc/resolv.conf, run this command to see more information on systemd-resolved configuration:

systemd-resolve --status

Also check the contents of the /run/resolvconf/interface/ directory: that is where the resolvconf tool collects all the DNS server information it gets from various sources. The /etc/resolvconf/interface-order will determine the order in which each source is checked, until either a loopback address is encountered or 3 DNS servers have been listed for real /etc/resolv.conf.

Since you are using dnsmasq to set up a wildcard domain, you'll want to keep 127.0.0.1 in your /etc/resolv.conf - but you'll want to configure dnsmasq to not use that file, but instead get the DNS servers it should use from somewhere else.

If /run/NetworkManager/resolv.conf contains those DNS servers you get from your ISP by DHCP, you can easily use that for dnsmasq by adding this line to its configuration:

resolv-file=/run/NetworkManager/resolv.conf

This tells dnsmasq where to get DNS information for those things it don't already know about. So if you want to use Google DNS, you could configure dnsmasq with

resolv-file=/etc/google-dns-resolv.conf

and put the DNS configuration lines for Google DNS in the usual format to /etc/google-dns-resolv.conf.

Improve this answer
7
  • Thank you so much for the detailed response! I'm testing a solution based on your recommendations and updating my question accordingly with additional detail (adding a "Follow-up" section). The changes I've made seemed to make a difference which I'll describe above. I'm going to run things for a little bit to see how it behaves over time (since this has been an intermittent issue). If everything checks out, the bounty is absolutely yours! Commented Jan 19, 2018 at 10:09
  • Still experiencing CPU spikes - the difference I noticed initially was just misinterpreted. I've added as much detail regarding all of this in the Folllow-up section above. Commented Jan 19, 2018 at 10:35
  • Also -- I discovered that when I restart dnsmasq, journalctl -f shows me "no servers found in /run/dnsmasq/resolv.conf, will retry". So, I decided to see what the contents of that file were and it is nameserver 127.0.0.53. Commented Jan 19, 2018 at 10:43
  • Preliminary update: I've replaced the contents of /run/dnsmasq/resolv.conf with nameserver 127.0.0.1 and the output of journalctl -f only shows 127.0.0.1 now. No CPU spiking since doing it and wildcard DNS is still functioning. I will update after I've let it run like this for a little bit. Commented Jan 19, 2018 at 10:51
  • With the info here, I was able to solve this issue! Please see "Update #3" in my question above for more detail. Bounty will be available in 15 hours at which point I will award it to you. Thank you again for your help!! Commented Jan 19, 2018 at 11:58
1

Fixed my dnsmasq systemd-resolve race with dnsmasq config.

  1. Use systemd-resolve (127.0.0.53) for external dns, as it does dnssec and has root servers.
  2. Only bind dnsmasq to loopback for some static configs, could add more interfaces if i need it for dhcp.

/etc/dnsmasq.d/myconfig

#PES 20180808 dnsmasq and systemd-resolve conflict. # dont use /etc/resolv.conf, go direct to systemd-resolve, only bind to lo no-resolv bind-interfaces interface=lo server=127.0.0.53
Improve this answer
1

On Ubuntu 18.04

1 - Create /etc/google-dns-resolv.conf

nameserver 8.8.8.8 nameserver 8.8.4.4

2 - Create a config file /etc/dnsmasq.d/my_dnsmaq.conf

resolv-file=/etc/google-dns-resolv.conf

3 - Insert at the bottom of /etc/default/dnsmasq

IGNORE_RESOLVCONF=yes

4 - Finally, restart your dnsmasq service

sudo service dnsmasq restart

I've used Johny's answer above but step nº3 in my answer was missing in order for my dnsmasq config file to be read properly. Thanks

Improve this answer
0

I originally documented this as a part of the question as I was trying various solutions. This has been cut/pasted from that question into this answer for clarity, but this answer does not stand-alone without the insight provided in telcoM's answer.

Solution:

I have made the following change according to telcoM's suggestion below.

Created /etc/google-dns-resolv.conf with the contents:

nameserver 8.8.8.8 nameserver 8.8.4.4

Added resolv-file=/etc/google-dns-resolv.conf to /etc/dnsmasq.conf

After making the change and running sudo service dnsmasq restart the CPU still spiked, but now rsyslogd entered into the mix of offenders. It settled down after about a minute, but I no longer saw 127.0.1.1 in journalctl -f.

Update: After ~10 minutes, the CPU has spiked again:

enter image description here

Update #2: Thanks to telcoM's excellent little DNS crash-course below and skepticism over where 127.0.0.53 was coming from, I dug in a bit (see my comments on his answer) and discovered it was coming from /run/dnsmasq/resolv.conf, which had the contents nameserver 127.0.0.53. I decided to update the contents to nameserver 127.0.0.1. Now, the only IP address I see output from journalctl -f is 127.0.0.1 and have not experienced a CPU spike since (going on ~10 mins now; which is at least as long as any period of rest my CPU has enjoyed prior)!

[SOLVED] Update #3: Using 127.0.0.1 in /run/dnsmasq/resolv.conf was foolish. It totally killed my DNS resolution (it appeared to work at first because I was only accessing cached DNSes). After switching to use Google nameservers (8.8.8.8 & 8.8.4.4) DNS resolution works again, my wildcard domains continue to work, and I am not experiencing CPU spikes. I'm going to see how this behaves for about an hour as I execute my typical day-to-day activities and update again accordingly; hopefully with a SOLVED!

Update #4: This appears to be solved now by following the steps in Update #3. Thank you telcoM for all of the excellent detail that steered me in the right direction. Bounty will be awarded when available in 15 hours.

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.