Periodic loss of all recursor backends — dnsdist 2.0.1 marks all recursors as ServFail once per 1–2 months

[chelezo]

Hello.

We are experiencing a recurring issue with dnsdist 2.0.1 running on Ubuntu 22.04 (IPv6 fully disabled on kernel level via GRUB).
Approximately once every 1–2 months, all three recursor backends in the recursor pool suddenly start failing health checks at the same time. dnsdist marks all three recursors as down with continuous ServFail responses.

Only the recursor pool is affected — authoritative pools and DNSCrypt pools continue working normally.

The problem persists until dnsdist or the recursors are restarted manually.

Environment details

• dnsdist version: 2.0.1
• OS: Ubuntu Server 22.04 LTS
• IPv6: Disabled via GRUB_CMDLINE_LINUX_DEFAULT="ipv6.disable=1"

Topology:

• 3× dnsdist instances
• Connected via keepalived to a single shared VIP
• Identical configuration on all three nodes

Recursor backends:

• 10.18.41.6:5301
• 10.18.41.7:5301
• 10.18.41.8:5301

Recursors themselves are configured very simply.

Problem description

Once every ~30–60 days, all three recursors begin failing health checks simultaneously.

dnsdist logs show continuous errors like:

Backend 10.18.41.6:5301 responded to health check with ServFail
Backend 10.18.41.7:5301 responded to health check with ServFail
Backend 10.18.41.8:5301 responded to health check with ServFail

Repeat every 5 seconds.

This state is restored automatically after 10 minutes.
Restarting either dnsdist or the recursor instances immediately resolves the issue.

Example of log output

Dec 12 07:57:48 dnsdist: Backend 10.18.41.6:5301 responded to health check with ServFail
Dec 12 07:57:48 dnsdist: Backend 10.18.41.8:5301 responded to health check with ServFail
Dec 12 07:57:48 dnsdist: Backend 10.18.41.7:5301 responded to health check with ServFail
Dec 12 07:57:53 dnsdist: Backend 10.18.41.6:5301 responded to health check with ServFail
Dec 12 07:57:53 dnsdist: Backend 10.18.41.8:5301 responded to health check with ServFail
Dec 12 07:57:53 dnsdist: Backend 10.18.41.7:5301 responded to health check with ServFail

... repeating infinitely

Full dnsdist configuration

setLocal('127.0.0.1')
addLocal('10.18.41.2')
addLocal('10.18.41.3')
addLocal('10.18.41.34')
addLocal('10.18.41.35')
setACL({'0.0.0.0/0'})
setSyslogFacility(0)
setVerboseHealthChecks(true)
controlSocket('127.0.0.1:5199')
setConsoleACL('127.0.0.1')
setKey("ej0NCggL+KefBk60w5BdJ/4v9RU42hNp2CrU/awltHg=")
setVerboseHealthChecks(true)
setRoundRobinFailOnNoServer(true)
setServerPolicy(roundrobin)
addDOHLocal("10.18.41.2:443",
            "/opt/doh_sert/fullchain.pem",
            "/opt/doh_sert/key.key")
pc = newPacketCache(4000000, {
    maxTTL=86400,
    minTTL=0,
    temporaryFailureTTL=60,
    staleTTL=60,
    dontAge=false
})

tap_logging = newFrameStreamTcpLogger("10.18.41.15:6000")
addAction(AllRule(), DnstapLogAction("dnsdist_server", tap_logging))
newServer({
  address="10.18.41.12:5454",
  pool="ext-dnscrypt"
})
newServer({
  address="10.18.41.13:5454",
  pool="ext-dnscrypt"
})
newServer({
  address="10.18.41.14:5454",
  pool="ext-dnscrypt"
})

specific_domains = newSuffixMatchNode()
-- specific_domains:add(newDNSName("mt.fppk.com"))
-- specific_domains:add(newDNSName("mx.fppk.com"))

addAction(
  AndRule({
    NetmaskGroupRule({'172.16.0.0/12'}),
    SuffixMatchNodeRule(specific_domains)
  }),
  PoolAction("ext-dnscrypt")
)

mgc_lan_domains = newSuffixMatchNode()
mgc_lan_domains:add(newDNSName("mgc.lan"))

addAction(
  AndRule({
    NetmaskGroupRule({'10.18.24.0/21'}),
    SuffixMatchNodeRule(mgc_lan_domains)
  }),
  PoolAction("geoip-auth-primary")
)

-- Пул гостевых вайфай
addAction(
  NetmaskGroupRule({'10.0.200.0/24'}),
  PoolAction("ext-dnscrypt")
)

newServer({
  address='10.18.41.6:5301',
  pool='recursor',
  checkName='google.com.',
  mustResolve=true,
  checkType="A",
  checkInterval=5,
  checkTimeout=1,
  maxCheckFailures=3,
  rise=2,
  sockets=16
})
newServer({
  address='10.18.41.7:5301',
  pool='recursor',
  checkName='google.com.',
  mustResolve=true,
  checkType="A",
  checkInterval=5,
  checkTimeout=1,
  maxCheckFailures=3,
  rise=2
})
newServer({
  address='10.18.41.8:5301',
  pool='recursor',
  checkName='google.com.',
  mustResolve=true,
  checkType="A",
  checkInterval=5,
  checkTimeout=1,
  maxCheckFailures=3,
  rise=2
})
newServer({
  address='10.18.0.2:53',
  pool='auth-primary',
  checkName='mgc.lan.',
  mustResolve=true,
  checkType="A",
  checkInterval=5,
  checkTimeout=1,
  maxCheckFailures=3,
  rise=2
})
newServer({
  address='10.18.41.20:5304',
  pool='geoip-auth-primary',
  checkName='vm1.mgc.lan.',
  mustResolve=true,
  checkType="A",
  checkInterval=5,
  checkTimeout=1,
  maxCheckFailures=3,
  rise=2
})

newServer({
  address='10.20.1.2:53',
  pool='lcc1-auth-primary',
  checkName='dc.lcc1.lan.',
  mustResolve=true,
  checkType="A",
  checkInterval=5,
  checkTimeout=1,
  maxCheckFailures=3,
  rise=2
})

newServer({
  address='10.18.1.2:53',
  pool='hhc-auth-primary',
  checkName='dc.hhc.lan.',
  mustResolve=true,
  checkType="A",
  checkInterval=5,
  checkTimeout=1,
  maxCheckFailures=3,
  rise=2
})
newServer({
  address='10.0.241.2:53',
  pool='local-auth-primary',
  checkName='s-f-dc.fx.local.',
  mustResolve=true,
  checkType="A",
  checkInterval=5,
  checkTimeout=1,
  maxCheckFailures=3,
  rise=2
})
newServer({
  address='10.20.4.42:53',
  pool='pr-auth-primary',
  checkName='dc.pr.lan.',
  mustResolve=true,
  checkType="A",
  checkInterval=5,
  checkTimeout=1,
  maxCheckFailures=3,
  rise=2
})
newServer({
  address='10.18.2.2:53',
  pool='bccb-auth-primary',
  checkName='dc.bccb.lan.',
  mustResolve=true,
  checkType="A",
  checkInterval=5,
  checkTimeout=1,
  maxCheckFailures=3,
  rise=2
})

newServer({
  address='10.42.0.2:53',
  pool='media-auth-primary',
  checkName='dc.media.lan.',
  mustResolve=true,
  checkType="A",
  checkInterval=5,
  checkTimeout=1,
  maxCheckFailures=3,
  rise=2
})
newServer({
  address='10.18.235.2:53',
  pool='ahcc-auth-primary',
  checkName='dc.ahcc.lan.',
  mustResolve=true,
  checkType="A",
  checkInterval=5,
  checkTimeout=1,
  maxCheckFailures=3,
  rise=2
})

newServer({
  address='10.18.236.2:53',
  pool='afcc-auth-primary',
  checkName='dc.afcc.lan.',
  mustResolve=true,
  checkType="A",
  checkInterval=5,
  checkTimeout=1,
  maxCheckFailures=3,
  rise=2
})

newServer({
  address='10.20.11.2:53',
  pool='est-auth-primary',
  checkName='dc.est.lan.',
  mustResolve=true,
  checkType="A",
  checkInterval=5,
  checkTimeout=1,
  maxCheckFailures=3,
  rise=2
})

newServer({
  address='10.18.234.2:53',
  pool='accg-auth-primary',
  checkName='dc.accg.lan.',
  mustResolve=true,
  checkType="A",
  checkInterval=5,
  checkTimeout=1,
  maxCheckFailures=3,
  rise=2
})

newServer({
  address='10.41.52.2:53',
  pool='gds-auth-primary',
  checkName='dc.gds.lan.',
  mustResolve=true,
  checkType="A",
  checkInterval=5,
  checkTimeout=1,
  maxCheckFailures=3,
  rise=2
})

newServer({
  address='10.41.192.2:53',
  pool='iqnd-auth-primary',
  checkName='dc.iqnd.lan.',
  mustResolve=true,
  checkType="A",
  checkInterval=5,
  checkTimeout=1,
  maxCheckFailures=3,
  rise=2
})




newServer({address='10.18.41.9:5300', pool='auth'})
-- newServer({address='10.18.41.10:5300', pool='auth'})
newServer({address='10.18.41.11:5300', pool='auth'})
getPool("recursor"):setCache(pc)
getPool("auth"):setCache(pc)
getPool("ext-dnscrypt"):setCache(pc)
getPool(""):setCache(pc)


msdcs_domain = newSuffixMatchNode()
msdcs_domain:add(newDNSName("_msdcs.mgc.lan"))
addAction(SuffixMatchNodeRule(msdcs_domain), PoolAction('auth-primary'))
msdcs_domain_est = newSuffixMatchNode()
msdcs_domain_est:add(newDNSName("_msdcs.est.lan"))
addAction(SuffixMatchNodeRule(msdcs_domain_est), PoolAction('est-auth-primary'))

msdcs_domain_lcc1 = newSuffixMatchNode()
msdcs_domain_lcc1:add(newDNSName("_msdcs.lcc1.lan"))
addAction(SuffixMatchNodeRule(msdcs_domain_lcc1), PoolAction('lcc1-auth-primary'))

msdcs_domain_accg = newSuffixMatchNode()
msdcs_domain_accg:add(newDNSName("_msdcs.accg.lan"))
addAction(SuffixMatchNodeRule(msdcs_domain_accg), PoolAction('accg-auth-primary'))

msdcs_domain_iqnd = newSuffixMatchNode()
msdcs_domain_iqnd:add(newDNSName("_msdcs.iqnd.lan"))
addAction(SuffixMatchNodeRule(msdcs_domain_iqnd), PoolAction('iqnd-auth-primary'))


msdcs_domain_media = newSuffixMatchNode()
msdcs_domain_media:add(newDNSName("_msdcs.media.lan"))
addAction(SuffixMatchNodeRule(msdcs_domain_media), PoolAction('media-auth-primary'))

msdcs_domain_ahcc = newSuffixMatchNode()
msdcs_domain_ahcc:add(newDNSName("_msdcs.ahcc.lan"))
addAction(SuffixMatchNodeRule(msdcs_domain_ahcc), PoolAction('ahcc-auth-primary'))

msdcs_domain_afcc = newSuffixMatchNode()
msdcs_domain_afcc:add(newDNSName("_msdcs.afcc.lan"))
addAction(SuffixMatchNodeRule(msdcs_domain_afcc), PoolAction('afcc-auth-primary'))

msdcs_domain_hhc = newSuffixMatchNode()
msdcs_domain_hhc:add(newDNSName("_msdcs.hhc.lan"))
addAction(SuffixMatchNodeRule(msdcs_domain_hhc), PoolAction('hhc-auth-primary'))
msdcs_domain_local = newSuffixMatchNode()
msdcs_domain_local:add(newDNSName("_msdcs.fx.local"))
addAction(SuffixMatchNodeRule(msdcs_domain_local), PoolAction('local-auth-primary'))

msdcs_domain_bccb = newSuffixMatchNode()
msdcs_domain_bccb:add(newDNSName("_msdcs.bccb.lan"))
addAction(SuffixMatchNodeRule(msdcs_domain_bccb), PoolAction('bccb-auth-primary'))

msdcs_domain_pr = newSuffixMatchNode()
msdcs_domain_pr:add(newDNSName("_msdcs.pr.lan"))
addAction(SuffixMatchNodeRule(msdcs_domain_pr), PoolAction('pr-auth-primary'))

msdcs_domain_gs = newSuffixMatchNode()
msdcs_domain_gs:add(newDNSName("_msdcs.gs.lan"))
addAction(SuffixMatchNodeRule(msdcs_domain_gs), PoolAction('gs-auth-primary'))

msdcs_domain_pr = newSuffixMatchNode()
msdcs_domain_pr:add(newDNSName("pr.lan"))
addAction(SuffixMatchNodeRule(msdcs_domain_pr), PoolAction('auth-primary'))



domain_names = newSuffixMatchNode()
domain_names:add(newDNSName("lan"))

-- addAction(SuffixMatchNodeRule(msdcs_domain), PoolAction('auth-primary'))
addAction(SuffixMatchNodeRule(domain_names), PoolAction('auth'))
addAction(AllRule(), PoolAction('recursor'))
-- tap_logging = newFrameStreamTcpLogger("10.18.20.145:6009")
-- addAction(AllRule(), DnstapLogAction("dnsdist_server", tap_logging))
webserver("0.0.0.0:8083")
setWebserverConfig({
  password="$scrypt$ln=10,p=1,r=8$RSYJ2QDmdlkNYMyqZF/FWw==$JQTftQCvAXR4Qtrg0lQmvrzgYEo3/PjEeuV4/2Oq1Vg=",
  apiKey="$scrypt$ln=10,p=1,r=8$RSYJ2QDmdlkNYMyqZF/FWw==$JQTftQCvAXR4Qtrg0lQmvrzgYEo3/PjEeuV4/2Oq1Vg=",
  acl="10.0.0.0/8"
})

Any advice is very welcome.
Thank you!

[rgacogne]

Dec 12 07:57:48 dnsdist: Backend 10.18.41.6:5301 responded to health check with ServFail
Dec 12 07:57:48 dnsdist: Backend 10.18.41.8:5301 responded to health check with ServFail
Dec 12 07:57:48 dnsdist: Backend 10.18.41.7:5301 responded to health check with ServFail
Dec 12 07:57:53 dnsdist: Backend 10.18.41.6:5301 responded to health check with ServFail
Dec 12 07:57:53 dnsdist: Backend 10.18.41.8:5301 responded to health check with ServFail
Dec 12 07:57:53 dnsdist: Backend 10.18.41.7:5301 responded to health check with ServFail

Have you tried:

1. capturing the traffic to check if the health-check queries and responses are looking as expected
2. querying the recursors directly to see if they are indeed sending ServFail for the checked name
3. looked at the recursors' logs after enabling tracing (https://docs.powerdns.com/recursor/running.html#tracing-queries) for these queries, to see what is happening?

[chelezo]

Dec 12 07:57:48 dnsdist: Backend 10.18.41.6:5301 responded to health check with ServFail
Dec 12 07:57:48 dnsdist: Backend 10.18.41.8:5301 responded to health check with ServFail
Dec 12 07:57:48 dnsdist: Backend 10.18.41.7:5301 responded to health check with ServFail
Dec 12 07:57:53 dnsdist: Backend 10.18.41.6:5301 responded to health check with ServFail
Dec 12 07:57:53 dnsdist: Backend 10.18.41.8:5301 responded to health check with ServFail
Dec 12 07:57:53 dnsdist: Backend 10.18.41.7:5301 responded to health check with ServFail

Have you tried:

1. capturing the traffic to check if the health-check queries and responses are looking as expected
2. querying the recursors directly to see if they are indeed sending ServFail for the checked name
3. looked at the recursors' logs after enabling tracing (https://docs.powerdns.com/recursor/running.html#tracing-queries) for these queries, to see what is happening?

 us === START OF TRACE 2025-12-12T10:21:44.577 ===
154 peoplestackwebexperiments-pa.clients6.google.com: Wants NO DNSSEC processing, auth data required by query for HTTPS
173 peoplestackwebexperiments-pa.clients6.google.com: No cache hit for 'peoplestackwebexperiments-pa.clients6.google.com|HTTPS', trying to find an appropriate NS record
177 peoplestackwebexperiments-pa.clients6.google.com: Cache consultations done, have 1 NS to contact
181 peoplestackwebexperiments-pa.clients6.google.com: Domain has hardcoded nameserver
186 peoplestackwebexperiments-pa.clients6.google.com: Nameservers: +10.18.41.12:5454(322.88ms)
195 peoplestackwebexperiments-pa.clients6.google.com: Resolved '.' NS (empty) to: 10.18.41.12
198 peoplestackwebexperiments-pa.clients6.google.com: Trying IP 10.18.41.12:5454, asking 'peoplestackwebexperiments-pa.clients6.google.com|HTTPS'
427 peoplestackwebexperiments-pa.clients6.google.com: Got 2 answers from (empty) (10.18.41.12), rcode=0 (No Error), aa=0, in 0ms
438 peoplestackwebexperiments-pa.clients6.google.com: Accept answer 'google.com|SOA|ns1.google.com. dns-admin.google.com. 843371293 900 900 1800 60' from '.' nameservers? ttl=57, place=2 YES! - This answ  er was received from a server we forward to.
450 peoplestackwebexperiments-pa.clients6.google.com: OPT answer '.' from '.' nameservers
458 peoplestackwebexperiments-pa.clients6.google.com: Determining status after receiving this packet
460 peoplestackwebexperiments-pa.clients6.google.com: Got negative caching indication for 'peoplestackwebexperiments-pa.clients6.google.com|HTTPS'
462 peoplestackwebexperiments-pa.clients6.google.com: Validation state was Indeterminate, state update is Indeterminate, validation state is now Indeterminate
471 peoplestackwebexperiments-pa.clients6.google.com: Status=noerror, other types may exist, but we are done (have negative SOA)
473 peoplestackwebexperiments-pa.clients6.google.com: Validation state was Indeterminate, state update is Indeterminate, validation state is now Indeterminate
=== END OF TRACE 2025-12-12T10:21:44.578 ===
 us === START OF TRACE 2025-12-12T10:21:45.417 ===
221 android.clients.google.com: Wants NO DNSSEC processing, auth data required by query for HTTPS
241 android.clients.google.com: No cache hit for 'android.clients.google.com|HTTPS', trying to find an appropriate NS record
245 android.clients.google.com: Cache consultations done, have 1 NS to contact
248 android.clients.google.com: Domain has hardcoded nameserver
254 android.clients.google.com: Nameservers: +10.18.41.12:5454(158.53ms)
262 android.clients.google.com: Resolved '.' NS (empty) to: 10.18.41.12
265 android.clients.google.com: Trying IP 10.18.41.12:5454, asking 'android.clients.google.com|HTTPS'
195898 android.clients.google.com: Got 3 answers from (empty) (10.18.41.12), rcode=0 (No Error), aa=0, in 195ms
195914 android.clients.google.com: Removing irrelevant SOA record 'l.google.com|SOA|ns1.google.com. dns-admin.google.com. 843371293 900 900 1800 60' in the AUTHORITY section received from .
195928 android.clients.google.com: Accept answer 'android.clients.google.com|CNAME|android.l.google.com.' from '.' nameservers? ttl=299, place=1 YES! - This answer was received from a server we forward t  o.
195940 android.clients.google.com: OPT answer '.' from '.' nameservers
195950 android.clients.google.com: Determining status after receiving this packet
195954 android.clients.google.com: Status=got a CNAME referral, starting over with android.l.google.com
195960  android.l.google.com: Wants NO DNSSEC processing, auth data required by query for HTTPS
195974  android.l.google.com: No cache hit for 'android.l.google.com|HTTPS', trying to find an appropriate NS record
195978  android.l.google.com: Cache consultations done, have 1 NS to contact
195980  android.l.google.com: Domain has hardcoded nameserver
195985  android.l.google.com: Nameservers: +10.18.41.12:5454(170.52ms)
195993  android.l.google.com: Resolved '.' NS (empty) to: 10.18.41.12
195995  android.l.google.com: Trying IP 10.18.41.12:5454, asking 'android.l.google.com|HTTPS'
196199  android.l.google.com: Got 2 answers from (empty) (10.18.41.12), rcode=0 (No Error), aa=0, in 0ms
196211  android.l.google.com: Accept answer 'l.google.com|SOA|ns1.google.com. dns-admin.google.com. 843371293 900 900 1800 60' from '.' nameservers? ttl=19, place=2 YES! - This answer was received from a   server we forward to.
196216  android.l.google.com: OPT answer '.' from '.' nameservers
196224  android.l.google.com: Determining status after receiving this packet
196225  android.l.google.com: Got negative caching indication for 'android.l.google.com|HTTPS'
196227  android.l.google.com: Validation state was Indeterminate, state update is Indeterminate, validation state is now Indeterminate
196236  android.l.google.com: Status=noerror, other types may exist, but we are done (have negative SOA)
196237  android.l.google.com: Validation state was Indeterminate, state update is Indeterminate, validation state is now Indeterminate
196240 android.clients.google.com: Updating validation state for response to android.clients.google.com from Indeterminate with the state from the CNAME quest: Indeterminate
196241 android.clients.google.com: Validation state was Indeterminate, state update is Indeterminate, validation state is now Indeterminate
=== END OF TRACE 2025-12-12T10:21:45.613 ===

It is practically impossible to intercept traffic at the moment of the crash due to the random nature of the crash.

[rgacogne]

I don't understand what these traces are, they do not seem to match the names checked by DNSdist to determine the health of the backends? In any case if you are not taking these traces while dnsdist considers the corresponding backend down there is not much we can learn from it.