Skip to content

pbr.user.tunnel-dns: resolve policy domains through the policy's interface - #185

Open
yitzhaq wants to merge 4 commits into
mossdef-org:1.2.4from
yitzhaq:tunnel-dns
Open

yitzhaq wants to merge 4 commits into
mossdef-org:1.2.4from
yitzhaq:tunnel-dns

Conversation

@yitzhaq

@yitzhaq yitzhaq commented Oct 2, 2026 •

Copy link
Copy Markdown

Problem. Geo-located CDNs (BBC, SVT, DR, Netflix and similar) answer according to where the resolver is. dnsmasq adds whatever it receives to the policy's nftset. So a domain routed out of a UK tunnel can still get addresses chosen for the router's home country, because its lookup went out the default way. One workaround could be to pin addresses by hand, or to add server= lines that then have to be kept in sync with the policies separately.

This PR adds a user file with a small map of interface → resolver reachable through that interface, for example a VPN provider's in-tunnel DNS. For every enabled policy on a mapped interface, it appends the following line to the dnsmasq nftset file:

server=/<domain>/<resolver>@<interface>
  • Resolution follows the policy. dnsmasq pins the upstream query to the interface, so the lookup takes the same path as the traffic. The lines are regenerated on every start/reload, so moving a domain to another policy, or pointing a policy at another interface, moves its resolution with it.
  • The pin survives output policies. dnsmasq pins with IP_UNICAST_IF, which only steers the first route lookup. If an output policy marks the query (for example a catch-all that sends the router's own traffic through a tunnel), the mark change in mangle_output makes the kernel reroute it by routing table and the pin is lost. The file therefore also inserts one rule at the head of pbr_output that returns, unmarked, the queries to each mapped resolver that are already routed out of their mapped interface (separate IPv4/IPv6 rules; addr#port resolvers use that port).
  • Fails closed. If the interface is down, lookups for its domains fail instead of leaking out another way.
  • Scoped. Domains on unmapped interfaces, and all non-policy names, resolve exactly as before. The map is empty by default, so installing the file changes nothing.
  • Any interface type. The map is explicit, so it works for WireGuard, OpenVPN, OpenConnect or anything else. Deriving the resolver from the interface's own dns option was considered and rejected:
    • tunnels started by their own daemons (proto none) do not expose their pushed DNS to netifd;
    • an interface's dns option may be set locally to point at LAN resolvers, which would create a loop.
  • Warning. It logs a warning when /etc/config/dhcp has its own server= entry for a policy domain, because dnsmasq would use both servers and the answers would mix.

It relies on the include running after the policies are processed and before pbr's dnsmasq hash comparison. That is the current order in start_service, so a change triggers the existing dnsmasq restart.

Tested on OpenWrt 24.10 with pbr 1.2.3-r71, using several WireGuard tunnels from two providers whose in-tunnel resolvers use the same address on every tunnel:

  • Each policy's domains resolve via its own tunnel; verified on the wire with tcpdump, and through whoami.akamai.net per tunnel.
  • The same resolver address on several interfaces is kept apart correctly.
  • A domain on a down interface times out without falling back.
  • The generated lines pass dnsmasq --test, and the generated nft rules pass nft -c.
  • With a router catch-all on 0.0.0.0/0, queries left via the catch-all's tunnel until the pbr_output rule was added; with it, each mapped resolver's queries leave via their own tunnel again (tcpdump).

This is offered as a user file because that is the smallest change. If you would rather have it built in, for example as a per-interface or per-policy resolver option emitted next to the nftset= lines, I'm happy to rework it that way.

🤖 Generated with Claude Code

@yitzhaq

yitzhaq commented Oct 5, 2026

Copy link
Copy Markdown
Author

A note on how this relates to DNS Policies (config dns_policy), since the two look alike at first glance.

A DNS policy is keyed by the client: it redirects every DNS query from src_addr to dest_dns, past dnsmasq. As the README says, that also means domain policies stop working for those clients, because their lookups never reach dnsmasq and the sets never fill.

This include is keyed by the policy's domains: dnsmasq forwards only those lookups to the resolver of the interface the policy routes them through, pinned to that interface. Every other lookup keeps the normal upstreams. The answers still pass through dnsmasq, so the sets fill as before, and they now match where the traffic actually goes (a geo-routed CDN answers for the tunnel's exit, not for the WAN).

So they complement each other: one picks the resolver a device uses for everything, the other makes a routed domain resolve where its traffic leaves.

I deliberately didn't derive the resolver from the interface, as dest_dns can. Some tunnel daemons run as proto none, so netifd never learns the DNS they push, and an interface's dns option may be set locally to point at LAN resolvers, which would loop. Hence the explicit interface → resolver map.

@egc112

egc112 commented Oct 18, 2026

Copy link
Copy Markdown
Collaborator

Thanks for this, and for the clear write-up, especially the comparison with dns_policy and the reasoning for an explicit map. Resolving a routed domain through the tunnel its traffic uses is a real gap for people sending geo-located services through a VPN, and as an opt-in user file it costs everyone else nothing. I'd like to take it, but three things need to be solved first.

1. Validate the map entries. The resolver string reaches two places that each read their whole configuration in one go, so one typo takes down far more than the feature. Checked on OpenWrt 25.12:

map entry pbr_output rule (nft -c) server= line (dnsmasq --test)
10.0.0.1 accepted OK
10.0.0.300 (typo) rejected, so pbr's ruleset fails to install bad command line options, so dnsmasq fails to start
a hostname rejected depends on whether it resolves

A user file's nft lines are installed as part of pbr's own ruleset, so the first case stops pbr starting, and the second leaves the LAN without DNS. Please accept only an IPv4/IPv6 address with an optional #port, and skip anything else with a logged warning, so a mistake only costs that interface's pinning.

2. Use the interface's device, not its name. The map key matches the policy's interface (the pbr/UCI name), but dnsmasq's @… and nft's oifname both need the Linux device. They're the same for WireGuard, but not in general: on the test router wan is eth0 and lan is br-lan, and an OpenVPN interface (proto none) would give @vpnclient instead of @tun0. dnsmasq accepts that without complaint, and the pin just never matches. Please look up each mapped interface's l3_device (ubus call network.interface.<name> status) for both places, and skip (with a warning) an interface that has none at start. That also makes the "any interface type" claim true.

3. Document it. Please add a short section to the 1.2.4 README in docs.mossdef.org: what the map is for, how it differs from dns_policy, that the domains fail closed when the tunnel is down, and the /etc/config/dhcp server= caveat. Please also add a comment in the file that pins its ordering assumption (it runs after the policies are processed and before pbr compares the dnsmasq file), since it appends to /var/run/pbr.dnsmasq directly and would break silently if that order changed.

With those three in, I'm happy to merge. Your offer to build it in as a per-interface option is appreciated, but a user file is the right size for this.

yitzhaq and others added 3 commits November 6, 2026 01:08
…rface

Geo-located CDNs answer according to where the resolver is, and dnsmasq
adds whatever it receives to the policy's nftset. A domain routed via a
tunnel can therefore get addresses that suit a different location,
because the lookup itself went out another way.

This user file maps interfaces to a resolver that is reached through
them, such as a VPN provider's in-tunnel DNS. For every enabled policy
on a mapped interface it adds server=/<domain>/<resolver>@<interface>
to the dnsmasq nftset file. dnsmasq binds the query to that interface,
so resolution follows the policy when domains move between policies,
and fails rather than leaks when the interface is down. Domains on
unmapped interfaces are not affected. The map is empty by default.

A warning is logged when /etc/config/dhcp has its own server= entry for
a policy domain, since dnsmasq would use both.

Signed-off-by: yitzhaq <17812841+yitzhaq@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dnsmasq pins a server=...@<interface> query with IP_UNICAST_IF, which
only steers the first route lookup. If an output policy then marks the
query, for example a catch-all that sends the router's own traffic
through a tunnel, the mark change in mangle_output makes the kernel
reroute it by routing table, and the pin is lost: the lookup leaves by
the catch-all's interface (or wherever main routes the resolver), so
geo-located answers are wrong again.

Insert a rule at the head of pbr_output that returns, unmarked, the
queries to each mapped resolver that are already routed out of their
mapped interface. They then leave exactly where dnsmasq pinned them.
IPv4 and IPv6 resolvers get their own rule, and a resolver given as
addr#port uses that port.

Signed-off-by: yitzhaq <17812841+yitzhaq@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A resolver entry reaches both pbr's ruleset and the dnsmasq file, and
each is read as a whole: a malformed address made nft reject the whole
ruleset, or stopped dnsmasq from starting. Accept only an IPv4/IPv6
address with an optional #port, and skip anything else with a warning,
so a mistake costs only that interface's pinning.

The map is keyed by the interface name used in policies, but dnsmasq's
@ pin and nft's oifname both need the device. They differ for anything
but WireGuard (a proto none OpenVPN interface would give @VPNclient
instead of @tun0, which dnsmasq accepts and then never matches). Look
up each interface's l3_device from netifd, and skip, with a warning,
an interface that has none when the service runs.

Also state the ordering the file relies on: it runs after the policies
are processed and before pbr compares the dnsmasq file.

The uci test mock now loads a package on foreach, as libuci does, so a
user file can be tested with its own cursor.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@yitzhaq

yitzhaq commented Nov 6, 2026

Copy link
Copy Markdown
Author

Thanks for the careful review. All three are in, plus a rebase onto current 1.2.4.

1. Map entries are validated. A value must be an IPv4/IPv6 address (checked with iptoarr()) with an optional #port in 1–65535. Anything else is skipped with a warning in the system log (tunnel-dns: <iface>: '<value>' is not an address[#port]; skipped), and that interface's domains resolve as before. 10.0.0.300, a hostname, 010.0.0.1, a scoped fe80::1%eth0 and a stray space are all rejected. The address is written back in canonical form (arrtoip()), so what reaches nft and dnsmasq is exactly what was checked.

2. The device, not the name. Each mapped interface's l3_device comes from network.interface.<name> status. Both the @ pin and the oifname in the pbr_output exemption use it. An interface with no device at that point is skipped with a warning (tunnel-dns: <iface>: no device (interface down or unknown); skipped). The header comment now says the map is keyed by the pbr/UCI name and the device is looked up when the service runs.

3. Docs and ordering. The docs are in mossdef-org/docs.mossdef.org#44: a bullet in the list of provided files, and a section covering what the map is for, how it differs from dns_policy, failing closed, what happens to a bad entry or a missing device, and the /etc/config/dhcp server= caveat. One thing the section states explicitly: an interface that was down when the service ran is only pinned on the next start/reload, since an interface update doesn't run custom user files. The file now has a comment pinning the order it relies on: after the policies are processed (the dnsmasq file holds this run's nftset= lines) and before the hash compare (so a change to the appended lines restarts dnsmasq).

Tests. New tests/06_user_files/05_tunnel_dns (16 checks) loads the shipped file with a filled-in map against the existing mocks, where vpn is tun0, as in your example:

  • @tun0 and "tun0" are used, never the UCI name;
  • an IPv6 resolver gets its own ip6 rule;
  • a non-default port appears as #5353 and in the nft concatenation;
  • a hostname, a bad address, a bad port and an interface without a device are each skipped with their warning;
  • first policy wins, disabled policies and IP literals are ignored, and the dhcp server= warning fires.

Against the previous version of the file, 10 of the 16 fail.

To test it, I had to make the uci mock's foreach() load a package on first use, as libuci does (uci.c, uc_uci_foreach). Before that, a user file with its own cursor saw no dhcp sections. Nothing else in the suite depended on that. Suite 76/76 and check-ucode-syntax.sh clean, both with ucode at the CI revision.

Live. I dry-ran the new file on my router (OpenWrt 24.10), against the real uci and ubus and with my seven WireGuard entries:

  • the generated server= lines are identical to what the installed version writes;
  • the pbr_output rule passes nft -c;
  • the lines pass dnsmasq --test.

One gap I'd offer to close, if you want it: a domain policy with interface 'tor' still resolves its domains through the normal upstreams, because pbr only redirects DNS addressed to the policy's destination addresses. So the lookup leaves outside Tor, and the answer is chosen for the WAN rather than the Tor exit. This file could take tor: '127.0.0.1#9053' in the map: the same validation, but no device lookup, no @ pin and no pbr_output exemption, since the resolver is on the router. I run that locally. I'm happy to add it here, send it as a follow-up, or leave it out. Your call.

@egc112

egc112 commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator

Thanks, this is a thorough update, and the extra tests are very welcome. I tested 60b84b3 on OpenWrt 25.12 (ucode 85922056) with a real WireGuard tunnel, and all three points are solved:

  • Validation: a bad address, a hostname and an unknown interface were each skipped with their warning, and pbr and dnsmasq kept running.
  • Device: wan became @eth0 and "eth0" in the rule, so a name that differs from its device works. The rule loads, and so does the ip6 version with an IPv6 resolver.
  • Docs: I confirmed what the section says on the router. With the tunnel down the pinned domain gets REFUSED while other lookups keep working, also after a dnsmasq restart. An interface that was down is pinned again on the next reload, and ifup alone does not do it.

tcpdump also shows the queries leaving through the tunnel from the tunnel's address, with nothing on the WAN device.

One thing left before merging. The file writes s.name unfiltered after the #, so a newline in a policy name becomes a real line in the dnsmasq file. With the name set to t185wg, a newline, then server=/injected.test/9.9.9.9, that server= line ended up as a live directive. pbr's own nftset= line stayed clean with the same name. It takes root to set such a name, but a junk line there could stop dnsmasq, which is the failure your validation is there to prevent. Please strip the newline or drop the comment, whichever you prefer, and add a check to 05_tunnel_dns.

Two optional points, neither of which holds up the merge. I found both by reading the code and did not test them:

Tor: thanks for the offer, but I'd like to leave it out and keep this file to its one job.

With the newline fixed, I'm happy to merge this together with the docs PR. We are busy getting 1.2.4 into OpenWrt at the moment, so it may take some time before I get to that.

…domain

A policy's name goes into the comment after each server= line. A newline
in it started a line of its own, so a name could add a live dnsmasq
directive, or a junk line that stops dnsmasq. Replace control characters
in the name with a space.

The /etc/config/dhcp warning only looked at the first domain of an entry
naming several, such as server=/a.example/b.example/1.1.1.1. Check them
all.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@yitzhaq

yitzhaq commented Nov 6, 2026

Copy link
Copy Markdown
Author

Thanks for testing it this thoroughly, including the injection.

  • Policy names: control characters in the name are now replaced with a space before it goes into the comment, so your example ends up as … # t185wg server=/injected.test/9.9.9.9 on one line. I kept the comment, since it's what tells you which policy a line came from. 05_tunnel_dns now has a policy named Inject + newline + server=/injected.test/9.9.9.9, and checks that its line stays whole and every line in the file starts with server=/.
  • The dhcp warning now checks every domain of an entry, so server=/a.example/b.example/1.1.1.1 warns for both. The test's dhcp entry now has the policy domain in second place.
  • src_addr: one bullet added to pbr 1.2.4 README: document pbr.user.tunnel-dns.uc docs.mossdef.org#44: dnsmasq can't tell clients apart, so the lookups are pinned for every device and only the routing follows src_addr.

Against the previous commit, the 3 new checks fail; with this one, 18/18 pass. Suite 76/76 and check-ucode-syntax.sh clean with ucode 85922056.

Understood on Tor, and no rush on the merge. Good luck with 1.2.4 in OpenWrt.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants