Hiding Services Behind Tailscale With Split DNS
The itch
I run a small VPS at Netcup. It serves a couple of public websites, and it also hosts three things that are for me only: a Nextcloud instance, a Gitea server and a little task tracker. The public sites need to stay public. The private three should be reachable from my laptop, my phone and my other machines, and from nobody else.
The lazy answer is “put them behind the VPN”. I already use Tailscale everywhere, so all my devices can talk to the server over its tailnet address. But I did not want to type 100.x.y.z:8443 into a browser, I wanted cloud.example.net to just work, with a proper certificate, from wherever I happen to be.
That creates a small puzzle. If public DNS points cloud.example.net at the server’s Tailscale address, the name is useless outside the tailnet, and Let’s Encrypt can no longer renew the certificate over HTTP. If public DNS points at the public IP, my devices connect over the open internet like everyone else, and I have to lock the thing down with passwords and fail2ban and hope.
What I wanted was both: the same hostname resolves to the public IP for the world and to the Tailscale IP for my own devices. That is exactly what split DNS is for, and it turned out to be about thirty lines of configuration. Here is how it fits together, plus the one evening I lost to a problem that had nothing to do with Tailscale at all.
The idea in one picture
My devices ask a small resolver on the server and get the Tailscale address. Everyone else asks public DNS and gets the public address, where the web server turns them away.
my laptop / phone the rest of the internet
| |
| cloud.example.net? | cloud.example.net?
v v
Tailscale split DNS -> dnsmasq on public DNS
the server (tailscale0 only) |
| |
| 100.111.22.33 | 203.0.113.10
v v
+------------------------------ VPS -------------------------------+
| Apache vhost cloud.example.net |
| Require ip 100.64.0.0/10 -> 200, Nextcloud |
| anything else -> 403 |
+------------------------------------------------------------------+
Step 1: a tiny dnsmasq that only talks to the tailnet
The server runs a second DNS resolver next to its normal one. It is dnsmasq, and it does two things: it answers a handful of names with the server’s Tailscale address, and it forwards everything else to a public resolver. The trick is that it listens only on the Tailscale interface, so nobody outside the tailnet can even reach it.
The whole configuration lives in one file, /etc/dnsmasq.d/tailnet-overrides.conf:
port=53
bind-dynamic
interface=tailscale0
except-interface=lo
no-resolv
server=1.1.1.1
server=9.9.9.9
address=/cloud.example.net/100.111.22.33
address=/cloud.example.net/fd7a:115c:a1e0::1234:5678
address=/git.example.net/100.111.22.33
address=/git.example.net/fd7a:115c:a1e0::1234:5678
address=/tasks.example.net/100.111.22.33
address=/tasks.example.net/fd7a:115c:a1e0::1234:5678
A few of those lines earn their keep in ways that are not obvious:
interface=tailscale0together withbind-dynamicmakes dnsmasq bind to the Tailscale addresses only, and cope with the interface appearing after dnsmasq has started. Withoutbind-dynamicit fails at boot becausetailscale0is not up yet.except-interface=lokeeps it off127.0.0.53, which systemd-resolved owns. This matters more than it looks, see the gotcha below.no-resolvplus explicitserver=lines means dnsmasq never reads/etc/resolv.conf. That rules out a loop where dnsmasq asks the system resolver, which asks Tailscale, which asks dnsmasq.- Both an A and an AAAA record per name. Modern clients ask for AAAA first, and if you only override the IPv4 record an IPv6-capable phone happily goes to the public address instead.
Every other example.net name, including the public websites, is simply forwarded upstream, so nothing else changes. I wrapped this in a small shell script that reads the current Tailscale IPs with tailscale ip, writes the file, runs dnsmasq --test and restarts the service. That way I can add a name by editing one array and re-running it.
Step 2: one rule in the Tailscale admin console
Now the tailnet has to know that this resolver exists. Tailscale calls this feature split DNS, and it takes about a minute to set up. In the admin console under DNS → Nameservers, add a custom nameserver with the server’s Tailscale IPv4 address, tick Restrict to domain, and enter example.net. Leave the global nameservers empty and keep MagicDNS on.
From then on every device in the tailnet that has “Use Tailscale DNS” enabled (the default) sends lookups for *.example.net to the little dnsmasq on the server and everything else to whatever resolver it normally uses. On my laptop it looks like this:
$ tailscale dns status
Split DNS Routes:
- example.net -> 100.111.22.33
$ resolvectl query cloud.example.net
cloud.example.net: 100.111.22.33 -- link: tailscale0
fd7a:115c:a1e0::1234:5678 -- link: tailscale0
Outside the tailnet the same name still resolves to the public IP, because public DNS was never touched. Let’s Encrypt is happy, the public sites are untouched, and my devices quietly take the private road.
Step 3: the web server is the actual gate
It is worth being honest about what the DNS trick does and does not do. It steers my devices to the Tailscale address. It does not stop anyone else from connecting to the public address, because the public IP is still in public DNS and Apache listens on it.
So the real access control lives in the web server. The virtual hosts for the three private services only accept requests that arrive over the Tailscale interface and return a plain 403 to everything else. In Apache that is a Require ip 100.64.0.0/10 fd7a:115c:a1e0::/48 block on the vhost, with the same idea available in nginx or Caddy. The certificate for the name is still issued and renewed over the public interface, because the ACME challenge path is explicitly allowed through.
The result is a nice division of labour. DNS makes the private road the default for my own devices without me thinking about it. The web server makes sure the private road is the only one that leads anywhere. Neither part has to be perfect for the other to hold.
One consequence I like: if I am on a network where Tailscale cannot connect, the name resolves to the public IP and I get a 403 instead of a login page. That is a clear signal that the tunnel is down, rather than a mysterious timeout.
The gotcha: the server must not drink its own water
There is one node in the tailnet that must not use the split DNS rule: the server itself. If it did, every example.net lookup it makes would go to Tailscale’s resolver, which would send it to dnsmasq, which runs on the same box. It works until it doesn’t, and when it doesn’t you get a resolver loop that is miserable to debug. So on the server, Tailscale DNS is switched off:
tailscale set --accept-dns=false
The server’s own lookups go through systemd-resolved, like any normal Debian box, and systemd-resolved has its own upstreams. Except that for three months, it didn’t, and I had no idea.
Here is what happened. Back in July I installed the systemd-resolved package while setting something else up. On Debian that quietly replaces /etc/resolv.conf with a symlink to the resolved stub. The old file had the provider’s nameservers in it. The new setup expects to learn nameservers from the network configuration, but the server’s interface is a static one in /etc/network/interfaces with no dns-nameservers line, and Debian builds resolved without any compiled-in fallback servers. Net result: a resolver that listens dutifully on 127.0.0.53 and has nobody to ask.
$ resolvectl query example.org
example.org: resolve call failed: No appropriate name servers or networks for name found
When I finally noticed that the server could not resolve anything, I did what you would do: I blamed the newest, cleverest thing on the machine, which was the Tailscale split DNS setup. I spent a good while reading tailscale dns status output before resolvectl status showed the actual problem, a Global section with no DNS servers at all. The fix is one drop-in file:
# /etc/systemd/resolved.conf.d/10-upstream.conf
[Resolve]
DNS=1.1.1.1 9.9.9.9 2606:4700:4700::1111 2620:fe::fe
FallbackDNS=8.8.8.8
followed by systemctl restart systemd-resolved. Two lessons from that evening. First, when DNS breaks, run resolvectl status before anything else; it tells you in ten lines who the system thinks it is allowed to ask. Second, the server resolving its own private names to the public address is fine and intended. It keeps the host’s DNS working even when Tailscale is down, and the host never needs to talk to those services through the tunnel anyway.
What you end up with
Three private services on a public server, reachable by their real names with real certificates from every device I own, invisible as a 403 to everyone else, and no extra client software beyond the Tailscale app that was already there. The moving parts are a 15-line dnsmasq config, one split DNS rule, and a Require ip block per vhost.
A few things to keep an eye on if you copy this:
- The Tailscale IP of the server is baked into the dnsmasq config and the admin console rule. If you ever remove and re-add the node, both need updating.
- Keep
accept-dnsoff on the server, and keepexcept-interface=loin dnsmasq. Those two lines are what stop the server from looping into itself. - Give systemd-resolved explicit upstreams on any Debian host with a static interface. It will not guess them for you.
- Add names to the override list and to the web server’s allow rule. One without the other is either a broken name or an open door.
And the checklist I run when something feels off, in the order that finds the fault fastest:
- On the server:
resolvectl statusshows DNS servers under Global, andgetent hosts example.orgreturns something. - On the server:
systemctl is-active dnsmasqandss -ulpn | grep :53show dnsmasq on the Tailscale address only. - On a tailnet device:
tailscale dns statuslists theexample.netsplit route. - On a tailnet device:
resolvectl query cloud.example.netreturns the Tailscale address viatailscale0. - From outside: the name resolves to the public IP and the site answers 403.
If step 1 fails, stop blaming Tailscale. I say this with some feeling.