Now that Starlink is working reliably I want to bridge the network between my two houses to make it easier to share stuff across. I have an elaborate setup of ssh tunnels and autossh for NAT busting and it’s awful. A VPN would be better. Some notes on getting there, this is very WireGuard 101 stuff.
The SF network is at 192.168.0.0/24 and is on fast fiber. I have a NAT router but I completely control it so with port forwarding it’s easy to set up a server. The GV network is at 192.168.3.0/24 and is behind double NAT; my own and also Starlink’s carrier grade NAT. The latter prevents listening on sockets pretty much entirely, so running a server behind it is awful and even elaborate NAT traversal seems dicey.
At first I thought I wanted a full bridging setup so the two houses appear in the same place. But in retrospect I think that is a mistake. It’d be bad for Sonos, for instance, it seems to use LAN broadcast to discover its own stuff and I’d much rather it found the local server for music files than tried to use the VPN. Same goes for Plex video too. OTOH I would like some discovery-based things to work; Windows filesharing for sure. I don’t really understand what I need.
To start simple I decided just to create a basic tunnel on a new subnet (192.168.7.0/24) between Linux boxes in the two houses. Doesn’t really solve my problem but those tunnels could later be gateways for house traffic. I am ignoring my Ubiquiti routers entirely. Folks have gotten various VPN solutions working on that, including Wireguard, but I’m concerned at the longevity of any hackery solutions for doing that. Maybe later.
Anyway setting up a simple tunnel between two Linux boxes is easy. I used WireGuard for this, apparently the modern VPN solution. It really is a breeze both to install and to understand. I mostly used this guide with some consulting of this other guide. Note WireGuard itself doesn’t really have good docs. For instance I cannot find anywhere a man page style reference for the conf files that wg-quick uses, never could find a definition of exactly what “Address” or “Endpoint” means. It’s sort of clear from context and examples but…
Anyway following the guides it all basically just worked. Very simple too. Wireguard’s model is peer to peer; peers are identified by a public/private key pair (and an internal ID). Tunnels between peers then get created as needed and each end of the tunnel is given an IP address. The guides are written in terms of a client/server setup so I’ll use that language from now on.
The main gotcha I had is when I first set up the server in 192.168.0.0/24 I gave the Wireguard tunnel itself an IP address in the same subnet, 192.168.0.0/24. That’s definitely a bad idea; my Linux box decided to use that new route instead of the ethernet and I had to reboot it. That’s why I put the Wireguard tunnel in a new subnet (192.168.7.0/24) instead. The other gotcha is I set SaveConfig=true. It’s a neat idea; you configure the thing on the fly with the command line and then the config file is updated. But at the same time I was editing the config file to fix things and restarting WireGuard and my changes kept getting clobbered. Pick one or the other style of creating a config, don’t do both.
Anyway, now I have a working tunnel so my two Linux boxes can talk to each other. The next step is using this tunnel to do “other things” like mount Windows fileshares, etc. Still not sure how I’ll go about that.
I’m going to keep updating this post as I hack more. I don’t feel like having the discipline to mark every update, so just expect stuff to be added over time.
I realize I don’t really know what I want from this VPN setup. I don’t want full bridging; I want some things like media servers to not be discovered over the WAN link. I do want a lot of transparency though. I’m going to start now by making sure every device in SF (192.168.0.0/24) can talk to every device in GV (192.168.3.0/24). I’ll worry about broadcasts and discovery later, if at all.
Several folks have asked “why not Tailscale”? That is probably an excellent answer; Tailscale is a nice VPN product built on top of WireGuard. I was under the impression it was client oriented, not network oriented, so you had to install Tailscale on every device. But that’s not really true; they have a “subnet router” capability that works for other devices. At this point I’m mostly not using it because I got WireGuard itself running OK and I want to tinker more.
Starting to think I should run Wireguard on my Ubiquiti boxes afterall. One of my routers is a Ubiquit Dream Machine which is sort of different software: here’s a port of Wireguard to Dream Machine.
Netcat for testing
I want to be able to test if packets get through but without the full round trip ICMP ping requires. That’s what netcat is for. On the server side;
nc -klu 3333. On the client side, testing sending a packet:
echo foo | nc -u 192.168.3.75 3333. Note the client netcat doesn’t hang up after sending the packet, it keeps the socket open looking for input.
Routing the subnet
The basic wireguard setup gets a tunnel going. You can ping the tunnel endpoint IP address (192.168.7.3). But not the “real” IP address of the machine at the other end (192.168.3.75). To add that, we need a route. Doing this manually is a mistake (wg-quick does this…) but it was the first thing I tried and half works, so I mention it here:
route add -net 192.168.3.0/24 gw 192.168.7.3 wg0
Doing that is enough to let UDP packets through. nc -u 192.168.3.75 3333 works! However traceroute (and presumably ping) fails with the mysterious message
send: Required key not available. Turns out that’s an error message from Wireguard and in my case is caused because 192.168.3.0/24 is not one of the “Allowed IPs” in the peer definition for the Wireguard tunnel. Wireguard is blocking the traffic.
The solution is to add the subnet to AllowedIPs.
AllowedIPs = 192.168.7.0/24, 192.168.3.0/24. Add that and restart and pings get through and get replies. Moreover there’s no need to manually add the route because wg-quick does it for you. “It infers all routes from the list of peers’ allowed IPs, and automatically adds them to the system routing table”. Nice! Long story short, to get a single machine on two different subnets talking to each other via Wireguard it’s sufficient to add each’s subnets to the other’s AllowedIPs.
Next step: get the rest of the LAN on each side to know to route traffic for the other subnet through Wireguard. I know this means adding a static route (probably in my router). I’m less clear what, if anything, I have to do to configure the Linux box to act as a router that forwards the packets for that destination. When searching for info on this I found a lot of advice about using IPTables masquerading for a NAT setup. Yuck, no!
Update: turns out basic routing just works already with no extra config. It may help that my Linux box which is the Wireguard endpoint already has IP Forwarding turned on. Anyway on my Windows box I just did
route add 192.168.3.0 mask 255.255.255.0 192.168.0.65 and now my Windows machine can talk to the other end of the Wireguard endpoint on its “real” address 192.168.3.75. Can’t talk to anything else on the remote network though, but that’s because they don’t know a route back yet.
Of course I don’t want to manually have to add this route to everything! A better solution is not to set the route on my Windows desktop but instead add a static route to my Ubiquiti Dream Machine. It’s hidden under Settings / Advanced Features / Advanced Gateway Settings / Static Route. The Destination Network is 192.168.3.0/24, the next hop is 192.168.0.65, where the WireGuard endpoint is.
C:\Users\Nelson>tracert -d 192.168.3.75 Tracing route to 192.168.3.75 over a maximum of 30 hops 1 <1 ms <1 ms <1 ms 192.168.0.1 2 <1 ms <1 ms <1 ms 192.168.0.65 3 60 ms 59 ms 59 ms 192.168.3.75
That’s it! Now everything on my 126.96.36.199/24 LAN can initiate connections to 192.168.3.75. Nothing else in 192.168.3.0/24 though because I haven’t added the route on that side yet. The big thing is my Windows box can now access the remote fileshare at
\\192.168.3.75\. I doubt it will show up as discoverable but a direct connection works.
This is all working well enough now I’m close to needing to document the final setup and making it persistent. It may actually be persistent; I just did the
systemctl enable command that gets wireguard to run at startup.