I liked the UniFi Security Gateway experience in Grass Valley so much I decided to replace my EdgeMAX with a UniFi router in San Francisco, too. I also wanted to add another WiFi access point and it’s nice to have a dedicated controller so I bought the home consumer UniFi Dream Machine, aka the UDM.
First impression: it all works out of the box! This is the first Ubiquiti device I’d be comfortable giving a random ordinary person and have them set it up. It was a bit daunting at first; there’s literally no instructions in the box, not even a single page “getting started”. That is dumb. But if you plug the device in and plug your laptop into one of the switch ports and open a browser window, it will grab you and direct you to setup (via a captive portal). The setup is very simple, too, setting up the basic home router folks need.
Too simple though; I really wanted to configure it before turning it on. No such luck. It demands an Internet connection, a remote access login(*), etc to even set up. I gave in and disconnected the old router and just lived with some downtime until I got the new one set up.
Really there wasn’t much to configure. I have my usual few tweaks: a static IP address for one host, a couple of forwarded ports. Fewer and fewer over time though and it was quite easy to get everything shifted over.
Except once again.. Ubiquiti fails to deliver local DNS. The various hacks for adding it to a USG all don’t seem to work either. I dunno, in Grass Valley the local DNS server at least started returning client names it had learned via DHCP after a few hours. I’ll give this time; I’ve seen some evidence it might be working.
More broadly the Dream Machine is lacking a bunch of features. Some of these are pretty fiddly / psecific things but a lot are fairly basic. It’s not clear if the OS can’t do it or if it’s just the UI doesn’t allow you to configure it yet. And the Dream Machine seems to lack a single config.gateway.json for advanced users to edit, the old back door way to enabling some features in UniFi routers. It’s a significant step back in flexibility. So far I’ve only run into one case where I’d want it, but…
(*) I’ve conceded to UniFi’s remote login. The selling point for me was installing the controller software on my mobile phone and being able to remotely look at my network even when not on site. Super handy. What’s weird about this UDM is it doesn’t even seem to support local login; I think it’s cached my cloud login credentials locally for when the network is down but I’m not sure. You can still log in as root too but that doesn’t even use the normal UniFi credentials, instead the login is username root and your cloud login password for the local password. Spooky.
We’ll see how the device is doing after a couple days’ use. I do like UniFi’s platform in general. Also impressed with the WiFi hardware here, a 4×4 antenna setup is nothing to sneeze at. It’s so good I may never bother setting up my old access point as a secondary in San Francisco.
Also I have an advanced project, creating a VPN tunnel between my two houses. Been wanting to do that for years, looks like Ubiquiti has a shortcut method for that. You can also set up a VPN server which would be a big help for me when travelling; I need to look into exactly what that gets you.
I just spent 30 minutes trying to figure out how to figure out where I was on a QGIS map. Specifically I wanted to be able to click on the map and copy the coordinates. The way to do this has changed in various QGIS versions so old docs are wrong. QGIS 3.14 (the new hotness) just changed it again.
Bottom line: install the QGIS Coordinate Capture plugin from the plugin repo. It’s at version 0.2 right now. Then select the hand tool (the pan tool) and right click on the map. A little option to copy the coordinate will pop up along with a chooser for which CRS you want. Note you have to be in the hand tool: the Identify Features tool doesn’t have it.
The confusion at the moment is because the Coordinate Capture function was removed in 3.14. The discussion looks well meaning, someone refactoring functionality and saying “this should be reimplemented in a nicer way”. Only instead of first shipping the nicer way they just removed the old way and then called it a day. The worst kind of open source fuckery.
That change means the method in the 3.10 docs doesn’t work; there’s no longer a Coordinate Capture panel. Nor is there a toolbar as some docs reference. Nor does it show up in “Derived” under Identify Features as some QGIS2 docs suggest. Apparently this really fundamental feature has moved around a lot over the years. This plugin seems like a temporary hack, I suspect the method will change again soon.
I’ve been wanting to make some print maps in QGIS using OSM data for roads. There’s a bunch of ways to do this from simple to complex, but I think I found a way that works for me.
Start with an OSM extract in .osm.pbf format. These are produced daily for countries and regions and are manageable size.
Import it as a layer into QGIS, which has native support for .osm.pbf format. However it will be slow because it is not indexed.
Convert to GeoPackage in QGIS, with the defaults. All the property names, etc will be carried over and you’ll get a spatial index.
Use the new GeoPackage layer in your maps, you can delete the old OSM PBF layer entirely.
The main drawback is you’re dealing with OSM schema. Ie, drawing roads involves deciphering all the possible tag values for “highway”. It’d be nice to use some processed dataset converted to a simpler schema but I don’t know of an easy way to get a free one.
If you don’t care about licensing, MapTiler has some very nice OSM Planet tiles you can easily import in new versions of QGIS using their MapTiler QGIS Plugin. You can import these as raster maps or vectors. The vectors come with some schema translation and styling so you start with an OSM-looking map right there in QGIS and then modify it as you wish. The downside is I’m not sure you can use this data for, say, a print project without a licensing agreement from MapTiler. Also I didn’t see any easy way to control the level of detail so if you want to override their idea of feature selection at a zoom level you may be out of luck.
I finally have a home network cellular backup option; I bought a UniFi LTE. This integrates with the rest of a Ubiquiti home network via their UniFi platform to give seamless failover to an AT&T cell tower when your main ISP is down. Price is $200, $15/mo, $10/GB. Right now there’s a sale where it’s $5/mo and $5/GB until May 2021; kinda hoping they keep it that way, we’ll see.
I’ll get the bad out of the way first. The LTE device is AT&T only, with a hardcoded SIM. You cannot use any other network provider and you must buy your contract via Ubiquiti on AT&T. Their network is high quality, but if you’re using a lot of LTE data, AT&T is expensive. My intention is to only use this as failover, hopefully never more than a few GB a month, so I’m willing to accept this limit. It’s a dumb limit though. If you want more flexibility there are a variety of LTE modems sold as hotspots. Also a bunch of routers have LTE failover built in, particularly things meant for installation in trucks and RVs.
The other bad is the UniFi LTE is definitely a Ubiquiti proprietary product. It will only work if you have one of their USG or Dream Machine routers. Many routers (including those two) support a more generic WAN failover, where you have a second WAN port and load balancing or failover rules to use it. The UniFi LTE is not that. In fact you don’t plug it into your router at all, it can go anywhere on your LAN and looks like a local ethernet device. (The docs say the router “makes a tunnel” to the LTE when it needs to route that way.)
One final bad: the numbskulls at Ubiquiti failed to include a power adapter. $200 and there was no way to turn the thing on. The docs are cryptic saying “use a UniFi switch to power it”. Digging deeper and I see the UniFi LTE is an 802.3at device; most UniFi switches that supply POE are 802.3af, a lower power standard. What’s weird is the LTE device only needs 8.5W, well within the capabilities of 802.3af, and some reports are that the device works on 802.3af. I didn’t chance it. (It does have the ability to pass another 8.5W through to a second POE port, that’d require more power than 802.3af requires, but it’s optional.)
Anyway this $110 8 port switch is the cheapest thing I could find that Ubiquiti sells that provides 802.3at. I was stubborn and didn’t want to buy a switch, particularly since I’m not sure where I’m going to locate the LTE modem, so I bought a $20 Shenzen special 802.3at injector instead. It seems to work.
The reason I put up with the bad is I wanted a no nonsense device with no confusion, something that would Just Work. And so far so good. I plugged it in and had it up and running within 5 minutes, including buying the contract service. The only fiddling I had to do is I was required to enable “remote management” on my USG; that’s some scary thing where your router state is accessible via the Internet with a password. I don’t love that idea but I relented.
I’m grateful for this simplicity. Doing it myself with an LTE modem would require researching a suitable modem, picking a carrier, picking a data plan for that carrier. Then getting it all working first on a laptop (for testing). And only then, plugging it in to the WAN port on my router and seeing if the failover actually worked. Probably in a way that violated the data contract I signed up for. Yuck. I’m happy I just bought my way out of all that effort.
Another bit of good; judging by the device’s shape and performance it has a pretty serious antenna in it. Ubiquiti got their start making excellent WiFi antennas and I’m hoping that extends to LTE. There’s also a coax connector for an external antenna but I think I’m unlikely to do better unless I put up a mast.
Like I said, I got it running with no stress at all. It seems to work.
There’s very little configuration on the device. About your only choice is the max data cap (3 GB / month by default) so you don’t get hit with an ugly bill if something goes wrong. I think I saw some reference to being able to set policy so some devices don’t get to use the cellular backup, probably involving a VLAN. Need to look into that.
I simulated a failure by unplugging my normal ISP (the simplest possible failure; detecting a flaky ISP is harder). The failover happened in under 15 seconds. The LTE connection was OK for this area: 14/1 Mbps and somewhat reliable and with lower latency than expected. (My experiments with a phone as hotspot are all over the place but often much worse). It switched back to the ISP when I plugged it back in, but took a minute or two. I did get email alerts about the outage but there was no obvious view on the Controller dashboard about the failover (there is one hidden status message). Anyway, it basically worked.
I’ve also had two unplanned failovers, both lasting less than a minute. I was away from the computer at the time so I didn’t notice. My ISP sometimes has hiccups where all my sessions are terminated and it’s out for a few seconds, I wonder if that triggered the failover? I don’t know that I mind exactly, but it’s something to keep an eye on.
I’ve read some complaints that networks were failing over to LTE, maybe on false pretenses, and then never recovering back to the preferred ISP. I don’t think I’ve seen that behavior yet. I did get a failure where I unplugged the LTE while I’d failed over to it. I then plugged my regular ISP back in expecting the network to go back to it. But it never did. I finally powered the router off and on and after 5 minutes (!) my network came back.
More generally I just wish the LTE failover state were much more prominent in the dashboard. Also that there be a big toggle switch I could hit manually to swap WANs.
What I really want is a tool to help me better situate the LTE antenna. I can’t find one; I’ve asked here. There are some diagnostics, including enough signal strength readings that I can probably experiment with different placements but there’s no live tool guiding me which direction to point in or anything. I think their long distance WiFi link utilities do have that.
I’m astonished at that “Good signal strength” measurement; we have terrible reception here at my house. Phones get 0 or 1 bars. All I did was put the antenna in a window pointed roughly towards where I think a tower is.
What do all those numbers mean? This article was helpful. The key number there is the RSRQ or “Reference Signal Received Quality”; it’s sort of like SNR but for multi-channel connections. That -11dB qualifies for “good”. The RSRP number is also useful, it’s the raw signal strength for a single channel and -115 is not so good. Band B12 is a 700MHz frequency.
Annoyingly, I can’t find a time series graph of the history of these readings. The Performance graph shows dropped packets, etc so that may be helpful but there’s no simple graph of RSRQ.
As for where the towers are, this article has lots of useful advice. Looking on CellMapper I think I found out which cell tower I’m connected to; there’s a A3E49 on the map north of me and the cell ID (blanked above) I have is 0A3E4911. The 11 stands for the cell to the south so that makes sense too. That tower is further away from me than I’d like but is way up high on a hill. Most of the other AT&T towers are lower, along the highway, which seems dumb. I need to do further experimentation with my phone as a signal tester.
I tried various locations in my house for the antenna. Some not very reliable data:
Office window: 0A3E4911, B12 RSRQ -12, RSRP -112 to -115
Office closet: 0A3E4911, B12, RSRQ -9 to -13, RSRP -114 to -125
Garage pointing north: 0A3E490A, B2, RSRQ -13 to -14, RSRP -119 to -120.
Garage point north after use: 0A3E4911, B12, RSRQ -10
Interesting the garage location uses a different cell (same tower) and band. The B12 bands are in the 700MHz spectrum, B2 is in 1900. I think that means the B2 option could be higher bandwidth. However it only stayed connected on B2 until I tried to use it, at which point it switched over to the other cell on B12. I don’t have a clear line of sight (including a metal garage door, among other things) so I suspect 700MHz is the better choice.
I’m pretty happy with the UniFi LTE, particularly at the current $5/GB sale price. It worked right out of the box once I found a power adapter for it. Now I need to run it for a few months and see what happens in practice.
Configuring failover behavior
Some extra notes written after the main piece.
After 48 hours I’m annoyed that the LTE is being failed over to for 5-15 seconds at a time. It’s happened maybe 8 times. These all might be legitimate primary failures, my ISP is flaky, but then it seems disruptive to fully change the routes since that terminates all active TCP connections. I’d rather the thing not respond to failures so quickly.
It tests whether it can ping out via that interface to ping.ubnt.com. So it will detect a failure upstream. It takes a little longer to failover to not be overly-touchy, it’s immediate when you lose link, 3 failed monitoring checks when link is still present (which is, at most, 40 seconds from failure to failover).
Create an upstream failure, and watch “show load-balance watchdog” via SSH to USG. After 10 seconds you should see “run fails” increment to 1/3, then later 2/3, later 3/3 and it’ll fail over at 3/3.
I wonder if that’s still true (this was late 2019). It’s a reasonable heuristic! And that show command shows “run fails: 0/3” which suggests 3 is still the threshold.
Group wan_failover eth0 status: Running pings: 3160 fails: 1 run fails: 0/3 route drops: 4 ping gateway: ping.ubnt.com – REACHABLE last route drop : Sat Sep 19 22:52:34 2020 last route recover: Sat Sep 19 22:53:07 2020
This gives me a way to monitor and/or simulate their own failover algorithm too, I can at least verify it’s doing what it thinks it’s doing.
I think I found the config JSON on the USG router that controls this too. Following these instructions about config files I ran mca-ctrl -t dump-cfg to inspect the config. Among other things I found this block:
Last night, after a couple of days of installation, we had the failure mode I was afraid of. The network failed over to the LTE at 9:30pm and didn’t switch back. Something else happened at 2:10am and again at 2:30am but it didn’t switch back to the ISP. Usually these last just a few seconds and it switches back to my ISP. This morning we were still on the LTE even though the ISP link was fine. Had to unplug the LTE and then reboot the router to get the thing back.
The main thing I’m seeing are clusters of 4 events; I’m thinking of these as failovers in response to a temporary short ISP outage.
15:48:06 LTE to state active
15:48:06 ISP to state inactive
15:48:40 LTE to state failover
15:48:40 ISP to state active
That’s what a normal working failover looks like. Not thrilled about having so many of them, but it seems to work. Here’s what happened last night
21:33:09 ISP to state inactive
21:33:10 LTE to state active
21:33:43 LTE to state failover
21:33:43 ISP to state active
02:09:39 LTE to state active
02:09:39 ISP to state inactive
02:30:21 LTE to state failover
That’s the event report.
First thing to note is that it was the 21:33 failover that actually started the problem. Even though events 3 and 4 make it look like the ISP should be active (not the LTE), I have independent logs of IP addresses that show me my home network was on LTE since 21:33. So that’s weird. The other weird thing about the 21:33 logs is that the events are backwards from normal; ISP to inactive first, then LTE to active. I don’t know how Unifi publishes events though so I’m not sure event ordering is strictly guaranteed. I suspect it is though and that timestamp difference (:09 vs :10) is evocative.
Then there’s the 02:09 events. No idea what those are about, it looks incomplete. I wonder if the router was using LTE but the watchdog infrastructure thought it was on the ISP? Whatever the case it’s a failure.
Ubiquiti is usually more reliable than this. I’m annoyed.
I bought this device because Wirecutter recommended it and it had a CO2 sensor. AQI or PM2.5 is the thing everyone cares about when there’s smoke, but I was also interested in CO2 levels in my house. This sensor does both. I also liked that Wirecutter said it had a good simple UI. And it has good cloud integration for data access. I’m looking forward to finally setting up Home Assistant to monitor it.
Setup out of the box is very easy. Charge the device (it has a battery!), turn it on. Install a mobile app, hold a button on the device a few seconds, and the mobile app walks you through the setup. Which mostly is just typing the wifi password for your network. I’m impressed with how these things get easier every time I get a new gizmo. I assume a Bluetooth handshake was exchanged.
Second half of the setup is getting the device to upload data to a cloud server in China somewhere. That was also easy; create a web account, then scan a QR code on that site on my phone and it automatically adopted my devices. From there I can publish a “kiosk URL” anyone can read. The URL is hilarious; just some JSON with the device ID and a web token signature all base 64 encoded. Eh, it works.
Now I want to get into the details. The docs don’t say a lot but there’s more info online here. By default it publishes an AQI number which I think is comparable to numbers like what PurpleAir or EPA publishes. (But which exactly? I don’t know other than it’s a US number.) It also publishes raw numbers for PM2.5 (in ug/m3) and CO2 PPM. And even more raw, there’s also “0.3μm per 0.1L of air”, a number which I guess is just a physical measurement and not adjusted at all.
Sitting outside in smoky air, The AQI and PM2.5 numbers are roughly in line with nearby sensors on Purple Air. The CO2 sensor read about 600 even when outdoors. I think that’s supposed to be 400 (which is normal outdoor a ir), so I went ahead and pressed the calibration button and now the reading says 400. It has some complex self-calibration but maybe this will help it along. I sure hope the smokey air doesn’t mean the CO2 level is naturally higher as well.
The CO2 sensor can directly measure global warming! Back in the 80s “normal” was 335 ppm; now it’s 415ppm. More details and history in this article.
Update: I got the device working with Home Assistant. Was as simple as configuring the HomeKit device that HASS found, which was apparently the Kaiterra. Type in the pairing code on the Kaiterra screen and I’m done.
For the first time I was able to verify that RCS messaging between two Android devices (a Pixel 3 and a Samsung 20) worked. Even better, my friend with the Samsung was using Samsung’s messaging app and I was using Google’s. Interoperability!
Here’s how to tell if your messages are being sent via RCS or SMS:
SMS are light blue and have the “SMS” tag. RCS are dark blue and say “Delivered”. They also have read status, and notification, and nice handling of longer messages and multimedia and stuff. When you’re sending a message it’ll tell you if it’s going out RCS. Not sure how to tell when messages you receive are RCS vs SMS, but my friend says on his Samsung app there was a blue dot next to my messages.
Basically RCS has all the nice things Apple has had in iMessage for 12+ years now. Only RCS is an open standard and not Apple proprietary. I’m hoping someone makes a nice native desktop app for RCS at some point. Google has clever browser-hosted access to your phone’s Messages app, but it’s kind of a hack.
Google and the carriers rolled out RCS for most people about a year ago. But looking back at my phone I could only find one other RCS message so far. Most of my friends are Apple captives so that may be why; Apple will presumably drag its feet in ever supporting the open standard.
One big drawback is RCS doesn’t support end to end encryption right now. I thought this was a political concession to various oppressive nation’s spy agencies. But more recently it looks like Google may be adding encryption support. I still prefer Signal for all messaging, really, but it’s hard to convince others to use it.
Setting up yet another new router, a Unifi USG. I’m finally buying fully into the Unifi platform. I like the concept; one central management process that configures and monitors all your network equipment. Proprietary of course, but I’m willing to bet on Ubiquiti.
Setting up the USG is awkward since it’s the backbone of your network. These docs helped. My catch is my LAN is configured to be 192.168.3.* but the default network is 192.168.1.*. And even though the Unifi Controller could see the USG when I plugged it in (via LAN port) and send it a firmware upgrade, it wasn’t able to actually provision (configure) it. So I had to follow their instructions; plug in a laptop to the WAN port, use the minimal web config to change the subnet, then use the Controller to configure the rest. Annoyingly the Internet access on the laptop worked fine on the 192.168.1.* subnet but not 3.*; I suspect that minimal config on the USG itself doesn’t even set the DHCP router address right when switching. Anyway once I’d switched the USG’s network to 3.* the Controller was able to send the full configuration and everything works.
Yet another router configuration system with its quirks to learn. Most of the defaults seem fine. The only thing I’m missing is static DNS entries for local static IPs, something Ubiquiti has never been great at. Turning on dnsmasq didn’t help. It may not be possible at all, or require editing a config file. Update I waited a day and now /etc/hosts on the router has a bunch of entries from DHCP, which it will also serve via DNS. I do not have dnsmasq enabled. They are FQDNs only, ie hostname.nelson.monkey.org and not the naked hostname. That’s reasonable but it does mean your host doing the query needs to be configured to search that domain.
One annoying thing in the USG: it does not have a switch. It has 3 ports: LAN1, WAN1, and then LAN2/WAN2. That last port can not be switched or bridged to LAN1, it is treated as a third network with its own subnet and routing. You could probably kludge it so it works like LAN1 but the router will be doing extra work. (Hilariously there’s a 4th RJ45 on the thing labelled “console”. It is literally a serial port.)
Another annoying thing in the USG: it’s slow. Not routing, that’s fast. But it takes 45+ seconds for the thing to boot. And re-provisioning takes a long time, although at least it’s conveniently orchestrated.
Figured it was time to document my LAN configuration for the next time I have to do this. Not that stuff doesn’t change.
We used our new generator in anger for the first time during a 42 power outage thanks to a PG&E PSPS, their response to their murderously unmaintained power network. It mostly worked fine and we didn’t even really notice any difference. Main problem was our ISP went down; that’s not our generator’s fault.
We used 10.7% of our propane tank for the outage; 54 gallons, or about 1.3 gal/hour. Or $150, if you look at it that way. The spec sheet says the generator uses 2.5 gal/hour at half load, 3.9 at full. It’s a 22kW generator and we’re using it nowhere near even half load. We average about 1500W over the day, so just 7% of capacity, but we’re apparently consuming 33% of max fuel. Not a big surprise but useful numbers to know.
Aside from the noise the main way I knew I was on generator is the lights flickered. And the cheap UPS in the server closet kept clicking off and on. (The fancy power conditoning UPS did not). Last time I looked at this I traced it to the well pump turning on; the momentary load generates a tiny brownout before the generator catches up. This time I noticed the oven heating also caused this kind of fluctuation a couple times a minute, a little weird since you’d hope the heating element was a steady load.
Ubiquiti’s Network Controller comes in an apt installable version for Ubuntu. For the most part their instructions just work. The one wrinkle I ran into with my Ubuntu 20 system is that the service wouldn’t start with systemctl having an error “Cannot locate Java Home”.
The solution for this was to manually install an older Java. apt install openjdk-8-jre-headless Their controller seems to require Java 8. The apt page has some confusing instructions about preventing Java 11 from being installed with apt-mark, but at this point we’ve moved on to openjdk-14 in Ubuntu land. Anyway Ubuntu lets you have multiple JREs installed with no drama.
The network controller is running as an HTTP interface on port 8080. Terrible choice of port (this is changeable). Once you do initial setup though it seems to start redirecting you to port 8443 instead.
I needed to migrate the devices from the previous network controller I’d installed. This all seems to be anticipated in the UniFi system, there’s instructions here on “adopting” the device on the new controller. I had to stop the old controller from running for the new controller to see the devices before adoption.
I also set it up to email me alerts for important things on the network. That requires an SMTP server. I grudgingly gave it a Gmail password, but only after creating a specific app password at Google only for this device.
Why am I doing all this? I’m upgrading some parts of my home network (finally installing a cellular backup for the ISP) and am finally fully embracing UniFi, Ubiquiti’s current platform for their networking equipment. I have UniFi access points and ran their controller software once, grudgingly, to install them. But I still have an old EdgeMAX router that predates UniFi. Or rather will for one more day.
The UniFi Network Controller has turned into a pretty good piece of software, btw. I was impressed when it helped me diagnose a WiFi performance problem I had with an old 802.11n device.
Wanted to give a shout-out to Tank Utility, a company that makes a gauge for monitoring your propane tank level via the Internet. Unlike most IoS devices it works great!
It’s very simple; $200 gives you a ruggedized battery powered outdoor thing you mount on your propane tank. There’s a small sensor that clips on the “remote ready dial” on your propane tank, I think it reads the needle setting via a Hall effect sensor. You do a one-time setup with a phone app for the WiFi and then the device will just quietly report the propane level a couple times a day to their free cloud service.
For end consumers they provide a decent mobile app for monitoring propane level, they also have a simple API for downloading the status. I think their real business is selling to propane companies, they have a commercial offering that I imagine tells the propane company when it’s time to deliver more fuel.
The clever thing in the device is it works off a few AA batteries but is designed for a battery life of up to 5 years. I think the thing is basically fully powered off most of the time and only powers up to connect briefly to the network to send a reading.
The main risk with this device is if the company disappears it will stop being useful. I wrote and asked them if they had a contingency plan for that and unsurprisingly for a small company, they didn’t have a good answer. The company seems just cool enough I could imagine them doing some one last update that enabled people to run their own monitoring server. Hopefully the company has a long life and the question never comes up.