Look at all this free electricity! Time scale is 6 hours. These are 410W panels which in practice seem to max out about 320W AC, or 350W DC before the inverter takes its share. This is from Jan 13; maybe I’ll get more in summer.
Solar sparklines
Author: nelsonminar
Installing Oculus on a PC: “Not Enough Space”
The Oculus Windows app installer has a bug; you can’t install it on an NTFS dynamic partition. If you try it reports there’s not enough space, despite there being plenty. Dynamic partitions are extremely common on Windows machines.
The workaround I used was to run OculusSetup.exe /drive=D. I happened to have a D drive that was a basic partition and while there’s no GUI option for choosing the install drive, there is this command line switch. If you’re not so lucky, there’s a lot of advice online suggesting making a 20GB virtual disk and installing there.
This bug is six years old. I always wonder how bugs like this happen; do the developers just not give a shit? Sometimes I think it’s incompetence but Oculus is a very impressive piece of tech in general. The linked Reddit discussion suggests this is some ham-handed attempt to avoid installing to removable media, so maybe it’s deliberate. Maybe it’s a DRM thing? What a stupid decision if so.
Debugging a WiFi problem
Still tinkering with Raspberry Pis and my Sunpower PVS6 solar controller. I had it all set up with a new RPi Zero 2 W when I had a new problem; the WiFi was flaky. It kinda sorta mostly worked well enough, I could get the HTTP requests through to it usually. But ping was showing 40% or more packet loss with a latency of 200+ms. I have a very clean WiFi environment, that should be 0% loss and 2ms latency.
A whole lot of testing later and I established that the problem is some sort of interference on 2.4GHz Channel 1, but only in my garage near the Sunpower box. That corner also has all my other ethernet equipment, is where the generator and solar feeds come in, and where the utility power and electrical panel are. So a lot of wires. None of that stuff should interfere with 2401-2423Mhz and there are a lot of mysteries still.
Switching my access point to channel 3 (2411-2433Mhz) seems to solve the problem. Part of the root cause of my troubles is I’d pinned the WiFi channel in the garage to 1 a year or more ago, I forget why. I’ve now set it to auto. I’ve had experiences where it seemed to be working for a few hours at a time before, so I don’t fully trust it yet. Edit: after rebooting the AP went back from channel 3 to 1. So I’ve configured it to hardcode channel 3 instead of “auto”. Whatever the interference is, the Ubiquiti hardware isn’t detecting it.
The most useful thing I did in the end was get 2 access points and 3 clients and try moving them around every few hours to see what triggered the problem. I used telegraf to ping them once a minute or so (but see below) for long term monitoring, and then occasional command line pings once a second for short term testing. Over two days I was able to narrow the problem down to the one channel in the one location, but it’s not related specifically to any particular hardware.
Some things I learned:
- wavemon is a helpful tool on Linux for showing WiFi status. so is /proc/net/status and learning all the arcane options to the iw command.
- The UniFi dashboard has some decent stats of its own hidden away under “Air Stats” for each individual access point device.
- WiFi has its own link layer retransmit; lots of those might account for higher latency. But I don’t know what timescale it is on: 1 ms? 500ms? You can see WiFi retransmits in wavemon as “failed” and in Air Stats as “retry”.
- Despite all these diagnostics none of them really showed me evidence the link was bad. All the basic link quality reports were 90% or better. No ridiculous number of retries (although it is 16% at the access point.)
- Linux machines, including RPis, need to be configured for the country for wifi to work at its best. Particularly important for 5GHz.
- Even with a perfectly good 5GHz channel available an RPi4 might still choose 2.4GHz. I could not find a way to reliably coerce it to use 5GHz.
- Raspberry Pis have a variety of low power optimizations that mean they may not respond well to a ping every minute when otherwise idle. My RPi4 regularly takes 100ms to answer the first ping, then is a solid 2ms after. My original RPi Rev B just misses pings entirely if nothing else is happening on it. The RPi Zero 2 W seems pretty solid but then I’ve got it doing other real computation. Anyway my once-a-minute ping was showing phantom errors where there were none. A once-a-second ping test worked better.
- There’s a power saving mode in the RPi radio hardware itself you can turn off with iw.
- Raspberry Pis may in general just not be very good at wifi.
- Don’t use mDNS names for ping testing. The mDNS query itself may get lost to a flaky / low power Pi and you never find the host.
- TCP retransmits hide a lot of sins. But it only works right until it doesn’t. When my WiFi link was bad about 5% of my requests wouldn’t get through at all.
WiFi on Linux: country registration (Raspberry Pi)
My Raspberry Pi 4 would not talk via 5GHz to my Ubiquiti Access Point. Turns out the problem is the AP was running on channel 157 (5785MHz, 80MHz wide) which is not a globally allowed WiFi channel. It is allowed in the United States. Ubuntu on RPi4 has no country configuration for the WiFi unless you set it explicitly. Setting it for US jurisdiction should fix it.
On Ubuntu the config file /etc/default/crda is where to set it. Note you have to set it to “US”; “us” does not work. On Raspberry PI OS it’s in raspi-config under Localisation options. It’s also configurable in the wpa_supplicant.conf file you create when setting up a headless RPi.
Once the system is running you can inspect the state of your current wifi config with the commands iw reg get and iw phy phy0 channels, see output below. There’s documentation on how this works here, here. The apt packages crda and wireless-regdb are also of interest, as is /lib/crda.
$ iw reg get
global
country 00: DFS-UNSET
(2402 - 2472 @ 40), (N/A, 20), (N/A)
(2457 - 2482 @ 20), (N/A, 20), (N/A), AUTO-BW, PASSIVE-SCAN
(2474 - 2494 @ 20), (N/A, 20), (N/A), NO-OFDM, PASSIVE-SCAN
(5170 - 5250 @ 80), (N/A, 20), (N/A), AUTO-BW, PASSIVE-SCAN
(5250 - 5330 @ 80), (N/A, 20), (0 ms), DFS, AUTO-BW, PASSIVE-SCAN
(5490 - 5730 @ 160), (N/A, 20), (0 ms), DFS, PASSIVE-SCAN
(5735 - 5835 @ 80), (N/A, 20), (N/A), PASSIVE-SCAN
(57240 - 63720 @ 2160), (N/A, 0), (N/A)
phy#0
country 99: DFS-UNSET
(2402 - 2482 @ 40), (6, 20), (N/A)
(2474 - 2494 @ 20), (6, 20), (N/A)
(5140 - 5360 @ 160), (6, 20), (N/A)
(5460 - 5860 @ 160), (6, 20), (N/A)
$ iw phy phy0 channels
Band 1:
* 2412 MHz [1]
Maximum TX power: 20.0 dBm
Channel widths: 20MHz HT40+
* 2417 MHz [2]
Maximum TX power: 20.0 dBm
Channel widths: 20MHz HT40+
* 2422 MHz [3]
Maximum TX power: 20.0 dBm
Channel widths: 20MHz HT40+
* 2427 MHz [4]
Maximum TX power: 20.0 dBm
Channel widths: 20MHz HT40+
* 2432 MHz [5]
Maximum TX power: 20.0 dBm
Channel widths: 20MHz HT40- HT40+
* 2437 MHz [6]
Maximum TX power: 20.0 dBm
Channel widths: 20MHz HT40- HT40+
* 2442 MHz [7]
Maximum TX power: 20.0 dBm
Channel widths: 20MHz HT40- HT40+
* 2447 MHz [8]
Maximum TX power: 20.0 dBm
Channel widths: 20MHz HT40- HT40+
* 2452 MHz [9]
Maximum TX power: 20.0 dBm
Channel widths: 20MHz HT40- HT40+
* 2457 MHz [10]
Maximum TX power: 20.0 dBm
Channel widths: 20MHz HT40-
* 2462 MHz [11]
Maximum TX power: 20.0 dBm
Channel widths: 20MHz HT40-
* 2467 MHz [12]
Maximum TX power: 20.0 dBm
Channel widths: 20MHz HT40-
* 2472 MHz [13]
Maximum TX power: 20.0 dBm
Channel widths: 20MHz HT40-
* 2484 MHz [14] (disabled)
Band 2:
* 5170 MHz [34] (disabled)
* 5180 MHz [36]
Maximum TX power: 20.0 dBm
Channel widths: 20MHz HT40+
* 5190 MHz [38] (disabled)
* 5200 MHz [40]
Maximum TX power: 20.0 dBm
Channel widths: 20MHz HT40-
* 5210 MHz [42] (disabled)
* 5220 MHz [44]
Maximum TX power: 20.0 dBm
Channel widths: 20MHz HT40+
* 5230 MHz [46] (disabled)
* 5240 MHz [48]
Maximum TX power: 20.0 dBm
Channel widths: 20MHz HT40-
* 5260 MHz [52]
Maximum TX power: 20.0 dBm
No IR
Radar detection
Channel widths: 20MHz HT40+
DFS state: usable (for 726 sec)
DFS CAC time: 0 ms
* 5280 MHz [56]
Maximum TX power: 20.0 dBm
No IR
Radar detection
Channel widths: 20MHz HT40-
DFS state: usable (for 726 sec)
DFS CAC time: 0 ms
* 5300 MHz [60]
Maximum TX power: 20.0 dBm
No IR
Radar detection
Channel widths: 20MHz HT40+
DFS state: usable (for 726 sec)
DFS CAC time: 0 ms
* 5320 MHz [64]
Maximum TX power: 20.0 dBm
No IR
Radar detection
Channel widths: 20MHz HT40-
DFS state: usable (for 726 sec)
DFS CAC time: 0 ms
* 5500 MHz [100]
Maximum TX power: 20.0 dBm
No IR
Radar detection
Channel widths: 20MHz HT40+
DFS state: usable (for 726 sec)
DFS CAC time: 0 ms
* 5520 MHz [104]
Maximum TX power: 20.0 dBm
No IR
Radar detection
Channel widths: 20MHz HT40-
DFS state: usable (for 726 sec)
DFS CAC time: 0 ms
* 5540 MHz [108]
Maximum TX power: 20.0 dBm
No IR
Radar detection
Channel widths: 20MHz HT40+
DFS state: usable (for 726 sec)
DFS CAC time: 0 ms
* 5560 MHz [112]
Maximum TX power: 20.0 dBm
No IR
Radar detection
Channel widths: 20MHz HT40-
DFS state: usable (for 726 sec)
DFS CAC time: 0 ms
* 5580 MHz [116]
Maximum TX power: 20.0 dBm
No IR
Radar detection
Channel widths: 20MHz HT40+
DFS state: usable (for 726 sec)
DFS CAC time: 0 ms
* 5600 MHz [120]
Maximum TX power: 20.0 dBm
No IR
Radar detection
Channel widths: 20MHz HT40-
DFS state: usable (for 726 sec)
DFS CAC time: 0 ms
* 5620 MHz [124]
Maximum TX power: 20.0 dBm
No IR
Radar detection
Channel widths: 20MHz HT40+
DFS state: usable (for 726 sec)
DFS CAC time: 0 ms
* 5640 MHz [128]
Maximum TX power: 20.0 dBm
No IR
Radar detection
Channel widths: 20MHz HT40-
DFS state: usable (for 726 sec)
DFS CAC time: 0 ms
* 5660 MHz [132]
Maximum TX power: 20.0 dBm
No IR
Radar detection
Channel widths: 20MHz HT40+
DFS state: usable (for 726 sec)
DFS CAC time: 0 ms
* 5680 MHz [136]
Maximum TX power: 20.0 dBm
No IR
Radar detection
Channel widths: 20MHz HT40-
DFS state: usable (for 726 sec)
DFS CAC time: 0 ms
* 5700 MHz [140]
Maximum TX power: 20.0 dBm
No IR
Radar detection
Channel widths: 20MHz
DFS state: usable (for 726 sec)
DFS CAC time: 0 ms
* 5720 MHz [144] (disabled)
* 5745 MHz [149] (disabled)
* 5765 MHz [153] (disabled)
* 5785 MHz [157] (disabled)
* 5805 MHz [161] (disabled)
* 5825 MHz [165] (disabled)mDNS, hostname.local, and WSL2
Modern computers can often be reached by the DNS name hostname.local. This is implemented via Multicast DNS, mDNS, which originated in the early 2000s as part of the zeroconf work to make networked computers usable without manual configuration. Apple’s Bonjour was the first implementation of it I saw working widely. Linux machines tend to use Avahi. For Debain and Ubuntu systems apt install avahi-daemon is generally enough to get the system announcing its name.
The magic of multicast
It works via multicast. mDNS-aware name resolvers know that if a domain name ends in .local, the way to resolve it is to send a UDP multicast request to 224.0.0.251:5353 or and see if anyone answers. (There’s an IPv6 equivalent). Multicast generally propagates across your whole LAN; if a host sees a query for their own name it answers back to 224.0.0.251 (not the requester directly) with a DNS record. It’s not exactly standard DNS packets, some slight modifications in place, but the basics are the same.
It works great! As long as you have a reliable LAN. I’m using it on some Raspberry Pi devices with a flaky WiFi connection and that’s not a good experience.
As an aside, can I say how much I love multicast? Ethernet broadcast too, and even the less common anycast. Being able to send a packet to the local network without being exactly sure who the recipient is can be very useful in all sorts of applications. A lot of stuff we do like NTP would work better if it regularly used multicast. Unfortunately multicast over the routed public Internet is pretty much a dead-end technology; no one has ever figured out how to manage it scalably.
mDNS services
I mostly use mDNS myself for .local domain name resolution. But mDNS is a full service discovery framework. You can use it to ask for things like “is there a printer on the network?”
Avahi comes with a discovery tool to get a quick dump of services on the network.avahi-browse --all --ignore-local --resolve --terminate
Here’s an example of the services my printer offers
+ eno1 IPv4 Samsung M267x 287x Series (SEC...) _privet._tcp local
+ eno1 IPv4 Samsung M267x 287x Series (SEC...) _scanner._tcp local
+ eno1 IPv4 Samsung M267x 287x Series (SEC...) UNIX Printer local
+ eno1 IPv4 Samsung M267x 287x Series (SEC...) PDL Printer local
+ eno1 IPv4 Samsung M267x 287x Series (SEC...) Internet Printer local
+ eno1 IPv4 Samsung M267x 287x Series (SEC...) Web Site localOn my LAN I have services for Sonos, Roku Airplay, the printer, my Samba fileshare, and some things Home Assistant has published.
I believe the service discovery is all implemented via mDNS queries, with some extra information placed in TXT records. It all looks fairly elaborate, RFC 6763 is one reference for how it works. Now I’m curious how carefully specified the metadata for things like printers are. Is there a clear standard or is it all freeform and customary publication?
WSL and mDNS
WSL2, Microsoft’s Linux VM for Windows, also supports .local domain names. It’s a little weird. My windows machine is named “Nelson-Win10”. On the rest of my network nelson-win10.local resolves to 192.168.3.114, the DHCP-assigned address for the host Windows system. Just like you’d expect. But inside a WSL2 shell it resolves to Nelson-Win10.mshome.net (172.27.48.1). That’s a private-use address and the domain name is something Microsoft has been doing for years as part of their Internet connection sharing stuff. This address is reachable from the host Windows system. It’s not accessible on the rest of my LAN because I don’t have routes for that subnet, I wonder if it would work if I added one? Anyway this is all a workaround for the fact that localhost doesn’t work very well (or at all) across the host Windows and WSL2). Honestly all the networking in WSL2 to the outside world is kind of confusing and maybe buggy. But hostname.local works on the Windows machine itself.
(Note that in a command shell on Windows itself, nelson-win10.local also works. It resolves to an IPv6 link local address in the fe80::/10 block.)
Customizing a Raspberry PI Zero 2 W for PVS6
Awhile back I set up a Raspberry Pi to be a gateway to get access to my SunPower PVS6 monitoring system. That’s going great; the extra status info is really helpful and interesting. (Bonus screenshot below). I just bought a Raspberry Pi Zero 2 W to be the new computer for this. It’s so tiny it’ll fit entirely in the PVS6 box, powered by the USB that’s in there.
It works! The wifi is pretty flaky, despite me getting an amazing -30dB signal. It seems to be a problem with one specific access point, need to tinker some more. (Update: it may be wifi power saving.) The cable mess is pretty janky but it’s going to be shut inside a box, not sure I care. A smaller ethernet adapter would sure be nice.
Here’s my detailed software notes on setting this up.
- Mostly I’m just following this guide
- Use the RPi image installer to burn RPi OS Lite to a new SD card. Note this is a 32 bit OS.
- Create and configure a wpa_supplicant.conf file on the DOS partition of the SD card
- Create an empty ssh file on the DOS partition
- Boot the RPi, find its IP address via my DHCP server. (The name raspberrypi.local might also work.)
- Log in with ssh as pi/raspberry. Ignore locale complaints.
- Change the password for pi
- Become root
- apt update; apt upgrade. (This takes awhile!)
- Enable console autologin using raspi-config
- Fix locale using raspi-config
- Change the hostname to “pvspi0” using raspi-config
- Edit /etc/dhcpcd.conf so the ethernet is not used as a gateway
- apt install chrony
- apt install avahi-daemon
- Disable wifi power saving
- Reboot the pi
- Log in to pvspi0.local using pi/newpassword.
Note mDNS takes awhile to start working, a minute or two after the RPi starts up - Install haproxy
- Configure
/etc/haproxy/haproxy.cfg. I did not include stats.
A brief note on power consumption. Idle the RPi Zero 2 W seems to use about 165mA (at 5W). Running stress-ng -c 4 peaked at 570 mA. The magic number here is 500 mA, that’s the minimum power for a USB port. I expect the system to basically be idle all the time so I think I’m safe with no extra measures. But some ideas on reducing power consumption.
- Turn off HDMI. Instructions don’t work.
- Turn off LEDs
- Use fewer than 4 cores
- Turn off unnecessary daemons. bluetoothd, hciattach (the serial console)
Here’s the bonus snapshot of some of my Grafana config. Not running on the RPi; all its doing is proxying requests. These graphs are centered on showing the status of all 28 separate panels, something you can only do by accessing the PVS6 data directly.
Faking Geolocation in Firefox
One mild annoyance with Starlink is geolocation doesn’t work very well. A lot of sites identify me by my IP address, which is the satellite base station somewhere near Winters, CA 100 miles from me. Even if they use the browser’s geolocation API to get my location my computer, since I don’t have a WiFi connection Firefox seems to fall back on that same IP based location.
Anyway, there’s a simple solution in Firefox. Open the URL about:config and search for “geo”. Edit the value of geo.provider.network.url and change it from the default (a Google service) to a static answer. Here’s what I use, for Grass Valley:
data:application/json,{"location": {"lat": 39.22, "lng": -121.06}, "accuracy": 1000.0}Note this JSON blob is not a Javascript Geolocation API response; it must be some special backend API. Mozilla has their own location API, I wonder why they use Google’s in Firefox?
It’d be nicer to have a browser extension to manage this for me. There used to be a consensus open source choice, Location Guard, but it hasn’t gotten an update for two years. There’s a bunch of spammy looking clones on the Firefox store I won’t trust. Location Guard might still work but when I briefly tried it I couldn’t see how to give out a specific location, only enable their privacy fuzzing.
Note that browser location spoofing won’t be good enough to fake out sites that really care what your location is. They’re probably using your IP address to locate you, or some other more devious means. But it’s a nice solution for ordinary apps that are just trying to do something simple.
Also note setting a single static location is probably a bad idea for a mobile device like a laptop or phone.
underscore DNS queries
Wanted to document this learning as more than some tweets.
The amazing Julia Evans launched Mess with DNS, a little sandbox where you can register some domain names and then see what queries their DNS server of record gets. Great tool for learning.
I learned something! When I did a single A query at home on Sonic.net using their DHCP servers, Mess With DNS sees six queries all at once
That seems like an awful lot. There’s several mysteries here, but the one I focussed on was that query for _.lily6.messwithdns.com. Underscores are generally understood to not be valid in domain names, although the underlying DNS protocols do seem to allow them in some places. But why would a query for a normal subdomain also produce a query for _?
A simple way to reproduce this is$ dig @50.0.1.1 nelson.lily6.messwithdns.com a
That’s a different Sonic resolver than I use. It only results in two queries, not six, but it still generates that underscore query.
I put the question out on Twitter and on Hacker News and got two useful replies.
First, a reply from a product manager at Cloudflare who very kindly looked in his logs for similar queries and reported.
Over the last ~5 minutes I see ~12 queries per second on average (globally) to 1.1.1.1 for _.www.google.com and ~3 QPS for _.www.cloudflare.com – which is effectively noise.
So that’s interesting. Those queries do exist but they are not very common.
As for what causes them, the Hacker News discussion came through. Several folks all pointed out this is something Bind9 does with the “QNAME optimization” option turned on. One Sonic engineer even explicitly confirmed that was the DNS server they use and the source of the queries.
So what do they mean? QNAME Minimization is a privacy thing in recursive resolvers. If you’re querying foo.bar.example.com at some point you end up asking the .com root server for information. But there’s no need for them to see the whole domain name, so you just send the minimum you need (example.com, I believe.)
That still doesn’t explain the underscore. TBH I don’t really understand it. The Bind9 code is here but there’s few comments. However I was able to find the commit that added it and the comment says
qname minimization, even in relaxed mode, can fail on some very broken domains. In relaxed mode, instead of asking for “foo.bar NS” ask for “_.foo.bar A” to either get a delegation or NXDOMAIN. It will require more queries than regular mode for proper NXDOMAINs.
So that seems to be the answer. It’s not great that this workaround for “some very broken domains” ends up doubling the name server traffic for the rest of the Internet, but here we are.
For completeness, several people pointed to RFC 2782 or RFC 8552. (It’s one of first Google search results for “underscore dns”.) That RFC is about using _ to mangle some other tokens to not collide with domain names; they are used when querying things like _tcp or _udp. I’m pretty sure that’s not what’s going on here.
I’m grateful to have gotten replies from someone at CloudFlare, my ISP, and the author of the QNAME Minimization RFC. It’s nice to be able to get an answer from actual experts.
Gaming: Starlink vs Path of Exile
I’ve been using Starlink since March. I’m also a gamer. While Starlink has gotten a lot better for gaming it’s still not perfect. See this lag graph for an example. Basically every few minutes the game freezes for a second or so and I’m 95% sure it’s Starlink packet loss. It doesn’t make the game unplayable but it is annoying.
Path of Exile is unusual in that the default network mode is lockstep. It only draws a frame if it receives a network packet from the TCP stream, so any packet loss and retransmit delays is immediately visible. Most games compensate for network flakiness by guessing at what probably should have happened and then catching up once the network resumes again. PoE has the option to do that too, but by default they show you exactly what’s coming in on the network. No packets, no updates.
So it’s a nice example of how Starlink’s momentary outages can cause problems. You won’t notice it streaming a video or loading web pages. You’ll see occasional hiccups in a live videoconference. In PoE you see little lag spikes.
I have more proper monitoring for similar loss: see my latency graph based on sending 5 pings at once every 5 seconds. Each red dot is packet loss. It’s really quite good but you see some latency spikes and a few lost packets here and there.
(I also posted this to Reddit.)
Small multiples in Grafana
I learned how to make lots of little graphs in Grafana 8 today. It’s a bit strange but I figured it out thanks to this video on variables and these docs. Here’s a visualization of the power production of all 28 solar panels on my house. Some of those panels aren’t doing a lot of work in the morning; hope it’s better in the summer!
The first step is to declare a variable. You do this in the dashboard settings. In this case I wanted to create a variable named “inverter” which has 28 possible values, one per serial number. I’ve already stored the serial numbers as tags in my Influx data so this Influx Query will get at them
show tag values from pvs6_inverter with key=SERIAL
When you do this Grafana will magically add a little selection UI to your dashboard. You can configure whether there’s an “All” option and whether multiple sections are allowed when you define the variable.
Then you need to make a Grafana panel for the graph that is parameterized by that variable. This is as easy as adding a where SERIAL =~ /$inverter$/ to the query for the graph data; you can do this via the graphical query builder. Once you’ve done this you can choose which graph you want to see, but there’s still only one panel with one graph.
The last step is to enable repetition. This is under “Panel Options / Repeat Options”. You choose which variable to repeat by, then what layout you want. “Horizontal” actually means vertical too; I have it set to 6 max per row and it creates multiple rows automatically. Then go back to the dashboard view and select “All”. You should see all your panels. It’s a little wonky about reloading on changes, you may have to do something to force the UI to refresh.
The last bit of polish I did was to remove all the decorations from the graph I could to get it to display more densely. No title, hide the legend, no Y axis. I switched to the Graph (old) widget because I couldn’t figure out how to hide the X time axis in the new Time Series graph. The only thing I couldn’t do was change the padding of the graphs themselves; there’s more whitespace than I’d like. But it’s not bad and I appreciate how much UI work Grafana does, so nicely.
Nelson's log 