Yet another new server setup post

My little projects server at is dying. The disk errored and remounted itself read-only two weeks ago and I only just noticed. Time to set up a new server. Last time I did this five years ago I took careful notes and they are useful to me now. So another boring blog post about the new server. These recent notes for my home server were still useful.

I’m still treating Linux servers as these little carefully customized hand crafted collection of installed packages, configurations, crontabs, etc. The real world has moved on to managed services but every time I look to do that for myself it’s either too much work for a little tiny service (docker, chef/puppet/whatever) or else too expensive (cloud hosting). This time around I got recommendations to look at Ansible for low-impact server setup automation, but given I set up one server every 5 years it seems overkill. Also recommendations for Dokku as a free hosted service manager. That in turn creates Docker images and also feels like overkill and a bad match for Apache.

Part of me would love to just push all this server management “to the cloud”. But it’s still not cheap to do that for anything custom / dynamic. My old server was $59/mo, my new server is $25/mo. For a dedicated bare metal machine. It’s hard to match that price using fancy cloud stuff. My long term goal is to get my services down to simpler static files so I can just host them that way, without complex software. And any new complex software I do make, do it with cloud management.

It’d be really nice to replace Blosxom with some static site generator; then literally everything I serve on the Web would be static files. Doing that requires resolving my other blogs too; this secret work blog, also my linkblog / Pinboard account.

What to install

Looking at my list from 5 years ago I still run most of the same services. Except now I feel it’s time to shut a few down. The live rivers map; it’s a tutorial for outdated technology. And logsoflag, it’s also outdated and not something I want to keep working on. The rivers system was the most complex thing I ran (postgres, tilestache, nginx proxy, …) so not reinstalling it saves a lot of time.

Most of what I serve is via Apache and virtual hosts. Things to get working:

  • somebits weblog: Blosxom CGI
  • static files
  • all static files
  • replace with shutdown notice
  • replace with shutdown notice, maybe leave the tool working in legacy status.
  • shutdown notice
  • shutdown notice
  • Let’s Encrypt certificates for *,, Separate certificates for the three this time.

I also have some other stuff running that needs installing:

  • autossh tunnel to my house
  • crontab entries for blog, linkblog
  • munin: default config
  • chrony: default config

Configuration Notes

apache. Default Ubuntu 18 config is OK; it uses the event MPM which is reasonable. I added the following modules: cgid expires headers rewrite ssl userdir. Configured charset to utf8, added my own nelson.conf.

blosxom. Had to install libimage-size-perl for one of the Blosxom plugins. Also had to patch a problem in the atomfeed plugin; the use of {{{updated}}} as a template fell afoul of a backwards incompatible language change in Perl 5. <3 ancient languages. Also had to install python-minimal to get a /usr/bin/python (that’s Python 2), for some ancient cron jobs.

munin. Has to be installed after apache, or the apache stuff doesn’t get installed.

autossh. Same as before.

rsnapshot. Same as before.


New server is all set up, remarkably easier since I could just look at the live running server and copy files from it. I had a backup, but nothing like a live system to clone…

Then I tried to get the disk back read-write so I could wipe it. What a mess. Trying to remount failed because the block device was marked read-only, presumably by the device driver. Trying to re-mark the block device just didn’t work. So then I ran fsck on the disk, the one mounted read-only as the operating system. Lol. Machine locked up solid. Also didn’t reboot. I guess I was hoping fsck would either refuse to run if the disk was mounted, or else somehow magically the OS would keep working even with fsck running. Nope.

RIP little server. 5 year at $59/month comes to about $3500. The hardware probably cost $1500, but then I’m playing for hosting too.

How PoB Item Tester works

I’ve been obsessed with the Diablo-like game Path of Exile for the past few months. Great game. For the code nerd in me, I’ve been fascinated by how the community has created a bunch of addons and tools to run with the game, despite the game having no support for them. The kludges are thick and woolly and fascinating.

The primary way most of these addons work is via AutoHotKey, a Windows-specific scripting system that occupies a role a little like AppleScript does on MacOS. AHK has been around forever; its main feature is that it’s very good at capturing global keyboard events and doing odd things in the background that don’t fit the usual GUI paradigm. I use it to remap CapsLock to search Google, for instance.

PoB Item Tester is the most complex addon I know. You hover over an item in the game, press Ctrl-C to copy its description to the clipboard, then press Ctrl-Win-C to have a window popup that shows what the item would do if you used it. Behind the scenes it’s using Path of Building to do calculations that simulate game behavor. PoB is a standalone Windows program that itself is a tour-de-force of reverse engineering. It is a full GUI program written in Lua (!).

Functionality aside, it’s the integration that I’m interested in. Here’s what the AutoHotKey script does:

  1. Hook Control-Win-C with AHK to get the item from the Windows clipboard and pass it to TestItemFromClipboard
  2. Write the item to a file on disk
  3. Invoke a Lua script from AHK
  4. Pop up a window based on some GUI setup done previously.
  5. Get the file created by the Lua script and display the contents

So the other half of the integration is in Lua. It has a whole Mock Lua UI system to allow PoB to run headless. Some details on what the Lua script run from AHK is doing:

  1. Set up the Mock GUI. Buried in there is the part that launches Path of Building’s Lua code.
  2. Tell PoB to load the build to test against
  3. Create an Item object based on the file written by AHK (from the clipboard). These seems to do the actual simulation calculations.
  4. Capture the item’s tooltip as HTML and write it to a file for AHK to display.

So there you have it. The game sure wasn’t designed to be scripted, but clever use of the clipboard and keyboard hooks lets people write scripts that feel like they’re running in-game. Path of Building wasn’t written to be scripted either, but someone figured out how to hack the Lua to get it to run headless and do calculations for them. That’s a lot of very motivated hackery, I’m impressed.

Amazon Fire 7 first impressions

My switch to an Android phone has gone so well I wanted to get an Android tablet, too. Only to learn Android tablets are universally considered to suck. OTOH they are also very cheap. So I picked up a brand-new Fire 7 9th generation. For $49. 16GB storage, 1GB RAM, two cameras, a full WiFi multitasking computer with a decent touchscreen display for less than the price of a fancy dinner. We live in an age of wonder. (Note 16GB storage is not enough, but it has a microSD slot. More wonders: 256GB of SD storage for $40.)

So how is the tablet? OK I guess. It seems fine for reading books and watching videos, which is what Amazon sells this for. The screen doesn’t feel quite responsive to my touches all the time but I’m not sure if that’s a hardware or software problem. 1GB of RAM really isn’t enough. CPU may be a bit sluggish. Camera quality is appalling. It comes with a bunch of adware (you can pay $15 to remove it). Did I mention it’s $49? Amazing.

The OS is FireOS, an Android 7.1 / Nougat variant. It’s ugly. It only supports the Amazon Fire store which has an anemic collection of apps. (No Firefox! No Kiwix!) The Amazon versions of apps aren’t great, like apparently the Silk browser is doing some MITM attack to try to save you bandwidth. However the device is unlocked, or at least it’s not hard to install the Google Play store and from there a bunch of normal Google Play apps. (One caveat; be sure to install the Google APKs to internal storage. Don’t format the SD card as an extension of internal storage and let the apps install to it, because then they can’t be updated via the Google Play store.)

So 30 minutes of installing later and I have a good generic Android device. Firefox for the browser. GBoard for the keyboard. Evie for the launcher, (which requires more work to launch via the launch button). Alles bueno.

I can’t find some things via the Google Play store even though they’re there for other devices; Tapet, 1Password, etc. Maybe these apps don’t explicitly mark they’re compatible with the Fire and so are excluded? I’ve been able to sideload these from APKMirror but I’m a little nervous about that, I like my trusted app stores. Particularly 1Password, brrr. All the apps I’ve sideloaded seem to work fine, so I’m not sure why Google Play isn’t giving me the choice to install them.

All in all I’m reasonably impressed. It’s not as nice as an iPad, nor even really as nice as my Pixel 3 phone. But it’s literally $49 and it works pretty well. And now I have an Android device to tinker with.

How much Mailchimp is spam? What should they do?

A discussion on Twitter led me to wonder; is all the mail I get from Mailchimp spam? Can I filter it? Turns out filtering Mailchimp in Gmail is hard. But I have a mediocre solution and some ideas on what Mailchimp should do to be kinder to email recipients.

I only notice Mailchimp mail when I get some crap I don’t want to see and click the Unsubscribe link. I get a lot of these same spams from the same company, like my old college or my local pizza place. The landing page for unsubscribing is distinctly Mailchimp, which is when I notice that they’re the source of the spam.

What seems to happen is a Mailchimp customer creates a new Mailchimp list for every marketing campaign. So I won’t get any more June 5 Campaign mails, but I’m bound to get the next June 12 Campaign mail and there’s no way to make it stop. It is spam. My question is; is all Mailchimp mail spam? (Spoiler; only 93% of mine in the last year is.)

Identifying Mailchimp in Gmail

My mediocre solution for identifying Mailchimp mail is a gmail search for three special phrases in the message body, the current copy Mailchimp includes to provide their (not-very-effective) opt-out mechanisms. This text could change at any time, but it seems to be consistent for several years now. Here’s the search term, all three phrases:

“view this email in your browser”
“want to change how you receive these emails?”
“You can update your preferences or unsubscribe”

The better way to filter would be on an email header like X-Mailer, Received, or the DKIM signature. Gmail’s filters don’t seem to allow filtering on any of those criteria. Here’s an example spammy mail; maybe someone else has a better idea how to get Gmail to identify these. (There is a possibility involving Google Script but I can’t tell if that’s really a good idea to use.)

My Mailchimp statistics

In the past year I’ve gotten 57 Mailchimp mails. Maybe more if they went to the Spam folder and were auto-deleted. I have deliberately subscribed to 0 Mailchimp mailing lists, so in some sense all 57 are spam.

However, a whole lot of these Mailchimp mails come from organizations I legitimately have business with. Reed College being the primary culprit, also my local pizza place. I’d rather not get any of those emails, but they are in some sense organizations that have a reason and a right to email me. 54 of the 57 Mailchimp mails I got are in this category, or 93%. I’d rather not get any more of them.

3 of the 57 Mailchimp mails were things I actually wanted to get. Two from the XOXO conference, one from OpenStreetMap. Those organizations use Mailchimp responsibly in a way that helps me and even if I didn’t explicitly opt-in to a mailing list. I want these mails.

0 of the 57 Mailchimp mails were pure spam. None of them were from an organization I’d never heard of before. So that’s good! Update: not a day after writing this I got a pure spam Mailchimp mail, an ad for a Kickstarter game called “Cracked”.

Still, a 93% feels-spammy rate is pretty awful and Mailchimp should do better.

Mailchimp for recipients

AFAIK Mailchimp currently has essentially no product for email recipients. We’re not customers, we’re the product Mailchimp is selling. There’s this polite veneer that somehow we’ve opted in to these communications, and Mailchimp does provide opt-outs. But they aren’t really working.

My proposed solution is Mailchimp should build a small product for email recipients. Leave the defaults as-is, to not decimate Mailchimp’s business. But give users the chance to say “I never want to receive any Mailchimp mail from this organization again”. Not just opt-out of a specific list, but all lists from a particular Mailchimp customer. It’s not simple; Mailchimp would have to create an account for each email recipient with some small Web UI, passwords, etc. It’s not a tiny project, but it’s not a huge one either.

Of course Mailchimp won’t do this. They have no incentive to. Right until the moment someone like me gets fed up enough and considers blocking all of Mailchimp as spam. Imagine if Gmail did that; it would absolutely crush the company. Mailchimp should have a solution in hand before their spam problem escalates to the level of existential crisis.

(Even better would be to require an explicit opt-in from each organization, or for each mailing list. That’s how it used to work in the old Mailman days. But no way would Mailchimp do this; it’d destroy their business.)

An alternate thing Mailchimp could do is get more aggressive at policing their customers. There’s a feedback form when you unsubscribe that includes lines “This is spam and should be reported” as well as “I never subscribed to this list”. I faithfully click one or the other every time I get spam via Mailchimp, in the hopes it creates some signal inside the company and maybe their customer will be reined in. So far I see it having zero effect though, and why would it? The spammer is who is paying Mailchimp, not me.

I have several friends who work at Mailchimp. It’s a good company and they have a good product for their customers. But as a company I suspect they are myopic, focussed only on their paying customers. I’d like them to do more to stop spammy use of their product before it becomes a crisis.

WSL access to Linux files via Plan 9

Microsoft shipped Plan 9 filesystems! Starting with Windows v1903, the Windows Subsystem for Linux now includes the ability to access your Linux files. To a casual user it’s very simple; you just open paths like \\wsl$\Ubuntu-18.04\home\nelson\.bashrc in explorer.exe or VS.Code or whatever and it all just works. Quite well too, I’m impressed. So far the only hiccup I’ve noticed is it doesn’t support symlinks, not even ones internal to the Linux filesystem.

Behind the scenes there’s a Plan 9 network filesystem involved. That’s not as weird as it sounds, I wrote some notes on it awhile back. Now Microsoft has released a technical overview video. Here’s some highlights:

  • Plan 9’s filesystem is a very simple network filesystem protocol to share files between systems. They are specifically using 9P2000.L.
  • They considered using Samba and SMB instead but can’t rely on Samba being installed and usable in the Linux guest OS and didn’t want to ship it because Samba is GPL licensed.
  • They picked Plan 9 because it’s much simpler to implement. Also Microsoft already had Plan 9 server code for some other Linux container project they’d done.
  • The \\wsl$\ path is handled in the Windows system by the MUP, an existing hook for network-like filesystems. They added a new one for Plan 9.
  • The $ is in the name so that it can’t be confused with a computer whose hostname is wsl.
  • The Plan 9 server in Linux communicates with the Windows Plan 9 client via a Unix socket. (Windows supports Unix sockets; who knew?)
  • Windows can access your Linux files even if no Linux is instance is running. There’s a new Windows service called LXSManagerUser that mediates user identity and permissions.

I continue to be impressed with the quality of WSL; it’s like a real Linux system in almost every way that matters. Microsoft has big plans for a second implementation now called WSL2 that will run a full lightweight VM. I’m not as excited about that but they say it will help improve filesystem performance, something that WSL definitely could use.

Pi-Hole problem with dnsmasqd, LXD

This is for the search engines. Installing Pi-Hole on an Ubuntu 18.10 system it wouldn’t start the DNS server. Looking at “service pi-hole status” I see this error:

May 31 15:00:05 gvl pihole-FTL[30005]: dnsmasq: cannot access /etc/dnsmasq.d/lxd: No such file or directory

For once Googling the error didn’t get me anywhere useful. I finally looked and found this symlink:

# ll /etc/dnsmasq.d/lxd
lrwxrwxrwx 1 root root 28 Apr 26 2018 lxd -> /etc/dnsmasq.d-available/lxd

However the entire directory /etc/dnsmasq.d-available/ doesn’t exist. So it’s a dangling symlink. Apparently dnsmasq.d refuses to start if it finds one of those, doesn’t just skip it. Removing that dangling symlink fixed the problem and now Pi-Hole starts.

No idea where the file came in. I tinkered with LXD on this machine in the past, maybe that’s a remnant of something I installed and then removed.

SMS vs iMessage vs Android Messages

I sure wish Android and Apple would get together and unify their messaging apps. The business with iMessage being different from SMS causes all sorts of problems. I mean mostly it’s great, it’s cheap/free to iMessage and it’s faster and more featureful. Right until the moment it fails and you really want SMS instead. Or you switch away from iOS. Apple has finally made it easy enough to de-register from iMessage, but now all my old Apple friends are sorta mystified how to message me. (Derp, the same way, it’s just green now.)

I’m more confused about what Messages from Google for Android is. It seems to be a lot like iMessage, including proprietary non-SMS transport for two users who are both using it. It does SMS too. It also has a webapp version that seems to work pretty well. (So does iMessage, a desktop app, but it’s MacOS only because Apple.) The webapp sends SMS if necessary; does it do that via my Android’s cell plan somehow? Directly? It is all a mystery.

Mostly it just feels like both companies have inserted them in front of standard SMS/MMS. And they’re adding value but in competitive ways that make it worse for consumers. No one wants that.