How PoB Item Tester works

I’ve been obsessed with the Diablo-like game Path of Exile for the past few months. Great game. For the code nerd in me, I’ve been fascinated by how the community has created a bunch of addons and tools to run with the game, despite the game having no support for them. The kludges are thick and woolly and fascinating.

The primary way most of these addons work is via AutoHotKey, a Windows-specific scripting system that occupies a role a little like AppleScript does on MacOS. AHK has been around forever; its main feature is that it’s very good at capturing global keyboard events and doing odd things in the background that don’t fit the usual GUI paradigm. I use it to remap CapsLock to search Google, for instance.

PoB Item Tester is the most complex addon I know. You hover over an item in the game, press Ctrl-C to copy its description to the clipboard, then press Ctrl-Win-C to have a window popup that shows what the item would do if you used it. Behind the scenes it’s using Path of Building to do calculations that simulate game behavor. PoB is a standalone Windows program that itself is a tour-de-force of reverse engineering. It is a full GUI program written in Lua (!).

Functionality aside, it’s the integration that I’m interested in. Here’s what the AutoHotKey script does:

  1. Hook Control-Win-C with AHK to get the item from the Windows clipboard and pass it to TestItemFromClipboard
  2. Write the item to a file on disk
  3. Invoke a Lua script from AHK
  4. Pop up a window based on some GUI setup done previously.
  5. Get the file created by the Lua script and display the contents

So the other half of the integration is in Lua. It has a whole Mock Lua UI system to allow PoB to run headless. Some details on what the Lua script run from AHK is doing:

  1. Set up the Mock GUI. Buried in there is the part that launches Path of Building’s Lua code.
  2. Tell PoB to load the build to test against
  3. Create an Item object based on the file written by AHK (from the clipboard). These seems to do the actual simulation calculations.
  4. Capture the item’s tooltip as HTML and write it to a file for AHK to display.

So there you have it. The game sure wasn’t designed to be scripted, but clever use of the clipboard and keyboard hooks lets people write scripts that feel like they’re running in-game. Path of Building wasn’t written to be scripted either, but someone figured out how to hack the Lua to get it to run headless and do calculations for them. That’s a lot of very motivated hackery, I’m impressed.

Amazon Fire 7 first impressions

My switch to an Android phone has gone so well I wanted to get an Android tablet, too. Only to learn Android tablets are universally considered to suck. OTOH they are also very cheap. So I picked up a brand-new Fire 7 9th generation. For $49. 16GB storage, 1GB RAM, two cameras, a full WiFi multitasking computer with a decent touchscreen display for less than the price of a fancy dinner. We live in an age of wonder. (Note 16GB storage is not enough, but it has a microSD slot. More wonders: 256GB of SD storage for $40.)

So how is the tablet? OK I guess. It seems fine for reading books and watching videos, which is what Amazon sells this for. The screen doesn’t feel quite responsive to my touches all the time but I’m not sure if that’s a hardware or software problem. 1GB of RAM really isn’t enough. CPU may be a bit sluggish. Camera quality is appalling. It comes with a bunch of adware (you can pay $15 to remove it). Did I mention it’s $49? Amazing.

The OS is FireOS 6.3.1.2, an Android 7.1 / Nougat variant. It’s ugly. It only supports the Amazon Fire store which has an anemic collection of apps. (No Firefox! No Kiwix!) The Amazon versions of apps aren’t great, like apparently the Silk browser is doing some MITM attack to try to save you bandwidth. However the device is unlocked, or at least it’s not hard to install the Google Play store and from there a bunch of normal Google Play apps. (One caveat; be sure to install the Google APKs to internal storage. Don’t format the SD card as an extension of internal storage and let the apps install to it, because then they can’t be updated via the Google Play store.)

So 30 minutes of installing later and I have a good generic Android device. Firefox for the browser. GBoard for the keyboard. Evie for the launcher, (which requires more work to launch via the launch button). Alles bueno.

I can’t find some things via the Google Play store even though they’re there for other devices; Tapet, 1Password, etc. Maybe these apps don’t explicitly mark they’re compatible with the Fire and so are excluded? I’ve been able to sideload these from APKMirror but I’m a little nervous about that, I like my trusted app stores. Particularly 1Password, brrr. All the apps I’ve sideloaded seem to work fine, so I’m not sure why Google Play isn’t giving me the choice to install them.

All in all I’m reasonably impressed. It’s not as nice as an iPad, nor even really as nice as my Pixel 3 phone. But it’s literally $49 and it works pretty well. And now I have an Android device to tinker with.

How much Mailchimp is spam? What should they do?

A discussion on Twitter led me to wonder; is all the mail I get from Mailchimp spam? Can I filter it? Turns out filtering Mailchimp in Gmail is hard. But I have a mediocre solution and some ideas on what Mailchimp should do to be kinder to email recipients.

I only notice Mailchimp mail when I get some crap I don’t want to see and click the Unsubscribe link. I get a lot of these same spams from the same company, like my old college or my local pizza place. The landing page for unsubscribing is distinctly Mailchimp, which is when I notice that they’re the source of the spam.

What seems to happen is a Mailchimp customer creates a new Mailchimp list for every marketing campaign. So I won’t get any more June 5 Campaign mails, but I’m bound to get the next June 12 Campaign mail and there’s no way to make it stop. It is spam. My question is; is all Mailchimp mail spam? (Spoiler; only 93% of mine in the last year is.)

Identifying Mailchimp in Gmail

My mediocre solution for identifying Mailchimp mail is a gmail search for three special phrases in the message body, the current copy Mailchimp includes to provide their (not-very-effective) opt-out mechanisms. This text could change at any time, but it seems to be consistent for several years now. Here’s the search term, all three phrases:

“view this email in your browser”
“want to change how you receive these emails?”
“You can update your preferences or unsubscribe”

The better way to filter would be on an email header like X-Mailer, Received, or the DKIM signature. Gmail’s filters don’t seem to allow filtering on any of those criteria. Here’s an example spammy mail; maybe someone else has a better idea how to get Gmail to identify these. (There is a possibility involving Google Script but I can’t tell if that’s really a good idea to use.)

My Mailchimp statistics

In the past year I’ve gotten 57 Mailchimp mails. Maybe more if they went to the Spam folder and were auto-deleted. I have deliberately subscribed to 0 Mailchimp mailing lists, so in some sense all 57 are spam.

However, a whole lot of these Mailchimp mails come from organizations I legitimately have business with. Reed College being the primary culprit, also my local pizza place. I’d rather not get any of those emails, but they are in some sense organizations that have a reason and a right to email me. 54 of the 57 Mailchimp mails I got are in this category, or 93%. I’d rather not get any more of them.

3 of the 57 Mailchimp mails were things I actually wanted to get. Two from the XOXO conference, one from OpenStreetMap. Those organizations use Mailchimp responsibly in a way that helps me and even if I didn’t explicitly opt-in to a mailing list. I want these mails.

0 of the 57 Mailchimp mails were pure spam. None of them were from an organization I’d never heard of before. So that’s good! Update: not a day after writing this I got a pure spam Mailchimp mail, an ad for a Kickstarter game called “Cracked”.

Still, a 93% feels-spammy rate is pretty awful and Mailchimp should do better.

Mailchimp for recipients

AFAIK Mailchimp currently has essentially no product for email recipients. We’re not customers, we’re the product Mailchimp is selling. There’s this polite veneer that somehow we’ve opted in to these communications, and Mailchimp does provide opt-outs. But they aren’t really working.

My proposed solution is Mailchimp should build a small product for email recipients. Leave the defaults as-is, to not decimate Mailchimp’s business. But give users the chance to say “I never want to receive any Mailchimp mail from this organization again”. Not just opt-out of a specific list, but all lists from a particular Mailchimp customer. It’s not simple; Mailchimp would have to create an account for each email recipient with some small Web UI, passwords, etc. It’s not a tiny project, but it’s not a huge one either.

Of course Mailchimp won’t do this. They have no incentive to. Right until the moment someone like me gets fed up enough and considers blocking all of Mailchimp as spam. Imagine if Gmail did that; it would absolutely crush the company. Mailchimp should have a solution in hand before their spam problem escalates to the level of existential crisis.

(Even better would be to require an explicit opt-in from each organization, or for each mailing list. That’s how it used to work in the old Mailman days. But no way would Mailchimp do this; it’d destroy their business.)

An alternate thing Mailchimp could do is get more aggressive at policing their customers. There’s a feedback form when you unsubscribe that includes lines “This is spam and should be reported” as well as “I never subscribed to this list”. I faithfully click one or the other every time I get spam via Mailchimp, in the hopes it creates some signal inside the company and maybe their customer will be reined in. So far I see it having zero effect though, and why would it? The spammer is who is paying Mailchimp, not me.

I have several friends who work at Mailchimp. It’s a good company and they have a good product for their customers. But as a company I suspect they are myopic, focussed only on their paying customers. I’d like them to do more to stop spammy use of their product before it becomes a crisis.

WSL access to Linux files via Plan 9

Microsoft shipped Plan 9 filesystems! Starting with Windows v1903, the Windows Subsystem for Linux now includes the ability to access your Linux files. To a casual user it’s very simple; you just open paths like \\wsl$\Ubuntu-18.04\home\nelson\.bashrc in explorer.exe or VS.Code or whatever and it all just works. Quite well too, I’m impressed. So far the only hiccup I’ve noticed is it doesn’t support symlinks, not even ones internal to the Linux filesystem.

Behind the scenes there’s a Plan 9 network filesystem involved. That’s not as weird as it sounds, I wrote some notes on it awhile back. Now Microsoft has released a technical overview video. Here’s some highlights:

  • Plan 9’s filesystem is a very simple network filesystem protocol to share files between systems. They are specifically using 9P2000.L.
  • They considered using Samba and SMB instead but can’t rely on Samba being installed and usable in the Linux guest OS and didn’t want to ship it because Samba is GPL licensed.
  • They picked Plan 9 because it’s much simpler to implement. Also Microsoft already had Plan 9 server code for some other Linux container project they’d done.
  • The \\wsl$\ path is handled in the Windows system by the MUP, an existing hook for network-like filesystems. They added a new one for Plan 9.
  • The $ is in the name so that it can’t be confused with a computer whose hostname is wsl.
  • The Plan 9 server in Linux communicates with the Windows Plan 9 client via a Unix socket. (Windows supports Unix sockets; who knew?)
  • Windows can access your Linux files even if no Linux is instance is running. There’s a new Windows service called LXSManagerUser that mediates user identity and permissions.

I continue to be impressed with the quality of WSL; it’s like a real Linux system in almost every way that matters. Microsoft has big plans for a second implementation now called WSL2 that will run a full lightweight VM. I’m not as excited about that but they say it will help improve filesystem performance, something that WSL definitely could use.

Pi-Hole problem with dnsmasqd, LXD

This is for the search engines. Installing Pi-Hole on an Ubuntu 18.10 system it wouldn’t start the DNS server. Looking at “service pi-hole status” I see this error:

May 31 15:00:05 gvl pihole-FTL[30005]: dnsmasq: cannot access /etc/dnsmasq.d/lxd: No such file or directory

For once Googling the error didn’t get me anywhere useful. I finally looked and found this symlink:

# ll /etc/dnsmasq.d/lxd
lrwxrwxrwx 1 root root 28 Apr 26 2018 lxd -> /etc/dnsmasq.d-available/lxd

However the entire directory /etc/dnsmasq.d-available/ doesn’t exist. So it’s a dangling symlink. Apparently dnsmasq.d refuses to start if it finds one of those, doesn’t just skip it. Removing that dangling symlink fixed the problem and now Pi-Hole starts.

No idea where the file came in. I tinkered with LXD on this machine in the past, maybe that’s a remnant of something I installed and then removed.

SMS vs iMessage vs Android Messages

I sure wish Android and Apple would get together and unify their messaging apps. The business with iMessage being different from SMS causes all sorts of problems. I mean mostly it’s great, it’s cheap/free to iMessage and it’s faster and more featureful. Right until the moment it fails and you really want SMS instead. Or you switch away from iOS. Apple has finally made it easy enough to de-register from iMessage, but now all my old Apple friends are sorta mystified how to message me. (Derp, the same way, it’s just green now.)

I’m more confused about what Messages from Google for Android is. It seems to be a lot like iMessage, including proprietary non-SMS transport for two users who are both using it. It does SMS too. It also has a webapp version that seems to work pretty well. (So does iMessage, a desktop app, but it’s MacOS only because Apple.) The webapp sends SMS if necessary; does it do that via my Android’s cell plan somehow? Directly? It is all a mystery.

Mostly it just feels like both companies have inserted them in front of standard SMS/MMS. And they’re adding value but in competitive ways that make it worse for consumers. No one wants that.

GPSLogger detailed notes

I’ve been running Mendhak’s GPS Logger for a few weeks now on Android. It’s good! Some details on precisely what it’s logging. I’m using this as notes for building my own GPS tracker for Wanderings. GPS Logger is good but I think it’s better to have a custom one for us.

Bottom line: the CSV format is most complete, GeoJSON has most of what you’d want in a GIS-friendly format. It logs points once a minute by itself in a mix of network and GPS modes. When a location tracking app like Google Maps is also running, GPS logger will logs points as often as once a second. Now the details…

Some of this is configuration dependent; I have it set to Log all 3 sources: GPS/GNSS, network (cell/wifi), and also passive (other apps). Logging once a minute, GPS not turned on between fixes, no distance filter. Accuracy filter dialed down to 200 meters. 60 seconds to get an accurate fix, 120 second timeout on the GPS fix. I’m logging simultaneously to GPX 1.0, CSV, and GeoJSON.

Most of this data comes from 2017-05-26, when I started in Hickory, NC and flew home to San Francisco, CA. Most of these observations are based on the CSV file which the author has said is the most comprehensive.

The data files are one-per-day. They seem to contain data from local midnight to local midnight. There are sometimes gaps of several hours, I believe related to low power / sleep modes in the phone. Particularly common in the middle of the night when I’m not moving.

On 2017-05-26 the CSV file has 6975 data points.

When I’m sitting idle in my hotel most of the fixes are of type “network”. There’s typically one entry per minute (with frequently two duplicated rows). The CSV file for those populates time, lat, lon, elevation, accuracy (typically around 20), and battery. Occasionally a fix of type “gps” also shows up; those have an accuracy of about 10. They also add bearing and speed, although when not moving those are presumably nonsense. Also HDOP VDOP PDOP and GeoidHeight, GPS-specific measurements. There’s a field for ‘satellites’ but it always seems to be 0.

Once I start driving around 14:50Z I also turned on Google Maps for routing. At that point all the fixes switch to GPS and I’m getting one a second. Accuracy is around 3 and satellites still 0; no HDOP/VDOP etc though. Plausible bearing and speed data. I’m guessing these are points GPS Logger got passively? This pattern is repeated a second time on a longer drive. Google Maps for navigation again then although I’d made an effort to turn the screen off for part of the drive.

During my flight from CLT to SFO I had the phone in airplane mode and in various pockets, etc with varying ability to see the sky. GPS Logger seems to be trying to record 1 point a minute. When it succeeds there’s two duplcate rows in the CSV, one with HDOP/VDOP data and one without. Altitude, speed, and bearing all look plausible. I mostly have no data from when I had the phone in my pants’ pocket; shirt pocket is better for obvious reasons.

Last bit of the day was me in an Uber, no Google Maps running. Odd mix of GPS and Network fixes about once a minute. It’s possible the Uber app was doing location tracking in the background, I’m not sure.

The 2019-07-26 GeoJSON file also has 6975 data samples in it. Every point has lat, lon, time, time_long, provider, accuracy. Almost every point has altitude (6759), 5032 have speed, 4094 have bearing.

The GPX 1.0 file also has 6975 data samples in it. lat, lon, elevation, time, and provider for almost all points. course and speed for GPS points also. There is no accuracy logged.

A special note on elevation: these seem to be WGS84 altitudes, before the geoid correction. (I have Use MSL instead turned off in the app.) That’s putting me underwater frequently in San Francisco, even the points from network sources. Need to think more carefully about geoid correction if I want to use this data.

Summary of all fields available by log type:

  • lat / lon: all
  • time: all
  • provider: all
  • accuracy: csv, geojson
  • altitude: all
  • speed / bearing: all (GPS source only?)
  • battery: csv
  • hdop/vdop/pdop: csv
  • GPS satellites: none (csv has them, but always 0)
  • GeoidHeight: csv