Heatmaps for geopoints?

Trying to figure out how to generate actual heatmaps for a bunch of geopoints. The usual hack is to draw blurry dots at 10% opacity and let them overlap. I want the real deal. The Sethoscope tool works in Python, but is pretty slow, see example below. Mourner at Mapbox is working on a WebGL thing.

The hard part here is heatmaps aren’t scale invariant; you need to render it differently depending on the scale the user is zoomed too. Realistically that means it needs to be in browser Javascript, and therefore fast. Maybe GL is necessary.

Razer Blade 2017 opinions

I bought a laptop for my trip to Germany. Difficulty level: ultrabook, but able to play Windows games. Turns out you don’t have a lot of options. I ended up with a 2017 Razer Blade. Some thoughts:

It’s accomplished its goal, I like the machine. In ordinary use it’s a quiet nice laptop. Good keyboard, nice screen, and it’s silent. Light 2d games (Necrosphere, Heat Signature) also run silently on the Intel GPU. When I engage a demanding game (XCOM 2 mostly these days) the NVidia GPU kicks in and soon after, the giant ridiculous loud fans.

The fan noise is really awful. Like don’t make a phone call in the same room awful. it seems to mostly be turbulence sucking air in from the 1/8″ gap under the laptop. But with earplug-style headphones and in a different room from other humans it’s OK. Heat is acceptable. This is the compromise one has to live with putting a GPU this big in a laptop. I kinda wish I could have bought a lower power / cooler GPU. Or maybe figure out a way to underclock this one. Games autodetect highest settings and when i turned them down it’s a bit quieter.

My main complaint as a sorta-Ultrabook is the weight. 4 lbs is noticeably heavier than my old 3 lb Macbook Air. But then the machine is much more powerful, too. I also wish the screen didn’t have such an enormous bezel. I think most laptops are now sized to the panel, but I suspect this Razer machine is sized to something else and they picked the biggest panel that’d fit and it’ about an inch shy. The speakers in the machine aren’t great either, another unappreciated thing Apple does.

But pretty much this Razer Blade is a nice machine. I think I chose right. I continue to wish someone made a true ultrabook with just a slightly nicer GPU in it than stock. At this rate Intel’s gonna catch up before that happens.

MQTT, OwnTracks, and location tracking

I want a passive location tracker on my phone, something to tell me where I’ve been over the span of years. Google Timeline + Android does this pretty nicely and this blog also has previous notes on Fog of War and other gamified apps. I was using OpenPaths on my iPhone to do something similar, but it was abandoned. The backing server is already gone and the app will die with iOS 11.

So what to do? A friend suggested OwnTracks, an iOS tracker. This isn’t civilian software; there’s no public service. The docs are all about running your own custom server to collect your location data. But the iOS tracker is capable. In particular you can choose battery vs. accuracy. You can either record your position every N seconds / every M meters, which takes about 10% of your battery an hour. Or you can use a more passive “significant movement” mode that is less sensitive but uses very little battery, maybe 1% an hour. I think that’s what OpenPaths did; I ran it for years without a problem. (There’s also a newer iOS API for “Visits” that also looks useful).

OwnTracks publishes data via MQTT, a message queue protocol designed for Internet of Things apps. It seems awfully complicated and a bit fusty, but if they’ve thought through reliable, low power connections for limited devices I’m glad to use it. The OwnTracks docs have some info on running your own MQTT server but that seemed complicated, so I’ve started with CloudMQTT which has a nice free level.

I created a CloudMQTT account and then just plugged in the Instance Info username/password into OwnTracks and set it to logging. CloudMQTT has a “Websocket UI” that shows you the message stream to verify the plumbing is working. OwnTracks reports look like this:

{
"batt":98,
"lon":13.41240532107618,
"acc":1414,
"vac":10,
"lat":52.53526024421524,
"t":"t",
"_cp":true,
"conn":"m",
"tst":1505981237,
"alt":53,
"_type":"location",
"tid":"fi"
}

So that all works. Now what about capturing that data for myself? After some false starts I ended up at paho-mqtt, a Python client. The example client on that page more or less works, except you have to modify the on_connect() function to take a fourth flags parameter. (That’s a hint this code is musty.) The example connects to some public MQTT endpoint with a bunch of events flying by so you know it worked. Eventually one sent non-UTF8 and broke the Python code, but at least it was working.

Last step was to reconfigure the sample client to connect to my CloudMQTT instance. To get that working I had to create a new user and set it up with the ACL controls. (Wildcard access, read/write). For some reason the main user/login didn’t work; maybe because it was in use by OwnTracks? I also had to call client.subscribe() to get messages on my specific OwnTracks topic; not sure if that is supposed to be necessary.

Cloud MQTT also has support for forwarding events to Amazon Kinesis. I set this up hoping it was an easy way to get at the data but it just adds complexity and another system for me right now. There’s probably some way to feed the data into a database with AWS plumbing that might be nice. And it’ll scale!

Things to learn more about with MQTT:

• Is MQTT a living tech, or is it abandoned?
• CloudMQTT user management
• paho.mqtt: naked TCP vs TLS vs Websockets
• MQTT topics and subscriptions
• Is Amazon Kinesis interesting for my application? Would be nice to have less code + servers to manage

 

Berlin home Internet link

More details on my Berlin Internet sitch. My apartment has a Technicolor router that comes from PrimaCom’s cable modem service. It was down entirely for several days (and not just for me) when we got here, which is not so awesome. Mobile hotspots are a poor alternative.

The downtime revealed PrimaCom manages the router WiFi. When the Internet was down the WiFi was also turned off. You couldn’t even turn it on in the admin pages (default login: username blank, password admin). Once Internet service was working again and I rebooted the router, I had WiFi. Weird.

The bummer about the router, or the larger network, is that it just forgets about idle TCP sockets after about 5.5 minutes. Idle ssh sessions and other quiet TCP links just stop working. Not even a reset, just a timeout, and it’s consistently timed so it’s not a FIFO queue filling up. There’s nothing in the router config that’s obviously causing the problem.

Of course web browsers never even notice, so most of their customers have no idea. I wonder if it breaks notifications via WebSocket though. For ssh I work around it with ServerAliveInterval=50. I suppose TCP keepalives would fix it in general but that’s a half-abandoned tech and the default interval in Windows is 2 hours (!) so you have to fiddle the registry. Ken has some weird TN3270 sessions he uses, I think SSL and telnet. We fixed his timeouts by just using Tunnelbear VPN which papers over the problem. I really should run my own VPN server.

We never could connect to an old school PPTP VPN. The TCP control channel for setup seems to have worked (according to Wireshark) but no GRE packets ever arrived.

The fun thing is I have an honest-to-goodness IPv6 address and all of it works. Sometimes I seem to be connecting to Google via IPv6. I’m not quite sure what to do with this capability, but it’s novel for me. I wonder if the router is still doing NAT for IPv6? The IP address a test site shows me is the same one my Windows box tells me I have, so I guess not. The router does have an “IPv6 firewall” enabled though, perhaps that provides basic protection from outside attacks? I should set up a remote IPv6 box to play around with this.

Update: Primacom seems woefully underprovisioned. In the morning I get about 5000kbps, bursting to 10Mbps. In the evening I’m doing good to get 500kbps, 1/10th the speed. I assume that’s contention with my other cable modem users. 500kbps isn’t really enough to even stream video.

Here’s a traceroute on IPv4

|------------------------------------------------------------------------------------------|
|                                      WinMTR statistics                                   |
|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
|                             192.168.0.1 -    0 |    3 |    3 |    2 |    5 |   10 |    5 |
|                              10.195.0.1 -    0 |    3 |    3 |   10 |   20 |   41 |   41 |
|                            172.16.0.201 -    0 |    3 |    3 |   15 |   24 |   42 |   42 |
|                            172.16.0.109 -    0 |    3 |    3 |   15 |   24 |   42 |   42 |
|                             172.18.0.53 -    0 |    3 |    3 |   16 |   25 |   43 |   43 |
|        ae63.edge7.Frankfurt1.Level3.net -    0 |    3 |    3 |   24 |   30 |   43 |   43 |
|   ae-2-3210.edge4.Frankfurt1.Level3.net -    0 |    3 |    3 |   30 |   34 |   44 |   44 |
|             10ge10-15.core1.fra1.he.net -    0 |    3 |    3 |   27 |   35 |   50 |   50 |
|              100ge5-2.core1.par2.he.net -    0 |    3 |    3 |   74 |   81 |   93 |   93 |
|             100ge14-1.core1.nyc4.he.net -    0 |    3 |    3 |  114 |  124 |  143 |  143 |
|              100ge9-1.core2.chi1.he.net -    0 |    3 |    3 |  143 |  156 |  163 |  143 |
|             100ge12-1.core1.mci3.he.net -    0 |    3 |    3 |  141 |  154 |  162 |  141 |
|               10ge4-1.core1.mci2.he.net -    0 |    3 |    3 |  142 |  155 |  163 |  142 |
|wholesale-internet-inc.10gigabitethernet1-3.core1.mci2.he.net -    0 |    3 |    3 |  142 |  155 |  163 |  142 |
|lag-to-oak.edge-a.clay1.mci.us.as32097.net -   34 |    3 |    2 |  160 |  161 |  163 |  160 |
|lag-core-b.dist-1-3.clay1.mci.us.as32097.net -    0 |    3 |    3 |  146 |  156 |  163 |  146 |
|                         wk.somebits.com -    0 |    3 |    3 |  145 |  157 |  163 |  145 |
|________________________________________________|______|______|______|______|______|______|

Powerline ethernet

I’m spending a few weeks in an apartment in Berlin that’s a large sprawling place. The single wifi router at one end has no hope of reaching the other end and there’s no ethernet in the house. So I took a $50 gamble on a powerline ethernet kit and it seems to work, at least up to 30Mbps. We’ll see if it lasts; my friend says in his experience these things fail after a month or two.

The plug-n-play experience out of the box is pretty great. Plug both devices in, press the pair button, and you’re done. It acts more or less like a bridged ethernet. Plug one end into your router, another end into a computer, and you’re done.

One wrinkle here is security. Your network is being carried out on power lines and who knows how far they’ll go. So the products all have a shared key encryption mechanism (pairing is key exchange) to keep your network private. They probably leak all sorts of RF out the modulated wires, oops.

TP-Link makes a huge variety of these devices. My box says “AV600 TL-PA4020P Kit”, a variant I can’t even find anywhere, but it seems about the same as the AV500 versions only 100Mbps faster. The choices seem to be power passthrough or not, 1-3 ethernet ports per device, and then more expensive for higher bandwidth (up to 1.2Gbps). Different versions for different countries too, both the electrical plug shape and radio frequency interference concerns. Some versions also have a WiFi access point built in, I kind of wish I’d bought one but I wanted to keep it simple and cheap.

The relevant network standard is HomePlug, some vendor consortium for powerline networking. (They just dissolved last year and put all the specs in the public domain.) In theory there’s vendor interoperability. I don’t quite know what’s going on, the Wikipedia article talks about 1000+ channels from 2 to 86MHz.

The really interesting play here is if all your Internet of Things devices supported this kind of networking. Just plug them into the power; no wifi or ethernet required. I wonder why that’s not more popular, I suspect it’s because of the rumored flakiness of powerline ethernet.

 

Converting video to x265 with avconv

A simple command line for converting video to small x265. These options are chosen to produce acceptably low quality, low bitrate output.

avconv -i in.mkv -c:v libx265 -c:a copy -crf 29 -s 854×480 -preset fast out.mkv

• c:v libx265 means “encode using x265”.
• c:a copy means “copy audio, don’t re-encode”
• crf 29 means “slightly worse than default 28 quality”
• s 854×480 means “resize to 480p”
• preset fast means “I don’t have a hardware encoder so please don’t take forever”

Obviously better quality is possible, this is producing about 500-600kbit/second video or about 4x the same of the audio track.

Speaking of audio track it’d be nice to convert whatever the source is to 128kbps AAC stereo. I’m not clear if a simple “-ac 2” is sufficient or if more complex downmixing is necessary.

Windows: typing European on American keyboard

Apple has this great way to type non-ASCII characters like Ö or ß or € on both MacOS and iOS. You long-press the key that looks similar to what you want (an O, or an S, or a $) and then wait a bit and a UI pops up and you choose what you want from a UI picker. It’s not great for fast touch typing but for that you’d want a full language keyboard. But for wanting to type the occasional non-English word, it’s great.

Windows, not so much. The frequent advice is type it by character code as if I’m going to memorize 153 and 128 and 223. Not to mention my laptop doesn’t have a numeric keypad. The advice to use the Character Map in that post is not so awful, it works OK, but it’s pretty clumsy.

The best bet seems to be the US international keyboard. It’s an alternate American keyboard where ‘ ” ` ~ ^ are dead keys; they’ll modify the next letter to be accented, or else insert themselves if that makes no sense. It works pretty well. Even better the right Alt key can be used to type a bunch of other stuff like ß or €. It’s not perfect but it’s pretty good. Note the Windows 10 anniversary edition moved this; it’s now listed as an “Option” on the English (US) keyboard, not a separate keyboard.

I also tried Holdkey which promises to add an Apple-like typing UI. But it didn’t work very well, the way it grabs onto the keyboard is problematic. There’s some other options along these lines I didn’t try.

Update: if I type y’all I get yáll which I think is hilarious.

Update 2: by default Ctrl-Shift switches between keyboard layouts. That’s obnoxiously easy to press by accident. You can turn that off or change the key mapping.