Sometimes it’s useful to set a clock to UTC. Say, your camera when you’re travelling; better to label all the photos with one consistent time IMHO. Some consumer UIs don’t let you choose UTC or GMT since they aren’t attached directly to a geolocation. In that case I always set the clock to Atlantic/Reykjavik time; they haven’t done daylight savings since 1967.
The full list of UTC-like timezones are Africa/Abidjan, Africa/Bissau, Africa/Monrovia, Africa/Sao_Tome, Atlantic/Reykjavik, and America/Danmarkshavn. (Not clear many places would recognize that last one, it’s pretty obscure.) I found these by searching the database for ^\s+0:00\s+-
The timezone database is a thing of beauty as Jon Udell so nicely articulated. Sadly the text contents of all the fascinating comments aren’t easily available as a web page; I should do a project to make a static site of them.
I have a CO2 sensor, a Kaiterra Laser Egg. Home Assistant has support for getting data from it but I don’t much like HASS UI. Or data retention; I was hoping it had stored a year of data but I learned the little sqlite database has a 10 day expiration. Oops. Anyway I decided to roll my own collector using Telegraf, for InfluxDB and Grafana.
The Kaiterra doesn’t have any LAN API I could find. Instead the devices upload to the cloud and they have a decent API service. It’s mostly pretty straightforward REST. One trick is you have to find your device ID. I can’t find that in any UI anywhere, I finally got it by having Kaiterra mail me my data, the data dump included the device ID.
There’s an official Kaiterra Python client but it’s 2 years old and pretty wonky. It also uses an old API endpoint which among other janky things has invalid JSON (single quotes, ugh). So I just wrote my own code using urllib and json in Python. One goal here was to make this not depend on any virtual environment, so stock libraries only.
It’s 2021 and Python still has no built-in way to import the common ISO-8601 timestamps included in every JSON file, like 2012-04-23T18:25:43.511Z. Plenty of third party libraries to do it. There’s a datetime.datetime.fromisoformat but the joke is on us, it doesn’t support the trailing Z. There’s a years-long argument about why that’s OK, basically the Python authors don’t actually want to support all of ISO 8601 in the method named “from ISO format” nor do they recognize the value of parsing the most common text format in the world. Fortunately you can just chop the Z off and it’ll probably work.
I’m running this as a Telegraf exec agent, outputting data in Influx format. That’s mostly pretty easy with the wrinkle that they want unix-epoch-like timestamps only expressed in nanoseconds. So I’m appending 000000000 to my time outputs, lol.
I think it’d be possible to not write any Python code at all and use a cleverly configured Telegraf http agent with a JSON parser, but I didn’t want to try to debug that so I wrote Python code instead.
Anyway, here’s the telegraf config snippet and the python code. I’m still tweaking the Grafana layout and don’t feel like pasting a giant ugly wodge of Grafana config anyway.
#!/usr/bin/python3.8
"""Telegraf / InfluxDB client to import data from a Kaiterra LaserEgg
Does not require any venv or external libraries. Does require Python 3.7 features.
"""
import urllib
api_key = YOUR API KEY HERE
device_id = YOUR UUID HERE
endpoint = 'https://api.kaiterra.com'
def urllib_client():
"Simple HTTP request for Kaiterra Laser Egg data"
import urllib.request, json, datetime
url = f'{endpoint}/v1/devices/{device_id}/top?key={api_key}'
expected_types = {'rco2': 'ppm', 'rhumid': '%', 'rpm10c': 'µg/m³', 'rpm25c': 'µg/m³', 'rtemp': 'C'}
# Make the request
req = urllib.request.urlopen(url)
json_text = req.read()
json_data = json.loads(json_text)
# Parse the data into a simple Python struct
record = {}
record['ts'] = json_data['data'][0]['points'][0]['ts']
for d in json_data['data']:
assert d['units'] == expected_types[d['param']]
point = d['points'][0]
assert point['ts'] == record['ts']
record[d['param']] = point['value']
# Convert ISO time with appended Z to Unix timestamp
record['unix_ts'] = datetime.datetime.timestamp(datetime.datetime.fromisoformat(record['ts'][:-1]))
return record
r = urllib_client()
values = ((f'{k}={r[k]}') for k in ('rco2', 'rhumid', 'rpm10c', 'rpm25c', 'rtemp'))
field_set = ','.join(values)
ts = int(r["unix_ts"])
print(f'laseregg {field_set} {ts}000000000')
I switched note-taking apps to Joplin. It’s total overkill for what I do, all I really want is a markdown editor that doesn’t require me to remember to press “save”. Joplin has a zillion more features than that. Also it’s a 200MB Electron download, so that’s a bit much. But it seems to work well and simply.
Originally I was using SimpleNote. It’s really nice! The only reason I switched away is because it doesn’t support encrypting notes on their cloud storage and I was doing things like pasting passwords into my notebook.
Then I used Standard Notes for a few years. And paid for it. But they hiked the price to $50/year and that seems like a lot of money for something so simple that I use occasionally. It also reminded me just how bad and klunky so much of Standard Notes is. Its reliable on Markdown plugins for the editor, none of which quite work right. Mediocre cross-note search and no in-note search at all (except with one plugin). It often feels slow. Just overall not excited to be paying so much for it. (Base Standard Notes is free but you need to pay to use even the most simple advanced features like a decent Markdown plugin.)
Joplin is free software and free to use, including a variety of self-hosted syncing options. (Which are encrypted). They do offer a cloud sync service at 24€/year but they also have a robust set of free sync options you can set up yourself. You’re mostly paying for convenience. And they don’t lock other basic features behind the price.
I’m using SyncThing for sync. Joplin will sync via files in a directory; it is then the user’s problem how to sync those files. SyncThings works great for that. The sync is a little delayed (5 minutes?) and getting it set up on Android was awkward because of the opaqueness of the file system. But it all worked.
There’s a zillion note-taking apps out there. Unfortunately most of them seem to be too complex and offering too many features, just like Joplin does. Everyone’s like “Evernote is bad and too complex” and then is going down the road to becoming Evernote. To Standard Notes credit they are not doing that; neither is SimpleNote. Anyway some of the other options I didn’t try are QOwnNotes, Cryptee, and SilentNotes. Joplin seems to be the nerdy consensus for the best so I didn’t even try the rest.
Migration wasn’t too bad btw, thanks to Standard Notes for having an export feature. (Which reminds me; data portability matters, don’t ever use an app that traps your data.) There’s a way to convert Standard Notes exports into Evernote format, which Joplin can then ingest. Lost a bunch of metadata, most importantly which notes had been archived. So I had to go in and re-sort my notes. It was time well spent actually, but if I had thousands of notes that would have been a problem.
I’ve been using syncthing a couple of years now and finally trust it to manage all of my source code. Part of that is getting it to sync files on my WSL Ubuntu images. They’re more or less like standalone Linux VMs and need their own syncthing, it won’t work to just sync files on the Windows host system.
Syncthing mostly just works in WSL, no drama. The challenge is that WSL doesn’t have a systemd or cron so there’s no easy way to start syncthing at system boot. Also syncthing itself is not really written as a daemon; if you just run “syncthing” it writes to stdout and listens to signals and stuff. Awkward.
There’s lots of advice on how to deal with this problem and most of it doesn’t work. The real problem I ran into is something weird about WSL, at least the WSL 2 I’m using. This GitHub issue discusses it: even with nohup and other efforts, background tasks launched by wsl.exe will sometimes disappear. You can write a script that looks like it runs a background process fine and test it in Linux and it works. But if you invoke the script via wsl.exe whatever processes it spawned disappear. No, nohup doesn’t help. This is all very mysterious and it’s worth remembering that WSL only started supporting background tasks a couple of year ago.
The solution that worked for me is to use the daemonize program. There’s an Ubuntu package for it. It’s pretty straightforward, it does all the Unix stuff you’re supposed to do for a daemon but don’t bother with (like cd / so that filesystems can be unmounted.)
Here’s the Linux shell script I use to run syncthing. The weird GUI port is because I also have Syncthing running on my Windows host.
It’s not ideal. If I’m logged in and just launch the shortcut it works pretty nicely. A command shell window briefly pops up and disappears; if that annoys me enough there’s kludges to workaround that.
There’s a bigger problem on system startup: the console window pops up and hangs for ~30 seconds. I thought it was broken at first but it eventually goes away on its own. While that was going on I also tried to launch a WSL terminal window to see what was happening; that hung too, until the syncthing starter window closed. My guess is that WSL 2 actually has a significant hidden startup time at Windows boot. Normally a WSL window pops up the moment you ask for it, but maybe Windows actually sets up the VM first time in the background and hopes no one notices. Not really sure though, will need to keep an eye on it.
Update: I asked on Reddit and this delay at startup does seem to be a thing that happens to others.
It’s real nice having my source code auto-synced. I sure hope SyncThing never breaks and deletes all my files. (I do have backups.)
I normally frown on time spent customizing coding environments. But since I’m switching to VS.Code I figured it was time to also consider trying proportional fonts for coding again. It’s ridiculous how we still write code as if our I/O devices were line printers and ttys (or worse, literal punched cards.) Proportional fonts are certainly idiosyncratic for coding but there’s a growing community of folks doing it. GoLang gave it a lot of legitimacy; Rob Pike has been using proportional fonts for decades and who’s going to say he is wrong?
Here’s where I’ve landed so far. To get there I had to set up VS.Code to deal nicely with proportional fonts, then adopt a Python code formatter that put out code that looked good in my setup: YAPF, not Black.
The first chunk sets the font. Note you have to set the terminal font to a fixed width font: by default it will inherit the editor fontFamily and proportional fonts do not work at all in a terminal window. They never will work well, too much Unix console UI assumes fixed width.
The rest of the settings are for making proportional fonts look nice for my Python code. Read below for details.
Font Choice
The nice thing about proportional fonts is you have so many to choose from. Want to program in Zapf Chancery? Go right ahead. (No, don’t do that). One thing to look out for is a few characters special to programming. You want 0/O and l/1/| to be distinguishable. The { braces} and [ brackets ] are also important and often overlooked in non-coding font design. Another thing to think about is the overall width of characters; many programmers want a narrow font to get more on the screen. I don’t.
I started with Calibri;. I like how it looks, it’s calm and innocuous. It’s not designed as a coding font but it’s not bad for the purpose. I’m currently experimenting with the five new Microsoft candidate fonts that fell off the back of a truck. They all look pretty good. None are designed as coding fonts but they all seem to work OK. The screenshot above is Tenorite.
I’d still rather use a proportional font designed for coding. There are precious few of those. Input is the standout and it is a very thoughtful font, including proportional options. I just don’t like the way Input Sans looks. Purely a subjective opinion. The Go Fonts are another good option but again don’t like how it looks, it’s a very opinionated design.
(PS: font installation still sucks in every OS. After you install new TTF fonts in Windows 10, close and reopen the Settings program or else it will hang / crash randomly. You also have to restart VS.Code to make it aware of newly installed fonts.)
Update: since taking the screenshot I added the letterSpacing. It’s a kerning kludge, to add 0.6 pixels of space between all letters. Text was just too tight, particularly things like the empty string ”. A single fixed value isn’t ideal for this but it helps.
Tabs vs Spaces
You can just change the fonts in VS.Code and start coding, things will mostly work. But the indentation isn’t awesome. The space in a proportional font is so narrow that the typical 4 space indent ends up being too narrow (and 2 spaces is impossible).
So I switched to tabs. Yeah, really; this is my embarrassed face. I’ve spent nearly two decades expunging tabs from all my code (except Makefiles and other monstrous systems that require them.) I had clearly chosen sides in programming’s stupidest controversy. But in the back of my head I always knew tabs were “theoretically” better; tab is a semantic character, saying “this is a new level of indentation”, and the fact it’s 2 or 4 spaces shouldn’t matter at all. The tab character is the way to express that logical structure. It’s also the practical way to make Visual Studio work, because it lets you customize the look of the indentation without changing the bytes in the file.
So the second block of my VS.Code setup is setting the tab size to the size of 8 spaces. That “looks right” to me in Tenorite. It happens to also be the standard. For Calibri 12 looks better to me. 12 would look terrible in a fixed width font but you make sure your output only ever has tabs (not 12 actual spaces) and it’s fine. Wouldn’t it be nice if this setting were in terms of an em-space or en-space instead of the tiny space character? In real typesetting spaces are variable width.
Code formatting: YAPF, not Black
Automatic code formatting pairs nicely with proportional fonts. You’ve given up the ability (or requirement) to vertically align blocks of comments and code and stuff. Might as well use a tool to just format everything for you. Again the Go community is a guide here; gofmt is a standard tool. The same principle works for Python too!
I switched to using Black a few days ago and loved having a formatter make choices for me. However Black is very serious about making specific choices and they are very clear they won’t support tabs. Nor anything but 4 spaces. That’s Black’s whole shtick and I kind of like it, right until I don’t.
So I switched to YAPF, Google’s formatter. It’s a whole lot like Black but it’s more configurable. Out of the box there are some slight differences I prefer (fewer vertical lines for braces, etc.) but it basically does PEP8 more or less like Black does. But YAPF invites configuration. I tried to use a light hand here, I really wanted to not do anything too personalized.
The second chunk is all about using tabs instead of spaces for indents. You have to tell it to use tabs twice; for most things, and then specifically for continuations (with the cryptic value valign-right). I’m not positive that the indent_width setting even means anything.
The column limit is.. I don’t know what’s right. 80 columns is very narrow, one nice thing about a proportional font is it typesets a good deal narrower than fixed width. 132 would be a natural choice with precedent. But I’m going to go to (near) infinite. We’ll see how that works out. 999 might be a better choice to catch pathological cases.
The last two entries are preference things for me, idiosyncrasies. I figured hell I’m customizing YAPF already, why not go whole hog!
(PS: if you want to understand what YAPF styles are, look at the --style-help output. The code for making styles is also interesting.)
Drawbacks
As I write this I just set this up today; I haven’t lived with it. We’ll see how it goes.
The primary drawback is interacting with other programmers. Folks would right be mad at me if I submit a 2 line patch to some code when 800 other lines changed reformatting the code. It’s always more important to follow an existing style! But right now I have the luxury of mostly working alone on my own code so I can experiment a bit.
The other drawback is interop. Much of the Unix world (and my own environment) is designed around monospace fonts and 2 or 4 character indents. Things like my .bashrc look weird in this new setup, at least until I convert them to tabs. It’ll never look right in a terminal window which has to be fixed width for most things to work right. I think writing code in an editor is the important use case, so I’m OK optimizing for it, but we’ll see.
I’m also a bit uneasy that this setup is so Python specific, relying on the code formatter. I guess most languages have formatters now though.
After years of using Sublime Text 2 I decided to switch to a newer code editor. The tipping point was wanting to use the Black Python formatter; ST2 doesn’t have support for it. (ST3 does but I don’t own that). VS.Code seems to be the new hotness so I’m giving it a try. So far I like it! Some thoughts:
Good: it feels very fast. It’s amazing how slow so many programs feel on my very fast desktop computer. VS.Code feels fast.
Good: the extension store. There’s a rich ecosystem of addons. I’ve already lost track of what all I’ve installed: Python, WSL, SQLite … I’m still looking for a good JSON extension (the one I tried is very very slow). I’m still understanding the extension model; I keep having to install black and pytest and pylint in every project’s venv. That’s awkward but kinda makes sense.
Bad: It’s an IDE. I don’t like IDEs, I just want a tool to edit text.
Good: It’s an IDE! I can code with tools as good as the Turbo Pascal environment I used in 1986. VS.Code is notable for being a cross-language IDE. It’s also a fairly low impact IDE, you can mostly use it as a text editor to start and it’s fine. But I’ve started making peace with the Python IDE stuff and it works fine. It runs unit tests nicely (albeit without support for continuous testing on save.) The debugger works.
Mixed: linting and PyLance warnings. These are enabled by default and mostly don’t seem to show up. But I just got a warning from PyLance because os.getenv can return None and I was passing its output to a function that needs a string. But I was anticipating that and already had a try/catch block for that. At least you can suppress warnings on a line by line basis with the comment # type: ignore
Good: black. It’s instantly freed me about worrying about code formatting. My only complaint with the default black setup is it does weird multiline formatting if any lines go over 88 characters by default. You can change the line length but the whole point of black is one standard consistent style, so I hate to change that.
Mixed: Intellisense suggestions. It’s very nice when exploring but the default settings are awfully chatty. Currently trying out these settings to tone it down some:
Good: The command palette UI is nice. VS.Code didn’t invent that but seems to lean on it heavily. Emacs users used to typing M-x will feel right at home.
Bad: preferences. For years the only way to change preferences in VS.Code was editing settings.json. There’s now a GUI for it but it’s clearly generated from the JSON specs and it shows. It’s clumsy.
Good: Settings sync. You log in via Microsoft or GitHub and VS.Code syncs your settings, UI state, etc. It’s optional.
Mixed: the look and feel. Something about the very spare UI chrome and the color schemes feels wrong to me. Maybe I’ll get used to it. I’m using the Dark+ theme now and it’s mostly fine, I just feel a bit dislocated. I have a particularly hard time with the filename tabs. This customization might help:
Great: git integration. I don’t think it’s trying to be a full git client but it has a nice UI for quickly seeing what code has changed and staging and committing code. The diff viewer is terrific.
Good: I can launch VS.Code from a WSL shell with code . It automatically loads the VS.code project file for me. It also seems to detect my venv/ directory and use it.
???: security. Every time I load files from a new directory VS.Code asks me if I trust the files in the directory. WTF? I guess VS.Code or some extensions will run code snippets it finds in files you are editing in some circumstances. That seems like a terrible security risk and passing it off to the user is not good.
I have my TV plugged in via HDMI to my Windows machine for occasional gaming. When I want to play games I need to swap the display resolution to one for the TV and also change the sound output device to the HDMI device. I’ve got it set for display mirroring; gaming as a second display sorta works but gets confusing and can require trips back to the mouse and keyboard.
Anyway I got tired of doing it by hand so I scripted it. The key is Nirsoft’s program NirCmd. It gives you a command line interface for doing all sorts of Windows desktop and system manipulations. Here’s the batch file I use to get into desktop mode:
I still play the amazing generative graphics game No Man’s Sky. One nice thing is the game has 11 different kinds of planets with different biomes. Each biome has a list of different names the game might use to describe them. It’s not always obvious what biome a name describes. So here’s a list of them all for easy searching. See also here and here for other sources, or my data as a spreadsheet.
This data is current as of September 2021 including the Frontiers update of NMS (3.6).
Somehow I’ve done generative art experiments for 30 years without every really understanding noise functions. I mean I’ve certainly used some but never directly, generally by incorporating someone else’s noise-derived textures or something. So here’s some notes.
Noise functions
Processing’s noise function is probably how most generative artists first encounter noise. It is straight up Perlin noise, the classic 1985 algorithm. It’s defined for any number of dimensions but let’s just focus on 2D noise for now. You give it an x, y and it gives back a random number 0-1. The key feature of Perlin noise is it’s smooth; two x, y points near each other will have a similar value. Note there’s an implicit scale in Perlin noise; is (100, 100) “near” (0, 0)? Or do you have to go as close as (10, 10) or (1, 1)? This scale factor is not very interesting but you do want to dial in a scale that has features of the size you want. Perlin noise implementations often provide other tuning like how many octaves to calculate and how much each contributes.
Perlin noise has a problem in that it’s somewhat directional; you can see horizontal and vertical artifacts in the top 4 images above. (Taken from this blog post). The bottom 4 images are similar treatments of OpenSimplex noise. Simplex noise is a different algorithm Ken Perlin made in 2001; see here for an explication. Unfortunately Simplex noise has some worry about a patent (expires January 2022!). So that’s why there’s OpenSimplex: it has similar features but was designed to be avoid the patent. (Patents on math: so stupid.) In addition to being isotropic both kinds of simplex noise are also faster to calculate.
FastNoise is really a collection of different noise functions with an emphasis on fast implementations. Includes Perlin, Simplex, and a bunch of other choices. The project is ambitiously multi-language and has several iterations as the author has supported SIMD first and then rewritten everything as FastNoise 2. Note the author Jordan Peck works on No Man’s Sky, one of the most ambitious procedurally generated videogames out there. It’s full of noise functions.
There’s a bunch of other noise functions out there too. Worley noise, Fractal noise, Gabor noise… It’s like after Perlin got an Oscar for his noise function everyone wanted their own ;-) There’s also a lot of work in making fast implementations, running on GPUs, etc. I’m getting in the weeds now so I’m just going to look at some simple things to use with Python implementations.
Python implementations
perlin-noise, a basic Python implementation of Perlin noise. Works with any number of dimensions.
noise, a native code implementation of Perlin noise and true Simplex noise. 1-3 dimensions for Perlin, 2-4 for Simplex. Also includes a GLSL implementation for shaders that I did not test. This code is mostly 9 years old! But there’s no reason that should matter for something this simple. Note it has no API support for setting a seed. You can override the entire random integer function though which would let you also control seeding.
pyfastnoisesimd is a SIMD optimized, parallelized implementation of FastNoise. There’s no simple API for generating a single sample; it’s designed to generate large grids of numbers all at once (very quickly).
I’m currently using OpenSimplex in my projects, mostly because it’s the first library I found. It’s easy to use and fast enough for me.
Benchmarks
I don’t really much care about speed in my applications but I was curious, so here’s some simple timeit statistics to generate a single 2d noise sample.
Constant
50000000 loops, best of 5: 0.00469 usec per loop
math.random()
10000000 loops, best of 5: 0.0359 usec per loop
OpenSimplex
100000 loops, best of 5: 2.34 usec per loop
perlin-noise with 6 octaves
5000 loops, best of 5: 90.6 usec per loop
noise.pnoise2
5000000 loops, best of 5: 0.1 usec per loop
noise.snoise2
5000000 loops, best of 5: 0.0844 usec per loop
So there you have it. About 0.1 microseconds per sample for the C implementations, 2.3 for OpenSimplex (Python), 90 for Perlin (Python). To it it another way, that’s roughly 10M samples a second for the C code. For comparison pyfastnoisesimd says it’s a few nanoseconds / sample, or 200M+ samples / second.
Using noise for art
Noise functions are just random number generators; how do you make art with them? Ha, as if there were a simple answer to that question.
I did make one simple demo that a zillion people have done; making a grid of OpenSimplex noise and then visualizing the number with a short line segment rotated by the noise value. You have to fiddle the scaling to get it to look interesting. Here’s a result that looks pretty nice plotted, although drawing all ~5900 line segments took 90 minutes.
Another classic plotter use of noise is to draw “flow fields”. It’s a simple physics simulation; you drop a particle at a point and then interpret the noise value as an angle of force. Then you draw the trace of the particle as its buffeted around your grid. I’ll probably do that next, it’s a classic with a lot of interpretations.
Seeds
An aside on seeds. It’s nice to use words as seeds instead of numbers. There’s a zillion ways to do that but one easy way to turn a string into an int is zlib.crc32(seed.encode('utf-8')). Note that’s a 32 bit int; 64 would be nicer. Oddly there are precious few 64 bit checksum and hash functions, I don’t think any are installed into Python by default.
