Understanding Python 3.4 asyncio

Working with discord.py has made me realize how I really don’t understand asyncio, the modern way to do asynchronous code in Python. Truthfully I barely understand any sort of async programming at all. I can sort of bash away at NodeJS a little bit. And I once wrote an asyncore program that did some neat network stuff. But mostly I’ve used threads or processes to do concurrency. It doesn’t help that I generally am starting with a single threaded synchronous program and then trying to later shoehorn it into some async context.

But soldiering on, here’s some resources:

  • Python 3.4 asyncio docs: the core reference. As with all Python docs it is well written, but unlike some Python docs it’s hard to follow. Much more of a reference than a tutorial.
  • Module of the Week: asyncio: a heroic effort to provide an overview tutorial of the complete module with lots of examples. It’s good, but I’m still confused after reading it.
  • Exploring Python 3’s Asyncio by Example and The New asyncio Module in Python 3.4: Event Loops, two more tutorial articles that just hit the highlights.
  • PEP 3156 is the proposal that resulted in asyncio, written in the Python 3.3 era with a reference implementation called “Tulip”. I haven’t read this yet.
  • PEP 0492 is the proposal that resulted in the “async def” and “await” keywords being added to the language in Python 3.5. Also “async with” and “async for”. To normal coders, this is mostly syntactic sugar for stuff that worked in Python 3.4. But behind the scenes Python added a whole new concept of “coroutines” to the language, to accompany generators.
  • PEP 255, PEP 342, and PEP 380 are 5+ year old PEPs related to defining generators in Python. (The “yield” keyword.) Generators are the basis of coroutines, and coroutines are the foundation of asyncio.
  • aiohttp is the popular HTTP library for use with asyncio.

Here’s my understanding of today, with the massive caveat that I’m totally new to this and may have important aspects wrong.

The key concept in Python asyncio is the event loop. When you are writing an async program you no longer are executing functions imperatively. Instead you are creating objects that ask for your functions to be called. (The “async def” keyword or the @asyncio.coroutine decorator do this for you; they wrap your imperative functions with an object, so when you invoke the decorated function it doesn’t actually run but instead returns the callback object). You then supply these objects to the event loop through functions like “call_soon()”. Finally you set the event loop running with run_forever() or run_until_complete(), which gives the event loop control over the CPU where it blocks while it runs your code via all the callback objects you gave it.

The other key concept in Python asyncio is coroutines and/or generators. That’s where the “yield” keyword comes in, and it’s actually the same old yield keyword we’ve had since Python 2.3. It’s the way for a Python function to return CPU control back to the thing that called it, yet also keep its stack state so if that function is called again it picks up where it left off. asyncio in 3.4 is literally implemented just as a bunch of fancy generator functions and lots of yielding everywhere. 3.5 adds the new syntax support with “await”. It also makes async functions into “coroutines” instead of “generators”. But the basic concept is still the same.

So that’s the core async magic. It’s really just a sort of process scheduler for cooperative multitasking. The event loop is the CPU scheduler that invokes tasks in a single Unix process/thread. Tasks politely interrupt themselves by calling yield when they can.

But there’s a lot more in asyncio. A key thing is that there’s convenience functions for places where you are most frequently going to want to yield. Ie: waiting for a socket, or a stream, or a subprocess, there’s convenience code for saying “get me the next bytes from this socket and oh by the way go ahead and yield control until there’s something to read”. There’s also a higher level wrapper for “protocols”.

And beyond that, there’s a lot of building blocks. There are Tasks and Futures to wrap some asynchronous thing in a structure. There’s a “wait()” method for collecting results from multiple asynchronous things. There’s locks and semaphores. Many of these convenience objects seem to basically be wrappers for the basic yield/await syntax.

So I think that’s how it all fits together. Now to translate that theory into actual working code to see if it’s right.

Discord API first use

My gaming group is switching from Slack to Discord. It’s a nice product. It’s sort of a Slack clone with voice channels as the big added feature. It’s also a product aimed at gamers, not corporate use, so there’s some hope the pricing and features will make more sense for gamers. I like the product, it’s worth a look.

So now I need to make my Slack bot post to Discord, too. Discord currently has no formal API. There is a fantastic group of hackers doing an Unofficial API project, though, and the company seems OK with it. Compared to most hacking projects this one is particularly well documented and organized. I dove in this morning and figured out how to write my own bot that works with the Discord API.

First, some resources:

The Python client is very nicely documented and organized. If you’re starting afresh you want to get version 0.10.0a, the alpha, directly from GitHub. PyPI currently has 0.9. 0.10 has reorganized the API, no reason to use the old one now.

The client itself is entirely based on Python 3.4’s asyncio (with preference for the new 3.5 syntax if available). That means you basically have to do everything through an event loop, which is a bit of a nuisance for simple things but absolutely necessary for building real software that’s going to interact with a Discord server. it also means discord.py only works with Python 3.4 and later. I think that’s all the right choice.

Here is my demo program. It took me a couple of hours in total, half of which was me reminding myself how asyncio works. All in all pretty straightforward, once you figure out the basic paradigm of how it fits together. There’s one choice embedded in my code here, which is I subclassed discord.Client instead of using the function decorator syntax for declaring event handlers. Either works fine.

The unofficial API also makes me understand the Discord product a little better. Unlike Slack, Discord has a notion of a single User that is logged into Discord with an email/password. That user can then be a member of various Servers, each of which has its own Channels. Slack doesn’t really have a cross-Slack notion of User, every Slack is an independent domain. The Discord API’s handling of public/private/DM channels is also a bit more straightforward than Slack’s.

quicky python continuous tests

pytest-watch is a fine quick “continuously run my tests” thing for Python in a tty. It’s nothing fancy, and the default UI is not perfect. But with -c and –no-beep and using –runner to run a custom command for your tests, it’s fine. Nice to not have to switch from my editor window to a console just to run the tests.

I’m envious of the NodeJS folks and their hipster test runners, they are very nice. Alternately I wish there were a good way to turn SublimeText into a bit more of an IDE with debugging and test output parsing. But after years of trying IDEs and going back to plain text editors, I think I’m not going to try to board that ship again.


sqldiff for sqlite3

Learned about a nice tool today: sqldiff. It shows differences between two sqlite databases. Particularly nice with the “–schema” flag, so you can just compare schema differences. I’m in the middle of database versioning hell as I keep iterating my lolslackbot project. Beginning to regret rolling all the database stuff by hand instead of using some ORM.

Mac sound capture, take 2

Awhile ago I wrote up some notes on recording League of Legends videos using OBS. I still mostly follow that horribly manual workflow, but I was having a hard time getting sound to work reliably. I think I have it sorted out now.

  1. Install Matt Ingalls’ SoundFlower 2.0b2 from GitHub
  2. Reboot. Not clear this is necessary, but I think it may be.
  3. Follow these instructions for configuring audio. However, Soundflowerbed is no longer included. Fortunately you can ignore that if you Audio MIDI Setup yourself (standard Mac software)

The key thing here is you use Audio MIDI Setup to create a new “Multi-Output Device”. That’s a virtual sound device that can send output to more than one place, in this case both “Built-in Output” (your speakers) and “Soundflower (2ch)” (the capture device).

Then you use the Sound preference panel to tell the Mac to play output to that new Multi-Output Device. Now every sound your Mac plays will both come out the speakers and be sent to SoundFlower.

Then you set up OBS to accept audio input from the SoundFlower device.

Unfortunately setting the default sound output to Multi-Output Device will break the volume control from the Mac keyboard. You can still set the volume for each output in the Sound panel though, just be sure to leave Multi-Output Device selected when you’re done. You want to modulate actual volume using Internal Speakers’ control. Set the Soundflower and Multi-Output Device volume to maximum, so that sound is passed through at full volume to OBS.

In previous versions of this setup I found cases where OBS could record audio correctly, but only if it was launched after the program playing the sound. No idea what that was about. I don’t think that’s happening any more.

If you’re serious about doing more fancy audio routing, the commercial Loopback program is pretty good. At $75 though it’s a bit pricy for what I’m doing.

CNAMEs at a DNS root

It’s common wisdom that you can’t use a CNAME at the root of your domain. http://www.example.com can be a CNAME pointing to something, but example.com can’t. That makes hosting websites on naked domains awkward. Only confusingly you actually can do this, it’s just a bad idea. And some DNS services make it look like you’re doing it in a good way, but really they’re faking it. It’s all pretty confusing but I think I understand what’s going on in detail. This post was motivated by this Hacker News thread.

A reading from the holy text, RFC 1034

A CNAME RR identifies its owner name as an alias, and specifies the corresponding canonical name in the RDATA section of the RR. If a CNAME RR is present at a node, no other data should be present; this ensures that the data for a canonical name and its aliases cannot be different. This rule also insures that a cached CNAME can be used without checking with an authoritative server for other RR types.

So if you make example.com a CNAME, you can’t put any other DNS records on it. AFAIK you can actually do this and it will work as well as a CNAME will. The problem is you often want other DNS entries at the apex domain. You want an MX entry so you can redirect mail at example.com. You want a TXT entry to verify you own the domain name to various services. You might want special funky records for DNSSEC or other stuff. It really doesn’t work to make your apex domain an alias for some other domain.

Confusingly, a lot of the big DNS providers do let you set up a CNAME at the apex domain. But behind the scenes they aren’t really serving it as a CNAME; it’s more of a synthetic alias implemented by their DNS server. Typically it looks up the destination host address and quickly substitutes that IP in with an A record for your apex domain. That breaks the end-to-end principle of the DNS lookups, which causes problems for geographic load balancing, fast cache expiry, and probably DNSSEC. But it’s not a terrible solution.

Here’s a list of some of the DNS vendors’ aliasing technology

Long story short, if you want to serve a website from an apex domain it requires compromises and/or protocol fiddling.

Looking at all this I wonder if the world would be well served with a special DNS entry just for web servers. It’s weird to put something application specific in DNS, but there’s a strong precedent in MX entries, for mail delivery. Probably a bad idea to extend that to the web, but maybe it’d all some useful tricks?


sqlite3 vs postgres: uneasy feeling

I continue to write my LoL Slack bot with sqlite3. I can live with the primary drawback of sqlite3, which is that writes require a database level lock (ie: no table or row locking). In normal use I should only ever have one writer to the database anyway. But I still have a nagging feeling I should use “a real database”. Ie: Postgres

The nervousness is I’m writing more complex code that’s wedded to sqlite. I’m using sqlite’s upsert equivalent, for which there is no standard syntax. I’m also using sqlite’s Python convenience feature of not using cursor objects. And just now I’m starting to pay attention to Python’s weird transactional semantics, which I suspect might be different in Postgres. In short my code has been dating sqlite for awhile but now things are getting serious and I’m getting commitment anxiety. Also Postgres has some alluring features I could use, like JSON and array types.

The underlying question is whether Postgres is really “better” for my kind of application. I suspect not, and part of why I stick with sqlite is to test that assumption, but the cost of being wrong is getting higher. The flip side is managing a Postgres daemon is still a PITA. Particularly on Ubuntu where upgrading is scary.