Python metaprogramming

This Reddit discussion had me looking at a couple of metaprogramming tricks used in Python code.

The standard library’s namedtuple type is generated on the fly at runtime. When a new namedtuple is created the library generates it from a template string. It fills out the template with the necessary details and then passes the class definition to exec(). Exciting! But relatively straightforward and easy to understand. (Except for that extra bit to support pickling; yuck.)

pytest is an alternative unit testing library that has you just using the assert statement everywhere, no library functions. To report test failures in a nice way, it does some dark magic:

Reporting details about a failing assertion is achieved by rewriting assert statements before they are run. … pytest rewrites test modules on import by using an import hook to write new pyc files

I haven’t read the pytest code but from the way that’s written my impression is they’re inspecting compiled bytecode and rewriting the assert statements to instrument them. That sounds a little scary, in particular if it fails it must be completely confusing. They do warn you that “if you are messing with import yourself, the import hook may interfere”. Update here’s a description of the code rewriting as of 2011.

That second technique reminds me a bit of the way that Protovis rewrote Javascript at load time so you could use more modern Javascript syntax than the browser supported. Mike abandoned that idea in D3, I think because it made things too complicated.

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s