Python3 zip() is a hassle

A Python3 annoyance; zip() now returns an Iterable, not a list. That means you can’t easily slice it.

In Python 2, this idiom for getting back the first column of a database query was useful:

cur.execute('select ...')
r = cur.fetchall()
firstColumn = zip(*r)[0]

In Python 3, this gives you an error

TypeError: 'zip' object is not subscriptable

The workaround is to wrap the zip in a tuple or list

r = cur.fetchall()
firstColumn = tuple(zip(*r))[0]

Annoying, but it doesn’t actually run any slower. I imagine you could use slice() or itertools.islice() to produce an iterator view of a slice of the zip iterator.

6 thoughts on “Python3 zip() is a hassle

  1. firstColumn = [row[0] for row in r]

    Easier on the eyes and more than a couple of times faster (just measured it, post made me curious how well the standard recommendation of ‘use a list comprehension’ works in this case).

  2. Huh, definitely easier to read but I’m surprised it’s faster. What was the source of the data in your input r? How big is r? I guess this style means you never copy row[1], row[2], etc, that’s a win. I’m not a fan of microoptimization like this but I am curious what it reveals on Python internals.

    1. I just grabbed the chinook sample db and did a select * from the table Track

      for a result of something a bit over 3k rows.

      I don’t really know why the performance difference is there either but I’d guess it has to do with the extra copying (how does python argument packing/unpacking actually work, etc?).

      I have noticed that in most cases, the stuff that relies on iterators/generators/yaddanators and the syntactic sugar on top (comprehensions) tends to be the fastest. Which is nice, given the improved readability.

  3. So it’s a memory thing, and a surprisingly huge one:

    The zip way somehow manages to eat about 10x the memory. Nice one!

      1. Oh not at all, thanks for coming with an interesting and unexpected test case. My own use of python is almost entirely ‘ad hoc data munging’ so one person’s ‘silly micro-optimization’ is another’s ‘few hours of enlightening entertainment’.

Comments are closed.