Mavericks security update 2015-004 has a serious SSL bug

My Mac suddenly started throwing SSL errors when connecting to various sites, like search.twitter.com or support.apple.com. The App Store application refused to load content, too. Long story short, MacOS Mavericks 2015-004 has a bug where an incorrect certificate named “VeriSign Class 3 Public Primary Certification Authority – G5″ is placed on the user’s login keychain. The fix is to run Keychain Access and remove it. Note: remove the one in the login keychain, not the System Roots.

This error seems really serious to me. Macs that are affected can’t get new software updates. Also Chrome will refuse to load any websites with SSL certs signed by that VeriSign certificate, including Apple’s own sites. Safari will load the site but will display SSL errors. Apparently Chrome is more strict in enforcing SSL security.

(I thought it was particularly interesting that it was impossible to get Chrome to visit Twitter. Twitter only serves HTTPS, not HTTP. And they have HSTS enabled which means Chrome will refuse to load a page without a working SSL certificate. Well that all succeeded, but boy was that a bad experience.)

Here’s some links with more discussion: Ask Different, Security StackExchange, Apple forums. I exported the two Verizon certs that were on my login keychain that were the problem, there’s a zip file here along with some screenshots of failed SSL certs. (That file won’t be online forever.)

I seem to be hitting a serious bug like this in MacOS every couple of months. Along with some broken-by-design things like their SMB client and I really am tempted to try going back to a Windows desktop. Or maybe Linux, if it weren’t so damn ugly.

Complex QOS rules considered harmful

tl;dr: some router firmware has a catch-all rule that throttles all unidentified UDP traffic to 5% of bandwidth (labelled “Crawl”). This is a stupid rule, disable it.

I just fixed a bug in my router’s configuration that explained why Google QUIC was not working well for me. It may also explain bugs I’ve been seeing in League of Legends, OpenVPN, and other UDP protocols. I’m not entirely certain.

I’ve been running the Tomato v1.28 (Toastman) firmware for a year+ now. It’s an old build. It has 40+ default QoS rules identifying all sorts of protocols from important ones (DNS) to silly ones (RealAudio streaming), and then classifies traffic service level. Unfortunately some of the rules are harmful.

The problem rule in this case was the very last one. “UDP Dst Port: 1-65535, classify Crawl”. And Crawl by default is limited to maximum 5% of total bandwidth! There are a few higher priority rules that classify specific kinds of UDP traffic: DNS, for instance. But any new or unanticipated use of UDP is severely throttled. Such as QUIC, Google’s fancy new web protocol. And Cisco VPN. And maybe OpenVPN.

And maybe League of Legends; it’s a UDP protocol too, and hasn’t performed as well on my slow network as I expected. Just playing a game feels about the same, maybe a little less laggy, but there’s still the same unexpectedly high packet loss. But I think one reproducible bug is gone now. Jayce gates cause a brief surge of UDP packets; it used to be that caused significant lag even when playing alone. Now they don’t cause lag.

The simple fix is to adjust the Crawl class to also get up to 100% of bandwidth (both inbound and outbound). That may still have lower queue priority though. You can also try adding more rules for UDP protocols you care about; QUIC is on ports 80 and 443, for instance. But trying to label all known UDP protocols is a Sisyphean task.

I can’t imagine why anyone ever thought a 5% cap for default traffic was a good idea. Particularly for a UDP protocol which may not even be able to interpret those dropped packets as a signal to rate limit itself. Judging by the comment they were trying to catch unidentified BitTorrent traffic, which must have its own rate limiting. But still, what a dumb rule.

After several years of using QoS on home routers I’m of the opinion that QoS rules cause as much trouble as they fix. It’s certainly caused me a lot of problems. In a home network there’s no meaningful way to shape incoming traffic at all. You can shape the outgoing traffic a bit, and I think prioritizing ACK is probably a good idea. (Although weirdly this behavior is not the default). But in general the QoS implementations out there complicate things a lot and don’t provide a lot of value.

It’s time to go back and look at what the Bufferbloat guys have accomplished recently, and whether fq_codel or something similar has gotten traction. Their approach seems much simpler. Last I checked no Tomato variant supported it.

QUIC problem

Update 2: this was a bug in my router, not QUIC. See below, I also posted to the QUIC protocol group about it.

The Google is broken on my computer. This happened once before. Web sessions to every site on the Internet work fine except Google properties. Those hang for ~15 seconds for every web request in Chrome. Safari requests work fine.

Both times this has happened to me have been up in Grass Valley, the place with the shitty fixed wireless Internet. And both times when I’ve looked in Wireshark there’s some QUIC protocol stuff happening, I don’t know what. But my guess is that Chrome is deferring all requests to QUIC and they have a bug / bad interaction with my ISP where it doesn’t work right.

I could swear I once saw some QUIC diagnostic info inside Chrome. Or at least a way to disable QUIC entirely. Can’t find it now. Of course it’s hard to find things when Google doesn’t work. Edit: found the disable, it’s in chrome://flags. The statistics are in chrome://net-internals/#quic. The moment I disabled QUIC Google searches worked better. Interestingly, simply restarting Chrome didn’t help, at least not completely. I lack the patience to test this carefully, no point if no one’s going to pay attention to the result.

In desperation I’ve tried using Bing. It’s not as good as I remember. In the past I didn’t like the UI but the relevance was mostly good. But now the relevance is bad. They were returning URLs for RSS feeds on Metafilter for searches. WTF?

Update: I have a theory! It may be my router QoS settings. Tomato / Toastman 1.28 doesn’t have a rule for QUIC, but does have a default rule that all UDP traffic otherwise not classified is assigned the category “crawl”, which is limited to just 5% (40kbps) on my network. D’oh! I wonder if this also is responsible for my worse-than-expected League of Legends performance here. I added a rule for QUIC (UDP ports 80/443, classify WWW). Also raised the bandwidth cap on “Crawl” to 100% traffic, that was a dumb default rule. Now to try it out for a few hours…

Python3 zip() is a hassle

A Python3 annoyance; zip() now returns an Iterable, not a list. That means you can’t easily slice it.

In Python 2, this idiom for getting back the first column of a database query was useful:

cur.execute('select ...')
r = cur.fetchall()
firstColumn = zip(*r)[0]

In Python 3, this gives you an error

TypeError: 'zip' object is not subscriptable

The workaround is to wrap the zip in a tuple or list

r = cur.fetchall()
firstColumn = tuple(zip(*r))[0]

Annoying, but it doesn’t actually run any slower. I imagine you could use slice() or itertools.islice() to produce an iterator view of a slice of the zip iterator.

Mac fan control

My ~3.5 year old iMac is making noise. Just a small noise, but it’s high pitched and annoying. Something mechanic, presumably a fan bearing. So I started trying to isolate the source of noise. It isn’t the internal spinning drive, you can turn it off by unmounting all the volumes. I’m not using the optical drive. So maybe a fan?

The tool to test a Mac’s fans is smcFanControl, a simple tool to read the temperature and fan speed settings and alter fan speed. When you run it it requires root access, then goes into your menubar. You can set the minimum fan speed for each fan separately. It works pretty well. I used it to run each fan at full speed and crikey, my Mac sounds like a vacuum cleaner. Never heard those fans run before! Didn’t really notice a change in the sound but it’s hard to be sure. Maybe my optical drive fan is to blame. Unfortunately you can’t set turn the fans off entirely; the apps docs say this is possible but not enabled for safety. A quick search didn’t turn up any easy hacks, either.

On a side note, apparently there’s no good CPU/GPU/Memory burn-in tool for Macs. At least not a free one. The #1 suggestion from Google searches is “run yes > /dev/null”. Which, well, that works a tiny bit but it’s not even exercising the whole core of a single CPU, much less the whole system. There’s a bunch of good tools that run under Windows, too bad no one’s bothered to port one. The core load generator doesn’t need to change (much?) but the GUI is a problem.

Simple Javascript time series charts

I want to draw a time series graph in Javascript. This is one of the simplest kinds of regular data visualizations and all of the libraries I find to do it suck. I looked at this a year ago, is anything better now?

What I want is a library that will automatically assign axes and draw correct time series graphs. The time intervals are not regular. Also sometimes data is missing. Not zero, null. Remarkably, very few charting libraries seem to be able to handle these common requirements. Here is some very quick evaluation, I spent ~3 minutes on each library so may have mischaracterized something or overlooked some greatness.

  • Charts.js: too low level, really a graphics library for drawing colors in canvas
  • Flot: good ol’ flot, just seems too old. Maybe that’s foolish and it’s solid?
  • D3.js: great library but too low level. I don’t want to build my own axis and don’t really want to program at all, just declare data.
  • Metricsgraphics: promising library at the right level, but seemed buggy. I spent about 90 minutes trying to get a time series to render and depending on my Date range, either got a decent graph or got confusing SVG rendering errors involving NaN.
  • RickshawxCharts, Ember, and d3.chart look to all have very slow or no development. Rickshaw was promising, I wonder if it’s just “mature”.
  • Dimple’s time series seems to assume equal spacing
  • NVD3 has been around a long time and is still active. It’s kind of a sloppy project though. The example code on the site doesn’t match what’s live. Some of the examples kill Google Chrome when run. There are no reference docs. I give up.
  • Vega looks too toolkitty and not enough libraryish. It frames itself as an analog to Protovis or D3, and I already know one of those. Also no time series example.

I need to investigate Highcharts. It’s not free software, but is free for non-commercial use.

dygraphs is sort of promising. It actually plots my data correctly but the default style is ugly. Also I’m not wild about canvas. It does a lot of fancy time series stuff out of the box though.

dygraph sample

Python3 est arrivé

I’m now using Python3 by default for all new projects. Almost entirely because I prefer sane Unicode handling. But really because Py3 has turned the corner and is now useful for most things. Probably it turned that corner a year or more ago, but it was the OpenAddresses work that made me start using Python 3 regularly. It helps that Homebrew and Debian both make it very easy to install ‘python3′ and ‘pip3′. And of course that most packages I care about are now ported over.

It took Python five years longer than I expected for the Py3 transition to get to this point. Let’s hope we don’t have to do it again.