Documenting obscure bugs helps!

Had a real pleasure today; someone found a description of a League of Legends networking problem I wrote up a year ago and it gave him the clue to fix a problem he was having. (Long story short; specific actions in the game cause bursts of high network traffic, which can trigger lag or disconnects.)

I spend a lot of time writing up details of bugs and software problems. I do this partly for clarity, writing it up is a way to make sure I really understand the problem. I also do it so I can find my notes later in case I have a similar problem. And finally I put those descriptions online in case they help other people. And about once a year, they do!


Windows 7 update fixes

I dusted off the old Bootcamp partition to run Windows to play Civilization VI, impatient for the Mac release. Every time I do this I’m reminded how bad Windows is. This time around it was 90 minutes spent trying to install and run the game and failing. I gave up and went to bed, then fixed it in the morning.

The symptom was the installer would hang while installing the Microsoft VC 2015 redist. Cancelling the installer didn’t help. Manually killing the wuas.exe process that was hung did allow the install to complete. But then the game wouldn’t run; you’d launch it and it’d quit immediately with no error. Launching the program manually (bypassing Steam) revealed the error, “api-ms-win-crt-runtime-l1-1-0.dll is missing”.

So then begins the cargo cult debugging. Googling for answers and trying various incantations to fix the mysterious problem.

The first thing I tried was manually installing the VC redists. These installers have always sucked. The end result is it’s copying 20MB worth of shared DLLs into a system directory. Why does it take five minutes? Why does it hang entirely? Running by hand (as opposed to via the game installer) sometimes helps. It didn’t this time, it kept hanging.

So the second thing I tried was fixing that error about the crt-runtime. There’s a consensus in the cargo cult about how to fix this; run Windows Update. But Windows Update wasn’t working for me. This is the moment I confess my copy of Windows 7 isn’t registered with a valid license, I’ve been using the grace period for the very rare times I want to run Windows. My bad. So is it the lack of activation that’s causing the problem? Windows is popping up a nag when I try to run the installers. But there’s no explicit error, the installer just hangs. Trying to install it manually didn’t work either, same hanging on Windows Update stuff.

And then the scales fell from my eyes. Windows Update is broken. A fix was released in April 2015; or maybe not until July 2016. Not sure. Anyway the first step in installing that fix is to stop the Windows Update service. Once I did that the crt-runtime installer worked, copying it’s 800kb file successfully without hanging. Congratulations, l’il fella! And that was enough to fix the problem.

So let me rant and describe all the failure-by-design, from top down:

  • The game crashes on startup with no message.
  • The game installer hangs with no explanation. You can quit the install, but then it’s broken.
  • The underlying VC distributable installers hang with no explanation. Its job is to copy 20MB of files, but it can’t do this.
  • The underlying underlying CRT runtime installer is hanging with no explanation. Its job is to copy 800KB of files, but it can’t do this.
  • The root cause is Microsoft screwed up Windows Update somehow and fixing it is awkward, because it requires Windows Update to install the Windows Update update.
  • And worming its tendrils through all this is Windows Activation DRM, which may or may not be interfering and causing problems. With no explanation.

It really is an awful system.


certbot bug and fix (NoInstallationError)

I’ve got a Let’s Encrypt SSL certificate on They expire every three months, so I set up certbot to renew the certificate. It wasn’t working though, the cron job got the error


I think this bug is now fixed in certbot. But if you follow common instructions, the cron job runs certbot with “–no-self-upgrade” so you won’t get the update. Running certbot by hand will have it upgrade itself with the fix.

The underlying bug was that certbot couldn’t find the apache2 command in the PATH, because it’s in /usr/sbin and that’s not in root’s path. (:facepalm:). The committed fix has certbot checking a couple of extra directories in addition to PATH.

League of Legends stats: some findings

I’ve been noodling with some League of Legends data as an excuse to learn more data analysis tech. I’ve blogged about the software, here’s the actual results I found.

League of Legends is a huge game, some 100M a month playing 1M+ games a day. And the data for those games is readily available. I took a look at third party summaries of game statistics from and and tried to divine some meaning. The dataset mostly is for ranked games by Platinum+ skilled players in North America for patch 6.17. But I also looked at players of different skill and trends across game patches.

The most interesting conclusion I found was that some champions win much more in the hands of skilled players. Nidalee, for instance, has a 45% win rate in Silver but a 54% win rate in Diamond. That’s not exactly a surprise, she’s known to have a high skill cap, but it’s good to see hard data. And there are surprises; I would not have guessed Pantheon also was highly improved or that Dr. Mundo suffered at higher skill levels.

Despite looking, mostly I didn’t find any unusual trends or big shifts in the game over patches. To me that’s a testament to the quality of the game, in particular Riot’s game balance team.

Below find some analysis and graphs. I’ve put my notebook output online for viewing here:

Basic stats for patch 6.17

The first thing I did was look at the basic stats for patch 6.17. Each row in my dataset is one summary line for all players of a champion in a role. For instance Nami / Support. Note there’s a bit of bias here; omits rare cases like Nami / Jungle.

There’s relatively little diversity in ADC and Support champions. Simple result: there are only 17 regularly played ADC champions and 25 Supports. There’s ~50 champions played Top/Middle/Jungle. This fact skews a lot of my later statistics, particularly with the ADC pool where a single champion (Lucian) dominates with a 32% play percentage. The top mid, by contrast, is Ahri with only 13%.

download (1).png

Players generally don’t pick champions with higher win rates. This surprised me a bit; I thought there’d be a clear bias towards the OP champs. There isn’t.


Players do tend to win more on champions they have more experience with. No big surprise here, but more games = more wins. The stat “experience” here is average games played on that champion. (It’s in the dataset, but not displayed on the website.) The correlation coefficient is +0.24 with a p value of 0.0005, which seems likely to be significant to me. The causation is less clear. For example Galio-Top players have an average of 107 games on the champ and a 55% win rate. Do people win on Galio because they play him a lot, or do they play him a lot because they win? (Or in the specific case of Galio; do relatively few people play him but the ones who do are really good?)


Stats for players of different skills from

I was curious how these statistics varied by tier, so I went to and downloaded stats for players at all skill levels. I mostly focussed on Silver through Diamond, since those are the most “normal” player populations. This data is for the month of Aug 15 – Sep 15 2016 and spans patch 6.17 and 6.18.

Some champions have much higher win rates at higher tiers. This result is the most interesting I’ve found. I feel like each player at the extremes has a story. I’m not surprised that Nidalee’s win rate goes up 9.37% from Silver to Diamond; she’s hard to play well and so presumably does better in the hands of a skilled player. Aurelion Sol goes up 5.8%, maybe because he depends on skilled teammates? But why does Pantheon go up 7.5%? He seems so simple! I’m guessing Aatrox and Garen do worse in higher tiers because players know how to exploit their weaknesses. And Amumu’s 4.1% drop may be as simple as Diamond players being less vulnerable to Amumu cheese. But there’s more to say here and I’d like to develop this data further. Here’s the 10 most improved and 10 who suffer the most.

Screen Shot 2016-09-21 at 4.32.54 PM.png

Some champs are significantly more popular at higher tiers. The companion result to win rates; what about pick rates by tier? Janna is the most more frequently picked, she goes from 0.8% to 2.9%. (As a frequent Janna player my theory is she is much more rewarding to play when your team is more skilled.) Leona drops from 1.6% to 0.6%. The ADC champion pool attenuates at higher tiers too; Lucian dominates and Jinx drops off, for instance.

Screen Shot 2016-09-21 at 4.37.11 PM.png

Silver-to-Diamond win rate is correlated with Silver-to-Diamond pick rate. In the previous section I reported that for Platinum players, win rate is not correlated with pick rate. However the champs that have the most improvement rate from Silver-to-Diamond also tend to have higher pick rates from Silver-to-Diamond. This suggests that higher tier players are picking champions that have more impact at their skill level. Well maybe; I have a suspicion this effect is pretty small.

download (2).png


Trends across patches

Finally I was curious how stable the game statistics were over patches. I collected historical data from’s GitHub going back to patch 5.10, over a year ago. These are all ranked Platinum games.

Stats are pretty stable between patches. In general there’s not a lot of change in the game since 5.10. For instance, players averaged a 2.55 KDA in 5.10 and a 2.48 KDA now in 6.17. There is a general trend up in some of the cumulative stats like “goldEarned” or “totalDamageTaken”. That could easily be a symptom of games just taking longer over time; unfortunately I don’t have stats on game length to normalize with. Two highlights though. Experience has a big drop at the beginning of Season 6. I take this to be new accounts and/or people changing up how they play. Also totalHeal had a big bump up around patch 5.22. That could be the same general cumulative stat effect, but I think it may also indicate the addition of more healing abilities and the rise of healing supports like Soraka and Nami.

download (2).png

Champion diversity is pretty stable between patches. Up above I showed the distribution of playPercent across 5 roles for patch 6.17; here’s an animation of that distribution across patches. It’s pretty stable. So are the distributions for all the metrics I’m most interested in; goldEarned, KDA, winPercent, in general the distribution across champs looks the same between patches. I find that a bit surprising because I think of LoL game balance being pretty dynamic with a constantly changing meta. But it’s not really reflected in the stats.


Better Jupyter charts: animation

Well this is neat; matplotlib supports generating animations, and you can inline HTML5 video right in Jupyter. The only wrinkle I hit is the stupid Ubuntu ffmpeg/avconv kerfuffle. Trying to produce the animation by default produced the error “KeyError: ‘ffmpeg'”; I solved it by setting

matplotlib.rcParams[‘animation.writer’] = ‘avconv’

There’s a GitHub comment saying that avconv may not be reliable, but it seems to be working for me in Chrome.

Now to figure out how to make this work with Seaborn. Some hints here.

Update: the Seaborn hints mostly just worked. Hilariously the inline video solution is to encode the whole MP4 file in a data64 blob right in the HTML, some 130k worth in my case. Hey, it works.

Better Jupyter charts: Seaborn

Took a quick tour through Seaborn, the enhancement library for matplotlib. It’s very good! It does two basic things. It makes the default matplotlib charts prettier, and it gives you an easy API to do some fancier types of statistical visualization. I’d say it’s a no-brainer to use Seaborn if you’re doing exploratory data visualizations. It still just renders raster images in Jupyter, so it doesn’t fulfill my goal of having nice SVG interactive charts in the browser, but Seaborn is solving a different problem.


“Drawing attractive figures is important”, the docs say, and I couldn’t agree more. Seaborn reconfigures matplotlib so the default charts look better. I don’t just mean nice anti-aliasing, but also reasonable grid ticks and color choices. Seaborn has good perceptual palettes which are really important. I believe stock matplotlib has recently improved in part with input from Seaborn.

The other part of “attractive figures” is the Seaborn API is DataFrame-aware and will label your plots using the labels in your DataFrame. Getting nicely labelled axes and titles and stuff takes several lines of manual code with matplotlib; with Seaborn it’s a single line of code.

Fancier statistics

The meatier part of Seaborn is it has more complex chart types built in.

Distributions are mostly what I’ve used. In detail:

  • distplot: 1 dimensional distributions, an enhancement of matplotlib.hist. It adds a continuous kernel density estimate to the bars, and also has a rug-plot option.
  • jointplot: 2 dimensional distributions, an enhancement of matplotlib.scatter. Adds correlation coefficient, histograms on the side, a sort of quicky ggplot. Can also do continuous contour plots. One drawback is there doesn’t seem to be any easy way to set the hue of the individual dots, so no sneaking in a third dimension of data. Pass in kind=”reg” to have it try to fit a regression curve to the scatter.
  • hexbins: 2 dimensional histograms
  • pairplot: pairwise jointplots for your N dimensional dataset. So nice to have this in a single command, although it’s unwielding for N > 5 or so. (Note this does have a way to set the hue of individual dots.)

Regressions are for fitting various kinds of statistical models to your dataset.

Categorical graphs are for looking at statistics for qualitative categories, plots like swarm plots and violin plots. They’re really quite beautiful. Here’s a sample visualization of the distribution of a variable (y axis) for each of 5 categories (the x axis). I’ve used a violin plot which shows a KDE continuous approximation, along with a swarm plot to show the actual dots on top of it. (The data set is # of games players have for each of 5 roles in League of Legends. That one crazy outlier is Fiddlesticks Jungle; players tend to have 220+ games on him!)


Finally data-aware grids are used for making small multiples plots of the same dataset. I already mentioned pairplot, for doing NxN visualizations of N variables, pairwise. FacetGrid is a tool for quickly generating a bunch of graphs comparing across multiple categorical variables.


I like Seaborn. I like that it looks good out of the box. I also like that it allows me to make more sophisticated graphs very simply, with little effort. It may be a little too easy; I’m not sure a violin plot is really the right treatment of that data above, for instance. But it sure looks good!

Better Jupyter charts: mpld3

As much as I’m liking Jupyter, I don’t much like that matplotlib is the default charting library. I mean matplotlib gets the job done, and it’s powerful and popular. But end of the day you’re rendering raster images with 10+ year old software. (Even if seaborn has made the raster images better.) We’ve got a fancy browser viewer, why not have fancy SVG/Javascript graphs?

That’s what mpld3 provides. Using it is as simple as importing it and calling mpld3.enable_notebook(). All (most?) of your old matplotlib API code will still work. mpld3 emulates that charting API but renders to SVG using D3, instead of rendering to a static image. It’s kinda nutty that it just works. The resulting pixels look good, although admittedly matplotlib’s raster backend (agg?) looks pretty good too. 

The real advantage is the graphs are manipulable with Javascript. Out of the box every graph is now pannable / zoomable, right in the browser window. You can do fancy linked brushes across multiple figures.

There’s also plugins to add more capability. PointLabelTooltip plugin is particularly nice, it adds an easy way to have HTML tooltips on a scatterplot when you hover the points. There’s also MousePosition (useful for images).

And… and that’s about it. After a bunch of work a couple of years ago development seems to have stalled. That’s OK, it’s useful as is. Particularly since it’s such a simple drop-in upgrade for matplotlib.

Probably time to look at alternatives. Bokeh is the “better matplotlib” and includes a bunch of browser / notebook stuff. However it’s still rendering raster images (albeit in an HTML5 canvas). The examples sure are pretty though. I also see frequent references to VisPy, which is GL based, but although there’s some noises made about WebGL and notebooks I can’t find a working demo. Other options I’ve run across are pygalbqplot and Altair. People also talk about Plotly but it’s a hosted system and costs money.

Some other thoughts on this topic: The future of visualization in Python, In defense of Matplotlib, Overview of Python Visualization Tools

While I’m here, ipywidgets is kind of nutty. It lets you add interactive controls to a notebook like sliders and input boxes, to turn your notebook into an interactive app.