I figured out what put the rogue SSL certificate on my system, the one that breaks MacOS Mavericks. It came back yesterday 2015-04-27 between 15:38:32 and 16:41:02. The system Console helped me narrow it down to the application Cyberduck, version 4.6.3. I use it sporadically to browse S3 buckets. It’s just the right kind of unusual application that would cause a bug like this that some small group of users on the Internet finds but not everyone.
Cyberduck 4.6.3 placed a bunch of errors in the console, including the smoking gun “4/27/15 3:42:05.621 PM Cyberduck: Error adding certificate to Keychain”. I also manually verified the login keychain entries show up again as soon as I run Cyberduck after deleting them. (For completeness I should add Cyberduck also complains about an SSL hostname mismatch when connecting to S3, but I think that’s a legitimate and expected error and unrelated to the Verisign certs.)
Version 4.7 (17432) is the latest version of Cyberduck. It doesn’t cause the problem any more. There was a reported and fixed bug in Cyberduck. I’m not sure the Cyberduck authors understand the magnitude of the problem though, or that users of old versions now have broken Macs. We now have a bug report on file, my report text below.
I also don’t understand the root cause of the problem. Cyberduck says they were taking a certificate offered by the server; Amazon S3, in this case. Were they verifying it was a valid cert first? If so, then why is it no longer a valid cert? And why did MacOS security update 2015-004 break it? I’m content to let that all remain a mystery, but I’m curious.
Update: this AWS discussion contains complaints about S3’s SSL certificate. Apparently it’s a key that is weakly signed and various software is deciding it’s no longer valid as they update to stricter requirements. That may explain why 2015-004 changed things.
For the search engines: one of the two bad certificates placed on my keychain by Cyberduck was “VeriSign Class 3 Public Primary Certification Authority – G5”
Bug report for Cyberduck
Prior to version 4.7, Cyberduck had code where it wrote some SSL certificates to the user login keychain. This behavior is documented in ticket #8741 and the code was changed to no longer do that.
However, the certificates old versions of Cyberduck wrote to the Keychain are now causing fairly serious problems with MacOS. Affected Macs can no longer verify Verisign-signed SSL certs in any application. Symptoms are the App Store refuses to load, MacOS software updates won’t get installed, Chrome refuses to load websites and Safari throws errors. It’s pretty bad. The problem seems to be triggered by Mavericks security update 2015-004 (released last week).
The fix is pretty simple: manually delete the spurious entries in the login keychain (so that the system entries are used instead). But users aren’t going to figure that out on their own. There’s no indication to the user there’s a problem with their keychain or that Cyberduck was the app that created the problematic entry. I only figured it out thanks to some lucky timing and a message on the system console.
While Cyberduck 4.7 no longer causes the problem, anyone who used an older version of Cyberduck may still have a broken Mac. Could Cyberduck do something to notify affected users? Maybe a new version of Cyberduck that checks for the bad entries and warns the user, pointing them to a help page?
It’d also be nice to figure out exactly what entries Cyberduck might have written. For me and a bunch of other users it’s two Verisign certs, one named “VeriSign Class 3 Public Primary Certification Authority – G5”. They seem to have come from Amazon S3.