Cyberduck is responsible for my bad SSL certificate

~ nelsonminar

I figured out what put the rogue SSL certificate on my system, the one that breaks MacOS Mavericks. It came back yesterday 2015-04-27 between 15:38:32 and 16:41:02. The system Console helped me narrow it down to the application Cyberduck, version 4.6.3. I use it sporadically to browse S3 buckets. It’s just the right kind of unusual application that would cause a bug like this that some small group of users on the Internet finds but not everyone.

Cyberduck 4.6.3 placed a bunch of errors in the console, including the smoking gun “4/27/15 3:42:05.621 PM Cyberduck[54126]: Error adding certificate to Keychain”. I also manually verified the login keychain entries show up again as soon as I run Cyberduck after deleting them. (For completeness I should add Cyberduck also complains about an SSL hostname mismatch when connecting to S3, but I think that’s a legitimate and expected error and unrelated to the Verisign certs.)

Version 4.7 (17432) is the latest version of Cyberduck. It doesn’t cause the problem any more. There was a reported and fixed bug in Cyberduck. I’m not sure the Cyberduck authors understand the magnitude of the problem though, or that users of old versions now have broken Macs. We now have a bug report on file, my report text below.

I also don’t understand the root cause of the problem. Cyberduck says they were taking a certificate offered by the server; Amazon S3, in this case. Were they verifying it was a valid cert first? If so, then why is it no longer a valid cert? And why did MacOS security update 2015-004 break it? I’m content to let that all remain a mystery, but I’m curious.

Update: this AWS discussion contains complaints about S3’s SSL certificate. Apparently it’s a key that is weakly signed and various software is deciding it’s no longer valid as they update to stricter requirements. That may explain why 2015-004 changed things.

For the search engines: one of the two bad certificates placed on my keychain by Cyberduck was “VeriSign Class 3 Public Primary Certification Authority – G5”

Bug report for Cyberduck

Prior to version 4.7, Cyberduck had code where it wrote some SSL certificates to the user login keychain. This behavior is documented in ticket #8741 and the code was changed to no longer do that.

However, the certificates old versions of Cyberduck wrote to the Keychain are now causing fairly serious problems with MacOS. Affected Macs can no longer verify Verisign-signed SSL certs in any application. Symptoms are the App Store refuses to load, MacOS software updates won’t get installed, Chrome refuses to load websites and Safari throws errors. It’s pretty bad. The problem seems to be triggered by Mavericks security update 2015-004 (released last week).

The fix is pretty simple: manually delete the spurious entries in the login keychain (so that the system entries are used instead). But users aren’t going to figure that out on their own. There’s no indication to the user there’s a problem with their keychain or that Cyberduck was the app that created the problematic entry. I only figured it out thanks to some lucky timing and a message on the system console.

While Cyberduck 4.7 no longer causes the problem, anyone who used an older version of Cyberduck may still have a broken Mac. Could Cyberduck do something to notify affected users? Maybe a new version of Cyberduck that checks for the bad entries and warns the user, pointing them to a help page?

It’d also be nice to figure out exactly what entries Cyberduck might have written. For me and a bunch of other users it’s two Verisign certs, one named “VeriSign Class 3 Public Primary Certification Authority – G5”. They seem to have come from Amazon S3.

Some references:

http://apple.stackexchange.com/questions/180570/invalid-certificate-after-security-update-2015-004-in-mavericks
https://discussions.apple.com/thread/6984765
https://trac.cyberduck.io/ticket/8741

Mavericks security update 2015-004 has a serious SSL bug

10 thoughts on “Cyberduck is responsible for my bad SSL certificate”

  1. That’s a big pile of sleuthing, thanks for doing it. It raises a more general question – ‘can a userlevel program entirely subvert your online security without as much as a log line or request for authentication’. In OS X, the answer appears to be ‘yes’.

    security add-certificates -k login.keychain cert.der

    This ‘just works’ and you’d think it should maybe ask you if you mean it first.

  2. yeah I’m a bit surprised by this too. also the lack of an audit trail or even logging on keychain. I think keychain is pretty great in general, certainly way better than the competition. but could be better. OTOH with such a mess of root CAs in browsers all this stuff is a sham anyway.

  4. Huh, this is interesting: people have been complaining about this particular Aamzon S3 certificate since October 2014. https://forums.aws.amazon.com/thread.jspa?threadID=164095

    I don’t fully understand it, but it seems like there’s something wrong with the way the certificate is signed and that slowly a bunch of software is deciding it’s no longer valid. Maybe security update 2015-004 updated MacOS to also now decide it doesn’t like this cert. That’d explain why the security update finally triggered the problem.

  5. Do you still have that zip of certs sitting about somewhere? Also, assuming Cyberduck was (for reasons fathomless to man) just blasting the certificates into the login chain, where did it get that poopy root cert? The intermediate comes from Amazon but the remote end of an SSL connection doesn’t provide its own root certificate. Could it have dug it out of the built-in Java cert store (Cyberduck is a well-disguised Java program)?

  7. Thanks. Here is another odd thing. Yosemite doesn’t think there’s anything wrong with these certs. Both are 2048 bit RSA certs. The root one has a SHA1 signature. Those signatures are being phased out but I don’t think we’re anywhere close to where such certs are being rejected outright. And even if they were, one crappy cert should not just break everything – there was still a valid chain of trust to the system roots. So maybe there is something wrong with that Mavericks update after all. If you manually re-add these to the login chain (double clicking them should be enough), does everything break again for you?

  8. Ok, I was confused – both of these certs are intermediaries and, until recently, you could get both from talking to Amazon. Cyberduck was, for some reason, stuffing them into the keychain. This seems somewhat silly and pointless but it should have been harmless. The mere presence of a non-verifiable (as far as the client is concerned) cert in the keychain should not have broken everything because verifiable paths were available. That’s exactly how things were before the security update and that’s exactly how it appears to be in Yosemite (10.10.3 and up). Except for the security update that broke things in Mavericks.

    It may not be the same thing but here’s a very curious problem someone had with Python and those exact same (by fingerprint, not merely name) accursed certs a couple of months ago.

    http://stackoverflow.com/questions/27804710/python-urllib2-ssl-error

    It seems in some cases they can trigger trust chain verification failure in OpenSSL. That’s what your screenshots show as well – the chain ends at an intermediate cert (coloured blue, instead of gold, for a root) which could not be connected to a trusted root cert.

  9. So there’s three mysteries

    1. Why does the old cert on my login keychain look invalid to some applications?
    2. Why does the application complain if it has a valid signature path using the new cert on the System keychain?
    3. Why does Keychain show the cert is valid when applications are rejecting it for signing certs?

    Problem 2 about multiple paths seems to be a known bug in OpenSSL, see the answer to that Stack question. Dating back to 2012 :-( MacOS does use OpenSSL but I don’t know if it uses it for this component. Or even what’s going on exactly.

    Problem 1, why the old cert causes a problem in the first place… I verified that placing just the “VeriSign Class 3 Public Primary Certification Authority – G5” on the login keychain is enough to cause it; the G3 one isn’t needed. I also compared the two versions of that cert using Keychain, screenshot is at http://imgur.com/Gwg1ED1

    The key difference seems to be the problem cert has a 128 byte signature, the working one has a 256 byte signature. Not sure whose signature this is; is it self-signed? It’s the exact same 2048 bit public key, in any event. The problematic cert also has some other extensions defined on it, I don’t know if they matter.

    Problem 3 just seems to be that Keychain’s UI is not very good. Or else I’m confused. Or maybe it’s that Keychain is doing its thing completely right but that applications like Chrome are enforcing their own new restrictions? But that wouldn’t explain why this problem got more severe only with the release of MacOS security update 2015-004.

    Speaking of that update, Apple’s release notes have two useful bits. “Update to the certificate trust policy”. And more directly, “Multiple vulnerabilities existed in OpenSSL 0.9.8zc, including one that may allow an attacker to intercept connections to a server that supports export-grade ciphers. These issues were addressed by updating OpenSSL to version 0.9.8zd.” There’s a list of CVEs for more details, I didn’t look.

    Frankly this is way more than I want to know about the details of certificate verification.

  10. >Frankly this is way more than I want to know about the details of certificate verification.

    Amen.

    I think, strictly speaking, there is nothing wrong with the cert. The signature is the signature of the issuing authority – only roots are self signed. For 1, are you sure the app complains the cert is invalid or the app complains it can’t construct a good trust chain with that cert? Same applies to for 3. A cert can be valid, but not usable to construct a blah blah, hwaghaghwahgah, etc.

    Yeah, the SO thing I linked primarily for the answer. It seems to show how this sort of thing can break. Whether the same specific way or not, I think that particular Mavericks update broke just like that and everything went to hell. The same certificate lives happily and harmlessly in other versions of OS X. If you find yourself less exhausted of this nonsense than I am, it might be worth filing the report on Apple’s bugreporter. ‘Random cert in login chain blows up OS X’. Bugs typed under ‘security’ seem to get reasonably prompt response by a human, these days.

Comments are closed.