Last week, after patching our servers against the Security Bug of the Month, we here at Gimlet got a new SSL Certificate, which your web browser uses to encrypt your traffic and verify that you’re sending it only to us. Unfortunately, this led to Firefox users being unable to connect for several hours.

One of the biggest dangers of the heartbleed bug is that it could allow Evildoers to download something called a server’s Private Key, and use it to do Enhanced Evil. In essence, your private key is a magic, randomly-chosen number that lets a a web site decrypt all the https traffic it receives. The only place that key should live is on the securing web server; operators never give it to anyone. Ever. Instead, operators use a program that does Clever Math to make a new number from the private key, and send it to a Certificate Authority who says “Ah, these people really are who they claim to be” and do yet more Clever Math and generate a Certificate. The operator then installs that on the web server and all is well.

However, with the private key, Evildoers could bypass the security provided by the certificate (using, you guessed it, Clever Math). In response, the right thing to do is get a new certificate from the Authority and have them tell everyone that the old certificate is bad. This process is called revoking a certificate. Where this gets tricky is telling everyone about the status of a certificate. In the past, every web browser was shipped with a list of revoked certificates; when there were a few thousand web sites, this worked fine. Now, there are millions and millions; maybe a third of them are going to revoke their certificates as part of this process. That would be one long list.

So instead, now, Certificate Authorities need to run something called an OCSP service. When your browser visits an https:// site, it’s supposed to contact the OCSP service and say “hey, is this certificate any good?” And it should say “Yes!” to good certificates and “No!” to revoked ones.

Our problem was that, after issuing our new certificate, our Authority’s OCSP service took several hours to start saying our certificate was legitimate — instead, it would respond “Unknown!” For added fun, every web browser treats OCSP responses differently. Chrome and Safari and Internet Explorer saw the Unknown response and decided it was good enough. Firefox decided that it was totally unacceptable. After about six hours, the OCSP server started returning a good response and all was well.

To make matters worse, the OCSP server decided to update its status for our old, revoked certificate within an hour or so. So, when we tried to roll back to our old certificate, Internet Explorer threw a “Revoked Certificate” error. Oddly, only IE seemed to think this was a big problem.

Going forward, if we ever do this again, we’ll make sure we have a certificate that’ll be recognized before we revoke the old one. This may mean switching Certificate Authorities, or just getting better service from the one we’re using now.