The phone rang at 2:11 a.m. — the alert SMS had beaten the user's call by four minutes. One line on the screen: TLS handshake failure on a client site.
The Scene
I dragged myself up, opened the laptop, and the first thing over ssh was checking the certificate:
openssl s_client -connect <domain>:443 2>/dev/null | openssl x509 -noout -dates
Valid until 2:07 a.m. Four minutes past.
Browsers were showing full-red warnings, and there were two messages in the client's group chat, both from their on-duty ops staff, asking whether the site had been hacked. We replied "certificate expired, working on it" — first, hold the line.
Where the Renewal Emails Went
The certificate was installed two years ago, and the renewal notice email on file belonged to that colleague — long gone, mailbox still there, unread. The CA had sent 3 reminders: 30 days, 7 days, 1 day. All sitting there.
This was almost a repeat of the 2017 incident, except back then we hadn't yet put the recipient in a config file.
The 20-Minute Swap
The new certificate had actually been issued long before, sitting in another colleague's Downloads folder — nobody had installed it. scp it up, assemble the chain, reload.
Two small traps. First, chain order: the site certificate goes first, intermediates after. Assembled backwards, Android devices reported it as untrusted until we fixed it. Second, after the reload curl still returned the old certificate — we assumed the config hadn't taken, but keep-alive connections were still being served. Waiting 30 seconds and retesting showed the new one.
Everything was done by 2:31 a.m.
Afterward
The next day we did three things. Every certificate in production went into a table with four fields: domain, expiry date, storage path, and owner. A daily script sends alerts at 30, 7 and 1 days before expiry, with recipients read from a group alias in the config file — never an individual's name — so staffing changes mean editing one place.
And one more rule: certificates live under /etc/ssl/<domain>/, never in someone's personal folder. If that certificate hadn't been findable in a Downloads directory, 20 minutes would have been a lot more than 20 minutes.
By the end of that year, the table was on the monthly ops checklist. Not for any grand reason — getting up at two in the morning is something you only need to do once.
