We swapped certificates by hand for years — the 2017 on-call rota, the 2 a.m. incident in 2019, all part of that history. In spring 2025 we finally moved to automated renewal. The first attempt didn't work; three things bit us.
1. The Validation Directory Was Blocked by Nginx
ACME validation requires placing a temporary file under /.well-known/acme-challenge/. One of our Nginx rules returned 404 for any dot-prefixed path — which also caught the validation requests, and issuance kept failing with 403.
We added a location block matching that directory exactly, pointing at a real webroot; every other dot-path stays denied.
2. Renewed, but Not in Effect
The certificate files updated, but Nginx never reloaded, so browsers kept receiving the old cert. The first version of the script didn't include a reload at all.
We moved reload into the renewal tool's post-hook, executed only when a new certificate is actually issued — not on every cron run. The hook also runs nginx -t first: if the config is broken, skip the reload rather than take the site down.
3. Did It Run? Did It Succeed? Nobody Knew
The cron job's default output went to some mailbox — exactly the trap we'd hit in 2019. Now the script appends a row to a cert_log table on every run: domain, execution time, result, and the new expiry date. Failures are flagged separately and wired into the alert channel that has exactly one group chat.
We also kept a small independent check, separate from the renewal script: read the certificate's actual expiry daily, with alerts at 30 days and 7 days, and flag any mismatch against the logged records. Don't trust the renewal tool's own report of success.
4. Don't Bundle Multiple Domains into One Certificate
The first version bundled 6 domains into one certificate for convenience. Later, when one domain needed to be decommissioned, reissuance touched the other 5 with it — the blast radius was too wide. We split into one certificate per domain with the script looping through them. A bit slower, but a problem with one never drags in the others.
Now
It has run for over half a year: 11 automatic renewals, human intervention exactly once — a temporary DNS resolution failure for one domain broke validation, and a rerun after the fix sorted it out.
That 2019 hand-filled certificate table still exists, exported to CSV and kept. The four fields are unchanged: domain, expiry date, storage path, owner. Only now the expiry column is written by the script itself.
