In the spring of 2017 we added Redis to a content site's homepage. Every page view queried 5 tables to assemble the data; under load, MySQL struggled. The idea was simple: serialize the assembled result into Redis under the key homepage:v1 with a 300-second expiry — on a hit, skip the database.
Day one went fine. Day two morning, trouble. Here's the postmortem.
The Symptom
Around 8 AM, MySQL CPU jumped from 20% to over 90%, and the slow query log filled with the same homepage aggregation SQL. The Redis hit rate dropped below 60%. In theory only one request should ever repopulate an expired key — this shouldn't have happened.
1. At the Instant of Expiry, Dozens of Requests Hit the DB Together
At the 300-second mark the key expired, right at a small traffic peak. Dozens of concurrent requests all found no cache, all queried the database, and all wrote the cache back. There was no mutual exclusion.
Fix: before repopulating, take a short-lived lock via setnx — say 10 seconds. Whoever gets the lock queries the DB and fills the cache; everyone else sleeps 50 ms and retries reading the cache. Less than 20 lines of code.
2. Someone Was Requesting Nonexistent Category IDs
What actually slowed the database was something else: the logs showed a batch of requests like /list?id=999999 — categories that don't exist, so nothing in the cache, so every request penetrated to the database and returned empty. Some crawler script was scanning.
Two-layer fix. One: when a query returns nothing, still write an empty value to Redis with a short TTL, 60 seconds. Two: validate the id range at the entry — anything beyond the current maximum category id returns 404 immediately, never reaching the cache logic.
3. serialize() Crossed a PHP Version and Started Warning
The cache stored PHP arrays serialized with serialize(). Later one machine got a PHP upgrade, the unserialized object structure didn't match, and notices piled up. We switched to json_encode / json_decode — also friendlier across languages, at the cost of slightly larger payloads.
4. No Version in the Key Name, So a Redesign Blank-Spotted the Site
The first key was just homepage. After a homepage template redesign, the new code read the old cache and a block went missing from the page — half an hour of debugging before we remembered. Since then, keys look like homepage:v2: bump the version when the structure changes and let old keys expire on their own.
The Current Setup
Homepage cache 300s, empty values 60s, lock 10s, versioned keys, id validation at the entry. Before launch we hit the expiry moment with ab and confirm only one request reaches the database.
Redis itself has never been the problem. The problem was the thought 'we added a cache, so we can stop caring about the database.'
