At 5:07 a.m., the on-call phone started buzzing. The order API of our fresh-grocery mini-program was reporting timeouts one after another; by 5:20 more than 40 orders had failed. The warehouse called in: some orders wouldn't submit, and drivers were waiting to load.
First, Rule Out What It Wasn't
Three of us split up and checked. Nginx QPS wasn't high — peaks of just over 90 per minute, normal for the morning rush, nothing like a traffic attack. PHP-FPM workers weren't saturated, Redis hit rate was normal, and the queue wasn't backing up.
So it wasn't concurrency.
It Was the Scheduled Job Next Door
The slow query log showed a batch of heavy queries starting at 4:55 a.m., one taking over 8 seconds. It was the nightly category aggregation, GROUP BY over 400,000+ detail rows to build the daily report. It usually finished by 3 a.m., but that day an upstream CSV arrived late, pushing the job's start to 4:55 — right on top of the 5 a.m. ordering window.
The heavy query held MySQL's IO and connections, small order writes queued behind it, and anything over the frontend's 8-second timeout failed.
What We Did on the Spot
Step one: kill the aggregation job. The morning market comes first. Orders recovered at 5:24 a.m.; the warehouse re-submitted 6 failed orders, but the rest had actually been written — only the responses never made it back. We verified by order number that nothing was double-counted.
Step two: change the schedule. From "run when the upstream file arrives" to "start no later than 3:40 a.m.; if the file is missing, skip and alert." No drifting toward 5 a.m. ever again.
Step three: route read-only heavy queries like this aggregation to the replica, never the primary. Replication had been set up after that accidental-deletion incident and had only ever served as a backup — this time it finally pulled real weight.
Afterward
We added one more check: every scheduled job's start time goes into a table, and nothing heavy may run within 30 minutes of 5 a.m. Listing them out, we found two lightweight jobs sitting right next to 5 a.m. and moved those too.
The late CSV turned out to be their export script stalling for 40 minutes. We added a line to the "external data" section of our maintenance checklist — the list is now written through 2026.
The morning market hasn't stalled since January 16. The warehouse still opens orders at 5 and cuts off at 6:30, and nobody mentions that morning anymore.
