On December 1, we posted the roster for the 7 days of Spring Festival: 11 people across two offices.
The hard part wasn't filling the shifts — it was keeping the same person off both New Year's Eve and New Year's Day. Three drafts before it worked: each shift is 24 hours with one primary and one backup. The Wuhan office volunteered for the first 3 days, saying their folks with short commutes weren't leaving until after Day 2 of the new year. Changping took the last 4.
Primary-and-backup doesn't mean both sitting at a computer. The primary answers alerts and makes the first call; anything they can't judge, they pull in the backup within 5 minutes. The backup has nothing else scheduled that day and doesn't travel far. That rule was set by a roster back in 2017 and has never changed.
Attached to the roster is a one-pager thicker than the roster itself: how to handle each common failure, which operations require a phone call for approval first, which database tables must never be touched, and the phone numbers for support and every vendor. Writing that page took two afternoons — longer than the scheduling.
Some ask: with monitoring this thorough, does anyone still need to be on watch? Yes. When something really breaks, the alert tells you 5xx is spiking — it doesn't tell you whether to roll back. A rollback is a decision, and decisions need a person.
On New Year's Eve, the primary was one of the Wuhan folks. At 8 p.m. he posted a photo of the office in the group chat — a single lamp on. No real alerts; he stayed until 10 and went home, and per the roster, the backup was on.
On Day 7 the holiday ended, the roster came off the whiteboard, and once again all that remained was the water-delivery number.
