Registration opened at 8 p.m.; by 7:40 the three of us were already in the client's conference room. Two windows on the projector: the Nginx access log and the backend QPS curve. Past events had burned us on single-server capacity, so this time we scaled up one machine in advance.
Before Scaling
The original setup: one 4-core 8GB ECS running PHP-FPM, with MySQL on another box. Under load testing past 300 concurrent users, response times started drifting. We added a 2-core 4GB spare with identical code and configuration, put Nginx in front, and gave both upstream servers a weight of 1.
Sessions were handled ahead of time: PHP sessions used to live in local files; once traffic split across two machines, a user might land on A for one request and B for the next, losing their login state. We moved sessions to Redis, shared by both. That was the only code changed before 8 p.m.
8:02 P.M.
Traffic climbed faster than expected. The first two minutes saw 400-plus submissions per minute, the QPS curve one straight climb. The log showed both machines taking requests, in roughly equal numbers.
The problem surfaced right then: machine A's response time held around 100 milliseconds, while machine B occasionally hit 500 milliseconds, bubbling up 502s at its worst. Checking the hardware: B has 2 cores, A has 4 — half the cores, the same weight. We had assigned half the workload to half the muscle.
We edited the configuration on the spot, set the weights to 3:2, and reloaded. B's 502s thinned out, though a few stragglers remained. We then added two lines to the upstream: max_fails=3 and fail_timeout=10s, letting Nginx temporarily pull out a node that kept failing instead of feeding it more requests.
The Unexpected Bottleneck
What actually kept us busy for ten minutes wasn't Nginx — it was SMS. A successful registration sends a confirmation text, and the third-party API rate-limits us; at peak, a few hundred messages queued up, arriving two to three minutes late. Someone asked in the group: 'did my registration not go through?'
Later we switched SMS sending from synchronous to queue-and-drain, so the page no longer waits. That change landed after the event — that night we just put the queue length on the monitoring screen and watched that it didn't breach.
Wrap-Up
After 9:20 the load tapered off. Neither machine crashed; Nginx forwarded more than 10,000 requests in total, and B was pulled out of rotation twice, each time rejoining on its own after ten seconds.
Before we left, we screenshotted the weight configuration line and posted it to the group with one caption: next time you add a machine, check the core count first.
