The rules for August's flash sale were simple: 200 tea gift sets, on sale at 9:00 a.m. sharp, one per person, first come first served. It sounds like a routine event; we went through three versions and several rounds of load testing. Here is the process.
V1: Collapsed in Load Testing
The code reused the regular checkout flow: a request comes in, stock is read from MySQL, and if there's enough the order is INSERTed and the stock UPDATEd. Load testing with ab at 150 concurrent users brought out both problems at once.
First, overselling. 200 sets sold 217. In the log's most extreme second, nine requests all read 'one left.' The classic ailment: no mutual exclusion between read and write.
Second, 502s. Every request ran read stock, insert order, update stock, write log; the MySQL connection pool filled up, all 32 PHP-FPM workers were waiting, and the requests behind them timed out outright.
That afternoon we circled three steps on the meeting-room whiteboard: check stock, deduct stock, create order. Spread across the application layer and the database layer, separated by the network — no way to guarantee atomicity.
V2: Move Inventory into Redis
The idea: don't let flash-sale requests touch MySQL.
Ten minutes before the sale starts, a scheduled task writes the number 200 into Redis under the key seckill:20210820:stock, caching the product info and purchase-limit rules alongside.
Purchase requests hit Redis directly. But not 'GET then DECR' — still two steps. We wrote a Lua script doing three things in one shot: read the remaining stock, check it's above zero, DECR, and push the user id into a claimed-set to block duplicates. Redis executes Lua single-threaded — atomic by nature.
The script returns 1 for claimed, 0 for sold out, 2 for already claimed. Whatever the result, the application layer drops a single message onto RabbitMQ and immediately tells the frontend: 'claimed, generating your order.'
Order persistence is left to consumers working at their own pace, off the critical seconds.
This version held at 500 concurrent users. Redis was steady and overselling was gone. But the new problem was the async part: three messages failed to consume — users saw 'claimed,' and half an hour later no order existed.
V3: Patching Both Ends
On the async chain we added three things.
- Messages carry a unique msg_id; the consumer deduplicates on it, so retries never create a second order.
- Failed consumption isn't dropped: it enters a retry queue, up to five attempts; beyond that it lands in a fail_order table, a scheduled task scans it and raises an alert for manual handling.
- The user side gets a 'result check' endpoint: after the frontend hears 'claimed,' it polls every 2 seconds for up to 30 seconds, and jumps to the payment page once the order appears. If nothing shows, it suggests checking back later — never a hard failure.
The MySQL layer wasn't left unattended either. The stock table got a version column, and when the consumer actually deducts, it uses optimistic locking:
UPDATE goods SET stock = stock - 1, version = version + 1 WHERE id = ? AND version = ?
Zero affected rows means someone else changed it first — roll back that message. Under normal traffic this step almost never conflicts; it's the last line of defense against 'Redis and the database disagreeing.'
Purchase Limits and Bots
One per person, enforced by the Redis claimed-set. But someone asked before launch: what about one person with several phone numbers? We didn't deploy heavyweight fraud tooling — just two plain layers: the API requires a login state with the user id in the token, and more than 20 requests per minute from one IP gets a 429 immediately, rate-limited at Nginx so it never reaches the application.
Later we went through the logs: 300-plus requests were blocked by 429, mostly one script probing. We didn't chase it, and no regular user was affected.
Those Three Minutes at 9 O'Clock
August 20, 8:50: the stock number on the monitoring screen read 200. Three of us, one screen each: one watching Redis QPS and remaining stock, one watching the RabbitMQ queue length, one watching MySQL and the error logs.
At 9:00 sharp the QPS curve went vertical. At 9:00:08, stock hit 0 — sold out in eight seconds. Then something looked off: the claimed-set's expiry had been set too short, a few people were judged as 'haven't claimed' and claimed again, generating 203 messages in total.
The queue peaked at 400-plus; six consumer processes drained it in ninety seconds. At 9:03, all 200 orders were persisted; the extra three were blocked by the optimistic-lock step and written into fail_order — three rows, refunded by support following the sheet.
No negatives, no overselling, and no calls about money taken without an order.
The Argument at the Retrospective
Someone asked: if it sold out in eight seconds, were all those load tests wasted? No conclusion after a while. My own take: the load tests weren't about surviving eight seconds; they were about those eight seconds producing no negatives, no 502s, and no 'claimed' without an order. None of the three happened.
Before the meeting broke up, we changed the claimed-set's expiry from five minutes to two hours and reran the script. That config line is still there, with a comment: rather spend a bit of memory.
That afternoon the client sent a photo: the warehouse had packed the 200 tea gifts into chest-high boxes with shipping labels on. Very tangible.
