An order-creation endpoint averaged 1.2 seconds under load, up past 2 seconds at times. Breaking it down, writing the order table took barely 100 milliseconds; the rest was waiting on three things that don't need an immediate result: sending the confirmation SMS, pushing a message to a WeCom group, and writing a detailed log.
How We Split It
The endpoint now does two things: write the order and publish one message to RabbitMQ. The payload is tiny — just an order_id and an event type. The remaining three actions each got their own consumer, fetching and retrying on their own.
The endpoint now averages a little over 200 milliseconds. Users hit the success page instantly, and an SMS arriving a second or two later goes unnoticed.
1. No Persistence: a Restart Lost 6 Messages
The first version took the defaults for queues and messages to save time. One night the service restarted and 6 pending messages vanished — 6 users never got their confirmation SMS. We only found out from the logs; the users didn't even complain.
The fix: durable queues, delivery_mode 2 on messages, and publisher confirms on the producer. All three together make it solid. The exchange got its persistent flag too.
2. No Idempotency: the SMS Sent Twice
One consumer acked only after finishing its business logic. A network blip meant the ack never landed, the message was redelivered, and the SMS went out a second time. One user received two identical texts.
Moving the ack before the business logic? No — that loses messages. Instead we added a dedup table keyed on order_id plus event_type: only insert-then-send, and duplicates get acked immediately. Same idea as the idempotency keys in our points system.
3. The Dead-Letter Queue
Messages that keep failing can't retry forever — they'd clog the queue. We capped retries at 3, after which messages go to a dead-letter queue, counted in monitoring with an alert past 5. The ops on-call sheet gained a line: "check dead-letter depth".
4. Ordering Isn't Guaranteed
Occasionally the WeCom notification arrives before the order-success log. It looks odd but breaks nothing. We don't force ordering — messages carry an occurred_at field, and display sorts by it. Trading throughput for a single-partition queue just to preserve order isn't worth it here.
Afterward
The three consumers can now be paused independently. Once, when the SMS provider had an outage, we stopped that consumer for two hours — orders kept flowing, and everything queued up and sent after recovery. Before, one SMS outage would have stalled ordering with it.
Those 6 lost messages were re-sent from the order table afterward, and we noted it in the docs: never trust default configurations.
