On the morning of September 16, customer service forwarded us three nearly identical orders: same user, same product, less than two seconds apart. Same amount, same shipping address — only the order numbers differed.
Why It Happened Three Times
The frontend had a 3-second timeout; on a slow network it would auto-retry before the response arrived, up to two retries. The backend order endpoint had no deduplication at all — one request in, one row inserted. All three attempts succeeded.
This was different from the earlier loyalty-points incident: that time the user was clicking manually; this time the program was clicking on its own, and the user had no idea.
Where to Put the Idempotency Key
We discussed combining user id + product id + timestamp, but change the timestamp and it is no longer a duplicate — that protects nothing. In the end we used a client-generated request_id: a uuid is created as soon as the checkout page loads and sent with the submission, and retries reuse the same one.
The server puts a unique index on request_id in the orders table. A duplicate insert fails; we catch the conflict and — instead of returning an error — look up the existing order and return it as-is. The frontend gets the same order number, so it never assumes failure and tries again.
The key point remains: duplicate requests should return the original result, not an error.
Two Things We Fixed Along the Way
- The frontend timeout went from 3 seconds to 8. Most 'timeouts' are really just slowness, not a missing response. After the change, retries dropped visibly.
- Retries apply only to clear network errors; no retry on 5xx or business errors. The first version didn't distinguish — a business message like 'out of stock' also triggered two retries. Wasted effort.
Cleaning Up the Data
Of the three orders already created, we kept one and ran the other two through the cancellation flow — no hard delete, with a note in the cancellation record: 'merged as duplicate submissions.' Payment had only captured once, so no double charge. Lucky, all things considered.
We exported a detail sheet for customer service — seven columns — so they could walk the user through it.
Afterward
We went through every creation-type endpoint, nine in total, checking each for an idempotency key. Four had never caused problems; we added keys to them anyway. The review checklist gained one line: what happens if this endpoint gets called twice?
The screenshots of those three orders were not deleted. They live in the 'Duplicates' section of our internal docs, right next to 'Two Hours.'
