In the points module's second week after launch, support forwarded a complaint: one member had made 1 order, and the points ledger held 3 identical credit records.
First, Rule Out Concurrency
Our first thought was a concurrent-deduction problem. The logs showed the three requests were 4 and 7 seconds apart — not simultaneous. The user-agent and request path pointed to the same browser refreshing the order result page twice in a row.
The cause: the points API was written as "insert a ledger row on every request," with no deduplication at all. A slow tap, a spinning network indicator, two extra clicks — and the points got credited twice more.
Choosing the Idempotency Key
We didn't use the random nonce from the request — that doesn't survive a refresh. We used something the business itself determines: order_id plus change_type. An order's "earn points on purchase" record can only exist once; "return deduction" is a different type, independent of the first.
We added a unique index on (order_id, change_type) to the points ledger table. A duplicate insert raises a conflict; the backend catches it without erroring, fetches the first record, and returns it as-is. The front end still sees success, and the user can't tell the difference.
The critical detail here: a duplicate request must return the original result, not an error. Return an error and the user assumes the credit failed — and clicks again.
The Front-End Layer
On the front end, the button greys out after submission, and the result page redirects to a different address so a refresh won't re-POST. This layer just discourages extra clicks; we don't rely on it. The real safety net is that unique index.
We also store a request token in Redis with a 10-minute expiry; a repeated token short-circuits immediately. This layer catches the cases that never even reach the unique index — sending SMS, calling external APIs — at a lower cost than a database write.
Fixing the Data
The 2 extra records couldn't just be deleted — there had to be an audit trail. We added an adjustment record: consolidated the 3 rows per order into 1, offset the surplus with a negative entry, and noted clearly that it was a duplicate-submission correction. In all, 11 affected orders; a half-hour script run, and we exported a detail sheet for support to keep on file.
Afterward we listed every API that could change a balance — 6 in total — and added an idempotency key to each. Two had never caused trouble; we did them anyway.
Since then we've had a habit: in any review, for any API with "add," "subtract," "deduct," or "issue" in its name, the first question is — what happens if it's called twice?
