The retry you can't see
Our payment provider sends us a webhook when a parent pays, and retries up to three times if we don’t answer quickly enough. So the same 500 dalasi confirmation can genuinely arrive three times.
Handled badly, one payment becomes fifteen hundred dalasi in the books.
The reason retries exist at all is a problem with no clean solution. When you send a request and get no reply, you cannot distinguish between it never arrived and it arrived, worked, and the reply got lost. From the outside those look identical. Don’t retry and you lose real events. Retry and you risk doing the same thing twice.
The way out is to make repeating harmless — then the sender can be as paranoid as it likes. That property has a name: idempotency. Doing it twice has the same effect as doing it once.
Three layers, deliberately
The schema. Each payment stores the provider’s charge ID under a unique constraint. The first delivery inserts. The second cannot — not because we check, but because the database will not hold two rows with the same ID.
That distinction is the whole lesson. I could have written “check if it exists, then insert.” It looks equivalent. It isn’t: two retries arriving together can both pass the check before either writes, and both get a perfectly truthful “no, it doesn’t exist yet.” Like two agents selling the last seat on a plane — neither was told a lie, and the seat is still sold twice.
Asking permission leaves a gap between the question and the act. A constraint has no gap, because the check and the write are one indivisible thing. Push the guarantee down to the layer that can actually enforce it.
The surface. A parent can also scan the QR on a printed invoice twice. There is no key involved there at all — the page simply reads the bill’s current state, and a settled invoice shows no Pay button. Idempotency achieved by design rather than by token, and more robust for it: it holds regardless of how the parent arrived.
The failure. If a duplicate ever did slip past both, an overpayment becomes credit on the student’s account rather than vanishing. Not idempotency — that’s graceful failure — and it’s the reason the system doesn’t depend on any single guard being perfect.
The bugs this prevents share a nasty property: they fail perhaps once in a thousand, pass every test, survive the whole pilot, and then charge somebody twice on a busy morning in a way you cannot reproduce.