One afternoon, one provider timeout, and one support person trying to be helpful. The enquiry arrived twice, and the customer asked why.
What happened, in order
11:04:20 a visitor submits a quote request
11:04:21 the response is stored, receipt r_9d41
11:04:21 a delivery row is written, state queued
11:04:35 the worker sends the notification
11:04:35 the provider does not answer. no status code returned
11:04:35 the delivery row state becomes unknown
11:20 a person sees the queue, and sees "unknown"
11:21 that person retries by hand, without reading the state
11:21 the provider accepts the second attempt
11:22 two notifications for one request. The first attempt had in
fact been accepted, and its acknowledgement was lost.
11:40 the customer replies: "did you mean to send this twice?"
What the retry cost
Nothing was lost, and that is the trap. The damage is narrower than a lost email, and worse. A customer now doubts the process. A support person will hesitate next time. The record shows two attempts where the evidence supported none.
The rule that prevents it
Unknown means the request left and no answer came back. Nothing about that state says the message failed.
unknown reconcile first, then decide
queued safe to retry
failed safe to retry, with the same idempotency key
accepted never retry
If reconciliation is not available, the honest action is to leave the row and escalate. Holding a row for an hour costs nothing. A duplicate costs trust.
The corrected version of the same afternoon
11:20 a person reads the state: unknown
11:21 the row is left open, and the provider status page is checked
11:30 the stored response is read, and the visitor is answered by email
11:40 the provider recovers, and the row settles as accepted
one notification, one request, no duplicates
Acceptance: the rule is working when
- the queue shows the state beside the row, not only the row
- nobody can retry an unknown row without a reconciliation step
- a retry uses the same idempotency key
- consumers of a webhook deduplicate by the response id
- the incident record says who decided, and why
Troubleshooting
| Symptom | Likely cause | What to do |
|---|---|---|
| Two notifications for one request | A blind retry | Deduplicate, and record the incident |
| The provider never answered | A timeout after dispatch | Reconcile, and hold the row |
| A retry created a second stored response | No idempotency key | Send one, and honour it |
| Support hesitates to act at all | The rule is not written down | Print the state table beside the queue |
| The customer asks what happened | Nothing was explained | Tell them the response was received once |
Limits
Delivery is at least once, so a duplicate is always possible. The rule reduces the chance to the cases where a provider genuinely accepted twice.
Never delete a response because its notification failed. The response is the only part of this story that cannot be recreated.