The decision
A notification arriving is not the same as a business action completing. Design for a sender to deliver an event again and for delivery order to differ from business order. Stripe's webhook documentation describes these behaviours for its service; other providers need their own contract checked.
A worked example
Imagine an illustrative “invoice paid” event. Receive and validate it, persist its identifier with the intended work, and acknowledge only after durable acceptance. If the sender repeats it, recognise the accepted event and avoid posting the same credit twice. If processing later fails, retain a status the worker can retry.
Alternatives worth weighing
An inbox records incoming events; an outbox records work created by your own transaction. Both help separate durable intent from network activity. A scheduled reconciliation checks whether the source and destination disagree even when no notification reports a problem.
Where the plan breaks
Do not keep duplicate identifiers only in process memory or mark an event done before its business change commits. Handle an old event arriving after a newer one with state or version rules. Back off temporary failures, but route permanently invalid data for review instead of retrying forever.
Separate delivery identity from business identity
In the invoice example, retain the provider’s event identifier as well as the invoice reference. Two different events can concern the same invoice; blocking every later event because the invoice was seen once would discard legitimate changes. Conversely, receiving the same event twice should not apply the same credit twice. Decide which identifier protects reception and which business constraint protects the resulting state, and commit the relevant checks with the actual change.
Validate the sender using the provider’s documented mechanism before accepting work. For services that sign the raw request body, preserve those bytes for verification rather than reserialising parsed JSON. Keep signing secrets out of logs and source control. The exact verification, retry timing and event retention rules belong to the chosen provider’s contract; an implementation copied from another service may appear to work while checking the wrong thing.
Demonstrate crash recovery at the awkward moment
Test a worker stopping after it receives an event, after durable acceptance and while processing the business change. In each case, inspect both the inbox state and the business record after restart. If an external side effect is involved, acknowledge that the remote system may have accepted it before the local result was saved. Use supported idempotency or a reconciliation lookup where available; do not claim that a local status flag alone guarantees exactly-once delivery.
Give operators a controlled replay action that keeps the original event identity and records why it was retried. Permanently invalid data belongs in an exception path, not an endless retry loop. A reconciliation job should compare authoritative records over a defined interval and report unresolved differences. Agree who reads that report, how quickly urgent discrepancies are handled and what evidence closes an incident. A queue with no owner can preserve data perfectly while leaving the business unaware that its work has stopped.
Before you commission the work
Can you repeat the same event without a second effect? Can a worker restart after a crash? Can one unresolved record be investigated and replayed? Use those demonstrations as acceptance criteria for the integration.