A webhook endpoint is a publicly reachable URL, with no user session behind it, that changes your data when something posts to it. Written down like that it sounds alarming, and it should — because the first version everyone writes is a route that parses JSON and updates a record, which is exactly that description with nothing guarding it.
The gap between that and something you can leave running is a short list, but every item on it fails silently when you skip it. This is the list, in the order I'd add them.
1. Verify the signature — against the raw bytes
Anyone can POST to your endpoint. The sender signs each request with a shared secret, usually HMAC-SHA256, and you recompute that signature and compare. If it doesn't match, reject before you look at the body.
Two details do all the damage here.
Sign the raw body, not the parsed object. This is the single most common webhook bug I've seen. Your framework's JSON middleware parses the body before your handler runs, and if you re-serialise that object to check the signature you get different bytes — key order changes, whitespace goes, numbers get normalised. The signature fails for every legitimate request and you conclude the secret is wrong. Most frameworks need an explicit raw-body option on that one route.
Compare in constant time. A normal string comparison returns as soon as two characters differ, so how long it takes leaks how much of the signature was correct. Use your platform's timing-safe comparison. And if the sender publishes an official SDK, use it — the SDKs do both of these correctly, and this is not a place to demonstrate that you can write HMAC.
2. Answer immediately, work afterwards
The instinct is to do the job and then respond. It's wrong, and it's the reason most webhook systems start dropping events under load.
Senders have timeouts, usually a few seconds. If your handler verifies the signature, updates three records, syncs an ERP and sends an email, you're gambling that all of it finishes in time. When it doesn't, the sender records a failure and retries — and your handler was probably most of the way through, so now the work happens twice.
So: verify the signature, persist the raw event, return 200, and process from a queue. The endpoint's only job is "I have this, it's genuine, it's safely stored". Everything slow happens somewhere that can fail and retry on its own terms without a stranger's timeout deciding your fate.
This also fixes debugging. Because the raw event is stored before processing, a bug in your handler is replayable — you have the original payload and can run it again after the fix, rather than asking the sender to resend events they may no longer have.
3. You will receive the same event twice
Webhook delivery is at-least-once. Not exactly-once — that isn't on offer, from anyone. A network blip after your handler committed but before the response arrived looks identical to a failure from the sender's side, so they retry, and the second delivery is indistinguishable from the first.
Which means every handler has to be idempotent, and the reliable way is not clever logic — it's a unique constraint. Store each event's ID in a table with a UNIQUE index and insert it as part of the same transaction as the work. A duplicate hits the constraint and you stop, having changed nothing. Doing it in application code — check, then act — leaves a window between the check and the write where a concurrent retry slips through, and concurrent retries are precisely the case you're defending against.
Keep those records at least as long as the sender's retry window. Stripe retries with exponential backoff for up to 72 hours, so a deduplication table pruned after an hour will happily let a day-old retry through and refund someone twice.
Worth saying plainly what this protects: double fulfilment, duplicate refunds, repeated emails. It's the difference between a bug and an incident.
4. Order is not guaranteed either
This one is less known and produces stranger bugs. Events can arrive out of sequence — a retry of an earlier event can land after a later one that succeeded first time.
So subscription.updated can arrive before subscription.created, and "cancelled" can be overwritten by a stale "active" that was delayed in transit. If your handler is "take the payload, write the fields", you have just resurrected a cancelled subscription, and nothing anywhere reports an error.
Two defences. Compare timestamps or version numbers on the object itself and ignore anything older than what you've already stored. Or, for anything important, treat the webhook purely as a signal — "something changed about object X" — and fetch the current state from the sender's API rather than trusting the payload's contents. That costs a request and buys you certainty about ordering, which is usually the better trade. It's the same argument I made about sending an invalidation rather than the change itself over a WebSocket.
5. The replay window fights the retry window
Signature schemes usually include a timestamp, and you're meant to reject anything older than some tolerance — Stripe's SDK defaults to five minutes — so a captured request can't be replayed at you later.
Now notice the tension, because it isn't obvious until it bites: retries can arrive hours after the original. A tolerance window tight enough to be meaningful will reject legitimate retries of an event you genuinely failed to process.
The resolution is to be clear about which mechanism is doing which job. Idempotency is your replay protection — a replayed request is a duplicate event ID, and the unique constraint stops it changing anything regardless of age. The timestamp tolerance is a cheap extra filter, and it should be set generously enough not to reject your sender's real retries. Getting this backwards means dropping real events to defend against an attack the unique constraint already handles.
6. Decide what a failure means before you have one
If you return a 500, most senders retry. That's usually right — a transient database error should be retried.
But some failures will never succeed: a payload referencing a customer who doesn't exist in your system, an event type you don't handle, something that fails validation. Retrying those forever costs you the sender's goodwill and, on some platforms, gets your endpoint disabled after enough consecutive failures — at which point you stop receiving everything, including the events that were working.
So separate them. Retryable failure: 500, let it come back. Permanent failure: return 200, record it in a dead-letter table, and alert someone. You've accepted the event and admitted you can't process it, which is honest and keeps the pipe open.
Unknown event types should be a 200 and a log line, never an error. Senders add event types, and you don't want an endpoint that breaks because someone shipped a feature you haven't heard of.
7. Log everything received, forever-ish
Every event, with its ID, type, receipt time, signature result and processing outcome. This is not optional infrastructure, it's the only way to answer the question you will eventually be asked, which is always some form of "did we get it?"
Without that log, a disagreement with a payment provider about whether an event was sent is unresolvable, and you will lose it, because they have logs and you don't.
The shape
Verify against raw bytes with a timing-safe compare. Persist and return 200 fast. Do the work in a queue, keyed on the event ID with a unique constraint that outlives the retry window. Never trust ordering. Distinguish retryable from permanent failures. Log all of it.
None of this is difficult. All of it is invisible when it's missing — the endpoint returns 200, the sender's dashboard shows success, and the only evidence of the problem is a customer who paid and didn't get the thing.