Learn the theory
Webhooks: Delivery, Retries & Signatures: how it works
4 min read · Updated
A webhook is an HTTP request that a service sends to your server when something happens — a payment succeeds, a repo is pushed, an order ships. Instead of polling an API every minute, you give the provider a URL and it calls you. That simple idea hides hard problems: your server can be slow, down or deploying, the network can drop the reply, and anyone on the internet can send a request to your URL. Reliable webhook handling is mostly about three things: acknowledge fast, be idempotent and verify the sender.
In short
- Webhooks are delivered at least once — never exactly once. Expect duplicates, retries and out-of-order events.
- Providers time out after a few seconds. Do the real work in a background queue and reply
200immediately. - Make handlers idempotent: remember the event ID and ignore repeats, so retries are harmless.
- Verify the HMAC signature on the raw body, and reject old timestamps to stop replay attacks.
- Retries aren't a guarantee. Reconcile periodically against the provider's API for events you never received.
How a webhook works
- An event happens inside the provider (a customer pays).
- The provider creates an event with a unique ID and an HTTP POST to the URL you registered, with a JSON body and signature headers.
- Your endpoint verifies the request, records the event and answers with a 2xx status.
- If the provider doesn't get a 2xx quickly, it assumes delivery failed and retries later.
POST /webhook HTTP/1.1
Content-Type: application/json
X-Event-Id: evt_1
X-Signature: t=1700000000,v1=9f2c...
{ "type": "payment.succeeded", "amount": 5000 }At-least-once delivery (and why duplicates happen)
The provider can't know whether your server processed the event — only whether it got a timely 2xx. If the reply is slow, lost, or an error, it must assume failure and retry. Two outcomes follow:
- Lost events if the provider never retries (or gives up) and you have no reconciliation.
- Duplicate events when your handler did the work but the acknowledgement didn't arrive in time — the classic “slow handler” bug in the lab.
Acknowledge fast, process later
The endpoint should do only the cheap, essential steps — verify the signature, store the raw event, enqueue a job — then return 200 in milliseconds. A background worker (see the BullMQ and RabbitMQ labs) does the slow work and can retry on its own schedule.
app.post('/webhook', express.raw({ type: 'application/json' }), async (req, res) => {
if (!verify(req)) return res.sendStatus(401); // 1. authenticate
await queue.add('process-event', JSON.parse(req.body)); // 2. enqueue
res.sendStatus(200); // 3. ack in ms
});Idempotency: make retries harmless
An operation is idempotent if doing it twice has the same effect as doing it once. Every webhook carries a unique event ID; store processed IDs and skip repeats.
-- unique constraint makes the duplicate insert fail
INSERT INTO processed_events (event_id) VALUES ('evt_1');
-- if it succeeded, do the work; if it violated the constraint, ignore the eventDo the insert and the business change in one database transaction, so a crash can't record the ID without doing the work (or the reverse).
Verifying the sender: HMAC signatures
Your endpoint URL isn't a secret. To prove a request is genuine, the provider signs it with a secret you share: HMAC-SHA256(secret, timestamp + '.' + rawBody). You recompute it and compare in constant time.
- Verify against the raw request body, not re-serialised JSON.
- Use a constant-time comparison to avoid timing leaks.
- Include and check the timestamp: reject anything older than about 5 minutes so a captured request can't be replayed.
- Rotate secrets, and keep them out of logs and source control.
Ordering, retries and reconciliation
- Don't assume order.
refundedcan arrive beforesucceeded. Use the event's timestamp/version or fetch the current object from the provider's API before acting. - Retry schedules typically back off: seconds, minutes, then hours, for up to a few days. After that the event is marked failed.
- Reconcile: a nightly job comparing your records with the provider's list API catches anything delivery missed.
- Return
2xxfor events you don't care about, or you'll trigger pointless retries.
Frequently asked questions
- What is a webhook?
- An HTTP callback: when an event happens, a service sends a POST request to a URL you provided so your application can react without polling.
- Why do I receive the same webhook twice?
- Webhooks are delivered at least once. If your server is slow, errors, or the response is lost, the provider retries — so duplicates are normal and handlers must be idempotent.
- How do I make a webhook handler idempotent?
- Store each event's unique ID in a table with a unique constraint, in the same transaction as the work. If the insert fails because the ID already exists, skip the event.
- How do I verify a webhook signature?
- Recompute the HMAC (usually SHA-256) of the timestamp and raw body with your shared secret, compare it to the signature header using a constant-time comparison, and reject requests with old timestamps.
- What status code should a webhook endpoint return?
- Return 2xx quickly (200 or 204) once the event is safely stored or queued. Return 4xx for invalid signatures and 5xx only when you genuinely want the provider to retry.
- How long do webhook providers retry?
- It varies: Stripe retries for up to three days with exponential backoff; GitHub does not automatically retry failed deliveries. Check your provider's documentation and add reconciliation.