All Posts

How I Ingest Webhooks Without Losing Events

A webhook handler that does real work inline is a data-loss bug waiting for a traffic spike. The durable pattern I run on Keaz — persist, queue, process idempotently — and where it breaks.

9 min read

A webhook handler that does real work before it returns 200 is a data-loss bug waiting for its first traffic spike. Stripe, Shopify, and the WhatsApp Business API will retry a failed delivery for days, fan out events in an order you did not expect, and occasionally send the same event twice — and if your handler is busy writing to a database when the next burst lands, you drop signals you will never get back. The durable pattern is boring and it does not lose data: authenticate the request, write the raw event to a table, acknowledge it in milliseconds, and let a separate worker do the real work, idempotently, from a queue. I run exactly this on Keaz across four providers, and it is the line between a webhook system you trust and one you babysit.

Durable webhook ingestion means your endpoint has one job — prove the event is real, store it, and acknowledge it — while every piece of business logic happens later, in a worker that can safely process the same event twice. Everything in this post follows from that single constraint.

Why this matters now

The trigger is a quiet but telling shift at the provider layer. Stripe's newest API version, 2026-06-24, makes thin events and event destinations the default shape of its webhook system: the payload shrinks to an ID and a type, you fetch the full object yourself, and events can be routed straight to Amazon EventBridge or Azure Event Grid. Read that as a statement of intent — the largest payments API on the internet is nudging developers to treat webhooks as an event stream they consume, not a payload they trust.

Stack that on the guarantees Stripe already publishes and the design constraints are unambiguous. Stripe retries a failed delivery for up to three days with exponential backoff, explicitly does not guarantee that events arrive in the order they occurred, and tells you plainly to return a 2xx before running any logic that could time out. Those three facts — at-least-once delivery, no ordering, and a hard latency budget — are not Stripe quirks. They are the shape of every serious webhook source, and they rule out the handler most teams ship first: verify, do the work inline, then return 200.

The durable pipeline, as code

The fix is to split the handler in two. The endpoint proves the event is real and stores it; a worker does the work. Here is the whole pattern in about forty lines:

// A webhook endpoint has exactly one job: prove the event is real, store it,
// and acknowledge it in milliseconds. No business logic runs here — a spike of
// retries from Stripe or Shopify must never block on your database.
export async function handleWebhook(req: Request, provider: Provider) {
  const raw = await req.text()                       // raw body — needed to verify
  const event = provider.verify(raw, req.headers)    // throws on bad signature → 400

  // Idempotent insert: the provider's event id is the unique key. A duplicate
  // delivery (Stripe retries for up to 3 days) hits the conflict and is ignored.
  const inserted = await db.oneOrNone(`
    INSERT INTO inbox_events (provider, event_id, type, payload, status)
    VALUES ($1, $2, $3, $4, 'pending')
    ON CONFLICT (provider, event_id) DO NOTHING
    RETURNING id`,
    [provider.name, event.id, event.type, raw])

  if (inserted) await queue.enqueue({ inboxId: inserted.id })  // hand off to a worker
  return new Response(null, { status: 200 })          // ACK fast, before doing the work
}

// The worker does the real work — later, and safely. FOR UPDATE SKIP LOCKED lets
// many workers pull disjoint rows from one Postgres table with no external broker.
async function drainInbox() {
  await db.tx(async (tx) => {
    const job = await tx.oneOrNone(`
      SELECT * FROM inbox_events
      WHERE status = 'pending'
      ORDER BY created_at
      FOR UPDATE SKIP LOCKED
      LIMIT 1`)
    if (!job) return

    try {
      await process(job)                              // idempotent: safe to run twice
      await tx.none(`UPDATE inbox_events SET status='done' WHERE id=$1`, [job.id])
    } catch (err) {
      const next = job.attempts >= 5 ? 'dead' : 'pending'  // dead-letter after 5 tries
      await tx.none(
        `UPDATE inbox_events SET status=$1, attempts=attempts+1 WHERE id=$2`,
        [next, job.id])
    }
  })
}

Two design choices carry the reliability. The first is the idempotent insert: the provider's own event ID is a unique key, so a duplicate delivery hits the conflict and is ignored instead of processed twice. The second is FOR UPDATE SKIP LOCKED, which lets a pool of workers pull disjoint rows from one Postgres table with no Redis and no Kafka — the database you already run becomes the queue. This is the transactional outbox pattern turned inward: instead of reliably publishing events out, you reliably capture events in. I run this on Keaz, where Stripe, Shopify, HubSpot, and the WhatsApp Business API all land in the same inbox table and drain through the same worker pool across twelve services — one ingestion path, not four.

The five-stage pipeline

Strip away the syntax and every durable webhook system is the same five stages, in this order:

  1. Authenticate at the edge. Verify the signature on the raw body before you parse anything. A failed signature is a 400, not a 500 — you want the provider to stop retrying a request that can never be valid. Stripe's signature carries a timestamp with a five-minute tolerance, which also closes off replay attacks.
  2. Persist the raw event. Write the untouched payload to an inbox table keyed on the provider's event ID. This is the single most important line in the system: once the event is on disk, nothing downstream can lose it.
  3. Acknowledge immediately. Return 200 the moment the row is committed, before any business logic. The provider's latency budget is measured in seconds, and a slow handler under a retry storm is how you turn one late event into a cascade.
  4. Process from a queue, idempotently. A worker pulls pending rows and does the real work — updating orders, syncing contacts, charging accounts. Because the same event can arrive twice, every handler must be safe to run twice; key your writes on the event ID or use an upsert.
  5. Dead-letter and alert. After a fixed number of failed attempts, move the row to a dead state and page a human. A poison event should never block the queue behind it, and it should never disappear silently.

Notice what is not in the pipeline: a message broker you have to operate, an event bus, or a second datastore. At the scale most SaaS products live at, the inbox table and a SKIP LOCKED worker are the whole system. You add Kafka when you have a reason you can name, not before.

What this means for a founder

The cost of skipping this is invisible until it is expensive. An inline handler that fails halfway through does not throw an error you will see — it returns a 500, the provider retries, your half-finished write runs again, and now you have a double-charged customer or a duplicated order. Worse is the silent case: a paid invoice you never recorded because your handler timed out during a spike and the retry window quietly closed. These are not hypotheticals; they are the most common webhook incidents I get called in to clean up, and nearly every one traces back to a handler doing real work before it returned 200. The durable pipeline converts that entire class of bug into a non-event, which is worth far more than the afternoon it takes to build.

My perspective

Here is the opinion I will defend: I will not ship a webhook handler that does business logic inline, no matter how simple the logic looks today. The simplicity is the trap — the endpoint that just updates one row is the one nobody revisits before the traffic that finally overruns it. On Keaz the endpoint is deliberately dumb: verify, insert, acknowledge, and get out of the way. The intelligence lives in the worker, where I can retry it, rate-limit it, and reason about it in isolation. It also composes with the rest of the stack — the inbox table sits inside the same tenant-scoped Postgres I would design for any multi-tenant SaaS, so an event is isolated to its tenant from the moment it lands. One ingestion path, one place to make it correct.

Recommended action this quarter

Audit your handlers before you add features to them. This week, find every webhook route that writes to your database before it returns 200 — those are your data-loss risks, ranked by traffic. This month, put an inbox table in front of the busiest one: persist the raw event, return 200, and move its logic into a worker. By quarter's end, every provider you integrate should share that one path, with dead-lettering and an alert on the tail. If you would rather not learn these failure modes during your next traffic spike, mapping them out ahead of time is exactly what a fractional CTO engagement is for.

Is your webhook system losing events you can't see?

Most teams find out during a spike, when the retries pile up and the double-charges start. Book a time and we will trace your busiest integration end to end and find where events are quietly falling through.

Book a Time