finsilabs/webhook-architecture

Webhook Architecture

Overview

Webhooks are HTTP callbacks used by commerce platforms (Shopify, Stripe) to push real-time event notifications to your application. Reliable webhook infrastructure requires: HMAC signature verification to prevent spoofed events, idempotent handlers that tolerate duplicate delivery, exponential backoff retry logic, and a dead-letter queue for events that exhaust all retries. This skill covers building a reliable webhook receiver and, for custom platforms, a webhook sender using the Outbox Pattern.

When to Use This Skill

• When receiving webhooks from Shopify, Stripe, Square, or other platforms
• When debugging missed events or duplicate processing caused by webhook delivery issues
• When building a commerce platform or app that needs to notify external systems of events
• When designing event-driven architecture between commerce microservices
• When setting up webhook fanout (single event delivered to multiple consumers)

Core Instructions

Step 1: Determine your platform and what webhooks you need to handle

| Platform | Where Webhooks Are Configured | Most Important Topics to Subscribe | |----------|------------------------------|-----------------------------------| | Shopify | Settings → Notifications → Webhooks (or via Admin API) | orders/create, orders/paid, orders/cancelled, inventory_levels/update, refunds/create | | WooCommerce | Install WP Webhooks plugin (free, wordpress.org) or use WooCommerce's built-in webhooks under WooCommerce → Settings → Advanced → Webhooks | Order status changes (processing, completed, refunded), stock updates | | BigCommerce | Advanced Settings → Legacy API Settings → Webhooks or via API | store/order/statusUpdated, store/product/inventory/updated, store/cart/abandoned | | Custom / Headless | Build your own webhook system | Use the Outbox Pattern for sending; HMAC verification + idempotency for receiving; see implementation below |

Step 2: Platform-specific webhook setup

---

Shopify

Register webhooks via the Shopify admin:

1. Go to Settings → Notifications and scroll to Webhooks
2. Click Create webhook
3. Select the event topic (e.g., Order creation) and enter your endpoint URL
4. Choose JSON as the format

Get your webhook secret for HMAC verification:

The secret is shown when you create the webhook. Store it as an environment variable — Shopify signs each webhook with this secret using HMAC-SHA256.

For apps using the Admin API, register webhooks programmatically:

const res = await fetch(`https://${shopDomain}/admin/api/2025-01/webhooks.json`, {
  method: 'POST',
  headers: { 'X-Shopify-Access-Token': accessToken, 'Content-Type': 'application/json' },
  body: JSON.stringify({ webhook: {
    topic: 'orders/create',
    address: `${process.env.APP_URL}/api/webhooks/shopify/order-created`,
    format: 'json',
  }}),
});

---

WooCommerce

Use WooCommerce's built-in webhooks:

1. Go to WooCommerce → Settings → Advanced → Webhooks
2. Click Add webhook
3. Set Name, Status: Active, Topic (e.g., Order Created), and your Delivery URL
4. The Secret field generates an HMAC-SHA256 signature for each delivery — copy it for your endpoint's verification

Or use WP Webhooks plugin for more control:

1. Install WP Webhooks (free, wordpress.org) for advanced trigger conditions and payload customization
2. Go to Settings → WP Webhooks and configure triggers for WooCommerce order events
3. WP Webhooks supports retry logic and delivery logs out of the box

---

Custom / Headless

For custom storefronts, implement both reliable receiving (for incoming webhooks from Stripe, Shopify, etc.) and reliable sending (Outbox Pattern for notifying your own integrations).

HMAC signature verification (Shopify and generic):

// lib/webhooks/verify.ts
import { createHmac, timingSafeEqual } from 'node:crypto';

export function verifyShopifyWebhook(rawBody: Buffer, hmacHeader: string, secret: string): boolean {
  const expected = createHmac('sha256', secret).update(rawBody).digest('base64');
  const received = Buffer.from(hmacHeader);
  const expectedBuffer = Buffer.from(expected);
  if (received.length !== expectedBuffer.length) return false;
  return timingSafeEqual(received, expectedBuffer);
}

export function verifyStripeWebhook(rawBody: Buffer, signatureHeader: string, secret: string): boolean {
  const parts = signatureHeader.split(',');
  const timestamp = parts.find(p => p.startsWith('t='))?.replace('t=', '');
  const v1 = parts.find(p => p.startsWith('v1='))?.replace('v1=', '');
  if (!timestamp || !v1) return false;
  // Reject events older than 5 minutes (replay attack protection)
  if (Math.abs(Date.now() / 1000 - parseInt(timestamp)) > 300) return false;
  const expected = createHmac('sha256', secret)
    .update(`${timestamp}.${rawBody.toString('utf8')}`)
    .digest('hex');
  return timingSafeEqual(Buffer.from(v1), Buffer.from(expected));
}

Idempotent webhook receiver — deduplicate using the platform's event ID:

// app/api/webhooks/shopify/route.ts
export async function POST(req: NextRequest) {
  const rawBody = Buffer.from(await req.arrayBuffer());
  const hmac = req.headers.get('x-shopify-hmac-sha256') ?? '';
  const topic = req.headers.get('x-shopify-topic') ?? '';
  const eventId = req.headers.get('x-shopify-webhook-id') ?? '';

  // 1. Verify signature — reject invalid requests immediately
  if (!verifyShopifyWebhook(rawBody, hmac, process.env.SHOPIFY_WEBHOOK_SECRET!)) {
    return NextResponse.json({ error: 'Invalid signature' }, { status: 401 });
  }

  // 2. Idempotency check — deduplicate by event ID
  const alreadyProcessed = await db.processedWebhooks.exists(eventId);
  if (alreadyProcessed) return NextResponse.json({ received: true, status: 'already_processed' });

  // 3. Mark as received BEFORE processing (prevents duplicate on concurrent delivery)
  await db.processedWebhooks.insert({ id: eventId, topic, receivedAt: new Date(), status: 'processing' });

  // 4. Return 200 immediately, process asynchronously
  processWebhookAsync(topic, rawBody, eventId); // Don't await — return fast

  return NextResponse.json({ received: true });
}

async function processWebhookAsync(topic: string, rawBody: Buffer, eventId: string) {
  try {
    const payload = JSON.parse(rawBody.toString('utf8'));
    switch (topic) {
      case 'orders/create': await importOrder(payload); break;
      case 'orders/cancelled': await cancelOrder(payload.id); break;
      case 'inventory_levels/update': await syncInventory(payload); break;
    }
    await db.processedWebhooks.update(eventId, { status: 'processed', processedAt: new Date() });
  } catch (err: any) {
    await db.processedWebhooks.update(eventId, { status: 'failed', error: err.message });
  }
}

Outbox Pattern for reliable webhook sending — guarantees at-least-once delivery even if your sender crashes:

// lib/webhooks/outbox.ts
// Write to outbox in the SAME transaction as the business event
export async function publishEvent(trx: Transaction, eventType: string, payload: object) {
  await trx.webhookOutbox.insert({
    id: crypto.randomUUID(),
    eventType,
    payload: JSON.stringify(payload),
    status: 'pending',
    attempts: 0,
    nextRetryAt: new Date(),
  });
}

// Outbox poller — runs every 10 seconds, separate from your main app
export async function processOutbox() {
  const pending = await db.webhookOutbox.findPending({ status: ['pending', 'retrying'], nextRetryAt: { $lte: new Date() }, limit: 100 });
  for (const event of pending) await deliverEvent(event);
}

// Retry schedule: 1min, 5min, 30min, 2hr, 8hr → DLQ after 5 attempts
const RETRY_DELAYS_MS = [60_000, 300_000, 1_800_000, 7_200_000, 28_800_000];

async function handleDeliveryFailure(event: OutboxEvent, error: string) {
  const nextAttempts = event.attempts + 1;

  if (nextAttempts >= RETRY_DELAYS_MS.length) {
    await db.webhookOutbox.update(event.id, { status: 'dead_letter', lastError: error });
    await db.webhookDeadLetters.insert({ eventId: event.id, failedAt: new Date(), reason: error });
    await alertOpsTeam(`Webhook permanently failed after ${nextAttempts} attempts`, { eventType: event.eventType, error });
  } else {
    const nextRetryAt = new Date(Date.now() + RETRY_DELAYS_MS[nextAttempts - 1]);
    await db.webhookOutbox.update(event.id, { status: 'retrying', attempts: nextAttempts, nextRetryAt, lastError: error });
  }
}

Replay dead-letter events (after fixing the subscriber endpoint):

export async function replayDeadLetter(deadLetterId: string) {
  const deadLetter = await db.webhookDeadLetters.findById(deadLetterId);
  await db.webhookOutbox.update(deadLetter.eventId, {
    status: 'pending', attempts: 0, nextRetryAt: new Date(),
  });
  await db.webhookDeadLetters.update(deadLetterId, { replayedAt: new Date() });
}

Best Practices

• Always return 2xx immediately — a slow webhook handler blocks delivery and may cause the sender to time out and retry; enqueue events on receipt and process asynchronously
• Use the Outbox Pattern for reliable sending — writing to an outbox table in the same DB transaction as your domain event guarantees at-least-once delivery even if your webhook sender crashes
• Make handlers idempotent — use the event's unique ID to deduplicate; "at-least-once" delivery is the standard for all webhook platforms; you must tolerate receiving the same event twice
• Log every delivery attempt — store delivery attempts with response codes, timing, and errors; this is essential for debugging and provides audit evidence for compliance
• Implement a dead-letter queue with alerting — events that exhaust retries need human intervention; alert via Slack/PagerDuty and provide a replay mechanism

Common Pitfalls

| Problem | Solution | |---------|----------| | Duplicate order processing from retried webhooks | Implement idempotency using the webhook event ID as a unique key in a processed_webhooks table with a TTL of 30 days | | Webhook handler times out causing retries | Process webhooks async: write to queue on receipt, return 200 immediately, process from queue | | WooCommerce webhook delivery failures | Check the WooCommerce → System Status → Logs for delivery errors; common causes are SSL certificate issues and timeout on slow shared hosting | | Shopify webhook HMAC mismatch | Compute HMAC over the raw request body; do NOT parse the JSON first — body parsers may reformat the JSON and change the signature | | Dead letters pile up silently | Alert when the dead letter count exceeds a threshold (e.g., 10 events); dead letters indicate a systematic subscriber failure requiring investigation |

Related Skills

• @analytics-integration
• @erp-integration
• @marketplace-connectors
• @monitoring-alerting-commerce