Webhook Event Delivery & Retry

Distributed Systems intermediate Building with AI 15 min read

Overview

Webhooks are HTTP callbacks that let external services push events to your application in real time. Instead of polling an API every few seconds to detect changes, your server registers a URL and the provider (Stripe, Shopify, GitHub) sends a POST request whenever something happens. This inversion of control reduces latency, saves API quota, and simplifies event-driven architectures.

However, receiving webhooks reliably is harder than it looks. Providers may send duplicates, your server may be down when events arrive, and malicious actors can forge requests to your endpoint. A production webhook handler must verify signatures, process events idempotently, acknowledge quickly, and handle retries gracefully.

In the platform, webhook handlers are the primary integration point for Stripe payments, Shopify orders, and WordPress content updates. Getting these handlers right is the difference between an integration that “mostly works” and one that never loses an event.

Core Concepts

Key Concept

Always verify the webhook signature before processing the payload. Skipping verification opens your system to forged events that could create fake orders, trigger unauthorized refunds, or corrupt data.

Key Concept

Idempotency is non-negotiable for webhook handlers. Use a unique event ID as a deduplication key stored in your database. If the ID already exists, return 200 without reprocessing.

How It Works

Receiving Webhooks

The webhook endpoint receives an HTTP POST with a JSON body and signature headers. The handler reads the raw body (before JSON parsing, since signature verification needs the exact bytes), computes the expected signature using the shared secret, and compares. If valid, it persists the raw event to a durable store and returns 200 immediately. Actual processing happens asynchronously via a job queue.

Key Concept

The “receive fast, process later” pattern is critical. By persisting the raw event and deferring processing to a background job, you avoid provider timeouts and can retry processing independently of delivery.

Signature Schemes

Different providers use different signing schemes. Stripe uses HMAC-SHA256 with a Stripe-Signature header containing a timestamp and signature. Shopify uses HMAC-SHA256 in X-Shopify-Hmac-Sha256. GitHub uses X-Hub-Signature-256. Each scheme must be implemented separately, but the core pattern is the same: compute HMAC over the raw body with the shared secret, compare in constant time.

Delivering Webhooks to Subscribers

When your system is the webhook sender (e.g., notifying third-party integrations about events in your platform), you need a delivery pipeline. Each delivery attempt is a job in the queue. On failure (non-2xx response or timeout), the job is retried with exponential backoff. After exhausting retries, the event is marked as failed and the subscriber can be notified or disabled.

Replay and Recovery

Persisting raw events enables replay: if a bug in your handler caused events to be processed incorrectly, you can fix the handler and reprocess all events from the stored payloads. This is far more reliable than asking the provider to resend events.

Practical Applications

Implementation Patterns

Code Examples

Go: Stripe Webhook Signature Verification

func HandleStripeWebhook(w http.ResponseWriter, r *http.Request) {
    body, err := io.ReadAll(r.Body)
    if err != nil {
        http.Error(w, "read error", http.StatusBadRequest)
        return
    }

    sigHeader := r.Header.Get("Stripe-Signature")
    event, err := webhook.ConstructEvent(body, sigHeader, webhookSecret)
    if err != nil {
        http.Error(w, "invalid signature", http.StatusUnauthorized)
        return
    }

    // Idempotency check
    if exists, _ := eventStore.Exists(r.Context(), event.ID); exists {
        w.WriteHeader(http.StatusOK)
        return
    }

    // Persist raw event, process asynchronously
    if err := eventStore.Save(r.Context(), event.ID, event.Type, body); err != nil {
        http.Error(w, "storage error", http.StatusInternalServerError)
        return
    }
    if err := jobQueue.Enqueue(r.Context(), "process_stripe_event", event.ID); err != nil {
        log.Printf("enqueue failed for %s: %v", event.ID, err)
    }
    w.WriteHeader(http.StatusOK)
}

Go: Outbound Webhook Delivery with Retry

func DeliverWebhook(ctx context.Context, sub Subscriber, event Event) error {
    payload, _ := json.Marshal(event)
    sig := computeHMAC(payload, sub.Secret)

    req, _ := http.NewRequestWithContext(ctx, "POST", sub.URL, bytes.NewReader(payload))
    req.Header.Set("Content-Type", "application/json")
    req.Header.Set("X-Webhook-Signature", sig)
    req.Header.Set("X-Event-ID", event.ID)

    client := &http.Client{Timeout: 10 * time.Second}
    resp, err := client.Do(req)
    if err != nil {
        return fmt.Errorf("delivery failed: %w", err)
    }
    defer resp.Body.Close()

    if resp.StatusCode < 200 || resp.StatusCode >= 300 {
        return fmt.Errorf("non-2xx response: %d", resp.StatusCode)
    }
    return nil
}

func computeHMAC(payload []byte, secret string) string {
    mac := hmac.New(sha256.New, []byte(secret))
    mac.Write(payload)
    return hex.EncodeToString(mac.Sum(nil))
}

Common Pitfalls

Key Concepts Recap

Further Reading