Table of Contents
- The Production Incident
- Understanding Webhook Delivery Guarantees
- What is Idempotency?
- Implementation: 3 Patterns
- Webhook Security
- Testing Idempotency
- Best Practices
The 3 AM Wake-Up Call
Phone buzzes at 3 AM.
“Hey, merchants are reporting they’ve been charged payout fees twice. I checked and found several merchants with double charges…”
That message from our support team was my introduction to the harsh reality of webhook idempotency in payment systems.
Today I’m sharing the story of this production bug, why it happened, and more importantly - how to build systems that handle webhooks being sent “at least once” while ensuring your logic runs “exactly once.”
(Spoiler: It’s not as simple as checking the database.)
The Production Incident
The Context
Our payment system integrates with a payment gateway (let’s call it “PaymentProvider”). Here’s how payouts worked:
Flow:
1. Merchant requests payout (withdraw money to bank)
2. PaymentProvider processes payout
3. PaymentProvider sends webhook: "Payout successful"
4. Our system charges 0.5% payout fee
5. Update merchant balance
The Naive Implementation
Here’s what our code looked like initially:
@Controller('webhooks')
export class WebhookController {
@Post('payment-provider')
async handlePayoutWebhook(@Body() event: PayoutEvent) {
const payout = await this.payoutService.findOne(event.payoutId);
// ❌ What's wrong with this logic?
if (payout.status !== 'completed') {
// Charge 0.5% fee
const fee = payout.amount * 0.005;
await this.feeService.createPayoutFee({
payoutId: payout.id,
amount: fee
});
// Update status
await this.payoutService.update(payout.id, {
status: 'completed'
});
}
return { received: true };
}
}
Looks reasonable, right? We check the status before charging the fee. What could go wrong?
The Incident Timeline
10:15:00 - Merchant A requests payout of $10,000
10:15:30 - PaymentProvider processes payout, sends webhook #1
10:15:30 - Webhook #1 arrives at our server, starts processing
10:15:30 - Code checks status: 'pending' ✅
10:15:30 - Charges fee: $50
10:15:31 - PaymentProvider retries webhook (didn't get response in time)
10:15:31 - Webhook #2 arrives at our server
10:15:31 - Code checks status: still 'pending' ✅ (first request hasn't finished!)
10:15:31 - Charges ANOTHER fee: $50 ❌
Result: Merchant charged $100 instead of $50
Root Cause Analysis
Problem #1: Race Condition
- Webhook #1 is processing (hasn't updated status yet)
- Webhook #2 arrives and checks status
- Both see status = 'pending'
- Both decide "haven't charged fee yet"
- Both charge the fee → DOUBLE CHARGE
Problem #2: Webhook Delivery Semantics
Payment providers guarantee "at-least-once delivery"
- They don't guarantee "exactly-once"
- Webhooks can be sent multiple times due to:
→ Network timeouts
→ Retry logic
→ Provider-side failures
→ Our server being slow to respond
This is not a bug in the payment provider - it’s by design.
Understanding Webhook Delivery Guarantees
Three Types of Delivery Semantics
┌─────────────────┬──────────────────┬────────────────────┐
│ Type │ Behavior │ Real World │
├─────────────────┼──────────────────┼────────────────────┤
│ At-most-once │ Send once, no │ Unreliable, rarely │
│ │ retry if fails │ used for webhooks │
├─────────────────┼──────────────────┼────────────────────┤
│ At-least-once ← │ Send until gets │ Stripe, PayPal, │
│ (Most common) │ success response │ Shopify, Square │
│ │ May duplicate │ ← STANDARD │
├─────────────────┼──────────────────┼────────────────────┤
│ Exactly-once │ Guaranteed once │ Expensive/complex │
│ │ (theoretical) │ (Apache Kafka) │
└─────────────────┴──────────────────┴────────────────────┘
Why “At-Least-Once” is Standard
Imagine you’re a payment provider:
Scenario 1: At-Most-Once
Provider: "I'm sending webhook for $10,000 payout"
*Network timeout*
Provider: "Oh well, not retrying"
→ Merchant never knows payout succeeded ❌
Scenario 2: At-Least-Once
Provider: "I'm sending webhook for $10,000 payout"
*Network timeout*
Provider: "No response, retrying in 5 seconds..."
Provider: "Sending again..."
*Server responds OK*
→ Merchant gets notified (might receive duplicates) ✅
Conclusion: At-least-once delivery is the best trade-off for reliability.
Who’s Responsible for What?
Payment Provider's job: "Webhook will arrive at least once"
Our job: "Logic runs EXACTLY once"
→ This is called IDEMPOTENCY
What is Idempotency?
Definition
Idempotency: Performing an operation multiple times produces the same result as performing it once.
Real-world examples:
✅ Idempotent:
SET temperature = 25°C
(set 10 times → still 25°C)
❌ Not idempotent:
INCREMENT temperature +5°C
(increment 10 times → 50°C increase!)
✅ Idempotent:
UPDATE users SET name = 'John' WHERE id = 1
❌ Not idempotent:
INSERT INTO payments (amount) VALUES (100)
Idempotency in Webhooks
// ❌ Not idempotent
function chargePayoutFee(payoutId: string) {
await db.insert('fees', {
payoutId,
amount: 50
});
// Called twice → 2 records → wrong
}
// ✅ Idempotent
function chargePayoutFee(payoutId: string, idempotencyKey: string) {
await db.insert('fees', {
payoutId,
amount: 50,
idempotencyKey // ← Unique constraint
});
// Called twice → second fails with duplicate key → idempotent
}
The Idempotency Key
An idempotency key is a unique identifier for each webhook event.
Common implementations:
Stripe: event.id (e.g., "evt_1ABC...")
PayPal: event_id
Shopify: X-Shopify-Webhook-Id header
Square: event_id
→ Use this key to detect duplicate webhooks
Implementation: 3 Patterns for Idempotency
Pattern #1: Database Unique Constraint (Recommended)
This is the simplest and most reliable approach.
Step 1: Database Schema
CREATE TABLE payout_fees (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
payout_id UUID NOT NULL,
amount DECIMAL(10, 2) NOT NULL,
idempotency_key VARCHAR(255) NOT NULL UNIQUE, -- ← The magic
created_at TIMESTAMP DEFAULT NOW()
);
-- Ensure idempotency key is unique
CREATE UNIQUE INDEX idx_payout_fees_idempotency
ON payout_fees(idempotency_key);
-- Also ensure one payout = one fee
CREATE UNIQUE INDEX idx_payout_fees_payout_id
ON payout_fees(payout_id);
Step 2: Service Implementation
@Injectable()
export class PayoutFeeService {
constructor(
private readonly feeRepository: FeeRepository,
private readonly logger: Logger
) {}
async chargePayoutFee(
payoutId: string,
amount: number,
idempotencyKey: string
): Promise<PayoutFee> {
try {
// Attempt to insert with idempotency key
const fee = await this.feeRepository.create({
payoutId,
amount,
idempotencyKey
});
this.logger.log(`Fee charged successfully: ${fee.id}`);
return fee;
} catch (error) {
// Check if it's a duplicate key error
if (this.isDuplicateKeyError(error)) {
this.logger.log(
`Duplicate webhook detected and ignored: ${idempotencyKey}`
);
// Return existing fee (idempotent response)
const existingFee = await this.feeRepository.findOne({
idempotencyKey
});
return existingFee;
}
// Re-throw other errors
throw error;
}
}
private isDuplicateKeyError(error: any): boolean {
// PostgreSQL unique violation code
return error.code === '23505';
}
}
Step 3: Webhook Handler
@Controller('webhooks')
export class WebhookController {
constructor(
private readonly webhookService: WebhookService,
private readonly payoutService: PayoutService,
private readonly feeService: PayoutFeeService,
private readonly logger: Logger
) {}
@Post('payment-provider')
async handlePayoutWebhook(
@Body() event: PayoutWebhookEvent,
@Headers('x-webhook-signature') signature: string
) {
// Step 1: Verify webhook signature (security)
await this.webhookService.verifySignature(event, signature);
// Step 2: Use event ID as idempotency key
const idempotencyKey = event.id; // e.g., "evt_123abc"
// Step 3: Get payout details
const payout = await this.payoutService.findOne(event.payoutId);
// Step 4: Calculate fee
const feeAmount = payout.amount * 0.005; // 0.5%
// Step 5: Charge fee (idempotent!)
await this.feeService.chargePayoutFee(
payout.id,
feeAmount,
idempotencyKey // ← Magic happens here
);
return { received: true };
}
}
Why This Works:
First webhook arrives:
- Insert with idempotency_key = "evt_123"
- Success → Fee charged
Second webhook arrives (duplicate):
- Try to insert with idempotency_key = "evt_123"
- Fails: Unique constraint violation
- Catch error, return existing fee
- No double charge! ✅
Pros:
✅ Simple to implement
✅ Database enforces uniqueness (single source of truth)
✅ Atomic operation (no race conditions)
✅ No external dependencies (Redis, etc.)
✅ Works with any SQL database
Cons:
❌ Relies on database constraint
❌ Less flexible for complex scenarios
Pattern #2: Redis Distributed Lock
⚠️ Warning: This pattern is more complex than Pattern #1. Only use it if you have specific requirements.
When you might need this:
- You’re processing webhooks on multiple servers simultaneously
- Webhook processing takes >5 seconds (long operations)
- You want to wait for first request to finish instead of failing fast
Core concept:
async chargePayoutFee(payoutId: string, amount: number, idempotencyKey: string) {
const lockKey = `lock:payout-fee:${idempotencyKey}`;
// Try to acquire lock
const lockAcquired = await redis.set(lockKey, '1', 'EX', 30, 'NX');
if (!lockAcquired) {
// Another server is processing, wait for it to finish
await this.waitForCompletion(lockKey);
return this.feeRepository.findOne({ idempotencyKey });
}
try {
// We got the lock - check if already processed
const existing = await this.feeRepository.findOne({ idempotencyKey });
if (existing) return existing;
// Process fee
return await this.feeRepository.create({ payoutId, amount, idempotencyKey });
} finally {
// Always release lock
await redis.del(lockKey);
}
}
Pros:
✅ Handles concurrent requests across multiple servers
✅ Prevents race conditions completely
✅ Second request waits instead of failing
Cons:
❌ Significantly more complex than Pattern #1
❌ Requires Redis (external dependency)
❌ What if Redis goes down? (need fallback)
❌ Lock timeout needs careful tuning
❌ Waiting logic can be tricky
Reality check: Pattern #1 (unique constraint) is enough for 95% of use cases. Only implement Redis locks if you have a proven need.
Pattern #3: State Machine with Pessimistic Locking
For workflows with complex state transitions:
// Database schema with state tracking
enum PayoutState {
PENDING = 'pending',
FEE_CHARGED = 'fee_charged',
COMPLETED = 'completed'
}
// Entity
class Payout {
id: string;
amount: number;
status: PayoutState;
version: number; // For optimistic locking
}
@Injectable()
export class PayoutFeeService {
async chargePayoutFee(payoutId: string): Promise<PayoutFee> {
// Use database transaction
return this.entityManager.transaction(async (em) => {
// Lock the payout row (SELECT FOR UPDATE)
const payout = await em.findOne(Payout, payoutId, {
lock: { mode: 'pessimistic_write' }
});
// Check state - only charge if pending
if (payout.status !== PayoutState.PENDING) {
this.logger.log('Fee already charged, skipping');
return em.findOne(PayoutFee, { payoutId });
}
// Create fee
const fee = em.create(PayoutFee, {
payoutId: payout.id,
amount: payout.amount * 0.005
});
await em.save(fee);
// Update state atomically
payout.status = PayoutState.FEE_CHARGED;
await em.save(payout);
return fee;
});
}
}
Pros:
✅ State machine is easy to reason about
✅ Transaction guarantees consistency
✅ No separate idempotency key needed
✅ Works well for complex workflows
Cons:
❌ Tight coupling with payout entity
❌ Pessimistic lock can slow things down
❌ Harder to scale horizontally
Comparison Table
┌────────────┬─────────────┬──────────────┬─────────────┐
│ Pattern │ Complexity │ Performance │ Use Case │
├────────────┼─────────────┼──────────────┼─────────────┤
│ Unique │ Simple │ Fast │ ✅ Default │
│ Constraint │ ⭐⭐ │ ⭐⭐⭐ │ choice │
├────────────┼─────────────┼──────────────┼─────────────┤
│ Redis Lock │ Complex │ Medium │ High conc. │
│ │ ⭐⭐⭐ │ ⭐⭐ │ multi-server│
├────────────┼─────────────┼──────────────┼─────────────┤
│ State │ Medium │ Fast │ Complex │
│ Machine │ ⭐⭐⭐ │ ⭐⭐⭐ │ workflows │
└────────────┴─────────────┴──────────────┴─────────────┘
Recommendation: Start with Pattern #1 (Unique Constraint). It’s simple, reliable, and covers 90% of use cases.
Critical: Webhook Security
Before processing ANY webhook, you MUST verify its signature:
@Post('webhooks/payment-provider')
async handleWebhook(
@Body() event: WebhookEvent,
@Headers('x-webhook-signature') signature: string
) {
// Step 1: ALWAYS verify signature first
const isValid = this.verifyWebhookSignature(
JSON.stringify(event),
signature,
process.env.WEBHOOK_SECRET
);
if (!isValid) {
throw new UnauthorizedException('Invalid webhook signature');
}
// Step 2: Process webhook (idempotent)
await this.processWebhook(event);
}
private verifyWebhookSignature(
payload: string,
signature: string,
secret: string
): boolean {
const hmac = crypto.createHmac('sha256', secret);
const digest = hmac.update(payload).digest('hex');
return crypto.timingSafeEqual(
Buffer.from(signature),
Buffer.from(digest)
);
}
Why this is critical:
❌ Without verification:
- Attackers can send fake webhooks
- Charge fees arbitrarily
- Manipulate balances
- Steal money
✅ With verification:
- Only legitimate provider can send webhooks
- Webhooks are cryptographically verified
- System is secure
Never skip signature verification in production!
Handling Out-of-Order Webhooks
Webhooks can arrive out of order:
Timeline (reality):
10:00 - Payout created (webhook #1)
10:01 - Payout processing (webhook #2)
10:02 - Payout completed (webhook #3)
Timeline (webhooks arrive):
10:00:10 - Webhook #1 arrives ✅
10:01:05 - Webhook #3 arrives ❌ (too early!)
10:01:15 - Webhook #2 arrives ❌ (delayed)
Solution: Use timestamp or version number:
async processPayoutWebhook(event: PayoutWebhookEvent) {
const payout = await this.payoutRepo.findOne(event.payoutId);
// Ignore outdated webhooks
if (event.timestamp <= payout.lastWebhookTimestamp) {
this.logger.log('Outdated webhook, ignoring');
return;
}
// Process and update timestamp
await this.updatePayoutStatus(payout, event);
payout.lastWebhookTimestamp = event.timestamp;
await this.payoutRepo.save(payout);
}
Idempotency Window: How Long to Store Keys?
Question: How long should we keep idempotency keys?
Recommended: 24-72 hours
Why 24-72 hours?
✅ Long enough for all webhook retry scenarios
✅ Most providers stop retrying after 24 hours
✅ Prevents database bloat from old keys
Clean up old keys with a cron job:
@Cron('0 0 * * *') // Daily at midnight
async cleanupOldIdempotencyKeys() {
const threeDaysAgo = subDays(new Date(), 3);
await this.db.query(`
DELETE FROM payout_fees
WHERE created_at < $1
`, [threeDaysAgo]);
}
Most payment providers (Stripe, PayPal) stop retrying webhooks after 3 days, so this window is safe.
Testing Idempotency
Local Testing: Simulate Webhooks with curl
Before writing automated tests, verify your implementation works locally:
# Step 1: Start your server locally
npm run dev
# Step 2: Send the same webhook 5 times concurrently
for i in {1..5}; do
curl -X POST http://localhost:3000/webhooks/payout \
-H "Content-Type: application/json" \
-H "x-webhook-signature: test_signature" \
-d '{
"id": "evt_test_duplicate",
"payoutId": "payout_123",
"amount": 10000,
"status": "completed"
}' &
done
wait
# Step 3: Check database - should have only 1 fee
psql -d mydb -c "SELECT COUNT(*) FROM payout_fees WHERE idempotency_key = 'evt_test_duplicate';"
# Expected: 1 (not 5!)
Testing with a real payment provider:
# Use ngrok to expose local server
ngrok http 3000
# Update webhook URL in payment provider dashboard to:
https://your-ngrok-url.ngrok.io/webhooks/payout
# Trigger test webhook from provider dashboard
# Then check your logs and database
Unit Test: Duplicate Webhooks
describe('PayoutFeeService', () => {
it('should charge fee only once for duplicate webhooks', async () => {
const payoutId = 'payout_123';
const idempotencyKey = 'evt_abc';
const amount = 50;
// First webhook
const fee1 = await service.chargePayoutFee(
payoutId,
amount,
idempotencyKey
);
// Duplicate webhook (same idempotency key)
const fee2 = await service.chargePayoutFee(
payoutId,
amount,
idempotencyKey
);
// Should return same fee object
expect(fee1.id).toBe(fee2.id);
// Should only have 1 record in database
const count = await feeRepo.count({ payoutId });
expect(count).toBe(1);
});
});
Integration Test: Concurrent Webhooks
describe('PayoutWebhookController (Integration)', () => {
it('should handle 10 concurrent duplicate webhooks', async () => {
const event = {
id: 'evt_duplicate_test',
payoutId: 'payout_123',
amount: 10000,
status: 'completed'
};
// Send 10 webhooks concurrently
const promises = Array(10).fill(null).map(() =>
request(app.getHttpServer())
.post('/webhooks/payout')
.set('x-webhook-signature', generateSignature(event))
.send(event)
.expect(200)
);
await Promise.all(promises);
// Should only create 1 fee
const fees = await feeRepo.find({
payoutId: 'payout_123'
});
expect(fees).toHaveLength(1);
expect(fees[0].amount).toBe(50); // 0.5% of 10000
});
});
Manual Testing
# Test duplicate webhooks manually
for i in {1..5}; do
curl -X POST http://localhost:3000/webhooks/payout \
-H "Content-Type: application/json" \
-H "x-webhook-signature: <signature>" \
-d '{
"id": "evt_test_123",
"payoutId": "payout_xyz",
"amount": 10000,
"status": "completed"
}' &
done
wait
# Check database
psql -d mydb -c "
SELECT COUNT(*) FROM payout_fees
WHERE payout_id = 'payout_xyz';
"
# Expected: 1
Best Practices Checklist
Webhook Idempotency Checklist:
✅ Verify webhook signature BEFORE processing
✅ Use idempotency key (event.id) with unique constraint
✅ Handle duplicate key errors gracefully
✅ Return 200 OK even for duplicate webhooks
✅ Log duplicate webhooks for monitoring
✅ Handle out-of-order webhooks (timestamp check)
✅ Test with concurrent duplicate webhooks
✅ Clean up old idempotency keys (24-72h window)
✅ Use transactions for multi-step operations
✅ Monitor duplicate webhook rate
Security Checklist:
✅ Verify HMAC signature on every webhook
✅ Use timing-safe comparison (crypto.timingSafeEqual)
✅ Store webhook secret in environment variables
✅ Rotate webhook secrets periodically
✅ Rate limit webhook endpoints
✅ Log failed signature verifications (potential attacks)
The Results
After implementing idempotency with Pattern #1:
✅ Zero duplicate charges since deployment (6 months and counting)
✅ Handled 50,000+ webhooks without issues
✅ Processed 1,247 duplicate webhooks correctly (2.5% duplicate rate)
✅ No production incidents related to webhook processing
Monitoring metrics:
- Webhook success rate: 99.97%
- Duplicate webhook rate: 2.5% (normal)
- Average processing time: 120ms
- Failed signature verifications: 3 (potential attacks, blocked)
Key Takeaways
1️⃣ Payment providers send webhooks “at-least-once” → Prepare for duplicates
2️⃣ Idempotency key = event.id → Use it with unique constraint
3️⃣ Database unique constraint is simple & effective → Pattern #1 covers 90% of cases
4️⃣ ALWAYS verify webhook signatures → Security is not optional
5️⃣ Test with concurrent requests → Race conditions are real
6️⃣ Clean up old keys → 24-72 hour window is enough
7️⃣ Log duplicate webhooks → Monitor for abnormal patterns
8️⃣ Return idempotent responses → Same input = same output
Remember: “Webhooks being sent multiple times is normal. Your system processing them exactly once is your responsibility.”
Resources
- Stripe Webhook Best Practices
- Idempotency Keys - RFC Draft
- AWS Builders Library: Making Retries Safe with Idempotent APIs
Have you dealt with webhook idempotency issues? What patterns have worked for you? Share your experience in the comments!
]]>
The Django model offers various types of relationships such as OneToOneField, ForeignKey, ManyToManyField, and GenericForeignKey.
