Duplicate runs

Goal

Know where duplicate runs come from in production and apply the right defense to webhooks, jobs, cron and emails.

The problem

You added idempotency to the payment endpoint (2.2). Linkr still sends some users the same welcome email twice, the weekly report goes out in triplicate and, every now and then, a credit is granted twice.

These repeats come from the infrastructure, which repeats things by design.

The six sources of repetition

flowchart LR
  subgraph Source
    A[User<br/>double click]
    B[Client retry<br/>after timeout]
    C[Proxy or LB<br/>resending]
    D[Queue delivering<br/>again]
    E[Webhook<br/>resent]
    F[Cron running on<br/>N instances]
  end
  A & B & C & D & E & F --> X{{Your code runs<br/>2+ times}}

The trickiest case is the timeout that succeeded: the server ran the operation, but the response didn’t arrive in time. From the client’s side, failure and success look the same.

sequenceDiagram
  participant C as Client
  participant S as Server
  C->>S: POST /credits (5 s timeout)
  S->>S: grants 100 credits (takes 6 s)
  Note over C: 5 s: timeout → "failed"
  C->>S: retry
  S->>S: grants another 100 credits
  S-->>C: 200

A sender in a distributed system can only choose between at most once (may lose messages) and at least once (may duplicate them). Queues, webhooks and retries choose “at least once”, so the receiver has to tolerate duplicates.

Source → defense

Source	Defense
Double click	Disable the button (helps the experience) and an idempotency key (actually guarantees it)
Retry after timeout	Idempotency key passed end to end
Resent webhook	Processed-events table with the event ID as a unique key
Queue (at-least-once)	Idempotent consumer: conditional transition or dedupe by message ID
Cron on several instances	Distributed lock (Postgres advisory lock) or a single scheduler
Job interrupted and restarted	Idempotent steps and a progress checkpoint

In practice

Webhooks: dedupe by event ID

Every serious provider (Stripe, GitHub, Mercado Pago) resends webhooks when it doesn’t get a 2xx in time, and sometimes resends even when it did. Each event has a stable ID. Use it:

CREATE TABLE processed_events (
  event_id     text PRIMARY KEY,
  processed_at timestamptz NOT NULL DEFAULT now()
);

app.post('/webhooks/stripe', express.raw({ type: 'application/json' }), async (req, res) => {
  const event = stripe.webhooks.constructEvent(req.body, req.header('stripe-signature')!, WEBHOOK_SECRET);

  await db.transaction(async (tx) => {
    // Record the event and apply the effect in the SAME transaction:
    // either both happen, or neither does.
    const fresh = await tx.query(
      'INSERT INTO processed_events (event_id) VALUES ($1) ON CONFLICT DO NOTHING RETURNING event_id',
      [event.id],
    );
    if (fresh.rowCount === 0) return; // duplicate: ignore

    await applyStripeEvent(tx, event);
  });

  res.sendStatus(200);
});

The transaction is what makes this work. If you mark the event as processed and the process dies before applying the effect, the event is lost; if you apply the effect and die before marking it, it repeats. Put both in one transaction and neither can happen. (Payment webhooks have more details, such as event ordering, which are covered in M28.)

Cron: one instance at a time

A node-cron or setInterval inside the app runs on every instance. With three instances, the report goes out three times. A Postgres advisory lock solves this with no new infrastructure:

// The number identifies the job; any fixed bigint works.
const WEEKLY_REPORT_LOCK = 72_001;

export async function runWeeklyReport() {
  // The lock belongs to the session (connection), so use the same connection from start to finish.
  const client = await pool.connect();
  let got = false;
  try {
    const { rows } = await client.query('SELECT pg_try_advisory_lock($1) AS got', [WEEKLY_REPORT_LOCK]);
    got = rows[0].got;
    if (!got) return; // another instance is already running it

    await sendWeeklyReports(client);
  } finally {
    if (got) await client.query('SELECT pg_advisory_unlock($1)', [WEEKLY_REPORT_LOCK]);
    client.release();
  }
}

The lock prevents simultaneous runs. If the job fails and runs again later, it still needs to be idempotent, so combine the lock with a record of what has already been done.

Emails and notifications: record the send

CREATE TABLE sent_notifications (
  user_id  bigint NOT NULL,
  kind     text   NOT NULL,     -- 'welcome', 'weekly_report'
  period   text   NOT NULL,     -- 'once', '2026-W42'
  sent_at  timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (user_id, kind, period)
);

Before sending, insert with ON CONFLICT DO NOTHING. If no row was inserted, someone already sent it. The order matters: recording before sending can lose an email if the send fails; recording after can duplicate it if the process crashes in between. For email, a rare duplicate is usually better than a lost message. For money it never is, so decide the order case by case.

Long jobs: checkpoints

A job that processes 100,000 links and dies at item 60,000 shouldn’t start over from zero or reapply the effect to the first 60,000. Process in batches and record progress:

let lastId = await getCheckpoint('recalc-stats'); // 0 the first time
while (true) {
  const { rows } = await db.query(
    'SELECT id FROM links WHERE id > $1 ORDER BY id LIMIT 1000', [lastId],
  );
  if (rows.length === 0) break;
  await recalcStats(rows.map((r) => r.id)); // must be idempotent
  lastId = rows.at(-1).id;
  await saveCheckpoint('recalc-stats', lastId);
}

Pitfalls

• Assuming “exactly once” comes for free. Queues and providers that promise exactly-once delivery get there with deduplication within limits. In your own code, assume “at least once”.
• Marking as processed outside the effect’s transaction. This is the classic source of lost or duplicated events.
• Returning a 500 for a duplicate webhook. The provider will keep resending. A recognized duplicate is a success: respond 200.
• What AI tends to generate: webhook handlers with no dedupe, node-cron inside the web server and jobs that reprocess everything from the start.

Go deeper

Materials in English unless noted.

• Article
  You Cannot Have Exactly-Once DeliveryTyler Treat

  Why "exactly once" delivery doesn't exist, and what you get instead.

• Docs
  WebhooksStripe

  Official best practices: signatures, fast responses, duplicate and out-of-order events.

• Docs
  Advisory LocksPostgreSQL

  Session-level and transaction-level locks, used in the cron example.

• Article
  Transactional OutboxChris Richardson

  The pattern for reliably publishing events together with a database write.