By KYN AI Advisory Team — AI implementation specialists, Singapore
Treat every webhook as unreliable. It can arrive twice, late, out of order or not at all. Make your agent safe with four controls: verify the signature, record each event id before acting, make each action idempotent so a repeat has no extra effect, and retry failures with backoff into a dead-letter queue that a person reviews. Without these, a payment notice can create two invoices, or a reply can trigger two emails.
Why do webhooks cause duplicate actions?
A webhook is a message one system sends to another when something happens. The sender expects a quick success response. If your endpoint is slow, errors or times out, the sender assumes the message failed and sends it again. This is by design: delivery is "at least once", not "exactly once".
So the same event can reach your agent two or three times. If the agent's action is "send a confirmation email", the customer gets three. If it is "create a payment record", your ledger is wrong. Agents make this worse because their steps are slower and less predictable than a plain function, so timeouts are more likely.
What is idempotency?
An operation is idempotent when running it twice has the same effect as running it once. "Set the status to paid" is idempotent. "Add 100 to the balance" is not.
Payment providers built this into their APIs. Stripe documents idempotent requests: you send a unique key with a request, and a repeat with the same key returns the original result instead of acting again. Use the same idea in your own system.
How do you build it?
Follow these steps for each webhook endpoint.
- Verify the sender. Check the signature or shared secret in the header before reading the body. Reject anything unsigned. This also blocks forged events.
- Acknowledge fast. Store the raw event and return success within a second or two. Do the work afterwards, from a queue.
- Record the event id. Insert the id into a table with a unique constraint. If the insert fails, the event is a duplicate: acknowledge and stop.
- Act idempotently. Derive a key from the event, such as the event id plus the action name, and pass it to every downstream call that supports one.
- Check state, not memory. Before acting, read the current state. If the invoice is already marked paid, do nothing.
- Write the outcome. Log the event, the action, the result and the actor.
A minimal table:
create table webhook_events (
event_id text primary key,
source text not null,
received_at timestamptz not null default now(),
processed_at timestamptz,
status text not null default 'received',
payload jsonb not null
);The primary key on event_id does the deduplication for you.
How should retries work?
Retries are for temporary failure, such as a timeout or a rate limit. They are not for bad data.
- Backoff. Wait longer between attempts, for example 1 minute, 5 minutes, 30 minutes, with a little randomness so retries do not arrive together.
- Cap. Stop after a set number of attempts.
- Classify. A timeout or a 5xx error is worth retrying. A validation error is not, because the same input will fail again.
- Dead-letter queue. After the cap, move the event to a queue a person reviews. Show the payload, the error and a replay button.
The replay button should be safe because every step is idempotent.
What about order?
Events can arrive out of order. An "invoice updated" message may reach you before "invoice created". Two defences work:
- Fetch the current object. Treat the webhook as a nudge. Read the latest state from the source system's API before acting.
- Compare versions. If the event carries a timestamp or version, ignore any event older than what you have stored.
Where do agents add risk?
Three places stand out.
- Side effects inside a loop. An agent may call "send message" several times while reasoning. Put a guard on the tool, not in the prompt: the tool refuses a second send for the same key.
- Long runs. A run that takes two minutes will exceed most webhook timeouts. Always hand off to a queue.
- Actions that should not repeat at all. Money movement, filings and deletions wait for a named person. A duplicate approval request is harmless. A duplicate payment is not. See approval gates for high-risk agent actions.
Related issues, such as database connections running out when many agent runs start together, are covered in database connection pool exhaustion in async automation agents.
How do you test it?
Replay the same event ten times and confirm one effect. Send events in reverse order. Kill the worker mid-run and restart it. Return an error from a downstream system and watch the backoff. Then check that the dead-letter queue shows the failure with a payload you can replay.
What does a duplicate look like in practice?
Take a payment webhook that fires when a customer pays an invoice (illustrative). The sender does not get a reply in time, so it sends the event again.
Without a guard, the agent posts two receipts and sends two thank-you messages. With a guard, the flow is:
- The event arrives with an id, such as a payment reference.
- The system tries to insert that id into a table with a unique constraint.
- The first attempt succeeds and the work proceeds.
- The second attempt fails the constraint. The system logs it as a duplicate and returns success to the sender, so it stops retrying.
Choose the key with care. A payment reference is a good key. A timestamp is not. Two different payments can share a second, and one payment can arrive with different timestamps.
Test it by replaying the same event five times and sending two events out of order. The result should be one receipt, one message and a log showing four ignored duplicates.
What should you monitor?
Monitor four numbers per integration: events received, events processed, duplicates ignored and failures waiting for retry. Alert when failures wait longer than a set time, not on every failure, so people trust the alert. Keep a dead-letter list for events that fail after all retries, with the payload and the error, and assign a person to clear it each day. Review the duplicates count weekly. A sudden rise means the sender changed its behaviour or your response is too slow. Keep payloads for a fixed period so you can replay an event after a fix, and delete them when the period ends.
Where KYN fits
KYN builds integrations with these controls from the first release. In the manufacturer's finance ledger of 3,309 reconciled rows, imports are repeatable and produced zero duplicates. Agents have their own restricted accounts, every action is logged with the actor, and outbound messages, money, filings and deletions wait for a named person.
The first 14 days are a review of your stack and interviews with your team. A full build ships in under 90 days. See integrations and finance systems.
Frequently asked questions
What is the difference between a retry and a duplicate?
A retry is the sender trying again because it thought the first attempt failed. A duplicate is what you receive if the first attempt had actually worked. You cannot tell them apart, so you handle both with idempotency.
Do I need a message queue?
For anything beyond trivial volume, yes. A queue lets you acknowledge fast, process later, retry safely and hold failures for review. A database table can serve as a small queue at SME scale.
How long should I keep processed event ids?
Keep them at least as long as the sender may retry or replay events, and as long as you need the audit trail. Check the provider's documentation for its retry window.
Checked 30 October 2026.
Curious whether Marketing agents fits your business? Talk to KYN on WhatsApp — no forms, just a conversation.
