Assume your agent will run the workflow twice

The agent calls the payment API. The request leaves. Then the connection drops before the response comes back.

Now the workflow is stuck on a question that has no local answer: did the charge go through, or not? Retry and you might bill the customer twice. Give up and you might have taken money without recording it. The model cannot see the difference, because from inside the workflow both outcomes look identical: a call that did not return.

That gap is the reliability problem in connected agent workflows, and it is not a model problem. It is the same partial-failure problem distributed systems have always had. The useful design question is not “will this step run” but “is running this step a second time safe.”

The retry is not optional, so plan for it

Retries are built into the substrate agent workflows run on.

Message queues that many workflows sit behind deliver at least once by design. Amazon’s documentation is explicit that standard SQS queues “ensure at-least-once message delivery, but due to the highly distributed architecture, more than one copy of a message might be delivered” Amazon SQS standard queues. If your orchestration, tool-call transport, or background job runner has that property anywhere in the chain, duplicate execution is a normal event, not an incident.

Agents add their own retry surfaces on top of that:

Source of a repeat	What actually happens
Network timeout	The tool call is retried because the response never arrived, even though the action succeeded
Orchestrator restart	A resumed or re-queued run replays a step that already ran
Background task re-fire	An async step that was not collected gets re-issued on the next run
Model re-planning	The agent decides to call the same tool again after an ambiguous result
Operator re-run	A human retries a “failed” task that had already taken the action

OWASP’s agentic AI guidance frames these systems as autonomous ones that plan and act across workflows with expanded capabilities OWASP Agentic AI - Threats and Mitigations. More planning and more acting means more places where the same real-world action can be triggered twice. A workflow that assumes exactly-once execution is making a promise the infrastructure underneath it does not keep.

Idempotency is a property of the action, not the prompt

You cannot instruct your way out of this with a line that says “do not repeat actions.” The agent has no reliable way to know whether the last call landed. The property has to live in the acting step.

An operation is idempotent when running it more than once has the same effect as running it once. RFC 9110 defines a method as idempotent when “the intended effect on the server of multiple identical requests with that method is the same as the effect for a single such request,” and it classifies PUT and DELETE as idempotent while POST and PATCH are not RFC 9110: HTTP Semantics. Note the wording: idempotency is about server state, not about getting the same response every time. A second DELETE can return a different status code and still be idempotent, because the state it leaves behind is the same.

That distinction matters for agent tools. “Set the ticket status to closed” is naturally idempotent: run it twice and the ticket is closed. “Add a comment to the ticket” is not: run it twice and there are two comments. When you register a tool for an agent, sort its actions by this property before you sort them by anything else.

The idempotency key is the standard fix

For actions that are not naturally idempotent, the established pattern is a client-generated idempotency key that the server uses to recognize a retry.

Stripe’s implementation is a good reference because it is precise about the failure it solves. A client generates a unique key so the server can “recognize subsequent retries of the same request,” and Stripe works “by saving the resulting status code and body of the first request made for any given idempotency key, regardless of whether it succeeds or fails.” Subsequent requests with the same key “return the same result, including 500 errors,” and keys are retained for at least 24 hours Stripe idempotent requests.

Read that carefully, because it answers the timeout question the agent could not. If the first call succeeded, the retry returns the recorded success without charging again. If the first call failed, the retry returns the recorded failure. Either way the customer is billed at most once, and the workflow gets a definite answer instead of a guess.

To apply this in an agent workflow:

• Generate the key once, at the point the workflow decides to act, not each time it attempts the call. A key regenerated on every retry defeats the entire mechanism.
• Derive the key from the intent, so the same logical action reuses the same key across retries, restarts, and re-plans. A random key created fresh on a resumed run is a new action to the server.
• Prefer downstream systems that support idempotency keys natively. When a system does not, put the dedup in a service layer in front of it rather than trusting the agent to remember.
• Store the key with the task record, so a later run or a human retry reuses it instead of starting over.

This is opinion grounded in the sources above, not a vendor claim: the key belongs to the logical action, and the hardest part is choosing where that identity comes from.

Retries should back off, not hammer

Making an action safe to repeat is only half the design. The other half is retrying well.

When many agents or workers retry a struggling dependency at the same fixed interval, they synchronize and arrive together, which keeps the dependency down. Marc Brooker’s AWS analysis is blunt about the fix: “The solution isn’t to remove backoff. It’s to add jitter,” and full jitter, where the wait is a random value between zero and a capped exponential backoff, “cut down work substantially” compared to approaches without it Exponential Backoff And Jitter.

For agent workflows, add two rules on top:

• Do not blindly retry a non-idempotent action that has no idempotency key. A failed call with no dedup and no confirmed outcome should surface for review, not auto-retry into a duplicate.
• Cap the attempts and record why the workflow gave up. An agent that retries forever is a slow-motion outage with a log nobody reads.

Who needs this, and who does not

This applies to any workflow where an agent can take an action that touches the outside world: charging money, sending a message, creating or mutating a record, provisioning a resource, deploying, or triggering another system. It is most urgent when those actions sit behind a queue, a background task, or a tool call that can time out. It pairs directly with an approval boundary; see decide which agent tool calls need human approval for classifying which actions are worth this effort in the first place.

It is unnecessary overhead for read-only or draft-only workflows. If the agent only searches documents, summarizes, or proposes changes for a human to execute, repeating a step wastes tokens but changes nothing durable. Do not add idempotency machinery to a workflow whose worst-case retry is a slightly higher bill.

The assumption this design makes is that duplicate execution is possible somewhere in your stack. If you can prove exactly-once end to end, you do not need this. Most teams cannot, because the guarantee usually breaks at a network boundary they do not control.

What can still go wrong even with the pattern in place: keys reused across genuinely different actions collapse them into one, keys scoped too narrowly stop deduplicating, and a dedup store with a short retention window forgets an action before a slow retry arrives. Stripe’s at-least-24-hour retention is a reminder that the window has to outlast your longest realistic retry gap.

The security and privacy side

Deduplication is a small data store that records what the workflow did, so treat it as one.

Idempotency keys and their saved responses can carry sensitive context: account identifiers, amounts, message bodies, record contents. Stripe’s own guidance is to “avoid using sensitive data (for example, email addresses or personal identifiers) as idempotency keys” Stripe idempotent requests. Use opaque random keys, and apply the same access controls, retention limits, and audit expectations to the dedup store that you apply to the actions it protects.

There is a replay angle too. A stored key that returns a prior successful result is convenient for legitimate retries and useful to an attacker who can reissue requests. Scope keys to an authenticated caller and a single logical operation so a replayed key cannot act on behalf of someone else. This connects to the broader habit of keeping an evidence trail for agent workflows: the dedup record is part of what a reviewer needs to reconstruct what the agent actually did.

Alternatives worth considering

Idempotency keys are the general tool, not the only one.

• Push the write into a natively idempotent shape. Upserts keyed on a stable identifier, “set to state X” instead of “increment,” and PUT-style replacements remove the problem instead of guarding against it.
• Separate deciding from executing. Let the agent produce a proposed action, then hand execution to a conventional system with its own idempotency and transaction guarantees. This is the same split described in treat MCP servers like production APIs.
• Use a reservation step. For high-value actions, write an intent record first, act second, and mark it complete third, so a retry can detect an in-flight or completed action before repeating it.
• Do not use an agent for it. If the operation is deterministic and high-stakes, a queue worker or rules engine with a well-understood delivery guarantee may be easier to reason about than an agent that decides when to call the tool.

What to review again

Revisit this when the workflow adds a new acting tool, when a downstream API changes its idempotency support, when you move a step behind a queue or into a background task, or when you raise an action’s blast radius from internal draft to external effect. The reliability posture is tied to the infrastructure underneath it, and that infrastructure changes more often than the prompt does. It is also worth checking whenever you expand a coding agent’s workbench to touch systems outside the repository.

The short version: decide the action can be repeated safely before you let the agent take it. The retry is coming whether you planned for it or not.