Make AI Agent Retries Safe
AI agent idempotency means one intended business action has the same effect when its tool call is retried. Give that action a stable identifier, keep its state durably, use the destination’s idempotency feature when available and verify an uncertain result before trying again. A prompt that tells an agent “do not repeat yourself” cannot provide this guarantee.
Picture a support agent that creates an order adjustment. The provider commits the change, but the network response never arrives. The agent sees a timeout and calls the tool again. Without a reliable operation boundary, one customer may receive two adjustments. The scenario is illustrative, yet the underlying failure is ordinary distributed-systems behavior. AWS’s Agentic AI Lens explicitly recommends idempotent task execution because retries after partial completion can duplicate side effects.
One action, several attempts
Separate four things that are easy to collapse in an agent trace:
- Intent: the approved business operation, such as create one adjustment for order 4821.
- Attempt: a call to a tool or provider. The same intent may need more than one attempt.
- Effect: what the destination actually committed, such as a new adjustment ID.
- Acknowledgement: the response that reaches the agent or orchestrator.
An acknowledgement can disappear after the effect succeeds. That is why “the tool returned an error” does not prove the action failed. The orchestrator, rather than the language model, should own the operation identifier and retry policy. The model can propose a new action; it should not invent a fresh identifier for the same action every time a response is missing.
Classify the tool before adding a retry
| Tool action | Example | Retry approach |
|---|---|---|
| Read-only | Get order status | Retry transient failures with bounded backoff, subject to provider limits. |
| State-setting write | Set a record to a specified value | Use a stable operation ID and check the current record; avoid overwriting a newer change. |
| Create or irreversible side effect | Send an email, create a payment, open a ticket | Use the provider’s idempotency contract or reconcile the destination before another attempt. |
Authorization comes first. A perfectly idempotent unauthorized payment is still unauthorized. Powercode’s AI agent security guide covers permission and approval boundaries; this article covers the reliability of an already permitted action.
Give the logical action a stable key
Create an opaque operation ID when the workflow decides to perform a specific action. Persist it before the first attempt, together with the action type, validated arguments, owner and status. Reuse that same ID for every retry of that logical action. A second, intentionally new adjustment needs a new ID, even if some arguments happen to match.
A useful ID can derive from a stable workflow ID and step ID, or be generated once and stored with the intent. The important property is stability across attempts and uniqueness across distinct intended actions. Avoid email addresses or other personal data in the key. Do not derive a new random ID on every retry. AWS highlights that changing the key defeats deduplication across a multi-step workflow.
Record enough state for a worker restart: planned, in_progress, succeeded, failed or outcome_unknown, plus a destination receipt when one exists. Use a unique constraint or atomic conditional write so two workers cannot both claim the same operation. A separate “look in the table, then write” check can race under concurrency.
Use the destination’s idempotency contract
When the external API accepts an idempotency key, pass the same key and unchanged request parameters on every attempt. Stripe’s API documentation, for example, describes a key that lets clients repeat a request after a connection error without creating a second object. It also documents important limits: results can be retained only for a defined period, repeated requests with different parameters are rejected, and a stored failure response can be returned again. Check the exact rules of each provider before building the retry loop.
Do not assume your local ledger and a remote provider commit atomically. Amazon’s explanation of idempotent APIs describes why a service must connect the key and the mutation in a sound server-side contract. A client-side cache alone cannot guarantee that a remote action happened only once.
When the provider has no key, reconcile first
Some email, CRM or legacy endpoints offer no idempotency parameter. The next best route is to search the destination by a stable business reference or provider receipt, inspect its audit trail and determine whether the first attempt committed. If the result is confirmed, record it locally and return that result to the agent. If it is confirmed absent, the workflow may make a new attempt under its bounded retry policy.
If the destination cannot answer whether a consequential action happened, mark the operation outcome_unknown and pause it for review. Creating a second payment or sending a second customer message merely because the first call timed out is not recovery. The risk may justify a manual check or a redesign of the integration. When the unknown outcome involves production records or customer harm, follow our AI agent incident response runbook to contain the run and verify downstream effects.
Worked example: one order adjustment
Assume an approved agent action should create one adjustment for order 4821. The orchestrator stores operation op-4821-adjustment-1 with the exact amount, currency and reason. A separate policy check has already authorized the action.
| Moment | Local record | Destination evidence | Next step |
|---|---|---|---|
| Before first call | Operation reserved with its original arguments | No adjustment yet | Send the request with the same idempotency key. |
| Response lost | Outcome unknown | May have committed | Query the provider or repeat only under its documented idempotency contract. |
| Provider confirms adjustment ID | Mark succeeded and store receipt | One adjustment | Return the recorded result; do not create another. |
| Arguments changed | Original intent remains fixed | Existing result may belong to old arguments | Stop and obtain a new decision; do not reuse the key for different work. |
The key identifies the logical action, not the number of tries. This example also shows why the agent should not regenerate the amount after a timeout: the old request may already have committed. If a person approves a genuinely different adjustment later, that is a new operation with a new approval and key.
Test the failure at the boundary
A happy-path test that calls the tool once proves little about retries. Build failure tests around the point where the side effect and its acknowledgement separate. Powercode’s agent evaluations guide explains how to test observable outcomes and forbidden actions; add the following integration cases to that release gate:
- Drop the response after the provider commits, then restart the worker. Verify that the second attempt returns the original effect or pauses for reconciliation.
- Run two workers with the same operation ID at once. Verify that only one claims the action.
- Send the same key with different arguments. Verify a visible conflict rather than silently accepting a changed action.
- Expire a key in a test environment. Check what the provider does after its retention window and whether the workflow still has the receipt.
- Simulate a provider with no idempotency support and an unreadable result. Verify that the workflow stops in an explicit unknown state.
Use bounded retries and backoff for transient errors. Validation failures, permission denials and conflicting arguments need correction, not more attempts. Durable orchestration can help a workflow resume from recorded state; Temporal’s workflow documentation describes this pattern. Durability does not remove the need for an idempotent external action.
What idempotency cannot promise
Idempotency is a contract for a defined operation and window. It does not make a chain of unrelated systems one atomic transaction. It does not unsend an email, reverse a payment, authorize a tool call or prove that a provider never duplicates work outside its stated guarantee. A workflow that writes to a CRM, sends a message and updates a billing system may need per-step receipts, compensating actions and human review if it stops partway through.
The architecture choice comes earlier: if fixed rules can handle the task, a deterministic workflow may be simpler to operate than an agent. Our AI workflow versus AI agent comparison helps with that decision. If an agent is justified, its side effects still need ordinary software-engineering controls. The CRM automation guide applies those controls to CRM integrations specifically.
Safe retry checklist
- Identify each tool that can change an external system.
- Define one logical operation, stable ID and immutable approved arguments.
- Persist an operation record before the first attempt and claim it atomically.
- Pass the same key to a supporting provider on every retry.
- Record a provider receipt and read back the resulting state.
- Reconcile timeouts before another write; pause outcomes that remain unknown.
- Test response loss, concurrent attempts, process restart and changed arguments.
Frequently asked questions
Is checking for an existing record before writing enough?
No. Two workers can both see “absent” and then both write. The destination needs a unique constraint, conditional write or documented idempotency contract. When the effect is remote, a local pre-check also cannot prove that a timed-out request did not commit.
Does an idempotency key guarantee exactly-once execution?
Only within the provider’s documented scope and retention period, and only when the same logical request uses the same key correctly. Across a multi-system workflow, there is no universal exactly-once guarantee. Design each side effect to be verifiable and give unknown outcomes a safe recovery path.
When Powercode can help
Powercode Group can help make an agent or automation safe to operate across real business systems through custom software development and AI engineering. Bring one state-changing tool call, its provider contract and a failure you need to survive. A focused review can identify the right operation boundary, recovery evidence and tests. If fixed integration rules solve the job, you may not need an agent at all.
