airoweb

airoweb post

The agent updated the record. The notification never left.

A transactional outbox keeps an agent's database decision and its downstream side effect from drifting apart when a process crashes or a network call fails.

Audience
Platform teams, Agent infrastructure teams, Technical leads
Level
advanced
Risk
medium
Updated
August 3, 2026

An agent approves a refund, writes refund_status = 'approved', and calls the payment service. The database commit succeeds. The process dies before the HTTP request leaves.

The customer does not get the refund, but the operations screen says it was approved.

Reverse the order and the failure reverses too. The payment service accepts the refund, then the database write fails. The customer gets the money, while the case still looks unresolved and may be refunded again.

Both implementations look reasonable in a code review:

await cases.markRefundApproved(caseId);
await payments.issueRefund(paymentId);
await payments.issueRefund(paymentId);
await cases.markRefundApproved(caseId);

Neither order makes two independent systems commit atomically. An agent cannot reason its way across that crash gap. If one call succeeds and the next call has no confirmed result, the model has the same incomplete evidence as the rest of the application.

For connected agent workflows, the useful boundary is a transactional outbox: commit the business decision and a durable intent to act in the same database transaction, then let a separate relay perform the external action.

Two correct-looking orders, two broken states

This is the dual-write problem. The workflow needs to change state in one system and notify or command another, but the writes do not share a transaction.

AWS Prescriptive Guidance describes both failure directions: a database update can succeed while event publication fails, or a notification can be sent while the database transaction rolls back. The result is inconsistency, not merely a failed job.

Agent workflows make this old problem easier to miss. The model’s visible task might be “resolve the case,” so the application treats the database update, customer email, ticket comment, and payment call as one semantic action. Infrastructure still sees separate commits across separate failure domains.

Prompt instructions such as “only send after saving” specify order. They do not provide atomicity. A retry policy does not close the gap either. It only decides what happens after the system notices uncertainty, and retrying a side effect can produce a duplicate.

The same problem appears whenever an agent both records a decision and reaches outside that record’s transaction:

Durable decision External effect Failure someone eventually sees
Mark an incident escalated Page the on-call engineer Escalated incident with no page
Approve a customer response Send the email Approved response that stays in the application
Accept a generated code fix Dispatch a deployment Repository says deploy requested, but nothing runs
Close a compliance exception Notify the evidence system Conflicting audit state across systems
Record an account change Publish an event to downstream CRM One system changes while dependent records remain stale

The model can propose any of these actions. It should not be responsible for making a distributed commit appear atomic.

Make the decision durable before execution

The outbox changes the unit of work. Instead of updating the business record and calling the outside system, the application updates the business record and inserts an outbox row in one local transaction.

BEGIN;

UPDATE refund_cases
SET status = 'approved',
    approved_action_id = :action_id
WHERE id = :case_id
  AND status = 'awaiting_approval';

INSERT INTO workflow_outbox (
  event_id,
  aggregate_type,
  aggregate_id,
  event_type,
  payload,
  created_at
) VALUES (
  :action_id,
  'refund_case',
  :case_id,
  'refund.approved',
  :minimal_payload,
  CURRENT_TIMESTAMP
);

COMMIT;

This is an implementation sketch, not a schema to adopt unchanged. The important property is that the state transition and the intent record succeed or roll back together. After commit, a conventional relay reads refund.approved and calls the payment service.

The agent is now upstream of the reliability boundary:

  1. The agent gathers evidence and proposes the refund.
  2. Policy code or a human approval step decides whether the proposal may proceed.
  3. The application commits the approved state and outbox event together.
  4. A relay delivers the event, records the attempt, and retries according to policy.
  5. A reconciler compares the external result with the internal action state.

The language model can help with the judgment-heavy first step. The remaining steps benefit from deterministic state transitions, credentials scoped to one action, and infrastructure that can resume without reconstructing intent from a transcript.

A relay turns the crash gap into recoverable work

The relay may poll an outbox table, consume a database change stream, or use change data capture. AWS documents both an outbox-table approach for relational databases and a change-data-capture approach. The choice affects latency and operations, but the invariant stays the same: only committed intent becomes eligible for delivery.

A small relational implementation might claim unpublished rows in batches:

SELECT event_id, event_type, payload
FROM workflow_outbox
WHERE published_at IS NULL
  AND available_at <= CURRENT_TIMESTAMP
ORDER BY created_at
FOR UPDATE SKIP LOCKED
LIMIT :batch_size;

PostgreSQL documents SKIP LOCKED as unsuitable for general-purpose reads because it gives an inconsistent view, while noting that it can reduce contention among multiple consumers of a queue-like table PostgreSQL SELECT. That makes it a candidate for claiming relay work, not for reporting whether the outbox is empty or complete.

The relay still has a second uncertainty window. It can send the external request successfully and crash before marking the outbox row published. On restart, it will send again. A transactional outbox prevents lost intent; it does not create exactly-once delivery across the network.

Design the receiver or adapter for duplicate delivery:

  • Give every logical action a stable event_id or idempotency key.
  • Reuse that identity on every relay attempt; do not generate a new key after a timeout.
  • Have the receiver store the accepted identity or make the state change naturally idempotent.
  • Record the external request identifier and confirmed outcome separately from the agent transcript.
  • Preserve ordering where later events depend on earlier state, usually by aggregate or partition rather than across the whole system.

RFC 9110 defines an idempotent request method by its intended server effect: repeating the request has the same intended effect as sending it once RFC 9110. For non-idempotent operations, the same protection has to come from the application protocol. The design in Assume your agent will run the workflow twice covers that action-level contract in detail.

Redelivery is not a theoretical edge case. GitHub, for example, documents webhook redelivery and recommends using its stable delivery header to identify the event and protect against replays GitHub webhook best practices. Integrations should treat delivery identity and replay handling as ordinary protocol fields.

Keep the control plane outside the model

An outbox row should be created by an application service that validates an allowed state transition, not by giving the model arbitrary insert access.

That distinction protects more than the schema. It gives the team one place to enforce:

  • whether this action requires human approval;
  • which authenticated principal requested and approved it;
  • which destination and tenant the event may reach;
  • the payload fields allowed to leave the source system;
  • the stable action identity used for deduplication;
  • cancellation rules before and after delivery begins;
  • retention and access policy for the evidence record.

The agent can return a typed proposal such as ApproveRefund(case_id, reason, evidence_refs). Policy code then verifies the case, approval, limits, and destination before creating the outbox event. This is the same separation of decision from authority that should exist at an MCP server boundary: tool inputs are requests to a protected service, not permission to mutate whatever the model named.

Do not put the entire prompt, retrieved documents, or chain of thought into the event payload. The relay needs the minimum data required to execute or look up the approved action. Keep richer review evidence behind an access-controlled reference. An outbox is operational infrastructure, and duplicating sensitive context into it creates another retention, access, and incident-response surface.

Treat event types as an API. Version them deliberately, validate payloads at both ends, and reject unknown or unauthorized destinations. If an attacker can influence retrieved content, a durable outbox can otherwise turn one prompt-injection success into a reliable delivery mechanism for the attacker’s requested side effect.

Where the pattern is too much

Use an outbox when a workflow must keep a local state transition aligned with a downstream command or event and the inconsistency would matter. It is especially useful when the workflow runs asynchronously, retries after crashes, or sends through a broker or webhook.

It is unnecessary for read-only research, drafts that stay inside one database, or actions whose source of truth already lives entirely in a managed workflow engine. If the only result is a document awaiting human execution, there is no dual write to coordinate.

It can also be the wrong pattern when:

  • The two writes already share a real transaction. Adding a relay introduces eventual consistency without buying safety.
  • The workflow spans several independent business transactions. An outbox can publish each committed step, but it does not define compensation for a partially completed process. AWS recommends considering saga orchestration when updates span services.
  • The downstream API cannot deduplicate or expose current state. The outbox preserves intent, but retries may still repeat a harmful action. Keep a human reconciliation step or use a safer adapter.
  • The team cannot operate the relay. Backlog age, poison events, schema compatibility, retry exhaustion, and destination outages all need owners and alerts. A managed queue or durable workflow platform may be cheaper than another home-grown worker.
  • Immediate global consistency is mandatory. The outbox is an eventual-consistency pattern. A design that cannot tolerate the interval between local commit and external delivery needs a different system boundary, not a faster poll loop.

The cost is real: another table or stream, a relay, deduplication state, monitoring, retention, replay tooling, and an operational procedure for stuck events. Platform teams should centralize those mechanics when several agent workflows need them rather than letting every product team invent a slightly different outbox.

Run the crash drill before granting autonomy

The acceptance test is not “the happy path sent the notification.” Pause the relay at each uncertainty boundary and inspect what the system can prove.

Commit the business record and terminate the process before delivery. The outbox event should remain available. Deliver the external request and terminate before acknowledging it locally. The next attempt should reuse the same action identity and avoid a duplicate effect. Feed the relay a malformed or unauthorized event. It should quarantine the event without silently dropping it or letting the model rewrite policy. Revoke the destination credential. The event should remain diagnosable without leaking the credential or full sensitive payload into logs.

Those results belong in the workflow’s evidence trail: who approved the action, which event represented it, which relay attempts occurred, what the destination confirmed, and how the final state was reconciled. Queue ownership is a separate concern; if several workers may relay the same table, use the claim and stale-worker protections described in A running flag is not a lock for your agent queue.

Review the design again when a workflow adds a destination, an event schema changes, a downstream service changes its idempotency behavior, sensitive fields enter the payload, or a side effect moves from internal staging to a customer-facing system.

The outbox does not make an agent more reliable. It makes the agent’s approved intent durable, then hands execution to infrastructure that knows how to survive a crash.

Sources