A running flag is not a lock for your agent queue

Suppose Worker A claims an agent task and changes its status from queued to running. It starts a long tool sequence, loses its database connection, and disappears from the scheduler.

The task still says running.

An operator eventually changes it back to queued. Worker B claims it and finishes. Then Worker A wakes up, reconnects, and writes its result too. The queue now has two workers that each had a reasonable basis for believing the task belonged to them.

The status column did not fail. It answered the wrong question.

running describes what the system last observed. It does not say who may act now, how long that authority lasts, or whether a delayed worker still owns it. A queue that hands consequential work to agents needs a time-bounded claim: a lease.

Replace permanent ownership with borrowed time

A lease gives a worker exclusive responsibility for a task until a deadline. The worker renews the lease while it is healthy. If it crashes, stops heartbeating, or loses access to the queue, the lease expires and another worker may claim the task.

This is the same recovery idea exposed by established infrastructure. Amazon SQS keeps a received message in the queue but hides it from other consumers for a visibility timeout. If the consumer does not delete it before the timeout expires, the message becomes visible for another attempt. A consumer can extend the timeout while it is still working Amazon SQS visibility timeout.

Kubernetes uses Lease objects for node heartbeats and leader election. Its lease data includes a holder identity, acquisition time, renewal time, duration, and transition count Kubernetes Leases. An agent job table needs the same ideas even if it uses different field names:

Field	What it proves
status	Where the task is in its lifecycle
lease_owner	Which worker currently claims authority
lease_expires_at	When another worker may recover the task
claim_generation	Which acquisition of the lease this worker holds
idempotency_key	Which real-world action must not be duplicated across attempts
attempts	How often processing has begun
last_error	Why the most recent attempt stopped or lost its claim

Do not collapse those fields into status = running_by_worker_A. Lifecycle, current authority, and logical action identity change for different reasons. Keeping them separate makes recovery rules explicit.

Claim the row in one transaction

The claim operation has to select eligible work and assign the lease atomically. A read followed by an update leaves a gap in which several workers can choose the same row.

For a database-backed queue, the shape can look like this:

WITH candidate AS (
  SELECT id
  FROM agent_tasks
  WHERE status = 'queued'
     OR (
       status = 'running'
       AND lease_expires_at < CURRENT_TIMESTAMP
     )
  ORDER BY available_at, created_at
  FOR UPDATE SKIP LOCKED
  LIMIT :batch_size
)
UPDATE agent_tasks AS task
SET status = 'running',
    lease_owner = :worker_id,
    lease_expires_at = :lease_deadline,
    claim_generation = claim_generation + 1,
    attempts = attempts + 1
FROM candidate
WHERE task.id = candidate.id
RETURNING task.*;

This is an implementation sketch, not a copy-and-paste queue. The transaction boundary, indexes, ordering policy, isolation level, and batch behavior need load testing in the database you actually run.

PostgreSQL documents the relevant trade-off precisely: SKIP LOCKED returns an inconsistent view, so it is unsuitable for general-purpose work, but it can avoid lock contention when multiple consumers access a queue-like table PostgreSQL SELECT. That makes it a fit for claiming work, not for reports that promise a complete snapshot of the queue.

The query should return the new claim_generation. The worker must carry that generation through heartbeats, completion, cancellation, and any write whose correctness depends on the claim.

A heartbeat is a guarded renewal

A heartbeat should not mean “Worker A is alive.” It should mean “Worker A still holds this particular acquisition of this task.”

The renewal therefore needs a compare-and-set condition:

UPDATE agent_tasks
SET lease_expires_at = :new_deadline
WHERE id = :task_id
  AND status = 'running'
  AND lease_owner = :worker_id
  AND claim_generation = :claim_generation
  AND lease_expires_at >= CURRENT_TIMESTAMP;

If the update affects no row, the worker has lost the lease. It should stop starting new tool calls, stop publishing results, and enter its cancellation path. It should not “repair” the claim by writing a later deadline without checking the generation.

Choose the lease duration from observed task behavior and recovery needs, not from an arbitrary round number. A short lease recovers quickly after a crash but creates more renewal traffic and makes ordinary pauses look like failure. A long lease reduces heartbeat pressure but leaves abandoned work unavailable for longer. Tasks with unpredictable tool latency may need a renewable lease; tasks that cannot report progress may need a different execution model.

Amazon recommends extending visibility while a consumer continues working and warns that a timeout that is too short can make the message available to another consumer before the original processing finishes. It also notes that at-least-once delivery can still produce more than one delivery during the visibility window Amazon SQS visibility timeout. A lease is a recovery mechanism, not a promise of exactly-once execution.

The old worker can still wake up

Lease expiry creates an uncomfortable but necessary state:

• Worker A held the claim.
• Worker A paused long enough for its lease to expire.
• Worker B acquired the task with a newer generation.
• Worker A resumed with an older generation.

The queue must treat Worker A as stale even if it never noticed the pause.

Guarding the final task update is straightforward:

UPDATE agent_tasks
SET status = 'completed',
    result_ref = :result_ref,
    lease_owner = NULL,
    lease_expires_at = NULL
WHERE id = :task_id
  AND status = 'running'
  AND lease_owner = :worker_id
  AND claim_generation = :claim_generation
  AND lease_expires_at >= CURRENT_TIMESTAMP;

Again, no updated row means no authority to complete.

The harder case is an external side effect. Worker A may have passed the lease check and then stalled before sending an email, changing a customer record, or publishing an artifact. By the time it resumes and calls the external system, Worker B may own the task. A check inside the queue database cannot retract a request already in flight.

Where the downstream system can participate, send the claim generation as a fencing token and make the receiver reject operations older than the newest generation it has accepted. Microsoft’s state-store protocol describes exactly this stale-writer race: two clients may both act as if they are active, so a protected key rejects writes carrying an older fencing token Azure IoT Operations state-store protocol.

Fencing and idempotency solve different problems:

• A fencing token says, “This worker’s authority is newer than that worker’s authority.”
• An idempotency key says, “These retries represent the same logical action.”

Use both when an agent job can be retried and reassigned. The idempotency key should stay stable across attempts; the claim generation should change with each acquisition. See assume your agent will run the workflow twice for the action-level design.

If the downstream API supports neither conditional versions nor idempotency keys, the safe fallback may be to have the agent prepare an intent for a conventional executor rather than grant the agent a direct write tool.

Recovery needs a policy, not only an expiry query

An expired lease tells the queue that ownership may be reassigned. It does not tell the queue that retrying is wise.

Classify tasks before returning them to general circulation:

Task condition	Safer next step
Read-only work	Reclaim automatically and record the abandoned attempt
Idempotent write with a stable key	Reclaim with the same action identity
Ambiguous non-idempotent side effect	Pause for reconciliation before retrying
Repeated deterministic failure	Move out of the active queue for inspection
Missing or revoked credential	Block until access is repaired; a new worker will not fix authority
Human cancellation	Keep cancelled; expiry is not permission to undo an operator decision

That reconciliation state is important. A timed-out agent that may have sent a message is not equivalent to one that failed before it called the tool. The queue needs an evidence trail that records the claim generation, tool-call intent, idempotency key, external response when available, heartbeat history, and reason the lease ended. This is part of the broader evidence trail an agent workflow needs, not debug data to discard when the run finishes.

Cancellation should use the same guardrails. Mark the task cancelled in the queue, revoke or stop renewing the current lease, signal the worker, and prevent its old generation from completing. For workflows with material external effects, pair this with the off-switch design for stopping new actions and reconciling actions already in flight.

Where this design fits

Use leases when work is processed asynchronously by interchangeable workers and an abandoned task must become available again. They matter most when agent runs are long, tool latency varies, workers restart, deployments replace processes, or a scheduler can dispatch the same logical task after a timeout.

They are unnecessary machinery for an in-process queue with one worker and disposable read-only work. They may also be the wrong layer when a managed queue or durable workflow engine already provides visibility timeouts, heartbeats, retries, cancellation, and execution history. Rebuilding those semantics in an application table creates schema, polling, indexing, clock, monitoring, and incident-response costs that a platform team must own.

A database-backed lease is attractive when tasks and business state need transactional claiming, the volume fits the database, and the team already operates the database well. A managed queue is usually better when elastic delivery and isolation from the application database matter more. A durable workflow engine is stronger when the process has many long-lived steps, timers, compensations, signals, and human pauses. A single leader or partitioned ownership model can reduce claim contention when the work naturally divides by tenant or resource.

Avoid pretending a lease makes unsafe external systems safe. If correctness depends on excluding stale writers, the protected resource has to enforce the fence. If repeated actions would cause harm, the action needs idempotency or reconciliation. If no layer can provide either property, keep a human approval boundary before the side effect.

Treat claim data as security data

The lease owner, generation, heartbeat history, and error record can expose internal identifiers, workload shape, customer references, and operational failures. Store only what incident review needs, restrict who can inspect or modify claims, and set retention separately from the full model transcript.

Do not let workers choose arbitrary owner identities or generations. The queue service should issue them from authenticated worker identity and authoritative task state. A worker that can advance its own generation, extend an expired lease, or complete without a guarded condition can bypass the ownership model.

Also separate lease authority from tool authority. Owning a queue task does not automatically justify access to every downstream system the task might mention. Credentials should remain scoped to the run and action, and the tool layer should enforce its own permissions. A well-designed queue cannot compensate for an agent holding an unrestricted keyring.

The operational test is simple: pause a worker after it claims a task, let another worker recover the task, then resume the first worker. The first worker must be unable to renew, complete, or overwrite the newer result. If it can still trigger an external side effect, the design is unfinished.

A running flag records history. A lease grants temporary authority. Agent queues that can survive crashes, retries, and delayed workers need to know the difference.