Runner lease failures

A retained sandbox may still exist after the worker driving its turn loses its lease. Start with the attempt records and runner logs: container presence alone does not prove that the native session can safely continue.

Lease timing and retries

The lease and execution timeout govern different things:

Setting	Default	Purpose
Attempt lease	120 seconds	Fences the worker’s authority to write.
Heartbeat interval	30 seconds	Renews that authority on a separate thread.
Heartbeat HTTP timeout	10 seconds	Limits each renewal request.
Retry delay after heartbeat failure	5 seconds	Retries while the current lease remains valid.
Agent execution timeout	3,600 seconds	Limits the turn itself.
Ordinary control RPC timeout	30 seconds	Default for other control requests.

A successful heartbeat restores the normal interval. HTTP 409 rejects the lease permanently: renewal and execution stop, and retries cannot restore the expired owner. Renewal does not depend on model output, tool activity or progress events, so a quiet call can span many lease periods.

The runner reaches the same verdict without a 409 when the lease cannot still be valid: once its last acknowledged renewal is a whole lease old, or when the control plane refuses the worker credential itself (HTTP 401, 403 or 404, which no retry changes). It then stops the agent as for a 409 instead of letting it write to the shared workspace after the reaper has superseded the attempt. A lease must therefore outlast one heartbeat interval plus one request timeout (40 seconds): a shorter AGENT_LEASE_TTL is ignored in favour of the default, and a shorter --lease-ttl is refused. A cancellation does not end renewal; the lease is kept while the runner releases or drains the turn.

With the first heartbeat at 30 seconds and repeated ten-second read timeouts, retry requests start at 30, 45, 60, 75, 90 and 105 seconds after claim. This illustrates opportunities to reconnect before the original 120-second lease ends; it is not a wall-clock guarantee. HTTP timeout limits apply separately to connection, pool, write and read operations. An outage longer than the remaining lease still loses the lease.

The regression suite drives a silent six-minute turn through the real heartbeat thread, stream loop and HTTP client against a simulated lease endpoint. It covers healthy renewal and timed-out reads followed by recovery at 100 or 105 seconds. The previous cadence loses the latter case. These tests reproduce a failure mechanism; they do not establish the cause of a historical incident.

Diagnose from positive evidence

Distinguish a renewal-request timeout from an execution timeout, explicit cancellation or agent exit. lease_expired proves only that the database did not persist a timely renewal. Missing error events do not prove the worker was healthy: the control outage that blocks renewal can also block error reporting.

Use these ordinary process-log records:

Record	Evidence
agent_attempt_claimed	Job, attempt, worker and host identifiers.
agent_control_failure	Failed operation/stage, job ID, exception class and HTTP status when available.
agent_heartbeat_retry	Consecutive failures, time since the last acknowledgement (or heartbeat start), and next retry delay.
agent_heartbeat_recovered	Renewal succeeded after failures.

The failure diagnostics exclude request URLs, tokens, response bodies and exception messages. The heartbeat’s exception class/status are logged once per outage; retry records carry the changing count and timing.

Correlate logs with the attempt’s persisted lease_owner, lease_expires_at and event timestamps. Collect runner and control-plane logs outside their containers: replacing a container can delete its Docker log history. The runner does not configure an external collector or recover deleted logs.

An attempt_superseded event does not establish that another attempt ran. Retained sessions can become lost without a replacement, including when activation ownership is uncertain during preparation. The UI announces another attempt only when its events exist.

Reporting and acknowledgement failures

Error-event upload and the terminal finish request are independent. The runner attempts the terminal transition even if event upload fails. If both remain unavailable, the reaper is the final backstop.

Post-claim preparation outside the shared execution handler, such as image inspection or manifest validation, has an outer failure handler too. It attempts a terminal failure before stopping heartbeat renewal and logs the exception class even if reporting fails. An explicit lease rejection skips that write. This records early setup failures without repairing their cause.

After supervisor and control-plane settlement both succeed, a failed upload of the final shared_workspace_settled notice does not fail the turn: the runner still submits its successful terminal state. Lease rejection prevents that write. Earlier settlement failures remain failures.

Admission and settlement receipts allow up to three attempts, using ten-second HTTP timeouts and five-second delays. Only transport failures and HTTP 408/502/503/504 are retried; 409/410 are definitive. The prompt is sent exactly once, after its admission acknowledgement succeeds.

Supervisor settlement retains the last token’s receipt until another activation takes ownership. Control-plane settlement returns the recorded boundary only for the same still-live attempt and activation. Repeating it does not increment the workspace epoch, append a snapshot or change the native turn ordinal. A sandbox manager restart loses its in-memory receipt, so uncertain completion across a restart fails closed.

When a retained conversation can continue

A reported preparation failure before admission can leave an existing conversation usable. The runner abandons the activation and reports a failed turn; the next message can resume the retained native session. This requires a known retained container. Failure before its identity was recorded does not provide that proof, and error-event availability must not gate the report.

An admission request with no received acknowledgement is different: the database may have committed it even though the runner has not sent the prompt. The membership then records the job as admitted. abandon only releases pre-admission turns (admitted_job_id IS NULL), so it cannot release this ambiguous turn; the terminal transition marks its context LOST. The runner never replays an unconfirmed admitted prompt.

Recovery after an expired activation or killed runner requires a further protocol: durably quarantine the member, fence the old runner’s preparation commands and sandbox manager token, verify the exact retained container and native turn boundary, then resume that session or report it unrecoverable. A sandbox manager busy=false observation is insufficient because workspace setup commands run outside its agent-turn transport. Never requeue a turn solely because its admission field is empty, clear a lost flag, or replay an admitted prompt to repair a display state.

Owner-facing wording describes interrupted responses and unavailable conversations in plain language. Infrastructure details stay in API records and operator logs; task errors and assistant messages remain intact. Historical LOST memberships remain disabled after wording changes.

The safe release case: a child never accepted its turn

The host runs each turn in a child process. It can release a claim back to the queue only when it can prove that child never accepted the turn:

• Process creation failed.

• Payload delivery failed.

• The child closed its handover descriptor without accepting, including a failed read-only Docker/network preflight before acceptance.

A silent child that remains alive is not proof and is never released. After acceptance, crashes, kills and lost acknowledgements belong to the lease protocol; the conversation may become lost instead of repeating the turn.

Release uses POST /v1/agent/worker/jobs/<job>/release with that turn’s worker token. The control plane requires the same still-running attempt and lease generation and no membership that admitted the job. The database statement enforces the admission condition, so an incorrect host observation cannot requeue admitted work.

Consecutive launch failures stop claims on that host and back off. Heartbeats report launch_error, which appears beside the host on Admin → Servers.