Skip to content

0011. Run scheduling and delivery: the project store owns all retries

  • Status: Accepted
  • Date: 2026-11-08

Runs wait for days, sends go through a queue, and every step can crash. If both the queue and the store decide retries, a lost response can become a duplicate send. Cloudflare does not document whether setAlarm is transactional, and an alarm handler that throws on one item can block the rest.

The project store (the DO on Cloudflare, the Postgres transaction in containers) is the single owner of run and send state and of every retry. Queues and the delivery consumer only execute attempts.

Scheduling:

  • tick processes due runs, trigger fires, and outbox rows in bounded batches and re-arms the timer while work remains.
  • On a DO, setAlarm(min(wake_at)) is called inside the transaction callback. The alarm handler catches errors per item, quarantines an item after repeated failures (and alerts), and re-arms in finally.
  • Every state with a deadline registers wakeAt(deadline) in the same transaction that enters the state: queued (claim deadline), sending (lease expiry), unknown (72 hours after the attempt), accepted while its Send event has not arrived (72 hours after the attempt, for budget settlement, ADR 0013), and waiting:plan_limit (72 hours after the message first entered it). tick applies the transitions that are due: re-enqueue a queued job, sending → unknown, unknown → failed:outcome_unknown, release of an unbilled accepted message’s budget reservation, and waiting:plan_limit → skipped:plan_limit. No deadline depends on a queue message or an external event arriving.
  • Backstop: each DO exports its next wake time to a next-wake index in D1, and a cron sweeper wakes DOs whose wake time has passed. In containers, the scheduler service polls project_wakeups with SKIP LOCKED, so several replicas are safe.

Send-attempt state machine for a message (one message per (run, step), unique). The states are queued, waiting, sending, accepted, unknown, failed, and skipped. A state that needs an explanation carries a reason, written state:reason (for example waiting:plan_limit, skipped:plan_limit, failed:outcome_unknown); step logs use the same form. Mermaid state names cannot contain :, so the diagram shows the reason on the transition.

stateDiagram-v2
[*] --> queued: send step passes eligibility, outbox job written
queued --> queued: deadline passed, re-enqueued
queued --> sending: claimSend (budget reserved, eligibility rechecked, attempt_id persisted, lease)
queued --> skipped: claimSend finds ineligible, stopped, or kill switch
queued --> waiting: workspace budget exhausted, reason plan_limit
waiting --> queued: upgrade or period reset within 72 h of first entering waiting
waiting --> skipped: 72 h after first entering waiting, reason plan_limit
sending --> accepted: provider returns a message ID, or a Send event arrives (billed on the Send event)
sending --> queued: definite pre-acceptance error, retryable (new attempt_id, backoff)
sending --> failed: definite pre-acceptance error, permanent
sending --> unknown: timeout, connection reset, unparseable response, or lease expired
unknown --> accepted: Send event within 72 h
unknown --> failed: no Send event within 72 h, reason outcome_unknown
accepted --> [*]
  • The 72-hour window of waiting:plan_limit starts when the message first entered it and is not reset when the message goes back to queued and waits again.
  • queued has a deadline. A job not claimed by the deadline is re-enqueued; duplicates are no-ops because only queued can be claimed.
  • claimSend runs in one transaction: reserve budget (ADR 0013), then the final eligibility check (unsubscribe, suppression, goal, workflow active in the run’s activation epoch (ADR 0017), run not exited, global and workspace kill switches), then persist attempt_id and a lease (owner, expiry). If the check fails, the reservation is released. If the local budget lease is empty, claimSend returns without claiming and the caller refills the lease before claiming again (ADR 0013).
  • sending with an expired lease becomes unknown, never a resend.
  • Only a definite pre-acceptance error (the provider returned an error response) is retried, by the store, with backoff and a new attempt. Unknown outcomes follow ADR 0012.
  • The consumer always acknowledges the queue message after recording the outcome. Queue redelivery is crash recovery, not a retry policy.
  • The dead-letter queue drains back through claimSend, so a dead-lettered job is re-evaluated, never sent blindly.
  • No external I/O happens inside a transaction: claimSend, the provider call, and recordOutcome are three steps.
  • The deployed behaviour of setAlarm inside the transaction callback is still to be verified (ADR 0009); the cron sweeper keeps runs from stalling either way.
  • Split queues, rate buckets, and provider-event ingestion are listed in docs/architecture.md.

Amendment (2026-11-18, pending ADR 0044, Proposed): claimSend moves a queued message of a paused workflow or project to waiting:paused, before the budget reservation; resume returns it to queued, and 72 hours after it first entered waiting it becomes skipped:paused, as for plan_limit. A message of a withdrawn version (rollback) becomes skipped:version_withdrawn.