0011. Run scheduling and delivery: the project store owns all retries
- Status: Accepted
- Date: 2026-11-08
Context
Section titled “Context”Runs wait for days, sends go through a queue, and every step can crash. If both the queue and the store decide retries, a lost response can become a duplicate send. Cloudflare does not document whether setAlarm is transactional, and an alarm handler that throws on one item can block the rest.
Decision
Section titled “Decision”The project store (the DO on Cloudflare, the Postgres transaction in containers) is the single owner of run and send state and of every retry. Queues and the delivery consumer only execute attempts.
Scheduling:
tickprocesses due runs, trigger fires, and outbox rows in bounded batches and re-arms the timer while work remains.- On a DO,
setAlarm(min(wake_at))is called inside the transaction callback. The alarm handler catches errors per item, quarantines an item after repeated failures (and alerts), and re-arms infinally. - Every state with a deadline registers
wakeAt(deadline)in the same transaction that enters the state:queued(claim deadline),sending(lease expiry),unknown(72 hours after the attempt),acceptedwhile its Send event has not arrived (72 hours after the attempt, for budget settlement, ADR 0013), andwaiting:plan_limit(72 hours after the message first entered it).tickapplies the transitions that are due: re-enqueue aqueuedjob,sending→unknown,unknown→failed:outcome_unknown, release of an unbilledacceptedmessage’s budget reservation, andwaiting:plan_limit→skipped:plan_limit. No deadline depends on a queue message or an external event arriving. - Backstop: each DO exports its next wake time to a next-wake index in D1, and a cron sweeper wakes DOs whose wake time has passed. In containers, the
schedulerservice pollsproject_wakeupswithSKIP LOCKED, so several replicas are safe.
Send-attempt state machine for a message (one message per (run, step), unique). The states are queued, waiting, sending, accepted, unknown, failed, and skipped. A state that needs an explanation carries a reason, written state:reason (for example waiting:plan_limit, skipped:plan_limit, failed:outcome_unknown); step logs use the same form. Mermaid state names cannot contain :, so the diagram shows the reason on the transition.
stateDiagram-v2 [*] --> queued: send step passes eligibility, outbox job written queued --> queued: deadline passed, re-enqueued queued --> sending: claimSend (budget reserved, eligibility rechecked, attempt_id persisted, lease) queued --> skipped: claimSend finds ineligible, stopped, or kill switch queued --> waiting: workspace budget exhausted, reason plan_limit waiting --> queued: upgrade or period reset within 72 h of first entering waiting waiting --> skipped: 72 h after first entering waiting, reason plan_limit sending --> accepted: provider returns a message ID, or a Send event arrives (billed on the Send event) sending --> queued: definite pre-acceptance error, retryable (new attempt_id, backoff) sending --> failed: definite pre-acceptance error, permanent sending --> unknown: timeout, connection reset, unparseable response, or lease expired unknown --> accepted: Send event within 72 h unknown --> failed: no Send event within 72 h, reason outcome_unknown accepted --> [*]- The 72-hour window of
waiting:plan_limitstarts when the message first entered it and is not reset when the message goes back toqueuedand waits again. queuedhas a deadline. A job not claimed by the deadline is re-enqueued; duplicates are no-ops because onlyqueuedcan be claimed.claimSendruns in one transaction: reserve budget (ADR 0013), then the final eligibility check (unsubscribe, suppression, goal, workflow active in the run’s activation epoch (ADR 0017), run not exited, global and workspace kill switches), then persistattempt_idand a lease (owner, expiry). If the check fails, the reservation is released. If the local budget lease is empty,claimSendreturns without claiming and the caller refills the lease before claiming again (ADR 0013).sendingwith an expired lease becomesunknown, never a resend.- Only a definite pre-acceptance error (the provider returned an error response) is retried, by the store, with backoff and a new attempt. Unknown outcomes follow ADR 0012.
- The consumer always acknowledges the queue message after recording the outcome. Queue redelivery is crash recovery, not a retry policy.
- The dead-letter queue drains back through
claimSend, so a dead-lettered job is re-evaluated, never sent blindly.
Consequences
Section titled “Consequences”- No external I/O happens inside a transaction:
claimSend, the provider call, andrecordOutcomeare three steps. - The deployed behaviour of
setAlarminside the transaction callback is still to be verified (ADR 0009); the cron sweeper keeps runs from stalling either way. - Split queues, rate buckets, and provider-event ingestion are listed in docs/architecture.md.
Amendment (2026-11-18, pending ADR 0044, Proposed): claimSend moves a queued message of a paused workflow or project to waiting:paused, before the budget reservation; resume returns it to queued, and 72 hours after it first entered waiting it becomes skipped:paused, as for plan_limit. A message of a withdrawn version (rollback) becomes skipped:version_withdrawn.