0038. Operator alerts are outbox entries keyed by incident, raised through an extension point
- Status: Accepted (decided by the maintainer)
- Date: 2026-11-15
Context
Section titled “Context”ADR 0034 says several send outcomes alert the operator: an unavailable provider (sending paused, credentials, configuration) and a bad_request rejection, and, by ADR 0034’s amendment of 2026-11-13, unknown outcomes and accepted sends whose Send event never arrives. ADR 0012 and docs/architecture.md also alert an unknown outcome that expires unresolved after 72 hours. If the daily-quota text match ever misses, the quota shows up as throttling that exhausts the retry limit, and that must alert too. The amendment fixes the mechanism: an occurrence is written to the project store’s outbox in the transaction that records the outcome, never emitted as a side effect inside it, and it carries an incident key without the project, so one account-wide SES failure seen by many projects raises one alert. docs/architecture.md lists operational alerting as a Proposed extension point with a no-op default. B2 will raise OnAccountSuppressionList bounces and late Send events through the same port.
Facts:
- The outbox held only delivery jobs, drained to the delivery queue after each command and retried from the wake (ADR 0011). Commands see only
ProjectTx, so they cannot reach a port, and a throw rolls back every write, the outbox included. - Every message state change goes through
core/src/messages/transitions.ts, which logs it. - Alerts leave the project store, so they fall under ADR 0032: IDs and codes, never an address, content, or provider text.
- A missing Send event is not a state change: an accepted message stays accepted. ADR 0011 already gives an accepted message without its Send event a deadline 72 hours after the attempt, for budget settlement (ADR 0013), which is not built yet.
Decision
Section titled “Decision”- Port.
OperatorAlerts.raise(occurrences)incore/src/extensions/operator-alerts.ts, with the defaultnoOperatorAlerts, which raises nothing. Executors call it from the outbox drain of the store’s wake, after commit and outside any transaction, and never from the drain that follows a command, so a slow or hangingOperatorAlertsadds no latency to a command such asclaimSendorrecordOutcome. The Workers composition root takes it as theoperatorAlertsoption ofcreateProjectDOand defaults to the no-op; the cloud composition root passes its implementation. Public code never imports the cloud implementation. - Outbox. The outbox carries
OutboxEntry, a delivery job or an alert occurrence, told apart bykind; project-store migration 9 adds thekindcolumn. Writing an entry registers a wake at once. After a command the executor drains only delivery jobs; the wake drains both kinds, each to its sink in its own batch, so a failing or slowOperatorAlertsnever holds back delivery jobs, and is never waited for by a command. A rejectedraisekeeps the occurrences, and a later wake hands them over again: at least once, never inside a transaction. Rows left after a drain leave a retry wake:OUTBOX_RETRY_MS(5 seconds) while delivery jobs are left,ALERT_OUTBOX_RETRY_MS(one minute) when only alert occurrences are. Alert occurrences are handed over only from a due wake: at the latest by that retry, and sooner when another wake of the store is due. - Conditions. An occurrence’s condition is the message outcome that raised it, in the step-log vocabulary:
queued:provider_unavailable,failed:provider_unavailable,failed:provider_rejected(only withproviderReasonbad_request; other permanent rejections are about the message),failed:retries_exhausted(the provider’s only retryable rejection is throttling, so every exhausted message counts),unknown:provider,unknown:adapter_error,unknown:lease_expired, andfailed:outcome_unknown(an unknown left unresolved for 72 hours, alerted as ADR 0012 and docs/architecture.md say). The transition writer adds the occurrence whenever it writes one of these statuses, so no path can miss it. The last condition,accepted:send_event_missing, has no state of its own; see below.- Amended on 2026-11-15, while building B2: two more conditions without a state of their own, each written by the provider event that raises it, in the same transaction.
bounced:account_suppression_listis a Permanent bounce from the provider account’s suppression list (ADR 0034), which means the provider’s suppression is misconfigured; it is written with the message’s charge reversal, so once per message.accepted:lateis a Send event more than 72 hours after the attempt (ADR 0012): a message recorded as not sent was sent.
- Amended on 2026-11-15, while building B2: two more conditions without a state of their own, each written by the provider event that raises it, in the same transaction.
- Missing Send events. An accepted message without its Send event has two deadlines in turn. A message accepted by the provider’s response first gets one
SEND_EVENT_WINDOW_MS(one hour) later, never past ADR 0011’s. When it passes, the due-work pass writes anaccepted:send_event_missingoccurrence and moves the deadline to ADR 0011’s, 72 hours after the attempt, where budget settlement will release the reservation; until budget exists, reaching it only clears the deadline. The Send event clears either deadline, and a message accepted by its Send event gets none. So a message alerts at most once and stays accepted. - Incident key.
{ provider, condition, providerReason, windowStart }: theEmailProvider.nameof the attempt (recorded on the message byclaimSend, project-store migration 9), the condition, the adapter’s code, and the start of theALERT_WINDOW_MS(one hour) window the occurrence falls in. It holds no project or workspace.incidentIdgives the key as one string. A code that does not look like an adapter code ([a-z0-9_], at most 64 characters) is replaced withinvalid_provider_reason, so provider text can never reach an alert. - Occurrence. The key, the workspace, project, and message IDs, and the time. Nothing else.
- Deduplication belongs to the implementation: it raises one alert per incident key across all projects, from a store shared by all of them. Core only guarantees that every occurrence of one incident carries the same key.
Consequences
Section titled “Consequences”- An account-wide outage raises one alert per hour while it lasts, not one per message or project. An outage that crosses a window boundary raises a second alert at once.
- Self-hosted installations get the no-op until they wire an implementation. A self-hosted implementation needs a store shared by all project stores to deduplicate.
- Every alerting outcome adds an outbox row, also on installations with the no-op. Rows are deleted after each drain.
- Accepted messages now hold a deadline and a wake. The Send event usually clears the deadline within seconds, but not the wake: every message accepted by the provider’s response wakes its store about an hour later, and that wake usually finds nothing due. Without the Send event, the message wakes its store twice: at the end of the window and at ADR 0011’s deadline.
- Known edge: the first deadline is never past ADR 0011’s, so a message accepted by a late response within the last hour before 72 hours after its attempt reaches ADR 0011’s deadline first and never raises
accepted:send_event_missing. - Alert occurrences reach
OperatorAlertson the store’s next wake rather than as the command returns; the write registers that wake at once, so the delay is the platform’s alarm latency. - While
OperatorAlertskeeps failing, every store with alert occurrences keeps them and retries the drain once a minute, or at its other wakes; during an account-wide incident that is every store that sent. An implementation should therefore reject only for a failure that a retry can fix, never for one occurrence it cannot handle. - Messages claimed before migration 9 have no recorded provider; their occurrences carry
provider: null. - Noted on 2026-11-16 (B4): an incident’s
providerReasonis usually the adapter’s code, but two are made without a provider call:provider_tenant_missing(the delivery consumer, cloud mode, no provider tenant) andfake_in_cloud_mode(the cloud-mode fake provider). Theirprovideris the selected provider’s name, though nothing was sent through it. - Not alerted yet: quarantined wake items (ADR 0011), and system mail failures (ADR 0029). They can raise through the same port when needed.