Skip to content

0038. Operator alerts are outbox entries keyed by incident, raised through an extension point

  • Status: Accepted (decided by the maintainer)
  • Date: 2026-11-15

ADR 0034 says several send outcomes alert the operator: an unavailable provider (sending paused, credentials, configuration) and a bad_request rejection, and, by ADR 0034’s amendment of 2026-11-13, unknown outcomes and accepted sends whose Send event never arrives. ADR 0012 and docs/architecture.md also alert an unknown outcome that expires unresolved after 72 hours. If the daily-quota text match ever misses, the quota shows up as throttling that exhausts the retry limit, and that must alert too. The amendment fixes the mechanism: an occurrence is written to the project store’s outbox in the transaction that records the outcome, never emitted as a side effect inside it, and it carries an incident key without the project, so one account-wide SES failure seen by many projects raises one alert. docs/architecture.md lists operational alerting as a Proposed extension point with a no-op default. B2 will raise OnAccountSuppressionList bounces and late Send events through the same port.

Facts:

  • The outbox held only delivery jobs, drained to the delivery queue after each command and retried from the wake (ADR 0011). Commands see only ProjectTx, so they cannot reach a port, and a throw rolls back every write, the outbox included.
  • Every message state change goes through core/src/messages/transitions.ts, which logs it.
  • Alerts leave the project store, so they fall under ADR 0032: IDs and codes, never an address, content, or provider text.
  • A missing Send event is not a state change: an accepted message stays accepted. ADR 0011 already gives an accepted message without its Send event a deadline 72 hours after the attempt, for budget settlement (ADR 0013), which is not built yet.
  • Port. OperatorAlerts.raise(occurrences) in core/src/extensions/operator-alerts.ts, with the default noOperatorAlerts, which raises nothing. Executors call it from the outbox drain of the store’s wake, after commit and outside any transaction, and never from the drain that follows a command, so a slow or hanging OperatorAlerts adds no latency to a command such as claimSend or recordOutcome. The Workers composition root takes it as the operatorAlerts option of createProjectDO and defaults to the no-op; the cloud composition root passes its implementation. Public code never imports the cloud implementation.
  • Outbox. The outbox carries OutboxEntry, a delivery job or an alert occurrence, told apart by kind; project-store migration 9 adds the kind column. Writing an entry registers a wake at once. After a command the executor drains only delivery jobs; the wake drains both kinds, each to its sink in its own batch, so a failing or slow OperatorAlerts never holds back delivery jobs, and is never waited for by a command. A rejected raise keeps the occurrences, and a later wake hands them over again: at least once, never inside a transaction. Rows left after a drain leave a retry wake: OUTBOX_RETRY_MS (5 seconds) while delivery jobs are left, ALERT_OUTBOX_RETRY_MS (one minute) when only alert occurrences are. Alert occurrences are handed over only from a due wake: at the latest by that retry, and sooner when another wake of the store is due.
  • Conditions. An occurrence’s condition is the message outcome that raised it, in the step-log vocabulary: queued:provider_unavailable, failed:provider_unavailable, failed:provider_rejected (only with providerReason bad_request; other permanent rejections are about the message), failed:retries_exhausted (the provider’s only retryable rejection is throttling, so every exhausted message counts), unknown:provider, unknown:adapter_error, unknown:lease_expired, and failed:outcome_unknown (an unknown left unresolved for 72 hours, alerted as ADR 0012 and docs/architecture.md say). The transition writer adds the occurrence whenever it writes one of these statuses, so no path can miss it. The last condition, accepted:send_event_missing, has no state of its own; see below.
    • Amended on 2026-11-15, while building B2: two more conditions without a state of their own, each written by the provider event that raises it, in the same transaction. bounced:account_suppression_list is a Permanent bounce from the provider account’s suppression list (ADR 0034), which means the provider’s suppression is misconfigured; it is written with the message’s charge reversal, so once per message. accepted:late is a Send event more than 72 hours after the attempt (ADR 0012): a message recorded as not sent was sent.
  • Missing Send events. An accepted message without its Send event has two deadlines in turn. A message accepted by the provider’s response first gets one SEND_EVENT_WINDOW_MS (one hour) later, never past ADR 0011’s. When it passes, the due-work pass writes an accepted:send_event_missing occurrence and moves the deadline to ADR 0011’s, 72 hours after the attempt, where budget settlement will release the reservation; until budget exists, reaching it only clears the deadline. The Send event clears either deadline, and a message accepted by its Send event gets none. So a message alerts at most once and stays accepted.
  • Incident key. { provider, condition, providerReason, windowStart }: the EmailProvider.name of the attempt (recorded on the message by claimSend, project-store migration 9), the condition, the adapter’s code, and the start of the ALERT_WINDOW_MS (one hour) window the occurrence falls in. It holds no project or workspace. incidentId gives the key as one string. A code that does not look like an adapter code ([a-z0-9_], at most 64 characters) is replaced with invalid_provider_reason, so provider text can never reach an alert.
  • Occurrence. The key, the workspace, project, and message IDs, and the time. Nothing else.
  • Deduplication belongs to the implementation: it raises one alert per incident key across all projects, from a store shared by all of them. Core only guarantees that every occurrence of one incident carries the same key.
  • An account-wide outage raises one alert per hour while it lasts, not one per message or project. An outage that crosses a window boundary raises a second alert at once.
  • Self-hosted installations get the no-op until they wire an implementation. A self-hosted implementation needs a store shared by all project stores to deduplicate.
  • Every alerting outcome adds an outbox row, also on installations with the no-op. Rows are deleted after each drain.
  • Accepted messages now hold a deadline and a wake. The Send event usually clears the deadline within seconds, but not the wake: every message accepted by the provider’s response wakes its store about an hour later, and that wake usually finds nothing due. Without the Send event, the message wakes its store twice: at the end of the window and at ADR 0011’s deadline.
  • Known edge: the first deadline is never past ADR 0011’s, so a message accepted by a late response within the last hour before 72 hours after its attempt reaches ADR 0011’s deadline first and never raises accepted:send_event_missing.
  • Alert occurrences reach OperatorAlerts on the store’s next wake rather than as the command returns; the write registers that wake at once, so the delay is the platform’s alarm latency.
  • While OperatorAlerts keeps failing, every store with alert occurrences keeps them and retries the drain once a minute, or at its other wakes; during an account-wide incident that is every store that sent. An implementation should therefore reject only for a failure that a retry can fix, never for one occurrence it cannot handle.
  • Messages claimed before migration 9 have no recorded provider; their occurrences carry provider: null.
  • Noted on 2026-11-16 (B4): an incident’s providerReason is usually the adapter’s code, but two are made without a provider call: provider_tenant_missing (the delivery consumer, cloud mode, no provider tenant) and fake_in_cloud_mode (the cloud-mode fake provider). Their provider is the selected provider’s name, though nothing was sent through it.
  • Not alerted yet: quarantined wake items (ADR 0011), and system mail failures (ADR 0029). They can raise through the same port when needed.