Skip to content

Turning on real mail

The checks and procedures to finish before an installation sends real mail through Amazon SES, and to keep at hand afterwards. They are B4’s deploy gate in docs/architecture.md. Set SES up first (Setting up Amazon SES). Placeholders are in angle brackets, as there; $OPERATOR_TOKEN is the Worker’s OPERATOR_TOKEN.

Do these in a sandbox or staging account and Worker first, then confirm them for production. Record each result (date, who, and the evidence) where your operations notes live, not in the repository.

  • Operator alerts are wired and reach someone, and a 5xx notification or an external health check watches the Worker, because a refused configuration raises no alert.
  • No active sending workflow exists without an approval in a workspace that requires approval (see ADR 0044: approval is optional per workspace, off by default). Nothing deployed before send approvals existed, so a fresh installation passes this.
  • No workspace with send approval off sends real mail until pause and rollback, the workspace send budget (ADR 0013), and the per-tenant sender protection of ADR 0024 are built (ADR 0044). Required on the managed service, and recommended for self-hosted installations too: with approval off, a leaked admin key can send any content from the verified domains.
  • The deliveries queue’s concurrency fits the SES sending rate.
  • The runbooks for removing a sending domain and the emergency stop have been read and tried once in staging.
  • The smoke test confirmed the daily-quota message text, the tenant error codes, and that DKIM covers List-Unsubscribe and List-Unsubscribe-Post.
  • With tenants: the send policy’s ses:TenantName condition key (setup step 1) is accepted by IAM and a send without TenantName is refused (see What a send without a tenant returns), and put-configuration-set-suppression-options --suppression-scope TENANT took effect (“Check” in setup step 2).
  • A round trip worked: an SES send, its event through EventBridge, and the event stored against the message.
  • The SES event fixtures are recorded, and the adapter’s tests pass on them.
  • setAlarm atomicity is confirmed on a deployed Durable Object.
  • The platform log settings are set in the deployment configuration.
  • Only then: EMAIL_PROVIDER=ses.

The project stores raise alerts for an unavailable provider, bad_request rejections, unknown outcomes, retries exhausted after throttling, accepted sends whose Send event never arrives, and account-list bounces (ADR 0038). The public Worker’s default implementation drops them, so a self-hosted installation that wants them implements the OperatorAlerts extension point in its own composition root. Without it, the only signals of an account-wide SES failure are the EventBridge alarms and the messages’ states.

Alerts come from outcomes, so a Worker that refuses its configuration (500 on every request, refusing to serve: in its logs) raises none: watch its 5xx rate or probe a cheap route such as GET <app-origin>/api/openapi.json from outside. One alert stands for an incident across projects and names the first workspace it saw; the others are found as below.

Deliveries queue concurrency and the SES sending rate

Section titled “Deliveries queue concurrency and the SES sending rate”

Each consumer invocation sends its batch’s messages one after the other, one SendEmail request each. Its throughput is about one message per SendEmail latency, so the queue’s throughput is about max_concurrency / latency. Above the account’s sending rate, SES answers throttled, which the store retries with backoff only up to the retry limit; sustained overload fails messages as failed:retries_exhausted and raises an alert.

  1. Read the sending rate per Region: aws sesv2 get-account --region <region> --query SendQuota.MaxSendRate. A sandbox account allows one message per second.

  2. Measure the shortest SendEmail latency you see from the Worker (the round trip below gives a sample), in seconds.

  3. Set max_concurrency on the deliveries consumer to at most floor(MaxSendRate × shortest latency), and at least 1. When that is below 1 (a sandbox account at one message per second with a latency under a second), keep the volume low until the rate is raised; some sends are then throttled by design, so confirm in the round trip that throttled retries succeed well within the retry limit.

    "queues": {
    "consumers": [{ "queue": "deliveries", "max_batch_size": 10, "max_batch_timeout": 0, "max_concurrency": <n> }]
    }

    Unset, the consumer scales up to the platform’s limit with the backlog.

  4. Revisit it when SES raises the rate, and when a Region is added: the rate is per Region, and the queue is shared, so size it for the slowest Region.

A workspace stops sending from a domain as soon as every project store’s replica lacks it: a queued message from it is then skipped as sender_not_verified, and an activation that names it is refused.

  1. Delete it from the control plane, which replicates the workspace’s setup to its project stores:

    Terminal window
    curl -sS -X DELETE <app-origin>/operator/workspaces/<workspace-id>/sending-domains/<domain> \
    -H "authorization: Bearer $OPERATOR_TOKEN"
  2. A 200 means every project store took the new replica. A 503 means at least one did not: its failedProjectIds keep the older replica, and can still send from the domain. Resync the workspace:

    Terminal window
    curl -sS -X POST <app-origin>/operator/sender-replicas/resync \
    -H "authorization: Bearer $OPERATOR_TOKEN" -H "content-type: application/json" \
    -d '{ "workspaceId": "<workspace-id>" }'

    Retry the resync a few times with a pause between attempts. A resync is idempotent: a store that already has the version ignores it.

  3. If the resync still answers 503, raise it as an incident: name the failing project IDs, check the Worker’s logs for the replication error (logged by project ID), and keep resyncing until it answers 200. Until then, use the emergency stop for that workspace if the domain must not send at all.

  4. Then remove the identity in SES (and, with tenants, its tenant association), and its DNS records. Removing it in SES first would make its sends fail at SES instead of being skipped.

Until kill switches exist, stop sending at SES, at the deliveries queue, or by removing domains. Each keeps the Worker on ses. Never switch a self-hosted Worker that has real messages queued back to fake: outside cloud mode, the fake provider accepts them and reports them delivered, so they are marked sent and billed while nobody receives them. In cloud mode it refuses every send instead (below).

Reversible stops at SES: SES then answers SendingPausedException, the adapter records unavailable:sending_paused, and messages wait as queued:provider_unavailable (alerted) and send once sending resumes, if that is within 24 hours of their first attempt; after that they fail as failed:provider_unavailable.

  • The whole account in a Region: aws sesv2 put-account-sending-attributes --region <region> --no-sending-enabled (resume with --sending-enabled).
  • One stream: aws sesv2 put-configuration-set-sending-options --region <region> --configuration-set-name <set> --no-sending-enabled.
  • One workspace with a tenant: pause the tenant’s sending in SES. SES then answers MessageRejected “Sending disabled for ” instead, which the adapter records as unavailable:provider_tenant_paused, with the same waiting, alert, and 24-hour clock.

Pause the deliveries queue, so no send is attempted at all and no 24-hour clock starts: wrangler queues pause-delivery <deliveries-queue-name>, and wrangler queues resume-delivery <deliveries-queue-name> to resume. Messages stay queued; a job that outlives the queue’s retention is enqueued again by the store when its deadline passes. The API, unsubscribe, and approval links keep working. While the queue is paused, the store enqueues every queued message again each time its 10-minute claim window passes, so duplicate jobs build up and arrive as a burst on resume; claimSend makes every duplicate a no-op. Messages already queued:provider_unavailable keep their 24-hour clock from their first attempt, so a pause longer than that fails them.

Last resort, with no 24-hour clock either: make the Worker refuse its configuration, for example by setting EMAIL_PROVIDER to a name it does not know, or deleting the SES_EVENTS_SECRET secret. Every request then answers 500, including the API, one-click unsubscribe, and approval links; delivery jobs are retried every minute and nothing is sent (a job dropped once the queue’s max_retries run out is enqueued again by the store when its deadline passes, as in setup step 6); messages already queued:provider_unavailable keep their 24-hour clock; SES events get 500 and are retried by EventBridge for 24 hours, then wait in the dead-letter queue for replay. Undo the change to resume.

Final stop for one workspace: remove every one of its sending domains, as above. Every message it would send becomes skipped:sender_not_verified, which is final, and its activations are refused. Its runs, events, and approvals are untouched.

  1. List the workspace’s domains from the control plane database:

    Terminal window
    wrangler d1 execute <control-database-name> --remote \
    --command "SELECT domain, status FROM sending_domains WHERE workspace_id = '<workspace-id>'"
  2. Delete each one through the operator endpoint, then resync the workspace until it answers 200.

  3. To resume, put the domains back with "status": "verified". Messages skipped in between are not sent later.

In cloud mode, the workspaces whose messages wait for a missing provider tenant (provider_tenant_missing) are those without one in the control plane, and any workspace whose resync has not answered 200, whose project stores may still hold a replica without the tenant:

Terminal window
wrangler d1 execute <control-database-name> --remote \
--command "SELECT id FROM workspaces WHERE provider_tenant IS NULL"

The workspaces whose messages wait for a paused tenant (provider_tenant_paused) are those whose tenant SES or the operator paused. The alert names only one affected workspace per alert window, so list every tenant in each Region and check its sending status, either on the tenant or on its reputation entity:

Terminal window
aws sesv2 list-tenants --region <region>
aws sesv2 get-tenant --region <region> --tenant-name <tenant-name>
aws sesv2 get-reputation-entity --region <region> \
--reputation-entity-type RESOURCE --reputation-entity-reference <tenant-arn>

A tenant whose sending status is DISABLED is paused; match its name with provider_tenant in the control plane to find the workspace.

adapters/email-ses/README.md describes test/mailbox-simulator.smoke.test.ts and how to run it with the send credentials. It prints every raw response. Confirm, and record the responses:

  1. Daily quota. In a sandbox account (200 messages a day), send until SES refuses. Mailbox simulator sends do not count toward the daily quota, so this needs real sends to addresses you own and have verified, not the simulator. Record the error code (ThrottlingException, TooManyRequestsException, or LimitExceededException) and the exact message text. The adapter matches “Daily message quota exceeded” to classify daily_quota_exceeded; if the text differs, change the adapter and its test before going live.
  2. Tenant errors. Run the test with SES_SMOKE_TENANT set to a tenant that does not exist, then a paused tenant, then a tenant not associated with the identity or the configuration set. Record each status and error code. The adapter classifies 404 as provider_configuration, MessageRejected “Sending disabled for ” as provider_tenant_paused, and any other 400 by its code; if a tenant error comes back as a 400 that reads as a permanent rejection of the message, change the classification first.
  3. DKIM over the unsubscribe headers. Send one marketing message through the deployed Worker to a mailbox you control (a verified address in the sandbox). In the received message’s DKIM-Signature header for your domain, check that the h= list includes List-Unsubscribe and List-Unsubscribe-Post, and that the receiver reports dkim=pass. SES signs the headers it is given; if they are missing from h=, the gate fails: mailbox providers may ignore one-click unsubscribe. Do not go live; record the received headers and raise it, since the fix (signing the domain with your own DKIM key, or another way of sending the headers) needs a decision.

With the send policy of setup step 1 and configuration sets on TENANT suppression (setup step 2), which one refuses a send without TenantName depends on the configuration set:

Send without TenantName Response Adapter
Through one of the TENANT-scope configuration sets (the smoke test with SES_SMOKE_TENANT empty) 400 BadRequestException, “Tenant name is required when suppression scope is TENANT on the configuration-set”: SES’s own check answers first rejected, bad_request
Without a configuration set (aws sesv2 send-email with the send credentials and no --configuration-set-name) AccessDeniedException on the identity: the IAM condition unavailable, provider_auth

Record both. The second shows that IAM accepts the condition key and refuses a send without it; the first, that a TENANT-scope set never sends without a tenant either. A send that is accepted in either case fails the check.

With the rule, connection, and API destination in place and EMAIL_PROVIDER=ses on a staging Worker:

  1. Approve a workflow that sends from a verified domain to success@simulator.amazonses.com, and trigger it.
  2. Read the message (GET /api/messages/<message-id>): state accepted with a providerMessageId, then a deliveryStatus of delivered once the Delivery event arrives, and billableEmails 1 in GET /api/metrics once the Send event arrives.
  3. If no event arrives, check the rule’s Invocations and FailedInvocations metrics, the dead-letter queue and its message attributes, and the Worker’s logs for refusing to serve:.

The SES event fixtures in adapters/email-ses/src/testing/fixtures are written from the AWS documentation. A wrong detail-type or subtype string would drop events silently or change suppression and billing, so they are replaced with recorded events before real mail (the fixtures’ README says how to anonymize them).

  1. Capture the events. Adding a target to your events rule changes the rule, so prefer a temporary second rule with the same pattern and a CloudWatch Logs log group as its target. Run this before the first send, in a scratch directory outside the repository, with the credentials you set SES up with:

    Terminal window
    region=<region>; rule=<your SES events rule>; capture=ses-fixture-capture
    log_group="/aws/events/$capture"
    account="$(aws sts get-caller-identity --query Account --output text)"
    aws events describe-rule --region "$region" --name "$rule" --query EventPattern --output text > pattern.json
    aws logs create-log-group --region "$region" --log-group-name "$log_group"
    aws logs put-retention-policy --region "$region" --log-group-name "$log_group" --retention-in-days 3
    jq -n --arg r "arn:aws:logs:$region:${account}:log-group:$log_group:*" '{
    Version: "2012-10-17",
    Statement: [{
    Effect: "Allow",
    Principal: { Service: ["events.amazonaws.com", "delivery.logs.amazonaws.com"] },
    Action: ["logs:CreateLogStream", "logs:PutLogEvents"],
    Resource: $r
    }]
    }' > logs-policy.json
    aws logs put-resource-policy --region "$region" --policy-name "$capture" --policy-document file://logs-policy.json
    aws events put-rule --region "$region" --name "$capture" --event-pattern file://pattern.json --state ENABLED
    aws events put-targets --region "$region" --rule "$capture" \
    --targets "Id=logs,Arn=arn:aws:logs:$region:${account}:log-group:$log_group"

    After the first send, aws logs filter-log-events --region "$region" --log-group-name "$log_group" --limit 1 returns an aws.ses event. Export them all, and list what was recorded:

    Terminal window
    aws logs filter-log-events --region "$region" --log-group-name "$log_group" --output json > ses-events-raw.json
    jq -r '.events[].message | fromjson | [."detail-type", (.detail.bounce.bounceType // ""), (.detail.bounce.bounceSubType // "")] | @tsv' ses-events-raw.json

    The raw events hold recipient addresses, your domain, and account IDs: keep them outside the repository until they are anonymized (step 4). Delete the capture the same day:

    Terminal window
    aws events remove-targets --region "$region" --rule "$capture" --ids logs
    aws events delete-rule --region "$region" --name "$capture"
    aws logs delete-resource-policy --region "$region" --policy-name "$capture"
    aws logs delete-log-group --region "$region" --log-group-name "$log_group"
  2. Produce them through the mailbox simulator:

    Fixture How
    send, delivery Send to success@simulator.amazonses.com
    bounce-permanent Send to bounce@simulator.amazonses.com
    bounce-on-tenant-suppression-list Put success@simulator.amazonses.com on the tenant’s suppression list (PutSuppressedDestination with the TenantName), then send to it with that tenant. SES refuses that PutSuppressedDestination while the account is in the sandbox (BadRequestException, “Your account is still in the sandbox”), so record this one after production access
    bounce-on-account-suppression-list Put success@simulator.amazonses.com on the account’s suppression list, then send to it without a tenant through a configuration set that uses the account list (a scratch set, not one of the two above). The send policy of setup step 1 with tenants refuses a send without TenantName, so use a scratch IAM user in the test account whose policy allows ses:SendEmail on the scratch set without the tenant condition, and delete it afterwards
    bounce-suppressed Send to suppressionlist@simulator.amazonses.com, which SES treats as on its global suppression list. In a recording (2026-11) it came back as Permanent / General, not Suppressed; if yours does too, confirm the Suppressed subtype as in step 3 instead
    complaint Send to complaint@simulator.amazonses.com
    tenant-sending-status-disabled, -enabled Pause the tenant’s sending, then resume it

    Remove the suppression entries afterwards.

  3. For reject, delivery-delay, rendering-failure, bounce-transient, bounce-undetermined, and complaint-on-account-suppression-list, which the simulator probably cannot produce, confirm the detail-type, the bounce or complaint type, and the subtype strings against a real event or the EventBridge schema registry (aws.ses schemas), and update the hand-written fixture if they differ.

  4. Anonymize each recorded event as the fixtures’ README says, replace the file, and run pnpm test:contracts. Every adapter test must pass on the recorded events.

setAlarm atomicity on a deployed Durable Object

Section titled “setAlarm atomicity on a deployed Durable Object”

The project store arms its wake alarm with setAlarm inside the transaction callback, and relies on a rolled-back transaction leaving no alarm. Miniflare shows that; confirm it on Cloudflare once, with a throwaway Worker in a staging account:

// probe.ts: deploy with a Durable Object binding PROBE (new_sqlite_classes: ["Probe"]).
import { DurableObject } from "cloudflare:workers";
export class Probe extends DurableObject {
async fetch(): Promise<Response> {
await this.ctx.storage.deleteAlarm();
try {
await this.ctx.storage.transaction(async () => {
await this.ctx.storage.setAlarm(Date.now() + 60_000);
throw new Error("roll back");
});
} catch {}
const rolledBack = await this.ctx.storage.getAlarm();
await this.ctx.storage.transaction(async () => {
await this.ctx.storage.setAlarm(Date.now() + 60_000);
});
const committed = await this.ctx.storage.getAlarm();
// The store also moves an existing alarm: a rolled-back move must keep the original.
try {
await this.ctx.storage.transaction(async () => {
await this.ctx.storage.setAlarm(Date.now() + 120_000);
throw new Error("roll back");
});
} catch {}
const kept = await this.ctx.storage.getAlarm();
await this.ctx.storage.deleteAlarm();
return Response.json({ rolledBack, committed, kept });
}
async alarm(): Promise<void> {}
}
export default {
fetch: (_request: Request, env: { PROBE: DurableObjectNamespace }) =>
env.PROBE.get(env.PROBE.idFromName("probe")).fetch("https://probe/"),
};

Expect rolledBack null, committed a time, and kept equal to committed. If rolledBack is a time or kept differs, the alarm survives a rolled-back transaction: stop, and raise it before going live, because the store’s wake recovery assumes otherwise. Delete the probe Worker afterwards.

The Worker’s own log lines carry no personal data or secrets (ADR 0032). What Cloudflare records about each request is set in the deployment configuration, and request URLs carry secrets: the one-click unsubscribe links are /u/<project-id>/<message-id>/<token> (ADR 0036), and whoever reads a token can unsubscribe that customer. Approval link secrets travel in the URL fragment, which is never sent, and in request bodies.

Decide and set, in the Worker’s deployment configuration:

  1. Workers Logs (observability in the wrangler configuration). Invocation logs record each request’s URL. Either turn them off ("observability": { "enabled": true, "logs": { "invocation_logs": false } }), keeping only the Worker’s own logError lines, or keep them and accept that tokens are retained for Workers Logs’ retention period; check the current period for your plan in Cloudflare’s documentation.
  2. Logpush (logpush in the wrangler configuration and a Logpush job). Leave it off, or configure the job without the Event field of Workers Trace Events, which holds the request URL, and set the destination’s retention.
  3. Who can read them: limit dashboard access to the Workers Logs and Logpush destinations to operators. wrangler tail and Tail Workers see request URLs too.
  4. Uncaught exceptions are logged by the platform with their message. The Worker catches and redacts its own errors, so treat an uncaught one in the logs as a bug.