0039. Sending domains and provider tenants are provisioned at runtime by one paced job runner per Region
- Status: Proposed
- Date: 2026-11-18
- Amends: ADR 0035 (sending domains get a lifecycle, an ID as their key in place of
(workspace_id, domain), an ownership challenge, and one owner per installation;SenderSetupStoreis also written by requests and the job runner, not only the operator endpoints; replication after a runtime write is a retried job instead of a synchronous step in the request; the operator endpoints become repair tools), ADR 0038 (alert occurrences can also be written to a control-plane outbox, carrying a workspace and a sending-domain ID instead of project and message IDs), ADR 0034 (tenant and identity management gets its own port and credential), ADR 0024 (settles how the DNS checks and the 1 request per second management limit are used, and adds a daily integrity read of each verified identity), ADR 0005 and ADR 0026 (adds thesendingDomains.*actions, workspace-level actions in the principal and target model of ADR 0041) - Depends on: ADR 0041 (Proposed) for principals, the
workspacetarget, andWorkspaceScope
Context
Section titled “Context”Today the operator sets up each workspace’s sending by hand: creates the SES tenant, creates the domain identity, waits for verification, associates the identity and both configuration sets with the tenant, and records the result through the /operator/* endpoints of ADR 0035. The maintainer decided on 2026-11-18 that this is automated before launch: users add and verify their own sending domain in the dashboard (and, by ADR 0005, through the API).
Things already decided that this must keep:
- A workspace sends only from verified sending domains in its project stores’ replica, matched exactly, and SES tenant association is the third guard (ADR 0034). Replicas are ordered by
sender_version(ADR 0035), and the tenant is recorded with its Region (ADR 0037). - The sending credential may call
ses:SendEmailonly; management never runs on the send path (ADR 0034). - No blind repeat of a side effect whose outcome is unknown (ADR 0007, ADR 0012), no domain names, addresses, or credentials in logs (ADR 0032), and our own DNS checks before calling SES (ADR 0024).
- Every lookup is scoped by project or workspace; the only exceptions are those the Tenant model lists.
- ADR 0035 left open whether a domain is unique across workspaces, deferring it to “the domain management API together with its DNS checks”.
Facts checked on 2026-11-18 in the AWS documentation:
CreateTenanttakesTenantName(“up to 64 alphanumeric characters, including letters, numbers, hyphens (-) and underscores (_) only”),Tags, andSuppressionAttributes; it fails withAlreadyExistsException,BadRequestException,LimitExceededException, orTooManyRequestsException. Tenants are per Region and per account, an account may create 10,000 (adjustable), and a tenant can send only once it has at least one verified identity and one configuration set.DeleteTenantremoves its associations but not the resources.CreateTenantResourceAssociationtakesTenantNameandResourceArn(an identity, configuration set, or template) and fails withAlreadyExistsExceptionorNotFoundException, among others. “A single resource can be associated with multiple tenants.” An identity or configuration set associated with a tenant cannot be deleted. No limit on associations per tenant is documented.CreateEmailIdentitytakesTagsandDkimSigningAttributes(Easy DKIM withNextSigningKeyLengthRSA_2048_BIT) and returns the three DKIMTokensandSigningHostedZone; it fails withAlreadyExistsException,LimitExceededException,ConcurrentModificationException(HTTP 500), and others. An account holds 10,000 identities per Region (adjustable). In IAM,CreateEmailIdentityandCreateTenantacceptaws:RequestTagandaws:TagKeys, and depend onses:TagResource(Service Authorization Reference).- Identities exist per Region. A verified domain identity lets the account “send email from any subdomain or email address of the verified domain without having to verify each one individually”; a separately verified subdomain’s settings override its parent’s. The DKIM CNAME value is “the DKIM token followed by a hosted zone domain”, which “varies by AWS Region and identity” and is returned as
SigningHostedZone. GetEmailIdentityreportsVerificationStatusandDkimAttributes.Status(PENDING,SUCCESS,FAILED,TEMPORARY_FAILURE,NOT_STARTED),VerificationInfo, the MAIL FROM attributes, and the tags.- SES looks for the DKIM records “for up to 72 hours” (
DkimAttributes), then the status isFAILEDand verification must be restarted “from the beginning” (SESv1VerifyDomainIdentity). SESv2 documents no restart operation, so a restart is a delete and a new create. If the DKIM records of a verified domain are removed, SES stops DKIM signing and later revokes the setup (DKIM troubleshooting); no period is documented. - Without a custom MAIL FROM domain, SES uses its own, so SPF does not align for DMARC; DKIM alignment does (DMARC, which recommends starting with
p=none). - Quotas: “All actions (except for SendEmail and SendBulkEmail) are throttled at one request per second”, per account and Region, and this is not adjustable.
- SES publishes no EventBridge event for identity verification (SES events in EventBridge: message events, tenant
Sending Status EnabledandDisabled, and Advisor recommendations only). Verification can only be polled. EventBridge does carry “AWS API Call via CloudTrail” events, which show calls, not status changes. - A Cloudflare Worker has no instance role; it can use only credentials stored as its secrets.
Alternatives considered
Section titled “Alternatives considered”| Choice | Alternatives | Why this one |
|---|---|---|
A ProviderProvisioning port, apart from EmailProvider |
More operations on EmailProvider |
Different credentials, different limits, and never on the send path. A self-hosted installation can send through SES and provision by hand |
| Jobs in the control plane, run by one runner per provider Region (a Durable Object) that makes at most one management call per pacing interval | Cloudflare Queues with max_concurrency: 1 and delays; calls made in the request |
The limit is per account and Region and cannot be raised. A queue paces batches, not calls, and its retries would be a second retry owner beside the job table (ADR 0011’s rule). A call in the request would fail under concurrent adds and return a 429 for something the user cannot fix |
| A separate provisioning credential, its policy a subset of the management role’s with the same name and tag conditions | Reuse the send user; attach the full management policy to a user; a user that can only sts:AssumeRole the management role |
See Credentials. A shared key would let one leak both claim any tagged identity and send from it |
| A workspace-specific ownership TXT record checked before the claim and the SES identity exist | Claim at add, held unverified for 72 hours; first to verify DKIM wins | An identity is one per account and Region with one set of DKIM tokens, so DKIM cannot say which workspace proved control; and a claim taken at add lets anyone hold a domain they do not control. A per-workspace TXT value proves control by this workspace before anything is reserved or created |
Polling GetEmailIdentity, gated by our own DoH check |
EventBridge; CloudTrail events | SES publishes no verification event |
| The tenant is created lazily, before the workspace’s first identity | At workspace creation | Most of the 10,000-tenant quota then goes to workspaces that send. No workspace sends without a verified domain anyway |
| No custom MAIL FROM in the first version | Custom MAIL FROM from the start | DKIM alone passes DMARC alignment; MAIL FROM adds two records, a status, an IAM action, and a failure mode. It comes with the marketing-send DMARC gate |
Decision
Section titled “Decision”The ProviderProvisioning port
Section titled “The ProviderProvisioning port”- A core port, independent of
EmailProvider, with one operation per provider call:
| Operation | SES call | already_exists |
not_found |
|---|---|---|---|
createTenant(region, tenant, tags) |
CreateTenant with SuppressionAttributes scope TENANT, reasons BOUNCE and COMPLAINT (ADR 0034) |
getTenant, then adopt (the name is the workspace’s own) |
— |
getTenant(region, tenant) |
GetTenant |
— | the job is rejected (the tenant vanished; operator alert) |
associate(region, tenant, resource) |
CreateTenantResourceAssociation; resource is the sending domain or a stream’s configuration set, named by the adapter |
done | the job is rejected (tenant, identity, or configuration set missing; operator alert) |
disassociate(region, tenant, resource) |
DeleteTenantResourceAssociation |
— | done |
listTenantsOf(region, domain) |
ListResourceTenants |
— | the identity is gone; the caller decides |
createDomain(region, domain, tags) |
CreateEmailIdentity with Easy DKIM, RSA_2048_BIT, and the tags |
getDomain, then adopt or domain_in_use (below) |
— |
getDomain(region, domain) |
GetEmailIdentity: verification and DKIM status, the DKIM tokens and SigningHostedZone, the MAIL FROM attributes, and the tags |
— | the identity is gone: a verified domain goes to removing (reason revoked) with an operator alert; a pending_dns one goes to removing (reason failed), which finds nothing to delete |
deleteDomain(region, domain) |
DeleteEmailIdentity |
— | done (but see the unknown-create rule below) |
- Other outcomes:
ok(with the result),throttled,unavailable(credentials, permissions, configuration,LimitExceededExceptionon a quota),rejected(a definite refusal about this request, such as an invalid name), orunknown(a timeout, connection error, 5xx includingConcurrentModificationException, or an unreadable response). The adapter keeps provider codes in aproviderReason, as ADR 0034 does, and never returns an error message, an ARN, or an account ID. - Implementations:
adapters/email-ses: SESv2 overfetchwith aws4fetch,retries: 0, a 10-second timeout, one signer per allowed Region (ADR 0037), with the provisioning credential, never the sending one.adapters/email-fake: an in-memory fake with fixed DKIM tokens, whose verification and outcomes a test controls, and which records every call with its time so tests can check pacing. It is used by tests andpnpm dev.- No implementation (
manual, the default): the installation provisions by hand through the operator endpoints, as today.sendingDomains.add,verify, andremoveanswer a configuration error;listandgetwork. Nothing is claimed that no runner would provision or expire.
PROVIDER_PROVISIONINGselectsses,fake, ormanual.sesrequiresEMAIL_PROVIDER=sesandSES_PROVISIONING_ACCESS_KEY_IDandSES_PROVISIONING_SECRET_ACCESS_KEY; a missing value is a configuration error naming the variable (ADR 0034’s rule). In cloud mode onlysesprovisions;fakeis treated asmanual, so the managed service never claims a domain it cannot provision.- A contract suite in
testing/contractsruns on the fake and on the SES adapter with recorded, anonymized responses and a stubbedfetch. CI never calls AWS.
Credentials
Section titled “Credentials”- The provisioning credential is a third IAM user, apart from the send user and the operator’s management role. Its policy grants only what the runner calls:
CreateTenant,GetTenant,TagResourceon the installation’s tenants;CreateEmailIdentity,GetEmailIdentity,DeleteEmailIdentity,ListResourceTenants, and the identity side of the associations, on identities carrying the installation’s tag; and both sides of the associations on its own tenants and configuration sets. Requests that set tags may set only the two tag keys below (aws:TagKeys). It has noSendEmail, no MAIL FROM, DKIM-attribute,DeleteTenant, suppression-list, or sending-status actions, and no IAM actions. Its name and tag conditions are the management role’s; in the managed service’s infrastructure both policies are built from the same statements, so they cannot drift. - Why not reuse the send user’s key: a leak of either key today does limited harm. The send key sends only from identities already associated with a tenant; the management capability sends nothing. One key with both could verify an attacker’s domain, associate any of the installation’s identities with any of its tenants, and send as a customer’s domain with a valid DKIM signature.
- What a leaked provisioning key can still do: it sends nothing, but it can create, disassociate, and delete the installation’s identities. Deleting them stops every workspace’s sending and is not reversible by us: a re-created identity gets new Easy DKIM tokens, so every customer must publish new DNS records. A condition limiting deletion to identities marked for removal would not help, since the same key could set the mark. So:
- every mutating call of this credential is watched through EventBridge’s “AWS API Call via CloudTrail” events, with an alarm above the expected rate;
- the daily integrity read (below) checks each verified identity’s DKIM status, tags, and that no MAIL FROM domain is set;
- the key is rotated on any suspicion.
- Why not a user that may only
sts:AssumeRolethe management role: the management role also holds suppression and pause actions the runner must not have, and the extra STS call and cached session add a second failure path without narrowing the key’s reach. - Both keys live in the same Worker’s secrets, so a compromise of the Worker’s code reaches both. This is accepted: the separation protects against a leaked secret, not a compromised deploy, and ADR 0035 already rejected a separate operator Worker.
- Rotation follows the send key’s procedure (a second key, deploy, deactivate, delete; at least every 90 days). A provisioning call refused during a rotation is
unavailable, and its job retries later, so no domain or message is lost.
Pacing, retries, and unknown outcomes
Section titled “Pacing, retries, and unknown outcomes”- Jobs are rows of the control plane,
provisioning_jobs(id, workspace_id, domain_id, kind, step, version, attempts, next_at, first_attempted_at, lease_owner, lease_expires_at, provider_reason), written in the same D1 batch as the state change that needs them. A job carries IDs only; the runner of a Region takes the jobs of workspaces whose recorded Region is its own (a workspace without one gets the default Region with its first job, as every versioned write does, ADR 0037). Domain jobs arecheck_ownership,create_domain,associate,check_verification,check_integrity, andremove_domain; workspace jobs (nodomain_id) aretenantandreplicate. A finished job is deleted. Facts about SES that outlive a job are kept on the domain row:identity_created_atandlast_unknown_create_at. - One job per domain, and one job of each workspace kind per workspace, at a time (a unique index on
domain_id, and a partial unique index on(workspace_id, kind)wheredomain_idis null). Each step writes the next step’s job in place of its own.sendingDomains.removeor an expiry replaces whatever job the domain has with a newremove_domain, under a new job ID. - A step’s result is written only if its job is still current: the batch that records an outcome is conditional on the job’s ID and lease owner and on the domain’s expected status. If a removal replaced the job while its call was in flight, the batch records only the facts of the call (
identity_created_at, orlast_unknown_create_atfor an unknown outcome) and writes no transition and no follow-up, so a latecreateDomaincannot bring a removed domain back. A removal that replaces acreate_domainjob that is leased or was ever attempted setslast_unknown_create_atto now in the same batch, since that call’s outcome may never be recorded. Areplicatejob carries thesender_versionit was written for and is deleted after a run only if that version is still the workspace’s; otherwise it runs again. - The workspace job
tenantruns in steps (step):createTenant, thenassociatewith each stream’s configuration set, one call each; then it records the tenant (a versioned write, below), writescreate_domainfor every domain of the workspace inprovisioningthat has no job, and is deleted. A domain whose ownership is proven writestenantif the workspace has neither a recorded tenant nor atenantjob, andcreate_domainotherwise; a domain proven whiletenantruns waits for it with no job and no identity, and a removal of such a domain writes itsremove_domain(which then finds nothing to delete). If the workspace already has a provider tenant recorded (one the operator created), the runner uses it and writes notenantjob. Iftenantfails, every domain of the workspace inprovisioninggoes tofailedwith its reason (none has an identity yet), with one operator alert. - One runner per allowed Region, the only caller of
ProviderProvisioningfor that Region: on Cloudflare aProvisioningRunnerDurable Object named by the Region, woken by an alarm and backstopped by the cron sweeper (ADR 0011). The container target gets a runner later, with the same contract (theschedulerservice under a Postgres advisory lock per Region). Each tick claims the next due job with a lease, makes at most one provider call, records the outcome and the follow-up job in one batch, and arms the next tick no sooner thanPROVISIONING_INTERVAL_MS(default 1000) after the call. The time of the last call is stored durably, so a wake from the cron backstop or a restart cannot shorten the interval. A runner starts no call while one is in flight, even past its lease. Requests never call the provider; they write a job and wake the runner. Jobs that make no provider call (check_ownership, the DNS part ofcheck_verification,replicate) run outside the pacing. - The job table is the single owner of retries. The runner’s decision per outcome:
ok, oralready_existsandnot_foundas the table above resolves them: the step is done; the next step’s job is written.throttled: the same job again after backoff, and the runner doubles its interval until a call succeeds.unavailable: retried with backoff for 24 hours after the job’s first attempt; then the job fails.rejected: the job fails.unknown, or a lease that expired: the same call again after backoff (forcreateDomain, withlast_unknown_create_atrecorded on the domain). This is safe where a repeatedSendEmailis not (ADR 0012): every management call is keyed by name, so a create that took effect answersalready_existsthe second time, an association is the same state written twice, and a delete answersnot_found. No repeat creates a second resource, and an unknown outcome is settled by the next answer, not by waiting for an event.- After 10 attempts that were not
throttledorunavailable, the job fails. - A check that finds nothing yet (no TXT, SES
PENDINGorTEMPORARY_FAILURE) is not an attempt: it is a normal result that schedules the next check, and only the deadlines below end the checking.
- A failed domain job other than
remove_domainandcheck_integritysends its domain on: tofailedifidentity_created_atis unset and nocreateDomainoutcome is unknown, otherwise toremovingwith the reasonfailed, so the identity is deleted before the claim is released. It raises an operator alert, and the domain showsfailureReason. remove_domainnever fails. It keeps the claim and retries with backoff capped at one hour for as long as it takes, raising an operator alert once a day while it does; the operator repairs the cause. Releasing the claim without a confirmed delete could let a late delete remove another workspace’s new identity of the same name.- Adoption after
already_existsoncreateDomain: the identity is adopted only if its tags carry this installation’s environment and this workspace’s ID, andlistTenantsOfshows no tenant other than the workspace’s. Otherwise the domain goes tofailedwithdomain_in_useand an operator alert, and nothing at SES is changed. A permission refusal of thisgetDomainis read the same way, as an identity outside the installation’s reach (another installation sharing the AWS account, or an administrator’s), not asunavailable; the alert carries the reason so a credential error can still be told apart. remove_domain, for a domain that leftverified, waits for the workspace’sreplicated_sender_versionto reach the version that removed the domain (so the project stores skip its messages before SES would refuse them), at most one hour, then one delivery lease longer (thesendinglease of ADR 0011), so a message claimed just before the replica changed can finish. If the bound passes without full replication it proceeds with an operator alert; a store that still holds the domain then gets SES’s refusal for an unassociated tenant (unavailable,provider_auth, alerted; ADR 0034) until replication reaches it. A domain that was never verified was never in a replica and is removed at once. Then it disassociates and deletes. Anot_foundfrom the delete ends it, except within 15 minutes afterlast_unknown_create_at, when the job looks again later, so a create that SES applies late is not left without a claim.- Alerts: the runner writes each alert occurrence to a control-plane outbox,
provisioning_alerts, in the batch that records the outcome, and raises it throughOperatorAlertsafter the commit. Occurrences use ADR 0038’s incident key unchanged,{ provider, condition, providerReason, windowStart }, with conditions of the formprovisioning:<job kind>:<outcome>(for exampleprovisioning:create_domain:failed) and the workspace and domain IDs in the occurrence, never the domain name. This is ADR 0038’s mechanism with the control plane in place of a project store.
Tenant naming
Section titled “Tenant naming”- The provider tenant is
<TENANT_PREFIX>-<workspace id>.TENANT_PREFIXis configuration (on the managed service, the environment name); unset, the tenant is the workspace ID alone. - The name must fit SES’s rule: 1 to 64 letters, digits,
-, and_(confirmed above, so ADR 0035’s assumption holds).TENANT_PREFIXis limited to 1 to 20 letters, digits, and-, so a workspace ID may be up to 43 characters of letters, digits,_, and-. Workspace creation (ADR 0041) generates IDs that fit. A workspace ID that does not fit (an operator-chosen ID) makes thetenantjobrejectedbefore any call, with an operator alert; the operator records a tenant through the operator endpoint. - Tags on the tenant and on each identity: the environment tag and
customer-lite:workspacewith the workspace ID. Both are IDs, not personal data.
Domain lifecycle
Section titled “Domain lifecycle”sending_domainsis keyed by a new ID (sdm_…) and gainsstatus,ownership_token,managed_by(runtimeoroperator),claimed_at,expires_at,verified_at,last_checked_at,failure_reason,identity_created_at, andlast_unknown_create_at. A partial unique index ondomainover the claim-holding statuses makes a domain owned by at most one workspace per installation (on the managed service, per environment, since each has its own control plane), and lets other rows of the same domain stay beside a claim.
stateDiagram-v2 [*] --> pending_ownership: sendingDomains.add (no claim yet) pending_ownership --> provisioning: our DNS check sees this workspace's TXT, claim taken pending_ownership --> failed: TXT seen but another workspace holds a claim, or a verified claim on a parent or subdomain, reason domain_in_use pending_ownership --> expired: 72 h without the TXT (housekeeping, nothing at SES) provisioning --> failed: a job fails before the identity exists, or the identity is not ours provisioning --> pending_dns: identity created (SES window of 72 h starts), associated pending_dns --> verified: SES reports SUCCESS provisioning --> removing: a job fails after the identity exists, reason failed pending_dns --> removing: the identity is gone, reason failed pending_dns --> removing: no SUCCESS at the check 30 min before 72 h, reason expired verified --> removing: SES reports FAILED or the identity is gone, reason revoked pending_ownership --> [*]: sendingDomains.remove (row deleted) provisioning --> removing: sendingDomains.remove pending_dns --> removing: sendingDomains.remove verified --> removing: sendingDomains.remove removing --> expired: identity deleted, reason expired removing --> failed: identity deleted, reason failed or revoked removing --> [*]: identity deleted, reason removed (row deleted) expired --> [*]: sendingDomains.remove (row deleted) failed --> [*]: sendingDomains.remove (row deleted)- Ownership first.
sendingDomains.addcreates the row inpending_ownershipwith a randomownership_token(fromIdGenerator, per row, so per workspace and claim attempt) and shows one record: TXT_customer-lite.<domain>with the valuecustomer-lite-domain-verification=<token>. Several workspaces may have such rows for one domain; none holds a claim, and no SES call is made. When our DNS check sees the workspace’s own value, the claim is taken in one write (the partial unique index decides a race) and the domain moves toprovisioning. The token is not a secret: it is public DNS data, and only proves that whoever controls the domain’s DNS chose this workspace. The record may be removed after verification. - Claim-holding statuses:
provisioning,pending_dns,verified,removing.pending_ownership,expired, andfailedhold no claim and no SES identity. Re-adding a domain whose row isexpiredorfailed(including one SES revoked) resets that row topending_ownershipwith a new token; it later gets a new identity and new DKIM tokens, so its DNS records must be published again. A workspace has at most one row per domain. Adding a domain the workspace has inremovinganswersconflict. pending_ownershiprows expire after 72 hours as housekeeping. The SES window is separate: it starts whencreateDomainsucceeds, and onegetDomainruns 30 minutes before it ends; withoutSUCCESSthe domain goes toremoving(reasonexpired).removingdisassociates the identity from the tenant and deletes it before the claim is released, so no identity outlives its claim (SES cannot delete an associated identity). A domain leaves the control plane’s sender setup the moment it entersremoving, in the same versioned write, and the project stores follow through that version’sreplicatejob.- SES’s
TEMPORARY_FAILUREchanges nothing: the next check is scheduled as before. - Subdomains. SES lets a verified domain identity send from all its subdomains, and a separately verified subdomain overrides its parent. So SES cannot keep a workspace that holds
example.comfrom sending asnews.example.com; the exact-match check against the replica (ADR 0034) is the guard, and it stays exact. A subdomain is its own sending domain with its own challenge, identity, and claim. A domain may not be claimed while another workspace holds a verified claim on one of its parent domains or subdomains (domain_in_use); unverified claims of other workspaces do not block it. A domain that is a public suffix (ICANN or private section of the Public Suffix List, a dated snapshot in core) is refused; a name below a public suffix is accepted. - Records shown to the user, each with its observed state (
present,missing,mismatch, orunchecked):- the ownership TXT, required while
pending_ownership; - three DKIM CNAMEs
<token>._domainkey.<domain>→<token>.<SigningHostedZone>, required; the target is built from what SES returns, never written in our code, since it varies by Region and identity; - DMARC at
_dmarc.<domain>, recommended when our check finds none: TXTv=DMARC1; p=none;. We never change an existing one, and DMARC does not gate verification.
- the ownership TXT, required while
- Checking:
- Our DNS check (ADR 0024) is a
DnsResolverport over DNS over HTTPS (DOH_URL, a public resolver by default). It costs no SES call and runs at add, onsendingDomains.verify, and on each scheduled check. - While
pending_ownership,check_ownershipruns our DNS check every 15 minutes. - While
pending_dns,check_verificationruns in two internal phases, not shown as statuses: our DNS check every 15 minutes until it sees every DKIM record, thengetDomainat 1, 5, 15, and 30 minutes, then hourly. The check 30 minutes before the SES window ends runsgetDomainwhatever our DNS check saw, so a domain our resolver cannot see is still verified if SES can.sendingDomains.verifymoves the domain’s job forward rather than adding one. - Daily integrity read of each runtime verified domain, by its
check_integrityjob: written when the domain entersverified, rewritten for the next day after each run, and replaced byremove_domain. It is one pacedgetDomainthat checks DKIMSUCCESS, the two tags, and that no MAIL FROM domain is set. OnlyFAILEDornot_foundrevokes. A changed tag or a MAIL FROM it did not set raises an operator alert and changes nothing, and so does a failure of the job itself: an outage never removes a verified domain. Operator rows are not read. This is a deliberate exception to “call SES only when something changes” (ADR 0024), as the cost of the leaked-key risk above; at one call per second it covers up to 86,400 domains a day, less when environments share an account’s limit.
- Our DNS check (ADR 0024) is a
Replication
Section titled “Replication”- Every change of a domain into or out of
verified, and the recording of the workspace’s tenant, is a versioned write of ADR 0035:sender_versionis incremented in the same batch and the workspace’sreplicatejob is written with it. The runner runsreplicatewithout a provider call: it replicates the workspace’s whole setup to each live project of the workspace the control plane lists, as ADR 0035’s replication does, and runs again with backoff for the stores that failed. When every listed store has taken a version, it records the version as the workspace’sreplicated_sender_version. After 5 runs that left a store behind it raises an operator alert, and keeps retrying. Replicas stay ordered by version, so retries and overlaps are safe. - A run writes to at most as many stores as the Worker’s subrequest limit allows and continues in the next run; this is the queue fan-out ADR 0035 left open, done through the job.
- The runner acts on its job’s workspace only:
core/authmints aWorkspaceScopefrom the job row (scopeForProvisioningJob), a source written by core under aWorkspaceScope, as a queue message produced by core is; every read and write of the step goes through that scope, and replication mints each project’s scope as ADR 0035’s does. - The tenant is recorded in
workspaces.provider_tenant(with the Region, ADR 0037) only after it exists and is associated with both configuration sets, so no replica carries a tenant SES would refuse. - The operator endpoints stay, for repair and for what the runner does not manage: the operator’s own Terraform-managed domain, workspaces with a non-conforming ID, a self-hosted installation in
manualmode, a claim to release, and the resync. Rows they write aremanaged_by = operator: the runner never calls the provider for them and never expires them.PUT .../sending-domains/:domainwith{"status": "verified"}records a domain the operator verified;pendingis recorded aspending_dns. APUTon a domain that has a claim-holdingruntimerow in this workspace, or that another workspace holds (decided by the sameisHeldElsewherecheck and unique index as a runtime claim), is refused withconflict, so the endpoint cannot bypass SES verification of a runtime claim or take a claim from another workspace; the operator first deletes that row.DELETEof a runtime domain startsremoving, assendingDomains.removedoes; of an operator row, it removes the row as today.
API actions
Section titled “API actions”Five api actions (ADR 0005, ADR 0019) with target: "workspace" and principal: "any", in the principal and target model of ADR 0041, which is its source; REST under /api/sending-domains, MCP tools sendingDomains_*:
| Action | Who may call it | Annotations | Behaviour |
|---|---|---|---|
sendingDomains.add { domain } |
owner: by session, or by an MCP grant with scope admin ∩ role |
idempotent | Normalizes (ADR 0035), screens, and returns the domain in pending_ownership with its TXT record. It never answers whether another workspace holds the domain: that shows only after the workspace’s own TXT is seen, as failed with failureReason domain_in_use. A domain the workspace has in pending_ownership, provisioning, pending_dns, or verified is returned unchanged and does not count toward the daily limit; one in expired or failed starts again (above) |
sendingDomains.list |
member (session, or grant with read), or a project key with read |
read-only | The workspace’s domains with status, without records |
sendingDomains.get { domainId } |
as list |
read-only | One domain with its records and their observed states, expiresAt, verifiedAt, lastCheckedAt, failureReason |
sendingDomains.verify { domainId } |
as add |
idempotent | Runs our DNS check now and moves the domain’s job forward; returns the domain. A call within 60 seconds of the last returns the domain as it is, without a new check |
sendingDomains.remove { domainId } |
as add |
destructive, idempotent | Starts removing and returns it: the domain is out of the sender setup at once, and out of the project stores once the replicate job has run. A pending_ownership, expired, or failed row is deleted. Removing a domain already removing returns it unchanged, and its reason becomes removed, so the row is deleted at the end; a call after the row is gone is not_found |
core/auth, the only minter of scopes, mints theWorkspaceScopeas ADR 0041 decides. A project API key and an MCP grant act on their own project’s workspace: the key’s API scopes, or the grant’s scopes intersected with its user’s current role there. A dashboard session names the workspace in theCustomer-Lite-Workspaceheader, and the user’s current membership and role there apply. A workspace the session’s user is not a member of answersnot_found; too low a role or scope,forbidden. A project key has no role, so it reaches onlylistandget. ADR 0026’s owner row already lists sending domains.- So the user’s own assistant, through an MCP grant with
admin, can add, verify, and remove domains of its project’s workspace when its user is an owner. ADR 0035 kept provisioning out of MCP so that a project’s AI could not mark a domain verified; that reason no longer holds: no action setsverified, and only SES does, after the domain’s DNS proves control by this workspace. - Errors:
validation_error(400, not a domain, or a session without the workspace header);forbidden(403);not_found(404, an unknown domain ID or one of another workspace, never told apart);conflict(409,addwhile the domain isremoving, orverifyon a domain that isremoving,expired, orfailed);sending_domain_refused(422, a public suffix or refused by screening; the screening’s reason is not disclosed);domain_limit_reached(422);rate_limited(429, more than 10 adds per workspace per day); and a configuration error when provisioning is not available. - Responses carry no ARN, account ID, credential, or tag. The ownership token, DKIM tokens, and record values are public DNS data.
Amendment (2026-11-19, ADR 0041, Accepted): there is no owner role. sendingDomains.add, sendingDomains.verify, and sendingDomains.remove need role admin: an admin’s session, or an MCP grant with scope admin whose user is an admin. Where this ADR says owner, read admin; ADR 0041’s admin row lists sending domains. list and get are unchanged.
Abuse controls
Section titled “Abuse controls”- Core enforces, with values in configuration: at most
SENDING_DOMAINS_PER_WORKSPACE(default 10) domains in claim-holding statuses, at most 3 of them not yet verified, and 10 adds per workspace per day; and the public-suffix refusal.pending_ownershiprows hold no claim and are not counted in the 3 not yet verified; they are bounded by the 10 adds a day and their 72-hour expiry. - An
AbuseScreeningextension point incore/extensions/, with a no-op default that allows, getsscreenSendingDomain({ workspaceId, domain })insendingDomains.addand answersalloworrefusewith a reason code. It runs in process, without a network call. A refusal is logged by workspace ID and reason code only. - If it throws, the add is refused with
sending_domain_refusedand an operator alert. Extensions otherwise never block core behaviour; this one does, and only for this call on this action: a lookalike domain passes every other check, since its owner can publish its DNS, and the user can retry. The self-hosted default never throws. - The managed service’s implementation and its lists and rules belong to that implementation (see the decision on lookalikes below).
- The ownership challenge means a claim is held only by a workspace that controls the domain’s DNS; an unproven row holds nothing and is cleaned up after 72 hours.
Logs and isolation
Section titled “Logs and isolation”- Logs and alert occurrences identify a domain by
sdm_and a workspace bywsp_ID, never by the domain name, which can be personal data. Provider messages and ARNs never leave the adapter (ADR 0032). - Every read and write of a domain row goes through a
WorkspaceScope(fromcore/authfor an action, from the job for the runner, and the named workspace for an operator endpoint), within it by domain ID, or by normalized name foradd, the DNS checks, and the operator endpoints. - The one read across workspaces is the claim check, an exception added to the Tenant model:
SendingDomainClaims.isHeldElsewhere(scope, domain)answers only yes or no to “does another workspace hold a claim on this domain, or a verified claim on a parent or subdomain of it”, and the claim itself is taken by the partial unique index. The runner’s claim step and the operator endpoint’sPUTuse the same two. Neither returns another workspace’s ID or row, and nothing else in the workspace’s scope can name another workspace’s domain.
Decided with this ADR
Section titled “Decided with this ADR”- Fail-closed abuse screening applies only to
sendingDomains.add, as above. - Lookalikes: a small curated list held by the managed service, matched by confusable skeleton; comparison across workspaces is deferred.
- DMARC does not gate verification. When marketing sends ship, they are gated on the sending domain having a DMARC record (
p=noneis enough); transactional sends only warn. - Tenant deletion is deferred to workspace deletion, which queues
remove_domainfor every domain of the workspace and leaves the idle tenant.
Consequences
Section titled “Consequences”- Tests required, each failing first (ADR 0006), on the isolation fixture:
- isolation: a workspace cannot list, get, verify, or remove another’s domain (
not_found); cannot claim a domain another workspace holds, or one with a verified parent or subdomain in another workspace (it endsfailedwithdomain_in_use, andaddnever tells); cannot claim with another workspace’s TXT value, or without one; a project key canlistandgetbut notadd,verify, orremove(forbidden), and a key or grant reaches only its own project’s workspace; the claim check returns nothing but yes or no; and the runner never associates an identity with another workspace’s tenant, including on adoption; - subdomains: a workspace with a verified
example.comcannot send fromnews.example.comthat another workspace verified, or that nobody verified: the message isskipped:sender_not_verified; - idempotency: repeated
add,verify, andremove; a job run twice;already_existsandnot_foundon each operation as the table says; - the unknown outcome of
createDomain: a timeout after SES created the identity leads toalready_existson the repeat and adoption, so one identity; a timeout before leads to one identity too; an identity tagged with another workspace or installation, or one the credential cannot read, is never adopted and never changed; - no orphaned identity: a job that fails after
createDomainsucceeded deletes the identity before the claim is released; a removal while a create’s outcome is unknown leaves no identity behind; - races: a removal while
createDomainis in flight leaves no domain outsideremoving; areplicatethat runs while the version rises runs again; a failedtenantjob fails the domains waiting on it; aremove_domainthat keeps failing keeps its claim and alerts; - pacing: with several workspaces adding at once, the fake records no two calls for one Region closer than the interval,
throttledwidens it, and a wake from the cron backstop does not shorten it; - removal order: a domain that left
verifiedis disassociated only after every store has its new replica plus one delivery lease, or after the bound with an alert; - the operator endpoint refuses to mark a runtime domain verified or to take another workspace’s claim;
- no domain name, address, ARN, account ID, or credential in any log line, alert, or API or MCP response.
- isolation: a workspace cannot list, get, verify, or remove another’s domain (
- Implementation, one pull request each, in order:
- Control plane: the migration (ID as the key, statuses, claim index, job table, alert outbox,
workspaces.replicated_sender_version; existingpendingrows becomepending_dnsand every existing rowmanaged_by = operator) and the operator endpoints on the new vocabulary, with no change of behaviour. - The
DnsResolverport and its DoH adapter. - The
ProviderProvisioningport, the fake, and the contract suite. - The runner logic in core: jobs, pacing, outcomes, the ownership challenge, the lifecycle, the claim check, replication as a job, and alerts, tested with the fake.
adapters/email-sesprovisioning: the calls, the outcome classification, and recorded fixtures.apps/workers: theProvisioningRunnerDurable Object, its alarm and cron backstop, and the configuration. The managed infrastructure for the provisioning user and its alarms is a parallel, private pull request.- The public-suffix check and limits in core, and the
AbuseScreeningextension point with its no-op default, in its own pull request; then its implementation for the managed service. - The
sendingDomains.*actions, once ADR 0041’s principals andworkspacetarget are built; then the dashboard screens. - The gate: a staging run against real SES (challenge, create, verify, expire, remove, a lost response simulated by a short timeout) records the fixtures; the manual per-workspace runbook is kept for repair only.
- Control plane: the migration (ID as the key, statuses, claim index, job table, alert outbox,
- Open:
- Moving a workspace to another Region is still the operator’s procedure (ADR 0037); the runner provisions only in the workspace’s recorded Region.
- Custom MAIL FROM, with the marketing-send DMARC gate.
- Whether a domain refused by screening can be appealed in the product.
- Bring-your-own SES credentials would also need their own provisioning credential.