Skip to content

0048. A/B experiments on send steps: two variants and an optional holdout, sticky assignment, a matured fixed horizon, and a winner chosen by a person or their AI

  • Status: Proposed. A/B testing in the first release, a holdout in the first release, a winner chosen by a person or their AI with no automatic promotion, and fixed-horizon frequentist statistics are the maintainer’s decisions of 2026-11-19, Accepted and not reopened.
  • Date: 2026-11-19
  • Amends: ADR 0015 (A/B testing enters the first release), ADR 0033 and ADR 0044 (a conclusion that ends a holdout needs an approval of a new kind, conclusion, where approval is required, and a conclusion is a live change of kind concluded)

The maintainer decided on 2026-11-19 that A/B testing is in the first release, with a holdout, and that a person or their AI chooses the winner.

Facts and constraints:

  • A send step holds from, subject, and body; a message is unique per (run, step) (ADR 0011). Versions are immutable and approval binds a version’s definition hash (ADR 0033); approval is optional, off by default (ADR 0044).
  • Message reports attribute conversions by last touch (ADR 0047), which cannot compare a holdout that got no message with variants that did.
  • Indie developers have small lists, and an AI that checks hourly and stops at the first good number finds false winners.
  • Core is deterministic; time and randomness come through ports, and hashing is available in core (ADR 0018).
{
"id": "send-welcome",
"type": "send_email",
"from": "hello@example.com",
"experiment": {
"id": "welcome-subject-1",
"metric": "conversion",
"minSamplePerArm": 1000,
"variants": [
{ "id": "a", "weight": 45, "subject": "Welcome!", "body": "…" },
{ "id": "b", "weight": 45, "subject": "Your account is ready", "body": "…" },
{ "id": "holdout", "weight": 10, "holdout": true }
]
}
}
  • One or two variants, each with its own subject and body, and at most one holdout. At least two arms. from and tracking belong to the step. Integer weights sum to 100. Three or four variants are deferred.
  • Holdout: its customers get no message and the run continues. A holdout needs metric: "conversion", so the workflow must have a goal.
  • experiment.id is unique within the workflow.
  • Metric: conversion (the workflow’s goal, ADR 0047), human_clicks, or human_opens, which is allowed only when declared and always carries the caveat opens_unreliable (ADR 0046). Default: conversion when the workflow has a goal, otherwise human_clicks.
  • Minimum sample per arm: minSamplePerArm, or EXPERIMENT_MIN_SAMPLE_PER_ARM (configuration) when omitted. It is the size an equal split would need. With unequal weights, the smaller arm of each comparison must reach n × (1 + 1/k) / 2 and the larger k times that, where k is the ratio of the larger arm’s weight to the smaller’s, so the comparison keeps the variance of an equal split of n per arm. The response shows the required size per arm.
  • experiments(workflow_id, experiment_id, step_id, arms_hash, goal_hash, goal_condition, conversion_window, metric, min_sample, weights, status, verdict, winner, started_at, verdict_at, concluded_at), keyed by (workflow_id, experiment_id).
  • Frozen when it first goes live: in the transaction that first makes a version containing the experiment live, the record stores arms_hash (the arms’ IDs, weights, and content), the metric, the minimum sample, and what the metric measures: goal_hash (the conditionHash of the workflow’s goal, ADR 0045), a copy of the goal condition, and the conversionWindow. Saving a later version that changes any of them under the same experiment.id is a validation error naming the experiment and the changed part; a new experiment.id starts a new experiment. Assignment outcomes (converted_at, the window) are evaluated with the frozen goal and window, never with a later version’s.
  • Statuses: collecting, then one verdict, no_difference or leader, then concluded (a winner chosen) or ended (the workflow was stopped, or a live version no longer contains the experiment, before a conclusion). A concluded or ended experiment is never reopened.
  • Sticky hash: SHA-256 over the UTF-8 bytes of project_id, workflow_id, experiment_id, and customer_id joined by U+001F; the first 8 bytes as an unsigned big-endian integer, modulo 10,000, mapped onto the cumulative weights in the listed order. customer_id is the internal ID. The same customer gets the same arm every time, across re-entry and versions.
  • At the send step, for every arm, holdout included, the engine runs the same eligibility check a send would (unsubscribe, suppression, goal, sender, stop, pause; a goal that is unknown skips the customer with skipped:condition_unknown and they are not assigned, ADR 0045):
    • An eligible customer is assigned. A sent arm creates its message with variant_id; a holdout writes the message as skipped:holdout (not billed) and the run continues.
    • An ineligible customer is skipped with that reason, as without an experiment, and is not assigned.
  • experiment_assignments(workflow_id, experiment_id, customer_id, arm, assigned_at, window_ends_at, message_id, converted_at, first_human_click_at, first_human_open_at, post_conclusion), one row per customer and experiment, written in the transaction of the first assignment. The customer is the unit of analysis: a re-entry reuses the arm and adds no row. Rows are kept until the experiment is concluded or ended plus EXPERIMENT_ASSIGNMENT_RETENTION (configuration, proposed 30 days), then archived with the messages (ADR 0047 budgets them).
  • Outcomes are written to the assignment row with the same null-to-value rule as message facts: converted_at when the goal is met within the conversion window counted from assigned_at for every arm. The goal evaluation on an event (ADR 0047) also checks the customer’s assignment rows whose window is still open, through an index on (customer_id, window_ends_at) that is part of this read model, so a run that exits early (a later condition, a stop) does not lose its arm’s conversion. Click and open times come from the assignment’s message.
  • Assignments made after a conclusion are flagged post_conclusion and excluded from results.
  • Experiment results are read from this model only. Message reports use last touch; experiments use intent to treat. Reports label which one they show.
  • Maturity. An assignment counts only once its outcome has matured: for conversion, assigned_at plus the conversion window has passed; for clicks and opens, assigned_at plus EXPERIMENT_ENGAGEMENT_MATURITY (configuration, proposed P3D).
  • Denominators (intent to treat): matured assignments per arm. For human_clicks and human_opens, only assignments whose message was tracked; holdouts have no such metric.
  • The comparison family, at an overall significance level of 0.05 with Bonferroni across the family:
    • two variants, no holdout: B against A (one comparison);
    • one variant and a holdout: the variant against the holdout (one comparison);
    • two variants and a holdout: B against A, A against the holdout, and B against the holdout (three comparisons, each at 0.05 / 3).
  • Each comparison: rates with 95% Wilson score intervals per arm, the difference with a Newcombe hybrid score interval (at the family’s corrected level), and a two-sided two-proportion z-test.
  • One evaluation. The verdict is computed once, at the first tick after every arm reaches its required matured sample, over exactly the first N matured assignments of each arm in assigned_at order (N being that arm’s required size), so the sample is fixed by rule, not by when the tick ran. It is recorded with its numbers and verdict_at: leader names the arm that beats its comparator significantly (with the difference and interval), otherwise no_difference. Data that matures later is shown as “post-horizon” beside the verdict and never produces a new verdict.
  • Warnings: below_min_sample (with matured progress per arm); sample_ratio_mismatch when a chi-square test of assignment counts against the weights gives p < 0.001, meaning the split or the data is broken; opens_unreliable; low_event_count when an arm has fewer than 30 events of the metric at the verdict.
  • Before the verdict, results show progress and intervals but no significance. The server makes no other claim: no probability to be best, no projected uplift.
  • Classification bias in opens and clicks is shared by every arm (ADR 0046).
  • experiments.conclude { workflowId, experimentId, winner } (principal: "any", scope send, role admin, annotated destructive). From then on, new assignments at that step get the winner’s content; a holdout ends, so its share is sent the winner. Messages already created keep their arm. A customer who reaches the step again after the conclusion (re-entry, or a later run), including a former holdout or losing-arm customer, gets the winner’s content: the conclusion overrides the sticky arm, because the point of concluding is that everyone now gets the winner. Two things are flagged post_conclusion: the assignment row of a customer whose first assignment comes after the conclusion, and every message created at the step after the conclusion (a message dimension of ADR 0047, so per-variant reports can separate them). Results exclude both. Outcomes recorded after concluded_at never count toward the experiment: an assignment whose maturity ends after the conclusion is not matured for the verdict, so a holdout customer who re-enters after an early conclusion and is then mailed the winner does not change the holdout’s result.
  • It changes no definition. It is runtime state on the workflow, keyed by (workflow_id, experiment_id), and is written as a live change of kind concluded (ADR 0044) with its principal.
  • Without a holdout, it only narrows sending to content already approved in the live version: no approval and no confirmation.
  • Ending a holdout starts mail to customers the live version deliberately left out. It needs the confirmation of ADR 0050 in every workspace, and, where approval is required, a send approval of kind conclusion, effective when approved. A conclusion approval binds the workflow, the activation epoch, the live version and its definition hash, the experiment ID, its arms_hash, and the winner; its decision is refused with conflict if any of those changed. It follows ADR 0044’s rule of at most one pending approval per workflow, whatever its kind: requesting it cancels a pending activation or rollback approval, and the reverse.
  • Rollback does not revert a conclusion. A conclusion belongs to the experiment, whose arms are frozen, so it applies to every live version that contains the experiment, including one a rollback restores. A restored version without the experiment is unaffected.
  • The winner can be any arm whatever the status; the conclusion records the status at that time.
  • An ended experiment in a live version. If a version containing an ended experiment becomes live again (a rollback, or a publication that keeps the step), the step sends the first listed variant to every eligible customer: no holdout, no new assignments, nothing recorded in the experiment. The preview of ADR 0050 says so. An ended experiment is never reopened; a new test needs a new experiment.id.

api actions with full parity (ADR 0005), target: "project", principal: "any" (ADR 0041):

Action Scope, role Annotations
experiments.list, experiments.get (arms, required and matured sample, intervals, verdict, post-horizon data, warnings) read, member read-only
experiments.conclude send, admin destructive

Experiments are defined only in workflow versions (workflows.upsert, and broadcasts, ADR 0049).

  • All arms are part of the version, so the approval page and the confirmation preview show them all.
  • A sent arm’s message is an email under the usual rules (ADR 0012); a holdout message is a pre-send skip, not billed. No plan feature or price (ADR 0014).
  • An indie developer’s AI can test two subjects or measure an email against a holdout, read one honest verdict, and conclude.
  • Small lists stay in collecting for a long time; that is the honest answer, and the progress says how far each arm is.
  • The maturity rule delays the verdict by the conversion window; it is what makes the horizon fixed.
  • Open: three or more variants; automatic promotion; experiments on delays and paths; sequential tests; variants of from.