Start with the question a holdout can answer
Choose a specific decision: does the weekly newsletter generate additional purchases, does a win-back series change reactivation, or does an entire email program add value beyond natural repeat demand? Do not combine unrelated questions into one test without defining the scope. A campaign-level holdout and a global program holdout answer different questions.
Define the eligible audience, business outcome and observation window before assignment. For ecommerce, this may be completed orders or contribution margin; for lead generation, qualified inquiries or booked consultations. The objective is to measure outcomes for all eligible people, not only those who open or click, because those behaviours happen after treatment and create selection bias.
- Specify treatment, control and the exact messages withheld.
- Choose a business outcome independent of email clicks.
- Define the follow-up window and conversion lag in advance.
Randomize at the right level
Assign eligible people randomly to treatment and control before sending. Keep a person in the same group throughout the test so multiple campaigns do not contaminate the comparison. If household or company accounts share purchasing decisions, consider whether assignment needs to happen at that higher level. Document exclusions and deduplicate identities before randomization.
Check that the groups are similar on pre-test purchase history, recency, geography and other important variables. Randomization reduces systematic bias, but small samples can still differ by chance. Do not manually move high-value customers into treatment after assignment. If an operational exception is unavoidable, record it and explain its effect on interpretation.
- Persist group membership for the full test.
- Check baseline balance without selectively reshuffling groups.
- Keep group assignment independent of future engagement.
Protect essential and consent-based communications
A marketing holdout should not accidentally suppress order confirmations, security notices, password resets or other necessary transactional messages. Define the messages exempt from the test and document any overrides. Also preserve unsubscribe, consent and preference-centre rules for both groups; being in a test never overrides a person's communication choices.
Platform implementations differ. Klaviyo's global holdout feature, for example, has specific account eligibility requirements and can cover multiple messaging channels, not just email. Its documentation distinguishes global holdouts from A/B tests and explains transactional exemptions and overrides. Verify current platform capabilities rather than assuming every small business can enable the same native feature.
- Inventory exempt transactional and safety messages.
- Confirm consent and suppression rules remain active.
- Check platform eligibility and whether holdout spans other channels.
Calculate lift using all assigned people
Suppose 4,500 treatment customers generate 270 purchases, a 6% purchase rate, while 500 control customers generate 25 purchases, a 5% rate. The absolute difference is one percentage point; relative lift is 20%. A simple estimate of incremental purchases in the treatment group is 4,500 times 0.01, or 45. This is an illustration, not a statistically validated result.
Translate the incremental outcomes into contribution value after variable costs, then compare against sending and production costs. Do not calculate incremental revenue by subtracting attributed email revenue from total store revenue. Use comparable denominators, account for unequal group sizes and report uncertainty. A small apparent lift with few orders may be indistinguishable from random variation.
- Report absolute difference and relative lift separately.
- Estimate incremental outcomes using treatment-group size.
- Include confidence or uncertainty, not just a single uplift percentage.
Avoid common sources of contamination
If control members receive the same offer through another email automation, the treatment contrast weakens. If the campaign is promoted through paid social, both groups may see it; that may be acceptable for an email-specific question but must be documented. Coupon codes shared publicly can also blur the distinction between who received a message and who acted on it.
Do not exclude non-openers after the test or compare clickers with non-clickers as though those groups were randomized. Avoid stopping as soon as the result looks favourable. Set a reasonable sample and duration based on baseline conversion rates, expected effect and business constraints, and consult a qualified analyst when the decision carries substantial financial risk.
- Audit overlapping campaigns and flows.
- Keep the planned analysis population intact.
- Do not repeatedly peek and stop on the first positive result.
Interpret results in the context of customer value
A holdout may reveal that frequent promotional emails drive clicks but little incremental purchasing, or that a useful onboarding series improves retention without immediate revenue. Segment exploratory results by customer lifecycle only when sample sizes support it. Do not declare every small subgroup result a separate discovery; multiple comparisons increase false-positive risk.
Consider secondary outcomes such as unsubscribes, complaints, repeat orders and long-term customer value. A short-term sales lift can be offset by excessive messaging or discount dependence. If the primary result is inconclusive, report that clearly and decide whether a longer test or a more focused hypothesis is warranted.
- Report both intended benefits and potential harms.
- Keep exploratory segment findings labelled as exploratory.
- Avoid confusing attributed conversions with causal incrementality.
Turn the experiment into an operating decision
Document the randomization method, eligible population, exclusions, messages, time window, metric definitions and final analysis. Share the result with marketing, finance and customer support. Decide whether to retain the email program, adjust frequency, refine targeting or test a different value proposition, rather than treating the holdout as a one-time reporting exercise.
Repeat only when the business question changes materially or enough time has passed to justify a new evaluation. A small organization may not have enough volume for a reliable holdout; in that case, acknowledge the limitation and use directional evidence without claiming causal lift. Measurement integrity is more valuable than a precise-looking but unsupported percentage.
- Archive assignment rules and the analysis method.
- Translate findings into a documented messaging decision.
- Revisit when strategy, audience or economics change.