Is your brand ready for a marketing holdout test?

Attribution asks which observed interactions receive credit. Incrementality asks how outcomes differ because the treatment occurred. A customer who would have purchased anyway can appear in an advertising attribution report. A holdout design aims to estimate the difference between comparable groups with and without the defined treatment.

Ronny Kohavi and coauthors’ work on experimentation at Microsoft explains the value of randomization and proper design for causal evaluation. The transferable lesson is disciplined comparison, not that every small ecommerce brand can immediately reproduce a large platform’s experimentation infrastructure.

The treatment might be eligibility for a campaign, a specific customer message, or a regional media change. State what remains available to the control group. A test of one remarketing campaign does not estimate the value of all marketing. A platform-controlled holdout and a geography-based test also have different assumptions and operating requirements.

Write the assignment unit: person, household, region, or another defensible unit. Consider spillover. People can share messages, travel between regions, or see other campaigns. These effects can blur the difference between groups and should be considered before the test rather than invoked afterward to explain an inconvenient result.

A small number of orders can leave a test unable to distinguish a meaningful effect from random variation. Ask a qualified analyst to assess baseline volume, variability, minimum detectable effect, assignment ratio, and the planned analysis. There is no universal number of days or orders that makes every design reliable.

The review by Nicholas Larsen, Jonathan Stallrich, Srijan Sengupta, Alex Deng, Ron Kohavi, and Nathaniel Stevens describes statistical challenges in online experiments. It reinforces the need for an explicit analysis plan. A calculator’s sample-size output is not a substitute for checking whether the proposed design fits the business and data.

Purchases, net revenue, or contribution may be appropriate primary outcomes depending on the decision and data quality. Clicks can explain behavior but may be too far from the budget question. Include refunds or other delayed outcomes when they materially change the economics, and state how long they can be observed.

Guardrails might include customer complaints, delivery capacity, or a minimum service standard. A marketing experiment should not deliberately create a broken purchase path for the control group. Withholding a promotional treatment is different from withholding essential product or safety information.

Imagine a brand considering whether a remarketing program adds enough purchases to justify its cost. It can use an appropriate platform experiment that assigns eligible people under documented rules. The brand chooses a primary purchase outcome, records the spending difference, and asks the analyst to assess whether the expected sample can detect an effect large enough to matter.

If the feasibility review suggests the test would only detect an unrealistically large effect, the team should not proceed merely to obtain an “incrementality” badge. It may need a longer feasible observation, a different design, or another decision method. Those choices require analysis and commercial judgment, not a generic promise of certainty.

If the test runs, the report shows the estimated incremental outcome and uncertainty rather than only the treatment group’s attributed ROAS. A confidence interval spanning both a meaningful gain and no effect can be inconclusive. It should not be rewritten as proof that the campaign works or does not work.

Log campaign changes, stockouts, promotions, tracking changes, and assignment issues. Some changes may be harmless; others can undermine the comparison. Agree in advance who can modify the treatment and what triggers a pause. A team that optimizes the test aggressively mid-run may no longer be evaluating the original question.

Do not stop at the first favorable daily result unless the analysis explicitly supports a sequential stopping rule. Repeatedly checking and ending only when the result looks positive can distort interpretation. The analyst should explain the chosen approach in language the business owner can understand.

A useful readout states the estimated effect, uncertainty, cost, tested population, and period. It explains whether the result supports the planned decision and what would be needed to generalize beyond the test. A result from one market, offer, or season may not apply unchanged to another.

Also report whether the design was executed as planned. A statistically polished estimate based on broken assignment is not trustworthy. Keep the experimental record and the commercial recommendation together so the owner can inspect the chain of reasoning.

A test can be executed correctly and still leave uncertainty too wide for the planned decision. Agree in advance whether the response would be to keep the current policy, reduce the commitment, gather more data under a valid design, or test a more specific question. This prevents pressure to declare a winner simply because the experiment consumed time and money.

Also consider the cost of being wrong in each direction. Continuing an ineffective campaign and stopping a useful one may have different consequences for the brand. The decision rule should reflect those consequences and the owner’s constraints, with an analyst explaining the statistical evidence. Avoid turning a single significance threshold into an automatic business command. The estimate, uncertainty, execution quality, and commercial stakes belong in the same discussion.

Dahna can help define the business question, coordinate data readiness, and scope an experiment with an appropriately qualified analyst or platform partner. Bring the decision, baseline order volume, campaign setup, and operational constraints. The feasibility assessment should come before a commitment to run the test.

If the brand is not ready for a credible experiment, use transparent observational reporting and a bounded pilot while improving the foundation. Calling a weak comparison causal does not make it more useful.

Use these guides when the next question goes beyond the scope of this article.

Download the completed tables as a CSV, then replace the example inputs and add your evidence, owner and review date. Illustrative figures are teaching examples, not default targets.