How to plan an ecommerce landing-page A/B test

Start with a customer uncertainty the current page may not resolve. For a device, it might be whether the kit includes the accessory shown in the ad. For a moisturizer, it might be the difference between two textures. Use support questions, task research or observed behavior to explain why the proposed change could help. “Try a new hero section” is an implementation idea, not yet a testable reason.

Write the hypothesis in plain language: showing the complete kit near the purchase decision may reduce uncertainty about what arrives and improve contribution from eligible visitors. Specify the proposed change and the expected customer mechanism. Keep unsupported product claims out of both versions. A test cannot make misleading copy acceptable simply because it performs well.

Decide what action follows each possible result. A clear positive result may support rollout; a negative result may preserve the existing page; an inconclusive result may justify further research or no change. Agreeing those choices in advance prevents the team from treating every outcome as a reason to keep redesigning.

Ronny Kohavi, Thomas Crook, Roger Longbotham and their Microsoft colleagues describe controlled experimentation as a way to evaluate ideas through customer behavior. Their published work also discusses the practical and organizational challenges of trustworthy experiments. The relevant lesson is to treat a persuasive idea as a hypothesis that still needs a sound comparison.

For a brand, that means separating the design team's confidence from the evidence produced by the test. A page can look clearer and still have an uncertain commercial effect. Conversely, a modest change can matter when it resolves a specific obstacle. The experiment's job is to estimate the effect under defined conditions, not to validate the most senior person's preference.

The example from Microsoft concerns large-scale online experimentation, not an ecommerce conversion benchmark. A smaller store needs a design suited to its own traffic and decision size. It should not copy a large company's testing cadence or assume that a short run produces enough evidence merely because the software displays a winner.

Specify who can enter the experiment: market, device scope, landing route and any excluded sessions. Choose an assignment unit that the testing system can maintain consistently. A visitor who repeatedly changes versions may receive a mixed experience, while cross-device behavior can complicate interpretation. Ask the implementation and analytics owners to document those limits rather than hiding them behind a percentage split.

Choose a primary measure that matches the decision. Purchase rate can answer some questions, but a page that changes offer selection may need contribution per eligible visitor. Define the purchase source, refund treatment and observation window. Use secondary measures to explain the result, not as an unlimited menu of alternative success claims.

Ask an analyst to estimate required traffic using the store's baseline, the smallest meaningful effect and the planned statistical method. There is no universal rule that a test needs a fixed number of days or orders. If the available traffic cannot answer the question within a practical window, use task research or a larger, better-grounded change instead of running an underpowered test indefinitely.

Preview both versions on the devices that matter. Verify that the correct product and price appear, links work and the purchase path remains intact. Test slow connections and returning visits. A delayed visual swap can create flicker or expose both versions; an implementation defect can overwhelm the intended content change.

Confirm that assignment, exposure and purchase records can be reconciled. A page request is not always a visible exposure, and duplicate purchase events can distort the result. Run controlled test orders using the team's established process and verify that they are excluded from production analysis where appropriate. Do not transmit personal checkout information merely to debug an experiment.

Check that the observed allocation is consistent with the planned design. A substantial unexpected imbalance may indicate targeting, caching or logging problems. Ask the analyst to investigate before interpreting a lift. Keep a deployment and incident log so outages, stock changes and major campaign shifts can be considered in the readout.

Imagine a fictional test with 10,000 eligible visitors in each group. The control records 300 orders and the treatment 324. The observed purchase rates are 3.00% and 3.24%, a difference of 0.24 percentage points or 8% relative. Those arithmetic descriptions are not, by themselves, evidence that the treatment is a reliable winner.

The analyst needs to apply the planned method, account for the assignment and observation design, and report uncertainty. Contribution, returns and relevant guardrails may change the business interpretation. Do not repeatedly stop and restart based on a favorable daily chart unless the chosen method explicitly supports that decision process.

Also ask where the result applies. A test during a major sale, with one device type or one acquisition source, may not generalize to the whole year. Record those conditions in the decision memo. A rollout can be staged and monitored without pretending that one experiment established a permanent universal truth about product-page design.

Request the hypothesis, eligibility rules, assignment approach, sample-size rationale, implementation checks and final decision memo. The memo should say what changed, what was observed, how uncertain the estimate is and what the team recommends doing next. It should also identify defects or external changes that limit interpretation.

Keep the losing or inconclusive result useful. It may reveal that the proposed explanation was not the main obstacle, that the audience was too narrow or that the test could not detect a commercially useful effect. Record that learning so the next project does not repeat the same idea under a new headline. A testing program should improve decisions, not just accumulate screenshots of winners.

Dahna can discuss a page experiment that connects customer evidence, content, website implementation and measurement. Bring the page, the uncertainty you want to resolve and the traffic available for testing. The first step is to determine whether a controlled experiment can answer that question credibly and what needs to be repaired before it begins.

Use these guides when the next question goes beyond the scope of this article.

Download the completed tables as a CSV, then replace the example inputs and add your evidence, owner and review date. Illustrative figures are teaching examples, not default targets.