A/B Testing: How Businesses Compare What Works

Should an email subject line highlight a discount or a new product? Should a checkout page have fewer steps? Businesses can use A/B testing to compare alternatives and measure how a change affects consumer behavior.

What is A/B testing?

Imagine an online store wants more customers to click its promotional emails. It randomly assigns recipients to two groups. One receives a subject line highlighting a discount; the other receives a subject line highlighting a new product. The rest of the email stays the same. By comparing click-through rates, the store can estimate how the subject line affects engagement.

A/B testing, also called split testing, is a randomized experiment comparing two versions of an experience using a predefined metric. Version A is usually the existing experience, while version B introduces a change. A higher observed conversion rate (the share of users who take the desired action) does not automatically establish a winner. The analysis must also consider uncertainty and whether the difference matters in practice.

How Businesses Use A/B Testing

A/B testing supports controlled experimentation in product development, marketing, and user experience research. Businesses use it to evaluate proposed changes before deciding whether to introduce them more widely.

Enhancing User Experience

A company might test whether moving its ‘Buy’ button affects how often visitors complete a purchase. Version A places the button near the product description, while version B places it below customer reviews. Visitors are randomly assigned to a version, and the company compares purchase rates. This measures behavior rather than directly establishing which design visitors prefer.

Driving Business Decisions

A/B testing also helps businesses evaluate marketing choices. An online retailer might compare two advertisements using completed purchases as its primary metric. The goal is to measure a specific outcome rather than assume that more clicks necessarily mean better business results.

A mobile e-commerce app could compare its existing product page with a redesigned version that adds 360-degree product views, more prominent customer reviews, and a different purchase button. The test would estimate the combined effect of the redesign. It would not establish which individual change caused the difference.

The Role of Statistics in A/B Testing

Statistics helps researchers plan A/B tests, estimate the effects of changes, and assess uncertainty. Reliable conclusions also depend on sound experimental design and accurate measurement.

Hypothesis Testing: Before running an A/B test, choose the metric you’ll use to compare the two versions. For example, if you’re testing two checkout pages, you might measure the percentage of visitors who complete a purchase. The null hypothesis is that the two versions have the same purchase completion rate. The alternative hypothesis is that their rates differ.

After the test, use the results to decide whether to reject the null hypothesis. If you fail to reject it, you don’t have enough evidence to conclude that the rates differ.

Sample Size: Decide how many users you need before the test starts. The number depends on the smallest improvement worth detecting, the significance level (explained below), and how likely you want to be to catch a real effect if one exists. The probability of detecting a specified effect is called statistical power, often set at 80%. With too few users, you can miss a real improvement. With a huge sample, even tiny differences can be statistically significant without being worth acting on.

Random Assignment: In an A/B test, participants are assigned by chance to version A or version B. For example, if an online store tests two checkout pages, it randomly assigns each visitor to one version. This helps balance factors such as device type and whether visitors are new or returning, making it easier to estimate the effect of the checkout design on purchase rates. The groups can still differ by chance, especially in small tests.

P-value and Significance Level: A p-value tells you how likely you would be to see a difference at least as large as the one you observed if the two versions actually performed equally. For example, a p-value of 0.03 means that, under that assumption, a difference this large or larger would occur in about 3% of repeated tests.

If the p-value is below the significance level you set before the test, such as 0.05, the result is statistically significant. Then look at the size of the difference to decide whether it matters to the business.

Confidence Interval: A confidence interval expresses uncertainty around an estimated difference. A method that produces 95% confidence intervals would capture the true difference in about 95% of repeated experiments under its assumptions. The interval’s width helps show how precisely the effect has been estimated.

Understanding Errors: A Type I error is a false alarm. You conclude the versions differ when they don’t, and you might launch a design that isn’t actually better. A Type II error is a missed opportunity. The change really made a difference, but the test didn’t detect it. The significance level sets the false-positive threshold. A larger sample generally increases power and reduces the risk of missing an effect of a given size.

Limitations of A/B Testing

A/B testing is useful, but it has limitations.

Time Requirements: Tests need enough participants and an appropriate duration to support useful conclusions. Low traffic or rare outcomes can make this difficult. Repeatedly checking results and stopping as soon as they appear significant can increase false positives unless the analysis accounts for that stopping rule.

Measurement Requirements: Results depend on reliable tracking and clearly defined metrics. Missing events, inconsistent tracking, or a metric that poorly represents the business goal can lead to misleading conclusions.

Difficulty Isolating Multiple Changes: If version B changes several things at once, the test shows their combined effect, not which change mattered. To separate them, test changes one at a time or use a multivariate test, which compares combinations of changes.

Limited Insight into Why It Works: A well-designed randomized test can estimate whether a change caused an outcome, but it may not explain the underlying reasons. Interviews, surveys, and usability research can help investigate why people responded differently

Key Takeaways

• A/B testing uses random assignment to compare two versions against a predefined outcome.

• A higher observed conversion rate alone does not establish a reliable improvement.

• Statistical significance, effect size, and uncertainty should be considered together.

• Reliable results require sound design, adequate sample size, and accurate measurement.

• An inconclusive result is useful information, not proof that the versions are identical.