The team implemented the new checkout button on Friday because it showed a 14% increase after two days of testing. By the next Friday, the button performed 3% worse than the previous version. No one knew when the performance trend reversed because the team stopped monitoring the data once they reached their target metric on the second day.
A/B testing involves showing two versions of a page, email, or feature to randomly assigned groups. It measures which version performs better against a specific metric chosen before the test begins. A valid test requires a clear hypothesis, correct randomisation, and a calculated sample size. If any of these elements are missing, a result that appears significant is often just random noise.
A/B testing involves showing two versions of an item, such as a webpage, email, or model prompt, to randomly assigned groups. You then measure which version performs better based on a metric selected before the test begins. This final requirement is often the most difficult part of the process. Most errors in A/B testing occur before the experiment starts. Common issues include using a vague hypothesis, choosing a metric after seeing the results, or failing to calculate a necessary sample size. Some testers also use a stopping rule that is really just "whenever the number looks good."
We build data and AI systems for clients at Shipshape Data, and testing is where a lot of otherwise sensible teams quietly fool themselves. Not through bad intentions. Through a dashboard that updates in real time and a very human urge to check it on day two. This guide covers how to write a hypothesis worth testing, how randomisation needs to work, how to size a test properly, how long to run one, the mistakes that invalidate results without anyone noticing, and what changes when the thing you are testing is an AI feature.
Why a coin flip needs a hypothesis first
The common approach is to test all possible variations: two headline options, three button colours, or a shorter form, for example.
Testing without a hypothesis is a process of making two guesses and choosing the one that produces a higher number.
The requirements for a hypothesis
A functional hypothesis identifies the specific change, the reasoning behind it, and the expected impact on a particular metric. "We think reducing the checkout form from nine fields to four will raise completion rate, because our support tickets show abandoned checkouts citing form length" is a hypothesis. "Let's try a shorter form and see" is not a hypothesis, because it does not allow for a clear failure. You can justify any result after an experiment if you do not define success before you begin.
A hypothesis grounded in a real observation, a support ticket pattern, a session recording, a drop-off point in the funnel, gives you something to learn from even if the test loses. "The designer likes purple" teaches you nothing either way.
Select the metric before reviewing any data
Select the primary metric in writing before the test begins. Do not choose multiple metrics with the intention of picking the most favorable result later. Identify one figure that answers the specific business question. You may also select one guardrail metric to ensure that improving the primary goal does not negatively impact another area. For example, a checkout redesign that increases the completion rate but significantly lowers the average order value is not a clear success.
Randomisation: how to split traffic fairly
Most people assume that the randomisation in A/B testing works correctly, so they rarely verify it.
If you make a mistake at this stage, the rest of the process will be flawed. Even if you are careful, you will only be measuring random noise instead of actual results.
Assign based on the user rather than the visit
If a visitor sees version A on Monday and version B on Wednesday, they are counted in both groups. This behaviour ruins the comparison. Assignment must be consistent. You should hash a user or device identifier one time and keep that person in the same variant for the entire test. Most testing platforms handle this automatically. Home-built tests, cookie-based splits with short expiration dates, or tests running across logged-out and logged-in states often fail to do this.
Watch for the split that quietly is not 50/50
Random splits can result in imbalanced samples if upstream factors bias the participants. This can happen if a feature flag rolls out slowly, a caching layer favours returning visitors, or a bot crawler targets specific URLs. You should check the participant counts in each group early in the process. If you expect a 50/50 split but see a 61/39 distribution after one day, the assignment mechanism is likely broken. The test is compromised until you fix this technical issue.
Segment before you launch, not after you lose
If you believe the effect varies by device, visitor type, or region, decide this before you launch. You must size the test to detect results in those specific groups. Analysing the data across many different segments after a neutral result to find one successful area is a form of the multiple comparisons problem. This issue is explained in the following section.
How big a sample needs to be before you can trust it
Most teams skip this step because it requires calculations that often lead to difficult conclusions. You may find that the test needs to run for a longer period, or that your sample size is too small to reliably detect the effect you expect.
The three numbers a sample size calculation needs
To calculate your sample size, you need your current conversion rate, a minimum detectable effect, a significance level, and statistical power. The minimum detectable effect is the smallest improvement you want to identify, rather than the largest one you expect. Significance is typically set at 95% and power at 80%. When you enter these figures into a sample size calculator, it provides the number of visitors required for each variant to ensure the results are statistically valid. Determining the minimum detectable effect is the most practical part of this process. For example, a 1% improvement on a low-traffic page might require a sample size that takes months to collect. It is useful to know this before you start. If a test runs for months and produces a null result, that outcome was predictable from the beginning.
Pages with low traffic are a problem
A page that receives 200 visitors per week cannot reliably detect a 5% increase in performance, regardless of what a software dashboard indicates. You must acknowledge that only significant changes are measurable at this traffic volume. To address this, you can run the test over a longer period, group the test with a similar page, or move the experiment to a higher-traffic part of the website. If you run the test without these adjustments and report the results, the business will likely mistake random data fluctuations for actual trends.
How long to run a test, and why the calendar matters more than the dashboard
Reaching the required sample size is necessary but does not guarantee a valid result. A test must also run for a sufficient duration to observe patterns that emerge over time. This requirement depends on the calendar as much as it depends on mathematics.
Run in full weeks, and more than one of them
User behaviour on a Tuesday differs from behaviour on a Saturday. A test that begins on a Wednesday and ends on the following Monday includes one weekend and an uneven number of weekdays. The winning variation at the end of this period might only be the winner because of the specific days the test covered. The standard solution is to run tests in full one-week blocks. You should run a test for at least two full weeks even after reaching the required sample size. This ensures the result is not caused by a single unusual week.
Novelty and primacy effects fade, if you let them
A redesigned homepage often receives an initial increase in engagement because it is new and attracts attention. This effect typically decreases over one to two weeks as visitors become accustomed to the changes. The opposite effect also occurs. A returning user may hesitate when encountering a changed process, while a new user will not experience this because they have no previous version to compare it to. Stopping a test during the first week risks measuring these temporary reactions to novelty instead of the actual impact of the underlying change.
Seasonal and promotional noise
A test that occurs during a sale, a public holiday, or a marketing campaign in one group but not the other will produce inaccurate results. You should keep promotional activity the same for both variants when possible. You can also change the test dates to avoid these specific time periods.
Peeking, multiple comparisons, and the other ways teams fool themselves
Most poor decisions in A/B testing result from using a functional tool incorrectly. These errors often occur when people work under deadline pressure. These users typically understand the basic definition of statistical significance, but they do not realise how checking a dashboard daily affects the validity of that metric.
Peeking inflates your false positive rate more than it feels like it should
Monitoring a live test every day and ending it as soon as it reaches a 95% significance threshold is an incorrect practice. This approach is similar to rolling a die repeatedly until it shows a six and then claiming the die is biased. Random fluctuations frequently cross significance thresholds by chance. If you stop a test as soon as it appears significant, a test with no actual effect will show a significant result much more often than 5% of the time. You should instead decide on a fixed sample size and end date before starting and only check for significance once the test is complete. Alternatively, you can use sequential testing methods designed for continuous monitoring. These methods adjust the significance threshold to account for the fact that you are checking the data frequently.
If you stop a test as soon as the results appear favorable, you are simply identifying the specific day that random variation matched your expectations and labelling that result as the truth.
Multiple comparisons: the more you test, the more you will find something by accident
If you test twenty metrics simultaneously, one will likely reach the 95% significance threshold by chance alone, even if your changes had no actual effect. This outcome is a mathematical certainty rather than a matter of luck. To prevent this, you should select a single primary metric before starting the test and use it as your main criteria. If you must evaluate multiple metrics or variants at once, you should use a statistical correction such as a Bonferroni adjustment or a false discovery rate control. These methods raise the significance threshold based on the number of comparisons you perform. If you do not use these corrections and instead report only the metrics that showed movement, you are incorrectly presenting a null result as a success in your reports.
Simpson's paradox and the segment that appeared effective
Total results can obscure different patterns within subgroups if the compared groups have unbalanced compositions. A variant that seems to perform better overall may actually perform worse in every individual segment. This occurs when one segment is overrepresented in a specific test arm. This situation is uncommon in properly randomised tests. This risk is a reason to verify the randomisation mechanism while the test is running.
Statistical significance is not the same as a result worth shipping
Statistical significance does not guarantee that a result is relevant to the business. This distinction is often ignored during urgent projects because the word significant implies a definitive conclusion. However, the term only indicates that an observed difference is unlikely to be the result of random chance. It does not determine if the difference is large enough to justify taking action.
Practical significance is a business judgement, not a statistics one
A 0.1% increase in click-through rate on a high-traffic page can reach statistical significance because large samples detect very small effects. Whether a 0.1% change justifies the engineering time required for development and maintenance is a business decision rather than a statistical one. You should determine the minimum effect size needed to change a business decision before starting the test. This is the same value used to calculate the required sample size. Evaluate the final results against that specific threshold. A bare "significant, yes or no" does not provide that information.
Confidence intervals tell you more than a single number does
A point estimate of "up 4%" suggests a level of certainty that is not supported by the data. The confidence interval of "somewhere between up 1% and up 9%," provides a more accurate account of the test results. This range should be included with the primary figure rather than replacing it. Providing both numbers is useful for decision-makers who were not involved in the testing process.
Testing AI features properly
These principles apply to AI features in the same way they apply to button colours. However, models present specific problems that can surprise teams, even those with experience running hundreds of conventional A/B tests.
The output is not fixed, so neither is the "version" you are testing
A generative model may produce different responses to the same prompt each time it is used. Therefore, "variant B" in an AI feature test is not a single, consistent experience for half of your users, as a static page would be. Instead, it represents a range of possible outputs, some of which will be good, some average, and some clearly incorrect in ways that a human would notice but a standard metric might miss. Calculating an average across this range of outputs can hide the fact that a feature performs very well most of the time but fails in other instances. This is especially important for features that provide direct answers to customers, as opposed to something like colour selection. Relying solely on overall metrics to judge AI features can overlook this potential for failure. A thorough evaluation process, as detailed in our guide to evaluating large language models, should be used in addition to A/B testing, not as a replacement for it.
Latency and cost are metrics too, not footnotes
A recommendation from a machine learning model could increase conversions but also add one and a half seconds to load time, or cost significantly more per request than the rule-based system it would replace, given your current traffic levels. Both of these factors should be included as key measurements in the test from the beginning, and not as an afterthought mentioned in the results presentation when the finance team asks about costs.
Identify who is affected by the improvement
A model may improve the overall metric while performing poorly for a specific group of users. This usually affects groups that are underrepresented in the training or evaluation data. An aggregate A/B test result hides these differences. You should slice AI feature results by segment as a standard practice. Our piece on AI bias also covers this topic. Because a model can drift after launch, the initial test result does not guarantee performance after ninety days. This is different from a static page redesign. You should plan for this drift before the feature launches. Do not wait for a customer complaint to show you the issue.
Where to start
Select a single change that allows for a clear hypothesis. Choose a page or user flow with sufficient traffic to reach a required sample size within several weeks. Identify the primary metric and the minimum effect size that constitutes a success before reviewing any data. Calculate the necessary sample size before the test begins. Set a specific end date and follow it, or use a sequential testing method designed for interim analysis. Do not check results daily and stop the test only when the data matches your expectations.
If you perform this process correctly one time, the discipline usually spreads. It becomes difficult to justify shipping products based on guesses once a team has compared a valid test result to a lucky outcome. If you are creating an experimentation process or need an evaluation of results you do not trust, talk to us. We will tell you if a test requires two more weeks of data because we want to prevent you from treating a random result as a deliberate decision.
If you are currently addressing this issue as part of your roadmap, our data consultancy provides services for this specific area. We begin by demonstrating the capabilities of your existing data.
Frequently asked questions
What is A/B testing?
A/B testing involves showing two versions of an item, such as a page, an email, or a prompt, to different groups assigned at random. You then measure which version performs better against a metric chosen before the test begins. Most errors occur before the groups are split. These issues include having a vague hypothesis, choosing a metric after the test is finished, or failing to calculate a proper sample size.
What is A/B split testing?
A/B split testing is identical to A/B testing. Traffic is divided randomly between two versions so that each visitor sees only one version during the test. This split must be sticky and assigned to the specific user rather than the visit. If the same person is counted in both groups, they will contaminate the results.
What is A/B testing software, and is it necessary for you?
A/B testing software is the tool that assigns visitors to groups at random, ensures they continue to see the same version, and tracks results based on your selected metric. Most commercial platforms handle these assignments correctly. However, custom systems and tests using short-term cookies often fail to maintain consistent groups. You should check the actual distribution of your traffic early in the process.
What is A/B testing in marketing?
In marketing, A/B testing is the process of comparing two versions of a page, email, or campaign using a specific metric, such as a completion rate. This metric must be chosen before the test begins. A procedure is only considered a valid test if the hypothesis and the primary metric are recorded in advance.
How can you determine if your sample size is sufficient for an A/B test?
Before you launch a test, you must enter a baseline conversion rate, a minimum detectable effect, and a significance level into a sample size calculator. The standard significance level is 95% and the standard power is 80%. Pages with low traffic are a common problem. If a page only has 200 visitors per week, it cannot provide reliable data for a small increase in conversions, regardless of any positive indicators on a dashboard.