The new checkout button was up 14% after two days, so the team shipped it that Friday. By the following Friday it was down 3% against the version it replaced, and nobody could say exactly when the number had flipped, because nobody had been watching for that. They had been watching for the number they wanted, and on day two it arrived.
A/B testing is the practice of showing two versions of something (a page, an email, a model prompt, a pricing tier) to different, randomly assigned groups of people, then measuring which one performs better against a metric you picked in advance. That last clause is doing more work than it looks like. Most of what goes wrong with A/B testing happens before a single visitor is split into a group: a vague hypothesis, a metric chosen after the fact, a sample size nobody calculated, a stopping rule that is really just "whenever the number looks good."
We build data and AI systems for clients at Shipshape Data, and testing is where a lot of otherwise sensible teams quietly fool themselves. Not through bad intentions. Through a dashboard that updates in real time and a very human urge to check it on day two. This guide covers how to write a hypothesis worth testing, how randomisation actually needs to work, how to size a test properly, how long to run one, the mistakes that invalidate results without anyone noticing, and what changes when the thing you are testing is an AI feature rather than a button colour.
Why a coin flip needs a hypothesis first
The instinct is to test everything: two headlines, three button colours, a shorter form. Testing without a hypothesis is not really testing. It is guessing twice and calling the guess with the bigger number a decision.
What a hypothesis actually needs to say
A usable hypothesis states what you are changing, why you think it will help, and what you expect to happen to a specific metric, by roughly how much. "We think reducing the checkout form from nine fields to four will raise completion rate, because our support tickets show abandoned checkouts citing form length" is a hypothesis. "Let's try a shorter form and see" is not, because there is no way to be wrong. Anything can be justified after the fact if you never committed to what success looked like beforehand.
A hypothesis grounded in a real observation, a support ticket pattern, a session recording, a drop-off point in the funnel, gives you something to learn from even if the test loses. "The designer likes purple" teaches you nothing either way.
Pick the metric before you look at any data
Decide the primary metric in writing before the test launches, and decide it alone. Not three metrics you will pick the best-looking one from afterwards. One number that answers the actual business question, plus perhaps one guardrail metric you are watching to make sure you have not broken something else in the process of improving the first one. A checkout redesign that lifts completion rate but tanks average order value has not necessarily won.
Randomisation: how to actually split traffic fairly
The "random" in A/B testing is the part everyone assumes is handled and almost nobody checks. Get it wrong and every other step in the process, however careful, is measuring noise dressed up as a result.
Assign by user, not by visit
A visitor who lands on version A on Monday and version B on Wednesday has just been counted in both groups, and their behaviour has contaminated the comparison. Assignment needs to be sticky: hash a user or device identifier once, and keep that person in the same variant for the life of the test. Most testing platforms do this by default. Home-built tests, cookie-based splits with a short expiry, or anything running across logged-out and logged-in states, tend not to.
Watch for the split that quietly is not 50/50
A genuinely random split can still produce an imbalanced sample if something upstream biases who reaches the test: a feature flag rolling out gradually, a caching layer favouring returning visitors, a bot crawler hammering one URL and not its counterpart. Check the actual numbers in each arm early. If you expected 50/50 and you are looking at 61/39 a day in, something is wrong with the mechanism, not the users, and the test is compromised until it is fixed.
Segment before you launch, not after you lose
If you suspect the effect differs by device, by new versus returning visitor, or by region, decide that before launch and size the test to detect it in those groups specifically. Slicing the data twenty different ways after a flat result, looking for the one segment where it worked, is not analysis. It is the multiple comparisons problem in a different outfit, and it deserves the section below.
How big a sample needs to be before you can trust it
This is the part most teams skip, usually because it involves a calculator and a slightly uncomfortable answer: the test needs to run longer, or the effect you are hoping for is smaller than the sample can reliably detect.
The three numbers a sample size calculation actually needs
You need a baseline conversion rate, a minimum detectable effect (the smallest improvement worth caring about, not the biggest you are hoping for), and a significance level, conventionally 95%, alongside a statistical power, conventionally 80%. Plug those into any standard sample size calculator and it will tell you, per variant, how many visitors you need before the test can distinguish a real effect from noise. The honest part of this exercise is usually the minimum detectable effect. A 1% lift on a page with modest traffic can require a sample so large the test would run for months, which is worth knowing before launch rather than after a null result that was never going to be anything else.
Low-traffic pages are the trap
A page with 200 visitors a week is not a good candidate for detecting a 5% lift with any confidence, whatever the dashboard's green tick says. Either accept that only large effects are detectable at that volume, pool the test across a longer window, combine it with a similar page, or test something further up the funnel where the traffic exists. Running it anyway and reporting whatever number comes out is how a business ends up mistaking noise for a signal.
How long to run a test, and why the calendar matters more than the dashboard
Reaching the sample size on paper is necessary and not sufficient. A test also needs to run long enough to see the patterns that only show up over time, and that is a calendar problem as much as a maths problem.
Run in full weeks, and more than one of them
Behaviour on a Tuesday is not behaviour on a Saturday. A test that starts on a Wednesday and stops the following Monday has seen one weekend and skewed weekday coverage, and whatever won by the end may simply have won because of which days it happened to catch. The standard fix is to run in complete week-long blocks, and to run at least two full weeks even once the sample size is reached, so the result is not an artefact of one unusually good or bad week.
Novelty and primacy effects fade, if you let them
A redesigned homepage often gets an initial lift purely because it is new and people notice it, an effect that decays over one to two weeks as visitors get used to it. The opposite happens too: a returning user's first encounter with a changed flow can cause hesitation that a new user never experiences, because the new user has no prior version to compare it against. Stopping a test in week one risks measuring the novelty rather than the underlying change, in either direction.
Seasonal and promotional noise
A test that spans a sale period, a public holiday, or a marketing push in one arm and not the other is not measuring what you think it is measuring. Where possible, keep promotional activity identical across both variants, or push the test dates around a known anomaly rather than through it.
Peeking, multiple comparisons and the other ways teams fool themselves
Most bad A/B testing decisions are not caused by a broken tool. They are caused by a correct tool used the wrong way, usually under some deadline pressure, by people who know roughly what statistical significance means but have not internalised what checking the dashboard every day actually does to it.
Peeking inflates your false positive rate more than it feels like it should
Checking a live test daily and stopping the moment it crosses the 95% significance threshold feels like diligence. It is closer to rolling dice until you land on a six and declaring the die loaded. Random noise crosses a significance threshold briefly all the time, purely by chance, and a test with no real effect underneath it will produce a "significant" result at some point during its run far more often than 5% of the time if you are allowed to stop whenever you like. The fix is to commit to a fixed sample size and end date and ignore significance until you get there, or use a sequential testing method built for continuous monitoring, which adjusts the threshold specifically because you are looking early and often.
A test you are allowed to stop the moment it looks good is not a test. It is a way of finding the exact day random variation happened to agree with you, and calling that day the answer.
Multiple comparisons: the more you test, the more you will find something by accident
Test twenty metrics at once and, purely by chance, roughly one will cross the 95% threshold even if nothing you changed had any real effect on anything. This is arithmetic, not bad luck. The fix is to decide the primary metric before launch and hold to it, and if you genuinely need several metrics or variants at once, apply a correction, a Bonferroni adjustment or a false discovery rate control, that tightens the threshold in proportion to how many comparisons you are running. Skipping that step and reporting whichever metric moved is how a null result gets reframed as a win in a results deck.
Simpson's paradox and the segment that looked like it worked
An overall result can hide the opposite pattern in a subgroup, or vice versa, if the groups being compared are not actually balanced in composition. A variant that appears to win overall can be losing in every individual segment, if one segment happens to be overrepresented in one arm. This is rare in a properly randomised test, which is one more reason the randomisation mechanism deserves checking throughout the run, not just at launch.
Statistical significance is not the same as a result worth shipping
A result can be statistically significant and still not matter to the business, and this is the distinction that gets lost fastest under deadline pressure, because "significant" sounds like it settles the question. It settles a narrower one: that the difference you observed is unlikely to be pure chance. It says nothing about whether the difference is big enough to act on.
Practical significance is a business judgement, not a statistics one
A 0.1% lift in click-through rate on a page with enormous traffic can reach statistical significance easily, because a large enough sample can detect a vanishingly small true effect. Whether 0.1% is worth the engineering time to build and maintain is a separate question, and it is a business one, not a p-value. Decide the minimum effect size that would actually change a decision before the test runs, the same number that fed the sample size calculation, and hold the result against that bar rather than a bare "significant, yes or no."
Confidence intervals tell you more than a single number does
A point estimate of "up 4%" invites more confidence than it deserves. The confidence interval around it, "somewhere between up 1% and up 9%," is a more honest description of what the test actually found, and it is worth reporting alongside the headline number rather than instead of it, particularly to anyone downstream who will make a decision based on the result without having watched the test run.
Testing AI features properly
Everything above applies to an AI feature exactly as it applies to a button colour. It also runs into a few problems specific to models, and teams that have run hundreds of conventional A/B tests sometimes get caught out here anyway.
The output is not fixed, so neither is the "version" you are testing
A generative model can produce a different answer to the same prompt twice, so "variant B" in an AI feature test is not one fixed thing shown to half your users the way a static page is. It is a distribution of outputs, some strong, some mediocre, some wrong in a way a human reviewer would catch immediately and a metric would not. Averaging over that distribution can mask a feature that is brilliant most of the time and badly wrong the rest, which matters more for something giving a customer an answer than for a shade of blue. Judging AI features on aggregate metrics alone misses that failure mode, which is why a proper evaluation approach, covered in our guide to evaluating large language models, needs to sit alongside the A/B split rather than replace it.
Latency and cost are metrics too, not footnotes
A model-generated recommendation might lift conversion and also add a second and a half of load time, or cost meaningfully more per request than the rule-based system it replaces at the traffic volumes you actually run. Both belong in the test as guardrail metrics from day one, not as a caveat added to the results deck after finance asks about the bill.
Watch who the improvement is actually landing on
A model can lift the aggregate metric while performing worse for a specific segment of users, most often ones underrepresented in whatever data trained or evaluated it, and an aggregate A/B result will not surface that on its own. It is worth slicing AI feature results by segment as standard practice, in the same way the underlying question shows up in our piece on AI bias. And because a model can drift after launch in a way a static page redesign never does, the test result on day one is not a guarantee about day ninety, and that is worth planning for before the feature ships rather than after a customer complaint reveals it.
Where to start
Pick one change you can state a real hypothesis about, on a page or flow with enough traffic to reach a sample size within a few weeks, and write down the primary metric and the minimum effect size that would count as a win before anyone looks at a single number. Calculate the sample size before launch, not after. Commit to an end date and hold to it, or use a sequential method built for early looking, rather than checking daily and stopping when the dashboard finally agrees with you.
Do that once, properly, and the discipline tends to spread, because the alternative, shipping on a hunch and calling it validated, gets harder to defend once one test has been run the right way and everyone has seen what a genuine result looks like next to a lucky one. If you are building an experimentation process from scratch, or you want a second opinion on results you are not sure you can trust, talk to us. We would rather tell you honestly that a test needs another two weeks than watch you ship a coin flip as a decision.