A/B testing promises to replace opinion with evidence. It delivers on that promise only if the test is sized before it starts and read honestly once it ends. Most of the value destroyed in experimentation programs does not come from bad ideas; it comes from good tests read the wrong way — stopped early, judged on noise, or declared a winner on a difference that would not survive a second run.
Three disciplines prevent almost all of it: decide the sample size in advance, understand what significance does and does not tell you, and never peek.
Size the test before you run it
The most common mistake is launching a test with no idea how much traffic it needs, then eyeballing the dashboard until it looks decided. Required sample size is not a matter of taste; it falls out of three inputs: your baseline conversion rate, the smallest effect you care to detect, and the statistical power you want.
The relationship that surprises people is between effect size and sample. Sample size scales with the inverse square of the effect: halving the change you want to detect roughly quadruples the visitors you need. Detecting a 20% relative lift on a healthy baseline might take a week; detecting a 2% lift can take a quarter. Work out the number before you commit with the sample size calculator.
What significance actually means
A result is statistically significant when the difference you observed would be unlikely — conventionally, less than a 5% probability — if the two variants actually performed identically. That is all it means. It is a statement about how surprising your data would be under the assumption of no effect.
Two things it is not: it is not the probability that your variation beats the control, and it is not a measure of how big or valuable the effect is. A p-value of 0.03 on a 0.1% lift is a statistically real result that may not be worth shipping. Significance is a filter against chance, not a business case. When a test is done, check whether it cleared significance with the significance calculator — then ask, separately, whether the effect is large enough to matter.
The peeking trap
Here is the discipline that quietly decides whether your whole program is trustworthy: do not stop the test the moment it looks significant.
Every time you check an in-flight test and stop if it has crossed the line, you give random noise another chance to have wandered across that line. Check often enough and you will "find" significance in tests where nothing is happening. The false-positive rate you think is 5% becomes much higher — sometimes several times higher — purely from repeated looks. This is why a test that looked like a winner on Tuesday so often evaporates by Friday.
The fix is to commit to your sample size up front and read the result once, at the end. If you genuinely need to monitor along the way, use a sequential testing method designed for it — not the fixed-horizon p-value, read repeatedly.
Run full weekly cycles
Even when the math says your sample has filled, let the test run at least one or two complete weekly cycles. Conversion behavior differs by day of week, and a test that ran Monday to Thursday has quietly oversampled your weekday audience. Whole weeks keep the mix representative.
The takeaway
Trustworthy experimentation is mostly procedural, not statistical. Decide the sample size before you start, read significance for what it is — evidence against chance, not proof of value — and refuse to stop early. Do those three things and your winners will still be winners the second time you run them.