A/B test peeking simulator
Set a true effect (or none at all), choose how often you peek at the results, and run 2,000 Monte Carlo experiments to see for yourself how often 'peeking' declares a winner that isn't real.
How it works
A fixed-horizon A/B test controls its false-positive rate — with no real difference, it wrongly calls a winner about 5% of the time, exactly the 0.05 significance level you chose. The catch is that this guarantee only holds if you look once, at a sample size decided in advance.
'Peeking' means checking the test repeatedly as data arrives and stopping the moment p drops below 0.05. Every extra look is another chance to cross the threshold by luck, so the more often you peek, the higher the chance you stop on a random high point. The simulator runs 2,000 experiments for your settings and reports how often each strategy declared a winner: peeking at your chosen frequency versus checking only at the planned end.
Set the true lift to 0 to run an A/A test — two identical arms — and the peeking rate is pure false positives. Set a real lift and the tool shows a subtler cost: early stops often crown a 'winner' during a moment it was actually behind, and they systematically overstate the winning margin. Simulation uses a normal approximation to the binomial for speed; the qualitative lesson matches the exact math.
Assumptions and limitations
- This is a teaching simulation, not your test's analysis. It uses a normal approximation to the binomial and a fixed 0.05 two-sided z-test at each look, so exact numbers differ slightly from a precise sequential calculation — the pattern does not.
- It models the classic mistake (naive repeated significance testing with a fixed threshold). Proper sequential methods — alpha spending, group-sequential boundaries, always-valid p-values, Bayesian designs — are built precisely to let you look often without this penalty, and are not simulated here.
- Real experiments carry complications this ignores: novelty effects, weekly seasonality, multiple metrics, and unequal traffic splits, each of which adds its own risks on top of peeking.
Frequently asked questions
What is peeking in A/B testing?
Peeking is repeatedly checking an experiment's significance while it runs and stopping as soon as it looks significant. Because each look is another opportunity to cross the p < 0.05 line by chance, peeking inflates the false-positive rate well above the 5% the test appears to promise — often to 20–40% with frequent checking.
Why does stopping a test early cause false positives?
Significance testing assumes a single analysis at a pre-set sample size. Random noise makes the p-value wander up and down as data accumulates; if you stop the first time it dips below 0.05, you are selecting for lucky moments. The simulator makes this visible: with no real difference, frequent peeking declares a winner far more than 5% of the time.
How can I check my test early without this problem?
Use a method designed for it: group-sequential boundaries or alpha spending (which raise the bar at early looks), always-valid p-values and confidence sequences, or a Bayesian design with a pre-committed decision rule. All of them let you monitor continuously while keeping error rates honest — the naive fixed-0.05 peek this tool models does not.
Is an A/A test the same thing here?
Setting the true lift to 0 makes both arms identical — an A/A test. Any 'winner' it finds is by definition a false positive, so the peeking rate you see is exactly the error rate you would be shipping. It is the cleanest way to feel how much peeking distorts a test.
Does the simulation send my inputs anywhere?
No. All 2,000 experiments run in your browser with client-side JavaScript. Nothing you enter is transmitted or stored.
More free tools like this, by email
I publish new calculators and explainers regularly. Get them when they land — no spam, unsubscribe anytime.