A/B test significance calculator
Enter the visitors and conversions for two variants. The calculator runs a two-proportion z-test and reports the p-value, the 95% confidence interval for the difference, and the relative lift — with every formula documented below.
Statistically significant at the 95% level.
- Control rate
- 4.00%
- Variation rate
- 4.60%
- Relative lift
- +15.0%
- p-value (two-tailed)
- 0.0365
- z-score
- 2.091
- 95% CI of difference
- 0.04% to 1.16%
How it works
The calculator compares two independent conversion rates using a two-proportion z-test. It computes each variant's conversion rate, then a pooled proportion across both samples, and uses it to estimate the standard error of the difference under the null hypothesis that both variants convert identically.
The z-score is the observed difference divided by that standard error. The two-tailed p-value is derived from the standard normal distribution: it is the probability of observing a difference at least this large in either direction if the variants truly performed the same.
The 95% confidence interval for the absolute difference uses the unpooled standard error. If the interval excludes zero, the result agrees with a significant two-tailed test at alpha = 0.05.
Assumptions and limitations
- The z-test is an approximation that assumes reasonably large samples. A common rule of thumb is at least 10 conversions and 10 non-conversions per variant; below that, use an exact test.
- The calculation assumes visitors are independent and each is counted once per variant. Session-based counting, returning visitors, or interference between variants violate these assumptions.
- Peeking at results repeatedly and stopping when significance appears inflates the false-positive rate. Decide the sample size in advance or use a sequential testing method.
- Statistical significance is not business significance. A significant 0.1% lift may not justify implementation cost, and a non-significant result is not proof of no effect.
Frequently asked questions
What does statistically significant mean in an A/B test?
A result is statistically significant when the observed difference between variants would be unlikely (conventionally, less than a 5% probability, p < 0.05) if the variants actually performed identically. It quantifies evidence against chance — it does not measure the size or value of the effect.
What sample size do I need for an A/B test?
It depends on your baseline conversion rate and the smallest lift you care to detect. Detecting small relative changes on low conversion rates can require tens of thousands of visitors per variant. As a rule, plan the sample size before starting and do not stop early just because significance appears.
What is a p-value?
The p-value is the probability of seeing a difference at least as large as the one observed, in either direction, if there were truly no difference between variants. It is not the probability that the variation beats the control.
Is this calculator's data sent to a server?
No. All calculations run locally in your browser using JavaScript. Nothing you type is transmitted, stored, or logged.
Should I use a one-tailed or two-tailed test?
This calculator reports a two-tailed p-value, which tests for a difference in either direction and is the more conservative default. One-tailed tests are only appropriate when you decided before the experiment that only one direction matters.
More free tools like this, by email
I publish new calculators and explainers regularly. Get them when they land — no spam, unsubscribe anytime.