Statistical significance
A result is statistically significant when the observed difference would be unlikely to occur by chance if there were truly no effect — conventionally, a probability below 5%.
In an A/B test, significance quantifies evidence against the idea that two variants perform identically. A p-value below 0.05 means a difference this large would happen less than one time in twenty by chance alone. It measures surprise under the null, not the size or value of an effect.
Significance is necessary but not sufficient: a significant result can be trivially small, and stopping a test early the moment it looks significant inflates false positives. Sizing the test in advance and reading it once is what keeps significance meaningful.
Learn this stuff by email
New definitions, tools, and explainers when I publish them. No spam, unsubscribe anytime.