Statistical significance
Also called Significance, Confidence level
A test of whether a difference between variations is larger than random chance would usually produce, judged against a threshold set in advance.
Statistical significance helps you decide whether the difference between two versions in an A/B test is likely to be real or could easily be random noise. The usual tool is the p-value: the probability of seeing a difference at least as large as the one you observed if there were actually no difference between the versions. If the p-value is below a threshold you chose in advance, commonly 0.05 (often described as 95% confidence), the result is called statistically significant.
A p-value is often misread. It is not the probability that version B is better, and a significant result does not tell you how big the improvement really is. It also does not mean the result is important for your business: with enough traffic, even a tiny difference can be significant. Look at the confidence interval around the difference to see the range of plausible effects.
Significance depends on sample size. Small samples produce large random swings, so a test needs enough visitors and conversions to detect the size of change you care about. Calculate the required sample size before you start, based on your current conversion rate, the smallest improvement worth detecting and the desired statistical power, often 80%. Checking results repeatedly and stopping the moment they cross the threshold, known as peeking, greatly increases the chance of a false winner.
How to improve it
- Set your significance threshold and required sample size before the test starts.
- Check results with the free A/B test significance calculator (open tool).
- Do not stop a test early just because it looks significant at one moment.
- Report the size of the difference and its confidence interval, not only the p-value.
- Treat a single significant result with some caution and confirm big decisions with a follow-up test.
How PageScore checks it
PageScore does not run experiments or analyse test data. For that, use the free A/B test significance calculator (open tool) alongside the testing tool you already use.