What statistical significance tells you
When variant B converts better than A, the question is whether that difference is real or just the normal ups and downs of random visitors. A significance test answers: if there were truly no difference, how likely is a gap at least this large? That probability is the p-value. At 95% confidence, a result is called significant when the p-value is below 0.05.
How this calculator works
- It runs a two-proportion z-test on the conversion rates of A and B.
- It reports the confidence interval for the difference. If the interval includes zero, “no difference” is still plausible.
- It estimates the sample size per variant needed to detect a difference of the size you observed with 80% power.
- Two-sided tests ask “is B different from A?” and are the safer default. One-sided tests ask only “is B better?” and should be chosen before the test starts.
Common mistakes
- Stopping as soon as it looks significant. Checking repeatedly and stopping at the first good result inflates false positives. Decide the sample size in advance.
- Running too short. Run for at least one full week, ideally two, so weekday and weekend visitors are both represented.
- Testing tiny changes on low traffic. Small effects need large samples. On low-traffic pages, test bigger changes: a new headline or offer, not a button colour.
More background in the statistical significance and A/B testing glossary entries.