A/B test significance calculator: real win or noise?
Enter the visitors and conversions for each version. You get both conversion rates, the uplift, a p-value and a plain answer at the 95% confidence level.
Result
- Conversion rate A
- 3.00%
- Conversion rate B
- 3.45%
- Relative uplift (B vs A)
- +15.0%
- z-score
- 1.801
- p-value (two-sided)
- 0.0717
- Confidence
- 92.8%
Not significant at 95%. The gap could be random noise. Keep the test running to its planned sample size, or treat the versions as equal.
95% interval for the difference (B minus A): -0.04 pts to +0.94 pts.
How it works
The calculator runs a two-proportion z-test. It asks how far apart the two conversion rates are, measured in units of the random wobble you would expect at your sample size.
p = (conv A + conv B) ÷ (visitors A + visitors B)
SE = √( p × (1 − p) × (1 ÷ visitors A + 1 ÷ visitors B) )
z = (rate B − rate A) ÷ SE
p-value = 2 × (1 − Φ(|z|))
Φ is the standard normal distribution. We compute it with the Abramowitz and Stegun approximation of the error function, which is accurate to about seven decimal places. With the defaults, A converts at 3.00% and B at 3.45%, which gives z = 1.80 and p = 0.0717.
How to read the result
A p-value under 0.05 means the result is significant at 95%. In plain terms, a gap this large would be unusual if the two versions were truly the same. Above 0.05, you cannot rule out chance.
Then look at the interval for the difference. If it runs from a tiny gain to a large one, the effect is real but its size is uncertain. Judge it by money: a small uplift on a high-traffic page can be worth more than a big uplift on a quiet one.
Assumptions and limits
- Visitors were split at random, and each visitor counts once.
- The test is two-sided at a fixed 95% level. It does not correct for testing many variants at once.
- It is a frequentist test read once, at a sample size you set in advance. Peeking and stopping early make false wins more likely.
- Statistical significance is not business value. Check the revenue impact before you roll out.
Common questions
What does a p-value of 0.05 mean?
If there were no real difference between A and B, you would see a gap at least this large about 5% of the time by chance. A smaller p-value means chance is a less likely explanation. It does not tell you how big or how valuable the effect is.
Can I stop the test as soon as it turns significant?
Better not. Checking every day and stopping at the first significant reading inflates false positives. Decide the sample size before you start, then read the result once you reach it.
Why is my big uplift not significant?
With few conversions, even a large relative uplift can come from noise. The test weighs the gap against how much the rates would wobble at your sample size. More traffic or a longer test narrows that wobble.
Can I use this for SEO tests like title changes?
Only for tests where visitors are randomly split between versions, such as landing page or CRO tests. SEO title tests usually compare groups of pages over time, which needs a different method.
Turn test wins into revenue
We plan conversion tests on the pages that already earn organic traffic, size them properly and report the result in money, not just percentages.
Get a Free SEO ROI ConsultationSee pricing or SEO ROI forecasting.