A/B Test Significance Calculator
Enter the visitors and conversions for your control and variants to see whether the difference is statistically significant, how big it is and how much more data you need.
| Variant | Visitors | Conversions | Conv. rate | Uplift | p-value |
|---|
How to use the A/B Test Significance Calculator
- Enter visitors and conversions for the control (A) and the variant (B). Use + Add variant for A/B/n tests.
- Choose the confidence level and whether the test is two-tailed (any difference) or one-tailed (B better than A).
- Read the verdict: uplift, p-value, confidence intervals and whether the result is significant at 90%, 95% and 99%.
- Check the chart: the less the two curves overlap, the stronger the evidence.
- Plan your next test on the Sample size tab with your baseline rate and the smallest uplift you care about.
How the significance test works
The calculator uses the standard two-proportion z-test with a pooled standard error, the same test most A/B testing tools use for conversion rates:
- Conversion rates: pA = conversionsA ÷ visitorsA, pB = conversionsB ÷ visitorsB
- Pooled rate: p̂ = (conversionsA + conversionsB) ÷ (visitorsA + visitorsB)
- Standard error: SE = √(p̂ (1 − p̂) (1/nA + 1/nB))
- z = (pB − pA) ÷ SE, and the p-value comes from the standard normal distribution: 2 × (1 − Φ(|z|)) for a two-tailed test.
Example: control 500 conversions from 10,000 visitors (5.00%), variant 570 from 10,000 (5.70%). The pooled rate is 5.35%, SE = 0.003182, z = 0.007 ÷ 0.003182 = 2.200, so the two-tailed p-value is 0.0278. That is significant at 95% (p < 0.05) but not at 99% (p > 0.01), and the relative uplift is +14.0%.
Confidence intervals for each rate use the Wilson score method, which behaves well for low conversion rates. The interval for the difference uses the unpooled standard error √(pA(1 − pA)/nA + pB(1 − pB)/nB).
Sample size per variant
Visitors needed per variant at 95% confidence (two-tailed) and 80% power, from n = [zα/2√(2p̄(1 − p̄)) + zβ√(p₁(1 − p₁) + p₂(1 − p₂))]² ÷ (p₂ − p₁)²:
| Baseline rate | Relative lift to detect | Target rate | Visitors per variant |
|---|---|---|---|
| 2% | +20% | 2.4% | 21,109 |
| 5% | +10% | 5.5% | 31,234 |
| 5% | +20% | 6.0% | 8,158 |
| 10% | +5% | 10.5% | 57,763 |
| 10% | +10% | 11.0% | 14,751 |
| 20% | +10% | 22.0% | 6,510 |
Halving the effect you want to detect roughly quadruples the sample you need, which is why small sites should test bold changes rather than tiny tweaks.
Common A/B testing mistakes
- Peeking: checking results daily and stopping at the first significant p-value can push the real false-positive rate far above 5%. Fix the sample size first.
- Too short: run tests for whole weeks so weekday and weekend visitors are both represented, even if the sample size is reached sooner.
- Many metrics or variants: the more comparisons you make, the more likely one is significant by chance. Pick one primary metric and correct for multiple variants.
- Sample ratio mismatch: if a 50/50 split delivers very uneven visitor counts, check the tracking before trusting the result.
- Significant is not the same as important: look at the confidence interval for the uplift and ask whether the low end would still be worth shipping.
Frequently asked questions
What does statistically significant mean in an A/B test?
It means the observed difference would be unlikely if the two versions really converted at the same rate. At 95% confidence, a result is significant when the p-value is below 0.05. It does not prove the variant is better by exactly the observed amount; the confidence interval shows the plausible range.
What is a p-value?
The p-value is the probability of seeing a difference at least as large as yours if there were truly no difference. A p-value of 0.03 means such a result would happen about 3% of the time by chance alone.
Should I use a one-tailed or two-tailed test?
Use two-tailed when you want to detect a change in either direction, which is the safer default. A one-tailed test only asks whether B is better than A; it reaches significance sooner but cannot detect that B is worse.
Can I stop the test as soon as it becomes significant?
Repeatedly checking and stopping at the first significant result (peeking) inflates false positives. Decide the sample size in advance with the Sample size tab and run the test to that size, ideally over whole weeks.
Why use a Bonferroni correction with several variants?
Each extra comparison is another chance of a false positive. Bonferroni divides the significance level by the number of comparisons, for example 0.05 ÷ 3 ≈ 0.0167 for three variants against one control, so the overall error rate stays near 5%.
What is statistical power?
Power is the probability of detecting a real effect of a given size. 80% power means a true uplift of that size would be found significant 4 times out of 5. Low observed power means a non-significant result is inconclusive rather than evidence of no effect.