A/B Test Significance & Sample Size Calculator
Enter the visitors and conversions for each variant to test whether the difference in conversion rate is likely to be real, using a two-proportion z-test with pooled variance. You get the p-value, z-score, relative lift, and a confidence interval for the difference, with a one-sided or two-sided option. Switch to "Sample size" to estimate how many visitors each group needs before you start. Everything is calculated in your browser, and nothing you enter is sent to a server. For means and standard deviations, see the Statistics Calculator; for profit planning, try the Break-Even Calculator.
🔒 Runs in your browser · Your data is never sent to a server
Notes
- This is a normal approximation using a two-proportion z-test with pooled variance. It gets rough when groups are small or rates are close to 0% or 100%; a warning appears when any expected count is below 5.
- Significant only means the difference is hard to explain by chance alone. It does not guarantee B wins, or that the difference is large enough to matter for your business. Check the size of the effect in the confidence interval.
- Checking the results repeatedly and stopping as soon as they look significant makes false positives much more likely (the peeking problem). Decide the sample size first with the sample size mode and judge only after reaching it.
- Comparing several variants or metrics at once needs a multiple-comparison correction such as Bonferroni. This tool assumes a single A versus B comparison.
- The one-sided test only checks whether B is higher than A and misses a drop in B, so use the two-sided test when in doubt. The confidence interval is always two-sided.
- The sample size assumes equal traffic in both groups and ignores repeat visitors and weekday effects.
Glossary
p-value
The probability of seeing a difference at least this large by chance if there were really no difference. Smaller values are harder to explain by chance. It is not the probability that B is better.
Significance level (α)
The threshold for calling a result significant, usually 5%. If the p-value is below α the result is significant, which keeps the false positive rate (Type I error) at or below α.
Power (1 − β)
The probability of detecting a real difference as significant. Aim for 80% or more; low power means real differences are easily missed.
Confidence interval
A range likely to contain the true difference. If a 95% interval does not include 0, the result is significant at the 5% level. A wider interval means the size of the effect is less certain.
Relative lift
The improvement of B over A as a percentage of A. If A converts at 2% and B at 2.5%, the difference is 0.5 points and the relative lift is +25%.
FAQ
If the p-value is below 0.05, can I say B won?
A p-value below 0.05 means the difference is hard to explain by chance alone. It does not mean B is a sure winner or that the gap is big enough to matter. The p-value is not the probability that B is better; it is how likely a gap this large would be if there were no real difference. Check the confidence interval for the size of the effect.
Should I use a one-sided or two-sided test?
Use the two-sided test when in doubt. A one-sided test only asks whether B is higher than A, so its p-value is half as large and it becomes significant more easily, but it misses cases where B is worse. If you use one-sided, decide that before the test starts.
How is the required sample size determined?
It depends on your current conversion rate, the smallest lift you want to detect, the significance level (usually 5%), and the power (usually 80%). Smaller lifts and lower conversion rates need many more visitors; going from 5% to 6% takes roughly 8,100 visitors per group.
Can I stop the test as soon as it turns significant?
It is not recommended. Looking at the results repeatedly and stopping the moment they look significant greatly raises the chance of a false positive. Fix the sample size in advance and judge once after reaching it.
Can I trust the result when there are few visitors?
The tool uses a normal approximation, which loses accuracy when an expected count (visitors times the conversion rate, and so on) is below 5, and it shows a warning in that case. Collect more visitors or consider an exact method such as Fisher's exact test.
Share this page