Experiment planning
A/B Test Calculator
Plan an A/B test, calculate the required sample, check statistical significance and estimate duration before launch.
- Sample size
- Significance check
- Duration estimate
Verified June 23, 2026 · method and limitations
This is a simplified two-proportion calculator with pooled z-test - not a substitute for sequential testing platforms or Bayesian tools. Peeking and multiple comparisons are not corrected.
Why do A/B tests require statistics?
Randomness
Result differences can be random. 100 conversions from 1000 vs 110 from 1000 - is it noise or a real difference?
Decision confidence
Statistical significance (e.g. 95%) means that there is only a 5% chance that the difference is due to chance.
Avoiding errors
Without a proper sample you may deploy a worse variant or reject a better one.
Key concepts:
- Confidence level - probability that the result is true (usually 95%)
- Statistical power - ability to detect a real difference (usually 80%)
- MDE - minimum detectable effect, the smallest difference you want to detect
- p-value - probability that the result is random (we want p < 0.05)
How many people do I need for the test?
Test parameters
Required sample
Interpretation
-
Quick MDE comparison
| MDE | New CR | Sample/variant | Total |
|---|
Enter test results to check whether the difference is statistically significant.
Variant A (Control)
Variant B (Test)
Calculating...
Enter data to check significance.
Significance at different confidence levels:
Projected impact (if variant B is deployed)
Calculate how long your A/B test should run at your traffic level. CR, MDE and variant count sync from the Sample size tab.
Test planning
Test duration
Recommendations
How does MDE affect duration?
| MDE | Required sample | Duration |
|---|
Compare multiple variant results at once. Add up to 4 variants.
Comparison results
Winner: Variant B
-
| Variant | Users | Conversions | CR | vs Control | p-value | Status |
|---|
CR visualization
Glossary
| Term | Definition | Typical value |
|---|---|---|
| Confidence Level | Probability that the observed difference is real, not random | 95% |
| Statistical Power | Probability of detecting a real difference if one exists | 80% |
| MDE (Minimum Detectable Effect) | Smallest relative change the test can detect | 10-20% |
| p-value | Probability the difference is random. Lower is better | < 0.05 |
| Type I Error (False Positive) | Declaring a difference significant when there is none | 5% at 95% confidence |
| Type II Error (False Negative) | Failing to declare a difference significant when there is one | 20% at 80% power |
Common A/B testing mistakes
Stopping the test too early
Results fluctuate over time. Always wait until the required sample size is reached.
Peeking at results
Checking repeatedly increases false positives. Set the duration upfront.
Testing too many variants
Each extra variant needs more sample and time. Focus on 2-3 variants.
Ignoring seasonality
Run the test for full weeks to account for day-of-week differences.
Best practices
Set a hypothesis before the test
What are you testing and why? What outcome do you expect?
Calculate required sample size upfront
Do not start without a plan. Set MDE, sample size and duration.
Test one change at a time
If you change five things, you will not know what worked.
Document every test
Record hypotheses, results and learnings. Build a knowledge base.
Formulas and methodology
sampleSize | 2 × (Zα + Zβ)² × p(1-p) / effect² |
zScore | (p1 - p2) / √(pooled × (1/n1 + 1/n2)) |
pValue | 2 × (1 - normalCDF(|z|)) |
Original editorial calculator. Results computed locally. Last verified: June 23, 2026.