AB Test Calculator
A/B test significance calculator that returns the p-value, z-statistic, observed lift, and confidence interval from a...
What AB Test Calculator does
An A/B test significance calculator that takes visitor and conversion counts for a control and a treatment group and returns the p-value, z-statistic, observed lift, and confidence intervals. It translates the statistical output into a plain-English verdict, indicating whether the difference is real or just noise. Users input their test data and select a confidence level (90%, 95%, or 99%) and hypothesis type (two-tailed or one-tailed) to get an immediate assessment of statistical significance.
How to use the TrafficLoopback AB Test Calculator
- 1
Enter the number of visitors and conversions for Variant A (the control) in the designated fields
- 2
Enter the number of visitors and conversions for Variant B (the treatment) in the designated fields
- 3
Select the desired confidence level (90%, 95%, or 99%) from the dropdown menu
- 4
Choose between a two-tailed or one-tailed hypothesis based on whether you only care if B beats A
- 5
Review the results, which include the conversion rates, absolute and relative lift, p-value, z-statistic, and a plain-English verdict on significance
Best for
Marketers, web developers, and product teams who need a quick, reliable way to determine if their A/B test results are statistically significant before making data-driven decisions.
Limitations
- Results depend on the assumption of a fixed sample size set in advance; peeking at results and stopping early inflates the false-positive ra
- The calculator provides estimates based on a two-proportion z-test and does not account for sequential testing or bandit algorithms
- No unit switching or integration with external analytics platforms; users must manually input their test data
AB Test Calculator FAQ
- What does the p-value tell me about my A/B test?
- The p-value indicates the probability that the observed difference between your control and treatment groups happened by chance; a p-value below your chosen alpha (e.g., 0.05 for 95% confidence) suggests the result is statistically significant.
- How is the confidence interval for the lift calculated?
- The 95% confidence interval for the absolute lift is calculated using the standard error of the difference between the two conversion rates, providing a range within which the true lift is likely to fall.
- When should I use a one-tailed hypothesis instead of a two-tailed hypothesis?
- Use a one-tailed hypothesis only if you decided in advance that you only care whether the treatment beats the control, not if the control beats the treatment; otherwise, default to two-tailed.
- What happens if I keep checking my test results until I see significance?
- Sequential testing or peeking at results and stopping when you see significance inflates your real false-positive rate above the stated alpha level, making the results less reliable.
More AB Test Calculator tools
Similar tools
Based on shared tags