A/B Test Calculator

The A/B Test Calculator has two modes. Plan test estimates how many users each variant needs and how long the test will run for a given baseline conversion rate, minimum detectable effect, confidence level and power. Analyze results compares each variant with control and reports the conversion rates, absolute difference, relative lift, z statistic, p-value, confidence interval and a significance decision, after checking for sample ratio mismatch. Everything is calculated in your browser.

Plan test uses the two-proportion z-test with pooled variance under the null hypothesis: n per variant = (z_α · √(2p̄(1 − p̄)) + z_β · √(p₁(1 − p₁) + p₂(1 − p₂)))² / (p₂ − p₁)², rounded up, where p₁ is the baseline rate, p₂ the expected variant rate and p̄ their average. A relative effect sets p₂ = p₁ × (1 + MDE); an absolute effect in percentage points sets p₂ = p₁ + MDE. z_α is the normal quantile at 1 − α/2 for a two-sided test or 1 − α for a one-sided test, and z_β the quantile at the chosen power. With Bonferroni and several variants, α is divided by the number of variant-versus-control comparisons. Duration is the total sample divided by the users who enter the test per day, rounded up.

Analyze results computes z = (p_B − p_A) / √(p̂(1 − p̂)(1/n_A + 1/n_B)) with the pooled rate p̂. The two-sided p-value is 2 × (1 − Φ(|z|)); the one-sided p-value tests only whether the variant converts higher than control. The confidence interval for the absolute difference uses the unpooled standard error √(p_A(1 − p_A)/n_A + p_B(1 − p_B)/n_B). Bonferroni multiplies each p-value by the number of comparisons; Holm adjusts them step-down. When a correction is on, intervals use the Bonferroni-adjusted level.

Before any significance result, the visitor counts are compared with the planned split (equal by default) using a chi-square goodness-of-fit test with one degree of freedom less than the number of groups. A p-value below 0.001 is flagged, because a broken assignment, bot filtering or tracking loss can bias every other number. Investigate the cause before trusting the result.

A significant result means the observed difference is statistically significant at the chosen α; it is not the probability that a variant is better. A non-significant result does not prove the variants are equal. Choose confidence, power, test type and correction before the test starts and wait for the planned sample: changing them after seeing results or stopping early inflates false positives. The calculator also warns about small samples where the normal approximation is unreliable.

This is a fixed-horizon frequentist calculator for binary conversion metrics. It does not cover sequential testing, Bayesian analysis, revenue or other continuous metrics, CUPED variance reduction, or unequal planned allocation in the sample size estimate. Use these numbers as a planning and sanity-check aid alongside your experimentation platform.