Sample size scales with 1/effect². Detecting a 5% lift instead of a 10% one does not cost twice as much traffic — it costs four times. That single fact is why most tests are underpowered: the effect people hope for is small, and the cost of proving it is quadratic.
| Baseline | +2% | +5% | +10% | +20% | +50% |
|---|
A test checked repeatedly and stopped the moment p < 0.05 does not have a 5% false positive rate. Every look is another chance for noise to cross the line, and the error rate climbs towards certainty with enough looks. The figures below are the approximate false-positive rate for a test with no real effect at all.
Baseline rate, the effect you want to detect, how often you will accept a false positive, and how often you will accept missing a real effect. Fix those four and the sample size is determined — it is not a matter of judgement or of how long feels reasonable.
n = (zα/2 + zβ)² × [p₁(1−p₁) + p₂(1−p₂)] ÷ (p₂ − p₁)² per
variant. At the conventional 5% significance and 80% power the leading term is
(1.96 + 0.84)² = 7.85. Everything else is the variance of the two rates over the
square of the difference between them.
The difference is squared in the denominator. A 5% baseline moving to 5.5% is a difference of 0.005, and squaring it gives 0.000025 — so the numerator, which is around 0.1, is divided by a very small number. That is where 31,000 per variant comes from.
A 5% baseline, detecting a 10% relative lift, at 80% power and 5% significance, needs 31,231 visitors per variant — about 62,000 in total. At 2,000 visitors a day that is 31 days. Ask instead to detect a 5% lift and it becomes 122,121 per variant, or four months.
Most likely because it could not have. "No significant difference" from an underpowered test means the test was too small to see the effect, not that the effect is absent. Put the sample you actually had into the detectable-effect mode: if the smallest thing you could have found is a 25% lift, a null result rules out almost nothing worth knowing.
No — not with a fixed-horizon test. Checking daily and stopping at the first p < 0.05 pushes the false positive rate from 5% towards 30% or more depending on how often you look. Either fix the sample size in advance and wait, or use a method designed for continuous monitoring such as a sequential test or a Bayesian approach.
It is the convention, and it means accepting a one in five chance of missing a real effect. That is a lot for an expensive decision. Raising power to 90% costs about 34% more traffic, which is usually cheaper than shipping the wrong thing or rerunning the test.
Each additional comparison is another chance of a false positive, so the significance threshold has to tighten — a Bonferroni correction divides α by the number of comparisons, which increases the sample needed for each. Four variants tested against control is not one test, it is three, and pretending otherwise inflates the error rate.
Two independent groups, one binary outcome, random assignment, a fixed sample size decided before the test starts, and a normal approximation to the binomial — reliable when the expected count in each cell is above about five. Not modelled: multiple comparisons, sequential or Bayesian designs, novelty and primacy effects, day-of-week and seasonal variation, users appearing in both arms, non-binary metrics like revenue per user, or cluster effects when the unit assigned is not the unit measured.
No changes to this tool’s own behaviour since the earliest archived release (v1.21). The full history for the site is in the changelog.