Sample size scales with 1/effect². Detecting a 5% lift instead of a 10% one costs four times the traffic, not twice.
The difference between the two rates is squared in the denominator. A 5% conversion rate moving to 5.5% is a difference of 0.005, and squaring it gives 0.000025 — which is how a numerator of about 0.1 turns into 31,000 visitors per variant. Teams plan for the effect they hope for and budget for the effect they expected, and the gap between those two is quadratic.
It assumes you fixed the sample size before starting. Stop the moment a result looks significant and the arithmetic no longer applies — repeated peeking pushes the false-positive rate from 5% towards 30%, so the test that finished early was never the test you sized. Sequential and Bayesian designs exist precisely to allow looking.
Estimate with the rule, then check it against the calculator that models it properly.
Open A/B Test & Sample Size →Sample size scales with 1/effect². Detecting a 5% lift instead of a 10% one costs four times the traffic, not twice. The difference between the two rates is squared in the denominator.