A two-day test came back flat, is the feature useless?

Question

Under time pressure, a colleague ran a 2-day A/B test of a new feature, found $p = 0.4$ , and concluded "no effect, let's not ship it." The feature was expected to move the metric by a small but real amount.

What is wrong with reading a non-significant result from a short test as proof of no effect?

Accepted Answer

The problem: the test was underpowered. Statistical power is the probability of detecting a real effect of a given size. It grows with sample size, because the standard error of the estimate shrinks like 1/\sqrt{n}. A 2-day test has a large standard error, so even a genuine effect can easily land at p = 0.4; the test simply could not resolve it. "Not significant" is absence of evidence, not evidence of absence. Quantifying the gap. The number of samples needed to detect an effect scales roughly with (\sigma / 	ext{effect size})^2. Halving the effect you want to catch quadruples the required …

A two-day test came back flat, is the feature useless?

Your answer

More Statistics questions