Test 20 signals and one will "work" by accident

Question

A researcher backtests 20 unrelated signals, each tested at the 5% significance level. Suppose in truth none of them has any edge.

Compute the probability that at least one signal comes out "significant" anyway, and explain what this means for anyone who mines many hypotheses.

Accepted Answer

One test. With \alpha = 0.05, a single test on a truly null signal has a 5% chance of a false positive and a 95% chance of correctly staying quiet. Twenty independent tests. The probability that all 20 correctly stay quiet is 0.95^{20} \approx 0.358. So the probability of at least one false positive, the family-wise error rate, is 1 - 0.95^{20} \approx 0.642. Almost two-thirds of the time, testing 20 worthless signals hands you a "discovery." The per-test guarantee of 5% says nothing about the family: run enough tests and a spurious winner is nearly certain. At 100 tests the chance of at leas…

Test 20 signals and one will "work" by accident

Your answer

More Statistics questions