A hundred signals at the 1% level

Question

A researcher tests 100 independent candidate signals, each with a $1\%$ -level test. Assume in truth none of them has any edge.

What is the probability at least one comes out "significant," how many false positives do you expect, and how would you correct the procedure?

Accepted Answer

The family-wise blow-up. Each null test is a 1\% coin. With 100 independent tests, \mathbb{P}(	ext{at least one significant}) = 1 - 0.99^{100} \approx 1 - 0.366 = 0.634. Even at the stricter 1\% level, there is about a 63\% chance of at least one "discovery" from pure noise, because you took so many more shots. The expected number of false positives is 100 	imes 0.01 = 1: scan 100 dead signals and, typically, exactly one looks exciting. Bonferroni correction. To hold the family-wise error rate (the chance of any false positive) at 5\%, test each signal at level \alpha / m = 0.05 / 100 = 0.0…

A hundred signals at the 1% level

Your answer

More Statistics questions