You found 30 significant signals. How many are real?

Question

A researcher tests 500 candidate signals at the 5% level and proudly reports 30 that came back "significant."

Estimate how many of those 30 you'd expect to be false positives if most signals are truly worthless, and explain what the count does and doesn't tell you.

Accepted Answer

The back-of-envelope. If a signal has no edge, it still trips the 5% test 5% of the time. If we suppose (pessimistically but realistically for blind data mining) that essentially all 500 are worthless, the expected number of false positives is 500 	imes 0.05 = 25. So out of 30 reported "discoveries," you'd expect around 25 to be pure noise. At most a handful might be real, and you can't tell which from the p-values alone. Finding 30 hits is barely more than chance would hand you; the result is close to what you'd get from testing 500 random-number streams. The sharper reading. The excess of …

You found 30 significant signals. How many are real?

Your answer

More Statistics questions