How many of your discoveries are fake?

Question

A researcher screens m = 500 candidate signals. Suppose about 450 are truly worthless (null) and 50 carry a real edge. If you keep every signal with p < 0.05 and apply no correction, how many false discoveries should you expect? How does controlling FDR at 10\% change the picture?

Accepted Answer

No correction. Each of the 450 worthless signals independently trips the p < 0.05 wire with probability 0.05, so the expected number of false positives is 450 	imes 0.05 = 22.5. Suppose your test has reasonable power and correctly flags, say, 40 of the 50 real signals. Then you report about 40 + 22.5 \approx 62 "discoveries," of which roughly 22.5, more than a third, are pure noise. Acting on that list, you would deploy capital behind dozens of signals that are coin flips. This is the core danger of uncorrected mass testing: the false positives scale with the number of nulls, which is usuall…

How many of your discoveries are fake?

Your answer

More Statistics questions