A 40% lift appeared overnight. Be suspicious, not thrilled

Question

An A/B test reports a stunning 40% lift in revenue, far larger than any change the team has ever shipped. Everyone is excited and wants to roll it out immediately.

Invoke Twyman's law and explain what to do before believing an extraordinary experimental result.

Accepted Answer

Twyman's law. "Any figure that looks interesting or different is usually wrong." The more surprising and exciting a result, the higher the prior probability that it's an artifact, a bug, a data glitch, or a broken assumption, rather than a genuine effect. Extraordinary claims should raise your suspicion first and your champagne second. Why this is rational, not cynical. Real product changes almost never move a top-line metric like revenue by 40%; the base rate of such effects is tiny. Meanwhile the base rate of instrumentation bugs, logging errors, and analysis mistakes is not tiny. By Bayes'…

A 40% lift appeared overnight. Be suspicious, not thrilled

Your answer

More Statistics questions