The test isn't significant, can we just run it longer?
Asked at Two Sigma
Your planned 14-day A/B test of a new execution algo ends at , just short of significance. The PM says, "we're so close, let's just keep it running until it hits ."
What is wrong with extending the test this way, and what are the sound alternatives?
Show a hint
Stopping early the first time inflates the false-positive rate. Does continuing until do anything different?
Your answer
This one is open-ended. Work it through, then check your reasoning against the full solution.