Why does optional stopping without a sequential design invalidate classical p-values?
PICTURE THIS: DATA SPLIT
Fit on train, tune on val, report on test once.
Simple meaning
Classical p-values assume the sample size or stopping rule was fixed in advance.
WHY — P-value instead of guessing?
Why interviewers care about P-value:
on P-value.
the situation, the default choice, and one exception - that reads as experience.
Name the idea, why it exists, then one short example.
End with when you use it and one common pitfall.
STEPS — What happens step by step?
Before you speak the answer, walk the interviewer through these steps:
- 1Classical p-values assume the
sample size or stopping rule was fixed in advance.
- 2If you stop at
the first significant look, the chance of ever crossing 0.05 under the null is far above 5 percent.
- 3Sequential tests, alpha spending,
or always-valid inference methods exist specifically to allow monitoring.
- 4Give an example
One tiny concrete case you can say aloud.
- 5Common mistake
What juniors usually get wrong.
- 6Close
When you pick this over the alternative.
EXAMPLE — See it in action
Here's a short line you can speak, broken into clear beats:
Note: Adapt this scaffold to your own project — keep it under 60–90 seconds.
Key takeaway
Classical p-values assume the sample size or stopping rule was fixed in advance. If you stop at the first significant look, the chance of ever crossing 0.05 under the null is far above 5 percent.