Write the hypothesis, metric and decision rule before running an A/B test.
When to use itBefore any experiment, so results can't be reinterpreted after the fact.
Hypothesis
We believe [change] for [users] will cause [effect] because [evidence].
Primary metric
The one metric that decides the result, and the minimum change worth detecting.
Guardrail metrics
Metrics that must not get worse.
Design
Audience, split, sample size and run time.
Decision rule
What you will do if the result is positive, flat or negative.