Hypothesis and Metrics Selection
The metric you pick decides what the experiment can teach you
Every experiment starts with a falsifiable hypothesis, an overall evaluation criterion, and guardrail metrics. Choose the right primary metric and your experiment reads clearly; choose wrong, and the p-value is theater.
What you will learn
- Write a falsifiable experiment hypothesis with a direction and mechanism
- Select an OEC that captures value without gaming opportunity
- Define guardrails that bound collateral damage
A good experiment hypothesis is 'change X will move metric M by mechanism Y'. The OEC is the metric you optimize; guardrails are metrics that must not regress.
What is this?
Hypothesis and metrics selection is the design step that happens before randomization. You write a falsifiable hypothesis (if we do X, then metric M moves upward, because of mechanism Y), choose one overall evaluation criterion (OEC) that captures the value you care about, add guardrail metrics that must not regress, and pre-register both. The rest of the pipeline compares the OEC; the value of the experiment lives or dies here.
Why do you need it?
- An unfalsifiable hypothesis ('improve the UX') can't be tested or killed.
- The OEC decides what gets optimized; a click-based OEC produces click-farm style products.
- Guardrails convert 'did revenue drop?' from after-the-fact panic into a pre-agreed decision rule.
- Pre-registration prevents the 'I meant to measure this all along' hindsight bias that poisons experiment culture.
Mental model
Choosing a destination before driving
Choosing the OEC
If you don't decide where you're going before you leave, every road looks fine and you'll defend any stop. The OEC is the agreed destination; guardrails are the 'no hairpin turns' rules.
How it works
- Start from a real problem and a mechanism, not from 'we could test something'.
- Name the OEC: a metric where better = real value (e.g. 30-day retention, gross profit per user), avoiding metrics that reward manipulation.
- Set guardrails: metrics that reflect the product's other promises (latency, refund rate, existing-user churn).
- Record target group (which users are exposed), duration logic and pre-registered stopping rule.
- Post-launch, read the experiment against the OEC and guardrails, then decide.
From problem statement to a pre-registered experiment.
In production
- A shipping company's OEC is 'first-order on-time delivery per user' — clicking more buttons isn't the goal, delivery is.
- A search team tests result ranking with OEC = queries that lead to a user action + guardrail = zero-correction rate.
- A freemium vendor runs a paywall experiment with OEC=paid revenue per user and guardrail=uninstall & support volume.
Best practices
- Write the hypothesis with a direction: 'we expect X to increase M by at least Y%'.
- Prefer OECs that are product value metrics (retention, quality actions) over engagement proxies (clicks, time).
- Pick metrics with low measurement variance so the test needs a realistic sample size.
- Keep OECs and guardrails to a handful; a 12-metric dashboard ships ambiguity, not decisions.
Common mistakes
- Choosing the OEC after seeing results ('we found revenue moved too').
- An OEC that rewards harming users (spam notifications → engagement) without an effective guardrail.
- A compound OEC with no definition of the aggregate — you can't defend the weighting later.
- Testing two mechanisms in one variant so a null result can't teach anything.
Debugging tips
- If the OEC is flat, verify the variant was actually delivered to the target group (exposure check).
- If the OEC moves but guardrails move too, treat it as a trade-off question, not a launch.
- Recreate the OEC computation in SQL/pandas independently and compare to the experiment tool.
- Confirm the OEC's grain (per user? per session?) matches the randomization unit.
Interview checkpoint
Quick quiz
Which is the strongest OEC for a checkout redesign?
Choosing the OEC after seeing the results is a valid analysis practice.