Guardrail Metrics

• Identify which metrics deserve guardrail status in a given product
• Define thresholds that turn guardrails into decision rules
• Evaluate an experiment balancing OEC win vs guardrail breach

• Single-metric optimization is how products get ruined a percentage at a time.
• Guardrails convert values ('we care about reliability') into enforceable measurements.
• They pre-commit the team to honesty: you can't 'accidentally' miss a breach when the threshold was written before the test.
• They make the trade-off visible at review time so the business can consciously accept or reject it.

The OEC is maximizing value; the guardrail is the cap that says no matter how profitable the idea, it can't blow the line item. Both are written before the quarter starts.

• Inventory the product's promises beyond the OEC: reliability (latency, uptime), trust (fraud, refunds), retention (existing-user churn), cost (infra, support).
• Select the top 2–5 that could plausibly move in this experiment's domain.
• Define a threshold: absolute or relative cap (e.g. latency p95 +5% max, support tickets +3% max).
• Pre-register them alongside the OEC so the readout is mechanical.
• At review, publish a 5-line verdict: OEC result, each guardrail delta vs threshold, and the combined decision.

• A push-notification experiment ships only when uninstall rate stays flat — a classic mobile guardrail.
• A ranking change is approved with the guardrail 'zero-correction-rate must not rise', holding organic relevance hostage.
• A pricing pilot uses 'support ticket volume per user' as a guardrail, catching confusion-driven refunds before revenue metrics glow.

• Choose guardrails the experiment could plausibly touch — every test doesn't need all five.
• Pre-commit thresholds in the experiment's kickoff doc, with one decimal place of specificity.
• Use statistical plus business thresholds: guardrail deltas get CIs too, and review reads the whole bracket.
• Re-run the guardrail read after the novelty window; early 'no impact' can turn into late regressions.

• Declaring guardrails after seeing results, so breaches magically never count.
• Guardrailing only the metrics that are convenient to measure, missing the real risk (e.g., fraud).
• Treating a threshold breach as a veto without checking uncertainty — a +8% support bump with a huge CI needs a follow-up, not a panic.
• Letting a 'great' OEC win excuse a guardrail breach permanently instead of iterating on the fix.

• If a guardrail shows a breach, segment it (which plan, which flow) to locate the mechanism before freelancing 'improvements'.
• Guardrails can spike from experiment-side effects (a broken flow) — verify exposure before blaming the product change.
• Check guardrail computation timing; metrics computed on different windows can look like regressions that aren't.
• When OEC and guardrail conflict repeatedly, re-examine whether the guardrail is measuring the right risk.