Reading Results and Launching

• Structure an experiment readout for a decision meeting
• Balance OEC gains, guardrail health, effect size and segment signals
• Use holdouts and follow-ups to catch long-term and novelty effects

• A single p-value is only one piece of the decision; the rest is judged magnitude, health and risk.
• Short-run wins reverse: novelty vanishes, learning effects accumulate, holdouts reveal true long-term lift.
• A disciplined readout culture is what lets a company scale experiments without scaling bad decisions.
• Documenting verdicts (inclusive of nulls) is how organizations build institutional memory instead of reinventing each week.

The pilot doesn't 'hope the runway is long enough'; she runs checklists: airspeed, fuel, wind, runway length — each a separate line, all mandatory before touching down. Shipping an experiment is the same multi-line check.

• Report the primary result: point estimate, 95% CI, p and the MDE threshold — one line.
• Report guardrails: each delta with its CI against the pre-agreed threshold — no narration, just deltas.
• Check segments not for winners but for risk: did any large segment regress badly (concentration risk) or miss the ship entirely?
• Integrate: strong gain + clean guardrails + stable segments → ship with long-term monitoring; unclear → iterate or run a follow-up; worse or breaching → kill.
• Post-launch: keep a holdout or monitoring lane at 1–5% traffic for ≥30 days to catch surprise long-term effects.

• A search feature ships at a modest +1.5% click lift after a 30-day holdout confirms the effect persists once past novelty.
• A checkout variant looks insignificant overall but a 20%-of-traffic segment (international) regressed 4%; the team fixes the locale issue then re-tests.
• A monetization test shows +6% revenue but a guardrail breach in support volume; the team ships a fixed variant and abandons the original.

• Read effect size and CI before p; 'statistically significant but small' and 'insignificant but sized like our MDE' are different realities.
• Publish verdicts including nulls into a central log — an experiment that taught you a dead end is an asset.
• For ship decisions, quote the worst-case bound: 'even at the CI's lower edge, this clears break-even'.
• Create a 30/90-day post-launch measurement habit: metrics learned in experiments decay that you must catch.

• Shipping on significance alone while the lower CI bound sits below a business-critical threshold.
• Ignoring the novelty window: shipping a UI experiment at week-one 'significance' that flips negative at week three.
• Killing a variant because the primary is insignificant when an examined segment produced a clear, explainable win worth a follow-up.
• Never writing the verdict down, so six months later no one knows the test happened or what it showed.

• If post-launch metrics disagree with the experiment, check both the pipeline and the flag after launch — latched flags leak assignment.
• Re-run the readout with the novelty window excluded before finalizing a UI verdict.
• For holdout analysis, segment by holdout vs acquired cohorts; mixing cohorts muddies long-term effects.
• When a ship works in test but not production, look at product mix differences, not just flag config.