Running the Experiment

• Launch experiments safely behind feature flags with a ramp plan
• Monitor exposure, SRM and guardrails during the run
• Explain peeking and early-stopping rules and follow them

• A flag misconfigured mid-run (or whole segments flag-on for both groups) quietly ruins the comparison.
• SRM and exposure monitors catch machinery failures while they're fixable, not at the post-hoc review.
• Peeking converts the 5% alpha into a rolling firehose of false discoveries.
• Ramps protect users and the data: a full blast on hardware that can't handle it distorts every metric.

You set the timer when you start cooking and trust it. Opening the oven every minute to see if it's done doesn't cook faster — it makes the roast's doneness unknowable.

• Ship behind a flag; the bucketing hash and the flag use the same user identity so assignments are stable and consistent.
• Verify exposure: count users receiving each variant daily; logs must reflect actual deliveries, not target splits.
• Monitor balance and guardrails with dashboards, but keep the primary metric behind an 'analysts only' door to stop peeking-by-default.
• Let the planned duration elapse in full — no early stop for significance; use sequential designs only if pre-planned.
• At the end, run the pre-registered analysis, then follow a holdout or long-term tracking lane for the biggest decisions.

• A checkout experiment ramps 1%→100% over four days, and the exposure monitor catches a Safari-only flag bug at 12:30, pausing the run.
• An experimentation platform auto-pauses a test when SRM p<0.001, forcing the owner to diagnose before re-running.
• A mission-critical monetization change ships with a 30-day holdout of 1% of users for long-term effect checking.

• Ramp deliberately — small doses first to catch bugs, then scale; shutdown is cheaper than recovery.
• Log exposure at assignment-time and at event-time; the two differ is a monitoring signal itself.
• Pre-commit the stopping rule: fixed duration, or a pre-registered sequential design with spending boundaries.
• Keep a run log: launches, ramps, incidents, config changes — the review needs the trail.

• Stopping the moment a p dips below 0.05, then 'rerunning' when it wobbles back — the p-hacking treadmill.
• Turning the feature flag off mid-run 'just for that one incident' without resetting the experiment clock.
• Reading results before verifying both groups actually received their variant (exposure blindness).
• Running a week with no weekend, month with no end-of-month activity — missing the seasonality the sample-size plan assumed.

• Exposure flat while traffic grows? The flag may be evaluated post-redirect, losing users at a checkpoint.
• SRM on a single device type points to bucketing by user_id colliding with device-derived identities.
• Guardrail moving early usually means a partial rollout bug, not the product change — verify features first.
• If the platform shows 'significant' daily but the pre-registered test disagrees, trust the pre-registered test.