Experiment Design and Randomization

• Choose a randomization unit consistent with the metric's grain
• Detect and diagnose sample ratio mismatch (SRM)
• Plan for novelty and primacy effects with AA tests and holdouts

• Randomizing at the wrong unit turns a clean test into a confounded one (session-level bucketing leaks users across groups).
• SRM — observed 55/45 instead of 50/50 — means assignment was affected by the treatment; the whole comparison is suspect.
• Novelty (a new UI attracts clicks simply because it's new) makes early wins fake and early losses permanent.
• AA tests (control vs control) verify the tooling itself before you trust any A/B it produces.

A chemist calibrates the pipette with water before adding reagent. An AA test is the same: run the machinery against itself — if it reports a difference between two identical groups, the lab is dirty.

• Choose the randomization unit at the level of a stable identity that maps to your metric (user_id for per-user metrics).
• Bucket deterministically (e.g. crc32(user_id)%100) so users stay in one group across days instead of resegmenting.
• Check SRM: if the observed assignment ratio deviates beyond chance (χ² test), investigate before reading results.
• Run an AA test on the same metric to establish the expected baseline noise and validate the platform.
• Account for novelty/primacy with ramped rollouts, longer durations, or novelty-window analyses.

• A marketplace randomizes at 'account' level for a pricing test because revenue aggregates per seller account.
• A social app bucketed at user level so a user sees one consistent feed — cross-session consistency beats statistical purity here.
• An experiment platform automatically refuses to read results when SRM exceeds 2 sigma, paging the experiment owner.

• Randomize on the unit matching the metric grain and the product experience you vary.
• Add SRM monitoring as a mandatory pre-read: a χ² p<0.01 in assignment ratio blocks the analysis.
• Pre-run an AA test when adopting new tooling (bucketing, SDK, metric pipeline).
• Ramp slowly and let the test run through a full week (weekday+weekend cycle) to cover behavioral rhythms.

• Bucketing per session so an active user appears in both groups, voiding user-level metrics.
• Ignoring an SRM because 'the numbers look fine' — SRM is the canary that ingestion or assignment broke.
• A/B-testing a feature flag that other users can toggle on/off, contaminating exposure.
• Ending a test on a Friday after 3 days when the metric is strongly weekly-seasonal.

• SRM showing? Check three places: the bucketing hash stability, the flag delivery, and the analytics SDK dedupe.
• If an AA test turns out 'significant', debug the metric pipeline, not the experiment.
• Exposure drift? Verify the feature flag is evaluated for the full population the experiment claims.
• Novelty effects show as a spike that decays; compare week 1 vs week 2 before and after the spike.