Offline Evaluation Revisited

• Structure validated time-based splits for ML
• Backtest models on historical serving windows
• Avoid leakage that inflates offline purity

• Offline is the only place strictness is affordable — inspect everything, re-label, reseg.
• Time-split validation mimics production better than random folds.
• History can be replayed — the ultimate regression suite.
• A leaky offline metric ships a false champion.

You requalify in the simulator (offline) — repeatable, cheap, any scenario, fail loudly — long before the real flight. The simulator only helps if it faithfully models the aircraft (no leakage, right scenarios); a generous simulator breeds a reckless pilot.

• Split by time: train on the past, validate on the future.
• Walk-forward/backtesting: retrain-and-validate across many time cutpoints.
• Evaluate on windows that mirror serving pressure (seasonality, promotions).
• Leakage audit: target-scoped features, row shuffles, cached labels.
• Gate deployment on minimum strict thresholds with held-out history as the baseline.

• A fraud model gates on walk-forward backtests across 12 quarters before every release.
• A forecasting team replays historical Black Friday-related windows to test seasonality handling.
• A credit model only ships if leakage-audited offline default metrics survive a time-ordered holdout.

• Always time-split by event timestamp, never random folds.
• Backtest across many cutpoints to expose instability, not just a point estimate.
• Audit features for target leakage at every release.
• Mirror production-serving conditions in the validation window where possible.

• Random CV on temporal data — leaking the future into training.
• A single backtest cutpoint, blessing/starving a model by accident of window.
• Leaky engineering features (post-event joins) inflating offline scores.
• Ignoring seasonality: one window can't represent a year.

• Offline perfect, online abysmal — hunt leakage: temp features, cached labels, join-after events.
• Backtest oscillates wildly — the model is overfitting window-specific quirks; tighten regularization.
• Validation good, fresh-window bad — window drift; widen the backtest span.