End-to-End Architecture Case Study

• Trace a full prediction from event to log across every plane
• Justify each architectural decision against cost, latency and reliability
• Design the monitoring and rollback story for a real-time product

• Fragments hide integration cost; a full trace shows where the platform actually earns its complexity.
• Real constraints — latency, label lag, cost — force the design decisions blueprints don't spell out.
• A worked company-shaped example is the fastest way to internalize the planes and their interfaces.
• The case study doubles as a blueprint for building and as a source of interview answers.

Every flight has a flight plan (training), is tracked live on radar (monitoring), rerouted when weather shifts (drift response), and rejected at the gate if the plane isn't cleared (registry gates).

• Data plane: Kafka streams each transaction; labeled outcomes arrive on a 24h lag; a snapshot dataset is versioned nightly for training.
• Training plane: a nightly Airflow/KFP pipeline trains an XGBoost model on the latest snapshot, evaluates against the current validation window, and compares to the incumbent in the registry.
• Registry + governance: candidates are registered with metrics and lineage; a quality gate plus a human-on-pager review promotes the winner to the 'champion' alias.
• Serving plane: an API fetches customer/merchant features from the online store, dynamic-batches requests into the model server, and returns a fraud score with reasoned metadata within the 150ms budget.
• Monitoring plane: data-quality checks on inputs, drift alerts on feature distributions, proxy metrics (suspicious-approval rate), and outcome-lag reconciliation against charged-back labels.

• The champion model's alias lets serving stay constant while training promotes new versions — deployment is one alias flip with a canary window.
• Batching at the model server keeps GPU cost per transaction low, and the exporter-side latency budgets drive the batching limits.
• When feature drift trips on merchant categories, the monitoring plane pages the model owner and shadows the champion against the drifted data before any promotion.

• Separate 'training data freshness' from 'serving feature freshness' in the SLA; they update on different cadences by design.
• Alias the champion: services depend on 'champion', not a version string, so promotion is atomic.
• Instrument outages and rollbacks as first-class events in the same monitoring plane as drift.
• Reconcile outcome-lag metrics: chargebacks arriving late must update earlier optimistic proxy numbers.

• Training on a snapshot while serving on fresh features — the classic skew tripping even a perfect pipeline.
• No champion/alias indirection, so every promotion is a code change in the serving tier.
• Monitoring live quality before outcome labels arrive, then never reconciling — the dashboards lie forever.

• When live scores degrade, diff the serving-feature distributions against the training snapshot first.
• In latency incidents, trace the request — feature fetch is the usual suspect, not the model.
• Verify the reconciliation job ran; a charged-back label backlog silently invalidates your quality dashboards.