Migration and Modernization
Migrating data platforms without the 3am horror story
Moving a warehouse to the cloud is an engineering program: land-and-expand or re-platform, reconcile old vs new, cut over by workload, and decommission with evidence.
What you will learn
- Choose a migration pattern (lift-shift, land-and-expand, re-platform) per workload
- Run a reconciliation gate before every cutover
- Decommission legacy systems with an evidence-based runbook
What is this?
Warehouse/cloud migration is moving workloads — data, pipelines, reports, users — from a legacy system (on-prem Teradata, Oracle DWH, another cloud) to a modern platform. Strategies range from lift-and-shift (same semantics, new host) to re-platform (rewrite in dbt/SQL on Iceberg) to `land-and-expand` (land cloud copy first, migrate consumers gradually). The discipline that makes migrations succeed is the same one that makes pipelines reliable: reconcile before, migrate incrementally by workload, and gate every cutover on evidence.
Why do you need it?
- Legacy systems cost too much and scale poorly, but no migration is cheap unless computed.
- Interfaces (BI, APIs, third-party) must keep working while the engine changes underneath.
- A bad cutover silently breaks month-end reports the CFO reads.
- Decommissioning Frees floor full legacy maintenance and licenses — the real ROI.
Mental model
Renovating a house while living in it
Warehouse migration
You don't knock down the kitchen before the new one works; you build the new kitchen (land-and-expand), move the pots incrementally, and only then rip out the old — with the old still usable as fallback until evidence says otherwise.
How it works
- Inventory: enumerate tables, jobs, reports, users, transforms and their dependencies (lineage!).
- Choose pattern per workload: lift-shift quick wins, re-platform the high-value processes.
- Stand up target: schema, security, golden copies; configure ingestion into the new platform (land).
- Migrate incrementally by workload with reconciliation per workload before/after (expand).
- Cut over: freeze source writes, backfill delta, validate counts/keys/values, switch consumers, monitor.
- Decommission: archive logs, verify 0 failures, flip DNS/app config, retire the legacy, document the runbook evidence.
The migration gate: reconcile before you switch consumers.
In production
- A Telco migrates 300 PowerBI reports from a Teradata appliance: they land a Snowflake copy first, run 2 weeks of parallel reconciliation, then flip reports by business unit over a month.
- An insurance org re-platforms its monthly reserves in dbt-on-Iceberg incrementally, keeping the old engine for the first 3 months as fallback.
- A bank decommissions a legacy ODS only after 90 days of zero reconciliation failures on the new platform and a signed-off runbook.
Best practices
- Migrate by workload, not by table — a workload is a report/use case with a consumer who signs off on cutover.
- Reconciliation is the ONLY cutover gate: counts, key sets and value totals match within tolerance, or you don't flip.
- Keep the old system live as fallback during the parallel period; roll back is faster than debugging live.
- Track the decommission plan from day 1: what backing store, who removed connections, what evidence proves 'done'.
Common mistakes
- Big-bang 'migrate everything next weekend' — the classic source of 3am rollbacks.
- No baseline: nobody knows the old counts, so 'different' can't be distinguished from 'better'.
- Migrating reports but forgetting scheduled downstream extractions that point at the old engine.
- Decommissioning on a calendar date instead of on evidence, then resurrecting broken services for months.
Debugging tips
- When a consumer sees different numbers post-cutover, diff the reconciliation report for that workload before touching anything.
- Use lineage to find every job touching a migrated table — schedules are the top source of 'the new system broke it'.
- Run the golden 3-number test (totals, counts, key sets) daily during parallel period.
- Keep old-neww mapping tables with version even after cutover — auditors will ask how you proved parity.
Interview checkpoint
Quick quiz
Which is the correct cutover gate for a migrated workload?
Keeping the legacy system as a rolling fallback during parallel operations costs more but de-risks migration.