Cloud Data Services
Advanced 18 min read Cloud Data Services

Migration and Modernization

Migrating data platforms without the 3am horror story

Moving a warehouse to the cloud is an engineering program: land-and-expand or re-platform, reconcile old vs new, cut over by workload, and decommission with evidence.

migrationreplatformland and expandcutoverreconciliationdecommission
Before we start

What you will learn

  • Choose a migration pattern (lift-shift, land-and-expand, re-platform) per workload
  • Run a reconciliation gate before every cutover
  • Decommission legacy systems with an evidence-based runbook
Introduction

What is this?

Warehouse/cloud migration is moving workloads — data, pipelines, reports, users — from a legacy system (on-prem Teradata, Oracle DWH, another cloud) to a modern platform. Strategies range from lift-and-shift (same semantics, new host) to re-platform (rewrite in dbt/SQL on Iceberg) to `land-and-expand` (land cloud copy first, migrate consumers gradually). The discipline that makes migrations succeed is the same one that makes pipelines reliable: reconcile before, migrate incrementally by workload, and gate every cutover on evidence.

Motivation

Why do you need it?

  • Legacy systems cost too much and scale poorly, but no migration is cheap unless computed.
  • Interfaces (BI, APIs, third-party) must keep working while the engine changes underneath.
  • A bad cutover silently breaks month-end reports the CFO reads.
  • Decommissioning Frees floor full legacy maintenance and licenses — the real ROI.
Think about it

Mental model

Imagine

Renovating a house while living in it

Think of

Warehouse migration

You don't knock down the kitchen before the new one works; you build the new kitchen (land-and-expand), move the pots incrementally, and only then rip out the old — with the old still usable as fallback until evidence says otherwise.

Mechanics

How it works

  • Inventory: enumerate tables, jobs, reports, users, transforms and their dependencies (lineage!).
  • Choose pattern per workload: lift-shift quick wins, re-platform the high-value processes.
  • Stand up target: schema, security, golden copies; configure ingestion into the new platform (land).
  • Migrate incrementally by workload with reconciliation per workload before/after (expand).
  • Cut over: freeze source writes, backfill delta, validate counts/keys/values, switch consumers, monitor.
  • Decommission: archive logs, verify 0 failures, flip DNS/app config, retire the legacy, document the runbook evidence.

The migration gate: reconcile before you switch consumers.

Inventory
Land
Reconcile
Cutover
Decommission
Real world

In production

  • A Telco migrates 300 PowerBI reports from a Teradata appliance: they land a Snowflake copy first, run 2 weeks of parallel reconciliation, then flip reports by business unit over a month.
  • An insurance org re-platforms its monthly reserves in dbt-on-Iceberg incrementally, keeping the old engine for the first 3 months as fallback.
  • A bank decommissions a legacy ODS only after 90 days of zero reconciliation failures on the new platform and a signed-off runbook.
Guidelines

Best practices

  • Migrate by workload, not by table — a workload is a report/use case with a consumer who signs off on cutover.
  • Reconciliation is the ONLY cutover gate: counts, key sets and value totals match within tolerance, or you don't flip.
  • Keep the old system live as fallback during the parallel period; roll back is faster than debugging live.
  • Track the decommission plan from day 1: what backing store, who removed connections, what evidence proves 'done'.
Watch out

Common mistakes

  • Big-bang 'migrate everything next weekend' — the classic source of 3am rollbacks.
  • No baseline: nobody knows the old counts, so 'different' can't be distinguished from 'better'.
  • Migrating reports but forgetting scheduled downstream extractions that point at the old engine.
  • Decommissioning on a calendar date instead of on evidence, then resurrecting broken services for months.
When things break

Debugging tips

  • When a consumer sees different numbers post-cutover, diff the reconciliation report for that workload before touching anything.
  • Use lineage to find every job touching a migrated table — schedules are the top source of 'the new system broke it'.
  • Run the golden 3-number test (totals, counts, key sets) daily during parallel period.
  • Keep old-neww mapping tables with version even after cutover — auditors will ask how you proved parity.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Which is the correct cutover gate for a migrated workload?

2

Keeping the legacy system as a rolling fallback during parallel operations costs more but de-risks migration.

Go deeper

Official documentation