Multi-Cloud and Portability
Portability is a design decision, not an accident
Multi-cloud isn't about running everywhere at once — it's designing so data and pipelines can move: open formats, vendor-neutral execution, and an exit plan priced before you commit.
What you will learn
- Design data to live in open formats on any object store
- Keep compute portable (SQL, Python, open engines) across clouds
- Price and test the exit before committing
What is this?
Multi-cloud strategy ranges from 'active-active across clouds' (rare and expensive) to the more common 'design-portable': default everything to open formats (Parquet/Iceberg/Delta), portable engines (Spark, Trino, DuckDB) and neutral orchestration, so moving or failing over to another cloud costs a migration, not a ransom. The opposite — proprietary formats + vendor-exclusive APIs — makes the 'second cloud' a fiction and the exit a horror show.
Why do you need it?
- Negotiating power and resilience: your data isn't held hostage by one vendor's pricing.
- Regulatory/data residency: some datasets legally must live in a specific cloud or region.
- Cost arbitrage only works if you can actually leave — portable designs make the threat real.
- Failover: an open-format standby copy in a second region is a real DR story, not paperwork.
Mental model
Owning your furniture vs renting a furnished apartment
Portable data design
A furnished rental is easy today; moving day, you own nothing to take. Portable cloud design is owning the furniture — open formats, neutral code — so 'moving' is a logistics problem, not a data loss problem.
How it works
- Data portability: Iceberg/Delta/Parquet on object storage with a portable catalog (open/metastore-agnostic).
- Compute portability: SQL (Spark SQL, Trino, dbt) and Python/PySpark that run on any engine/cloud.
- Orchestration portability: DAGs defined in code (Airflow/dagster/Prefect) run against cloud-agnostic operators.
- Identity portability: use workload identity federation so keys aren't cloud-entangled.
- Exit test: run a dry 'read-everything-on-cloud-B' exercise quarterly to validate assumptions.
- Cost model: egress, compute and storage prices per cloud feed a continuous total-cost-of-platform burn-down.
Portable stack: one data universion, many execution substrates.
In production
- A fintech stores all raw+curated data as Iceberg on S3 and runs both Spark and BigQuery external tables against it — no vendor owns the format.
- A startup keeps its dbt models cross-engine and executes them on DuckDB locally, on Athena in dev, and on a warehouse in prod — the same manifest everywhere.
- Regulators require EU data residency; the org keeps a paired GCP region by exporting Iceberg snapshots there — the data remains accessible to the primary pipeline.
Best practices
- Default to open formats and open catalogs; treat proprietary-only features as a documented exception.
- Write business logic in the most portable language (SQL over cluster-manipulating glue).
- Run a quarterly portability drill: can you read this quarter's data with another engine/cloud?
- Publish an exit/potential-bolt plan: cost, timeline, migration steps — before the contract locks it down.
Common mistakes
- Two clouds but one space-shaped datalake: a laptop of vendor-native formats tied to each.
- Framing portability as 'no vendor features ever' — you lose legitimate value; allow a small vetted set.
- Testing portability only in theory; real drills find the hidden format/lifecycle links.
- Ignoring egress: cross-cloud scans can cost more than the switch you're planning.
Debugging tips
- If another engine can't read your delta file, check the writer/reader version skew and table format spec.
- Portability drill fails → grep for vendor-specific path prefixes and session/auth calls in jobs.
- Cross-cloud lag spikes → check for catalog read amplification across regions rather than raw object counts.
- Exit cost surprises → run your cost model with actual egress + compute lists for 30 days before trusting it.
Interview checkpoint
Hands-on lab
- Write a portable Iceberg-style table in Parquet
- Read it back through a different 'engine' API
- Prove the same query returns the same numbers across readers
Write Data That Another Engine Can Read
30 min · intermediate
Quick quiz
Which choice maximizes portability?
Running in two clouds only helps if the exit is actually feasible.