The Data Cloud and Managed Platforms
The platform that markets itself as easy
Modern cloud vendors (and vendors like Snowflake and Databricks) sell data platforms as a service: storage, compute, catalog, governance and integration in one product with consumption-based pricing.
What you will learn
- Describe the services a 'data cloud' bundles (storage/catalog/compute/governance)
- Compare compute-and-storage separation vs classic warehouses
- Evaluate vendor lock-in, egress and interoperability trade-offs
What is this?
A 'data cloud' is a managed platform that fuses storage, compute, catalog, governance and data sharing under one control plane — snowflake, Databricks Lakehouse, Google BigQuery, Microsoft Fabric. Compute and storage are typically separated (you pay per query-hour and per byte stored independently), an elastic catalog tracks every object, and governance (access, lineage, quality) is a first-class service rather than a bolt-on.
Why do you need it?
- It removes the 'build five tools and glue them' tax: catalog, warehouse and governance come integrated.
- Elastic, separated compute means you pay for what you run, not a fixed cluster that idles.
- Data sharing across org boundaries becomes an exchange (no one emails CSVs).
- Serverless consumption lets small teams get enterprise-grade quality without a platform team.
Mental model
Booking electricity instead of building a generator
A data cloud platform
You don't wire a house to your own power plant — you buy metered electricity from a grid you share. Data clouds meter queries and storage the same way and sell you a governance switchboard on top.
How it works
- Storage in a vendor-native format (or open formats like Parquet via Iceberg/Delta on object storage).
- Compute clusters spin up per query and scale horizontally; warehouses = virtual compute endpoints.
- Catalog centralizes schemas, tables, volumes, functions and lineage across the org.
- Governance layer unifies row/column access, masking, tagging and compliance policies.
- Data sharing/cleanrooms allow governed access to other people's data (and your own) without copying.
- Integrations/connectors ingest from SaaS and databases; open catalogs expose it to Spark/Athena/Trino.
Layers inside a typical data cloud.
In production
- A ten-person fintech stands up Snowflake, loads SaaS data via a connector, and serves CFO reports on day one — no platform team, no self-hosted orchestration.
- A retailer shares a governed product catalog with suppliers via a cleanroom, so exact revenue numbers cross orgs without copying files.
- An analytics team runs ad-hoc BigQuery queries on inbound events while a marketing warehouse auto-scales to zero at night.
Best practices
- Ground decisions in total cost of ownership: storage, compute, egress and idle-cluster behavior.
- Prefer open table formats (Iceberg/Delta) so a future exit doesn't ransom your data.
- Standardize on one catalog for lineage and access even if compute is multi-engine.
- Set up per-role cost allocation so each team sees its own consumption bill.
Common mistakes
- Assuming 'serverless' means free — runaway queries and overnight cubes burn real money.
- Copying everything into one vendor and then discovering egress is expensive to leave.
- Letting each team provision its own isolated workspace, fragmenting the catalog.
- Skipping cost alerting until the monthly invoice shocks the finance team.
Debugging tips
- Slow query → check if compute autoscale met the query or concurrency throttled it to a queue.
- Cost spike → run the vendor usage explorer by warehouse/time-of-day; overnight backfills are the usual suspect.
- Data invisible → the object is in storage but the catalog hasn't refreshed (external table metadata drift).
- Access denied → check the platform's tag/masking policy layer, which can hide columns even with table grants.
Interview checkpoint
Hands-on lab
- Build a cost model separating storage and compute
- Simulate autoscale to zero and backfill spikes
- Derive a per-query and per-month bill comparison
Model Data Cloud Consumption Pricing
30 min · beginner
Quick quiz
Which benefit best describes compute-storage separation in data clouds?
Open table formats are a hedge against platform lock-in.