Open Table Formats

• Explain what an open table format adds on top of Parquet-on-storage
• Use snapshot isolation and time travel to recover from bad writes
• Read a table from multiple engines (Spark/BigQuery/DuckDB) without copying data

• ACID on the lake: a failed write never leaves half of a table visible.
• Time travel and rollback: step back to the snapshot before the bad job ran.
• Multi-engine reads: Spark, Trino, BigQuery and DuckDB all read the same Iceberg table in place.
• Incremental processing: read only changed files since a watermark instead of re-scanning everything.

A folder of files is a cabinet: you can rifle through it, but 'what is the current state?' has no single answer. A ledger (metadata log) tells you the agreed current state and every previous one — that's the snapshot/commit model of Iceberg or Delta.

• Commit: a writer atomically swaps the current metadata pointer (e.g., Iceberg's latest.metadata.json ref) to a new snapshot.
• Snapshots: every commit records the exact file list; old snapshots stay readable for time travel.
• Manifests: small files listing data files and their column/statistics for pruning.
• Compaction: background jobs merge small files so snapshots don't accumulate fragmentation.
• Optimistic concurrency: writers pass the expected base snapshot and fail (or retry) if it moved.

• A retail analytics team writes daily loads as Iceberg; when Tuesday's job duplicated rows, they rolled back to Monday's snapshot in minutes and re-ran with a fix.
• A media company points both Trino (interactive) and Spark (batch) at the same Delta table; the lake replaces a separate warehouse.
• A fintech uses incremental reads to stream only changed events since the last checkpoint into feature stores.

• Adopt the format once and standardize: mixed Iceberg/Delta across teams recreates byte-copy silos.
• Compaction on a schedule so snapshots stay lean and queries don't chase thousands of files.
• Keep a default branch and require explicit snapshots for risky experiments.
• Choose your catalog intentionally (AWS Glue, Nessie, Hive Metastore, or the engine's own) — that is where the table 'lives'.

• Treating open table formats as a partitioning tool instead of a transaction layer.
• Forgetting to expire old snapshots until storage balloons and orphan files pile up.
• Mixing writers that don't honor the format's isolation rules (e.g., writing Parquet beside an Iceberg table).

• Reader 'sees old data': confirm which snapshot the engine was pointed at (catalog vs direct path) before blaming the write.
• Dry-run a rollback: cannot start rewriting the table in place on a broken commit without a safe failover strategy.
• Small-file explosion after UPSERTS: run compaction and rebalance to restore scan efficiency.
• Reading via the wrong engine/catalog connector yields metadata mismatches — check engine's supported table formats.