Lineage and Provenance

• Distinguish column/table/field-level lineage from row-level provenance
• Use lineage for impact analysis before changing a model
• Attach provenance (source, run id, timestamp) to outputs for audits and replays

• Before any schema change you can see exactly which consumers and transforms will be affected.
• When a number looks wrong, lineage tells you where to look upstream in minutes, not days.
• Auditors require provenance for critical data: which source, which run, which version produced this report line.
• Replay and repair jobs need lineage to know what to recompute without rebuilding everything.

Lineage is the map showing which factories feed which warehouses; provenance is the lot number stamped on each crate. You use the map to plan, and the lot number to recall a bad batch.

• Capture lineage at build time from the transform engine (dbt manifest, Spark query plans, orchestrator logs).
• Store in the catalog as nodes (tables/columns) and edges (dependencies) with direction and version.
• Auto-generate field-level lineage from SQL parsing where possible; fall back to table-level.
• Attach provenance attributes to outputs: source keys, run id, processing timestamp, schema version.
• Expose reverse lineage (impact) and forward lineage (root-cause) to users and CI.
• Recompute lineage on every deploy so the map matches the code that actually ran.

• An analyst spots an anomalous revenue figure and traverses forward lineage to the exact source column and transform step.
• A platform team runs impact analysis before renaming payment_status: the report lists every failing model and dashboard.
• A regulator asks for a report line's origin; the provenance attributes show source table, extraction run 2026-09-22T02:00 and schema v7.

• Persist lineage from the actual execution (build-time), not hand-drawn diagrams that rot.
• Keep field-level lineage where feasible; table-level lineage still wins over none.
• Stamp provenance at the earliest landing step so it survives every transform.
• Version lineage: regenerate on deploy so the map reflects current code, and keep history for audits.

• Storing lineage as static YAML that drifts from the code within weeks.
• Collecting table-level lineage only, so column-level questions stay unanswerable.
• Ignoring consumer edges — lineage that forgets 'who reads this' can't do impact analysis.
• Recording provenance for rows but not exposing it in the UI/tooling, making it invisible.

• Start from the symptom (a value) and follow reverse lineage until you find the first step where the value deviates from expectation.
• Compare lineage across two runs: a changed transform path often explains a changed number.
• When a join output doubles, look at the lineage: a new multi-valued source or a missing dedupe step.
• Use provenance run ids to replay one bad run's inputs for a minimal reproduction.