Output Architecture
Where scores land matters as much as how they're made
Batch predictions are only useful if well-written: versioned columns, metadata alongside scores, and a storage layout that consumers read reliably. Output architecture decides whether downstream tools trust the numbers.
What you will learn
- Define a prediction output schema with versioning and metadata
- Choose storage layout by how consumers read
- Keep outputs immutable and addressable
What is this?
Output architecture is the contract of a batch job's result: the columns (prediction, probability, model version, data version, run id, timestamp), their types, the storage layout (table, files, keys), and immutability guarantees. It decides whether a warehouse can trust and lineage the numbers.
Why do you need it?
- A score without version/metadata is a number nobody can question, fix or roll back.
- Consumers query differently — columnar tables vs files vs key lookups — layout follows them.
- Immutable, addressable outputs make re-runs and audits straightforward.
- Schema evolution needs a plan or every downstream read breaks at once.
Mental model
A lab report with lot numbers and units, signed
Prediction output schema
A bare number means nothing clinically; the report carries patient, units, lab, version and date. Downstream doctors (consumers) trust and re-verify it because the provenance is printed on the page.
How it works
- The output schema is defined: prediction columns plus metadata columns (model, data, run, ts).
- Storage layout chosen by access: partitioned table for SQL, object files for bulk, keys for point lookup.
- Writes go to versioned locations with no overwrite of prior windows.
- Consumers declare expected schema at read time; schema drift surfaces as a contract failure.
- Lineage metadata lets any output row be traced to a run and model.
Prediction output anatomy.
In production
- A warehouse table partitions by date + run id; analysts query 'latest run' via view.
- An object-store layout keeps last N runs; consumers pin a run id for reproducibility.
- A schema registry catches a drift before it silently misaligns a join.
Best practices
- Always emit metadata columns: model_version, data_version, run_id, ingested_at.
- Partition/name outputs by run so 'latest' is a view, not an overwrite.
- Validate consumer-visible schema at both write and read time.
- Keep a schema registry or manifest per run.
Common mistakes
- Scores without provenance columns — numbers no one can trace.
- Overwriting the previous run's data, destroying the delta view.
- Schema silently evolving between runs, breaking joins downstream.
- Pointing all consumers at 'latest' file path that changes semantics.
Debugging tips
- Joins mismatch — check whether the output schema version changed between runs.
- 'Numbers changed for no reason' — confirm the run id/model version in the metadata, not assumptions.
- Consumers read stale — the 'latest' view may reference an old partition; verify by run id.
Interview checkpoint
Hands-on lab
- Emit scores with provenance metadata
- Write run-scoped outputs without overwrite
- Read back a pinned run reproducibly
Version Prediction Outputs
35 min · intermediate
Quick quiz
Which of these belongs in every prediction output?
Overwriting last run's output is fine as long as the numbers are newer.