Training Pipelines
Intermediate 14 min read Training Pipelines

Data Pipeline Integration

Where data engineering hands off to ML

Training pipelines consume data pipelines: freshness, delivery semantics and backfills from the data side set what a training run can trust. Integration is cooperation on contracts, not shared servers.

data pipelinetraining triggerdata freshnesscontracthandoff
Prerequisites: Training Pipelines Dataset Versioning
Before we start

What you will learn

  • Define the handoff contract between data and training pipelines
  • Trigger training from data events correctly
  • Handle backfills and late data without double-training
Introduction

What is this?

Data pipeline integration is the deliberate handoff between the data plumbing (ingestion, cleaning, features) and training: a contract of freshness, schema and versioned references that the training pipeline validates before running. It answers 'when can I trust this data enough to train?'.

Motivation

Why do you need it?

  • Training on half-arrived or unvalidated data wastes hours of GPU on garbage.
  • Explicit contracts stop the 'data was late again' blame game.
  • Backfills and late-arriving rows are real events; sensors and idempotency handle them.
  • Schema changes on the data side need to reach training as a contract change, not a surprise.
Think about it

Mental model

Imagine

A handoff in a relay race

Think of

Data → training handoff

The baton (validated dataset) must cross the line at the right time in the right zone. If the incoming runner is late or drops the baton (schema/freshness broken), the next runner doesn't just run faster — they inspect the baton first.

Mechanics

How it works

  • The data pipeline publishes versioned datasets to a known location with a manifest.
  • A contract file records schema, freshness SLA and version.
  • The training pipeline's sensor waits for a new version or lands a bounded dataset.
  • Validation steps assert the contract before any training compute starts.
  • Backfills arrive as new data versions, re-triggering only the affected models.

The contract-guarded handoff between pipelines.

Data pipeline
Contract
Sensor
Validate
Training DAG
Real world

In production

  • A nightly batch publishes 'daily_training@v83'; the training sensor reads the manifest and gates on freshness.
  • A schema change bumps the contract version, blocking stale training configs automatically.
  • A backfill publishes a corrected version; models that consumed the old version get re-triggered with a reason tag.
Guidelines

Best practices

  • Give every handoff a manifest (schema, counts, freshness, version).
  • Validate the contract before spending compute, in the training pipeline.
  • Design for late/backfilled data as first-class events.
  • Tag training runs with the data version they consumed.
Watch out

Common mistakes

  • Training pipeline reaching into the data pipeline's internals to 'sniff' for files.
  • Missing sensors — training starts on whatever happened to be there.
  • Backfills overwriting a version instead of publishing a new one, confusing lineage.
  • Contract enforced by conversation, not code.
When things break

Debugging tips

  • 'Trained on the wrong data' — check the manifest version consumed vs the latest published.
  • Sensor stuck — the upstream publish step may have failed silently; check its logs.
  • Backfill double-trained — confirm new-version semantics, not in-place overwrite.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

What should the training pipeline validate before spending GPU time?

2

A backfill should be published as a new dataset version rather than overwriting an old one.

Go deeper

Official documentation

Feast — offline training data integration
docs