Metrics and Parameters
Log the inputs, measure the outputs, honestly
Parameters are what you set, metrics are what you measure — but only useful metrics are logged per iteration, comparable across runs and honest about the split they came from.
What you will learn
- Choose which parameters and metrics to log in a run
- Record metric curves over steps, not just final values
- Attach the evaluation split to every metric so comparisons stay valid
What is this?
Parameters are the configuration of the run — hyperparameters, data windows, seeds. Metrics are measurable outcomes — loss, accuracy, latency. Production-grade tracking logs parameters with defaults, metrics as curves over steps where relevant, and always attaches the evaluation context each metric came from.
Why do you need it?
- Final-metric-only logging hides how a model got to its answer (overfitting, plateau behavior).
- Comparisons are invalid if metrics come from different splits — context is part of the number.
- Logging defaults prevents the #1 investigation failure: 'I never set that'.
- Curves enable early-stopping analysis and regression hunting between checkpoints.
Mental model
A race car's telemetry
Tracked ML metrics
Lap time alone (final metric) can't explain a loss — you need RPM, speed and tire temp over each lap. Curves and per-step logs are the telemetry that explains the final number.
How it works
- Parameters are logged once at run start: kernel, C, lr, batch, data window, seed.
- Metrics are logged as scalar values and, where useful, with a step (epoch or batch number).
- MLflow lets you log the same metric at multiple steps to build curves that the UI plots.
- Tags mark run purpose: champion, candidate, debug, data-window-range.
- Evaluation metrics should record which split and data version computed them (test-v1 vs test-v2).
What a well-logged run records.
Examples
Logging a curve and a named eval
for epoch in range(epochs): loss, acc = train_step() mlflow.log_metric("train_loss", loss, step=epoch) mlflow.log_metric("train_acc", acc, step=epoch) final = evaluate(test_set)mlflow.log_metrics({"test_acc": final.acc})mlflow.set_tag("eval_split", "test_v1")The loss curve exposes overfitting; the eval-split tag keeps future comparisons honest. Final metrics are necessary but not sufficient.
In production
- A yearly retrain logs the same metric set so year-over-year comparison is apples-to-apples.
- An NLP team logs validation perplexity per epoch and selects checkpoints mid-run.
- An autoscaling study logs serving latency alongside model quality in the same run.
Best practices
- Always log defaults for every parameter, even untouched ones.
- Include a seed and data version as parameters — they change results materially.
- Log both train and validation loss so overfitting is visible in the curve.
- Make metric names stable across runs ('f1_at_0.5' not 'f1-today').
Common mistakes
- Logging only the best epoch's metric, hiding the overfit signature.
- Comparing metrics from different splits or data versions and drawing conclusions.
- Logging hyperparameters after training so the recipe records nothing.
- Name drift in metrics making cross-run charts meaningless.
Debugging tips
- Curve looks like a machine gun? You logged a metric for both train and val under the same name.
- Metric 'improves' every run but test is flat? Check whether eval comes from a changing split.
- Autolog double-logs? Disable conflicting manual logs or autolog.
Interview checkpoint
Hands-on lab
- Log parameter defaults and per-epoch metric curves
- Tag runs as champion vs candidate
- Compare curves across two runs from the store
Telemetry: Log Curves and Tags
25 min · intermediate
Quick quiz
Which of these is a parameter rather than a metric?
Logging the data version with each metric makes comparisons considerably more trustworthy.