Scaling Laws

• State the empirical scaling relationships
• Compute data/compute/size trade-offs
• Apply scaling reasoning to fine-tuning budgets

• Your judgment 'should we take a 3x model?' reduces to the same loss-vs-cost curve.
• It explains why data quality/quantity matters as much as size.
• It reframes fine-tuning: small-target tasks often get more from data than from a giant base.
• Emergent behavior traces to threshold scaling — capability cliffs rather than smooth gains.

The first 10 minutes of sanding transform the rock; the next 10 hours remove microns. Capability per dollar follows the same diminishing curve — spend at the elbow, not the asymptote.

• Loss ≈ a·S^α for size S, and similar in data D and compute C, with fitted exponents.
• Chinchilla optimality ≈ 20 tokens of data per parameter.
• Compute-optimal scaling grows params AND data together, not just one.
• Bigger models with identical data underperform better-fed smaller ones.
• Frontier behavior is bought with extreme compute; margins shrink per parameter.

• Choosing a checkpoint size on a budget — data/tooling often yield more than model size.
• Fine-tune spent per-task: collect more examples before scaling the base.
• Explaining 'bigger model sometimes worse' to stakeholders via data adequacy.

• Before buying a bigger model, estimate the gap with a few devset probes.
• Balance token budget with task data, not vanity parameter count.
• Use scaling intuition for evals: small runs extrapolate poorly; measure, don't guess.

• Buying the biggest model when the data/task fix is better.
• Treating Chinchilla as universal — it governs pre-training budgets, not fine-tuning.
• Ignoring that gains concentrate at the frontier, where latency/cost live.

• Larger model doesn't beat small on YOUR task → gap often data/format, not capacity.
• Overfit strong → the model is far larger than the fine-tune data justifies.
• Loss curves flatten early → you're on the data-limited side; add tokens before params.