Universal Approximation

• State the universal approximation theorem precisely
• Distinguish representation capability from optimization and generalization
• Use the theorem's limits to reason about model capacity needs

• It justifies why neural nets can, in principle, fit arbitrary input-output relationships.
• It reframes failures: if a net can't fit, it's likely optimization or data, not representation power.
• It explains the role of width: enough hidden units = enough 'basis pieces' to tile any function.
• It clarifies why depth, while not strictly needed, makes representation dramatically more sample-efficient.

With a big enough brick box you CAN build any shape. But that says nothing about the construction manual (optimization), how long it takes (learning complexity), or that you'll pick the right bricks by luck (random init).

• ReLU/tanh units chop space into regions; each region maps to a linear piece of the output.
• More units = more pieces = closer fidelity to the target function (finer approximation).
• The theorem covers all continuous f: pieces tile to any tolerance ε > 0.
• Depth adds compositional complexity — represent the same f with far fewer total units by composing features.
• Neither the number of pieces nor the guarantee extends to discrete/noisy/unbounded functions at training time.

• Choosing architecture width for arbitrary regression fits uses the 'pieces' intuition (input dims × output dims × complexity).
• It justifies adding hidden layers when the mapping requires composition (e.g. images → labels), not just tiling.
• Explainability and distillation arguments lean on expressiveness claims about MLPs as weak baselines.

• Use the theorem for model-selection reasoning, not guarantees: it sets the ceiling, LR sets the floor.
• When a net underfits a smooth function truly (loss plateaus high), add width first — cheapest expressiveness lever.
• Keep the domain compact: scale/normalize inputs so activation saturation doesn't silently break coverage.
• Prefer depth where features compose hierarchically; width where the function is inherently piecewise-linear.

• Quoting the theorem as 'my model CAN fit anything, so why is val bad' — that's optimization/regularization's fault.
• Expecting one-hidden-layer nets to beat deep nets in practice on high-dimensional data.
• Ignoring that 'compact domain' is about bounded, not unbounded inputs.
• Confusing approximation in theory with generalization on unseen data.

• If a smooth ground-truth regression function underfits at width 1000, check optimizer and init before width again.
• Fit a known analytic function (sin, 2D surface) — a working sanity check for representational ceiling.
• Measure the train loss's low-water mark; it's the representation+optimization ceiling, not the model's expressiveness alone.
• If ultra-wide fails but deep passes, the mapping was compositional — depth was the better lever.