Deep Learning
Intermediate 16 min read Deep Learning

PyTorch Basics

The framework that made autograd feel native

PyTorch is the de-facto framework for research and production deep learning: automatic differentiation, GPU acceleration and a Pythonic imperative loop. This concept covers tensors, autograd, modules and a minimal training loop.

PyTorchtensorsautogradnn.Moduledataloadertraining loop
Prerequisites: Python NumPy Deep Learning Basics
Before we start

What you will learn

  • Manipulate tensors and move data between CPU/GPU
  • Explain autograd and how .backward() computes gradients
  • Write a minimal nn.Module training loop
Introduction

What is this?

PyTorch is an open-source deep-learning framework exposing tensor operations with automatic differentiation. You define a model as an nn.Module, run forward computations imperatively, call loss.backward() to populate .grad on each parameter, then step an optimizer. Its eager, Pythonic style and reusable ecosystem (torchvision, transformers, lightning) made it the default for research and increasingly for production serving.

Motivation

Why do you need it?

  • Autograd removes hand-written backprop — you write the forward pass and the gradients appear.
  • Eager execution fits debugging and research (print anything), and exporting via TorchScript/ONNX still serves in production.
  • GPU/deploy portability: same python code runs on CPU or CUDA, and mmaps to mobile.
  • Deep ecosystem: pretrained models in torchvision/hub/transformers are one import away.
Think about it

Mental model

Imagine

A spreadsheet that remembers every formula's lineage

Think of

PyTorch autograd

You build the forward calculation like a formula chain; the engine traces each intermediate, and when you ask for the gradient everywhere, it walks that chain automatically — you never hand-write the derivative formulas.

Mechanics

How it works

  • torch.tensor/like: n-dimensional arrays with dtype, device ('cpu'/'cuda') and requires_grad flag.
  • Autograd builds a graph on tensors with requires_grad=True as you compute; .backward() fills .grad.
  • nn.Module wraps parameters (nn.Parameter) and forward(); optimizers (Adam, SGD) read .parameters().
  • Dataset/DataLoader batches data; potentially with distributed samplers; transforms drop in.
  • The training loop is five lines: zero_grad → forward → loss → backward → optimizer.step().

A minimal PyTorch training loop.

DataLoader
Forward
Loss
Backward
Step
Real world

In production

  • Image models export from nn.Module to TorchScript/ONNX and serve via FrameworkServer/elite.
  • Hugging Face transformers runs on PyTorch for fine-tuning and via ONNX Runtime for latency-bound serving.
  • Distributed training (DDP/FSDP) lets the same nn.Module scale from one GPU to many nodes.
Guidelines

Best practices

  • Move data to the device once (.to(device)) rather than per-step for speed.
  • Call zero_grad() before each backward or you accumulate gradients across steps.
  • Set model.eval() / model.train() to toggle dropout & batchnorm mode.
  • Wrap hot loops with torch.no_grad() for inference and eval to cut memory.
Watch out

Common mistakes

  • Forgetting .train()/.eval() flips — evaluation silently uses training-mode dropout.
  • Sharing the same tensor mutation in the graph (in-place ops) breaking autograd.
  • Training on CPU because .to('cuda') was skipped — silent 50x slowdown, no error.
  • Calling optimizer.step() without zero_grad and wondering why loss oscillates.
Root cause analysis

Common issues

Error
RuntimeError: Expected all tensors to be on the same device
Root cause

Input stays on CPU while model weights are on CUDA

Fix

x = x.to(device); y = y.to(device) after loading the batch.

Error
Gradients are None on step
Root cause

requires_grad=False, or zero_grad() wasn't called / loss detached from graph

Fix

Check requires_grad on inputs, then make sure backward runs on the final loss.

Error
Validation loss looks like training loss
Root cause

Model stuck in train() mode with dropout/batchnorm active

Fix

Wrap eval in model.eval() and torch.no_grad().

When things break

Debugging tips

  • Print shapes/dtype at each op — the most common PyTorch bugs are dimension mismatches.
  • Assert loss is finite and descends on one batch before enabling the scheduler.
  • Use torch.set_grad_enabled(False) for inference; detect accidental graph builds with a memory check.
  • Move batch to the same device as the model and confirm via x.device == next(model.parameters()).device.
Interviews

Interview checkpoint

Practice

Hands-on lab

Train Your First PyTorch Model
30 min · intermediate
In progress
Objectives
  • Build a tensor pipeline and check autograd works
  • Define, compile and train an nn.Module classifier
  • Evaluate on validation with model.eval() and torch.no_grad()

Train Your First PyTorch Model

30 min · intermediate

0/5 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

What does optimizer.step() do in a training loop?

2

You must call zero_grad() before backward() to avoid accumulating gradients.

Go deeper

Official documentation

PyTorch 60-minute blitz
docs
PyTorch docs
docs