Training with PyTorch
nn.Module, DataLoader, training loop, momentum/Adam, regularization
In this chapter, you will learn to
- Write a model and a full training loop in PyTorch
- Choose an optimiser, mini-batches and regularisation
- Diagnose a training run that does not converge
Tools: PyTorch
18 exercises
Each exercise is graded automatically, in your browser. The full statements are available once you have an account.
What this is for in real life
Nobody rewrites backpropagation for every project: researchers and engineers use a tool, and PyTorch is one of the most widely used. - AI labs. Created at Facebook (now Meta) and handed over to the PyTorch…
Lesson: tensors, NumPy arrays with two extra talents
Until now, your networks lived in NumPy arrays, and you wrote every derivative by hand. PyTorch stores numbers in tensors: arrays you handle almost like NumPy ones (
.shape, indexing,@,.sum(dim=0)instead of…Lesson: PyTorch just gives names to what you built
In the previous chapters, you built by hand every piece of a network: an autograd engine (your
Varclass, which remembers its operations and knows how to do.backward()), a forward pass…Farewell to numpy
In the previous chapter, you built a neural network by hand with numpy: forward pass, backward pass, weight updates... It works, but it is fragile and painful to maintain. Every new architecture means re-deriving the…
Data as a stream
Your network works on 4 XOR points. But in the real world, you will have 10 000, 100 000, even millions of examples. Impossible to load everything into memory and compute the gradient over the whole dataset at every…
Lesson: from "all at once" to mini-batches
In the exercise "Farewell to numpy", you passed all the data in one go:
model(X). With 4 XOR points, no problem. With a million points, it no longer fits in memory. You already know the solution: mini-batch…The network that will not converge
A colleague hands you his code: a small network that is supposed to classify points into 3 categories. It runs, never crashes, trains away quietly for 200 epochs... and stays stuck at 33% accuracy, barely the level of…
The pre-trained model
You have 50 images of cats and 50 images of dogs. Training a CNN from scratch with 100 images? Good luck getting past 55%. But someone gives you a pre-trained feature extractor: a network that has seen millions of…
Lesson: transfer learning, do not relearn what has already been learned
Training a vision network from scratch takes millions of images. You only have 100. The trick: reuse a network pre-trained on millions of images, which has already learned to turn an image into a vector of rich…
Save and load
You trained a model for 30 minutes. Your computer crashes. Everything is lost. Or: you train a model on your machine, but you need to use it on a production server. How do you transfer the model? The challenge…
Lesson: saving a model, its weights not the object
A trained model = an architecture (the layers) + weights (the learned numbers). We save only the weights: it is portable and robust. PyTorch exposes them as a dictionary, the
state_dict. Saving:Four-game tournament
A modelling tournament opens its four games to you. The rules are strict: a single training procedure, run as is on the four datasets, with no hand tuning from one game to the next. The datasets have the same 10…
The snowball effect
First day at the lab. You join a team that trains neural networks, and you are entrusted with their descent tool. Training a network means sliding its weights toward the bottom of an error landscape, knowing at each…
The smart step
Momentum helps, but it uses the same learning rate for all the weights. Yet in your network, some weights receive gradients 100 times larger than others. A single learning rate is either too large for some or too small…
The anti-cheat net
Your network reaches 100% on the training data, but only 60% on new data. It has memorized the examples instead of learning the general rule. The problem: how do you force a network not to rely too much on…
The luck of the draw
Same architecture. Same optimizer. Same dataset. And yet, depending on how you initialize the weights, training succeeds or fails completely. You are given
train(params), which trains a network (3 layers, 32…The normalizer
Your deep network trains well on some datasets, but on others the internal activations explode or collapse to zero from layer to layer, and training diverges. You are given
train_with_norm(norm_fn), which…The gradual brake
Your network learns too aggressively: with a fixed learning rate that is too large, training diverges (the loss explodes to NaN). If you lower it too much, learning becomes slow. It is as if the steps of the descent…