Backpropagation

Computation graph, backward pass, mini-autograd

In this chapter, you will learn to

• Break a computation into a graph and send gradients backwards
• Build a tiny automatic differentiation engine
• Check that a gradient is correct

Tools: NumPy

10 exercises

Each exercise is graded automatically, in your browser. The full statements are available once you have an account.

1. What this is for in real life

   A network has millions of settings, and to learn it must know which way to turn each one. Backpropagation answers for all of them at once, and it is what makes deep learning possible. - Every modern AI. The…

2. Sensors in cascade

   You are calibrating a measurement chain: a signal goes through a long series of processing stages, applied one after the other. Each stage is of one of two types: - ["lin", a, b]: it turns the value v into…

3. The split signal

   Your measurement chain, continued. To gain finesse, the calibration bench has been rebuilt: right after the input, the signal is now duplicated and sent into two processing branches mounted in parallel. Each…

4. The gradient spy

   You have a 2-layer network with ReLU. It produces an error on some data. You want to know: if I change one weight by a tiny bit, by how much does the error change? This is the fundamental question for training a…

5. The backward pass

   The previous problem required $2P$ forward passes for $P$ parameters. For a network with 1000 weights, that is 2000 evaluations. For a million weights, it is unusable. Challenge: find a way to compute ALL the…

6. The complete training

   You now have a fast way to compute the gradients. What remains is to use them so that the network learns on its own. The data forms two classes arranged in concentric circles, impossible to separate with a line.…

7. Lesson: your own Python objects, in five moves

   In the exercises that follow, you are going to build a new kind of number, one that remembers how it was computed. For that, Python lets you create your own types of objects: a class describes the blueprint, and…

8. The tensor that remembers

   When you compute z = x * y + w in numpy, the result z does not know where it came from. If you want to compute $\frac{\partial z}{\partial x}$, you have to redo everything by hand. The problem: what if the*…

9. The gradient rewinder

   In the previous exercise, your Var class learned to remember: each result keeps track of the operation that produced it and the Vars it came from. The full thread of the computation is recorded. You now want to put…

10. The autograd network

    You have an engine that computes gradients automatically. Put it to the test: can it train a neuron to classify data, without you writing a single derivative? The neuron computes…