Neural networks

Perceptron, activations, multi-layer: the XOR thread

In this chapter, you will learn to

• Understand what a neuron computes and what it cannot do alone
• Stack layers and choose an activation
• Train a small network on a non-linear problem

Tools: NumPy

9 exercises

Each exercise is graded automatically, in your browser. The full statements are available once you have an account.

1. What this is for in real life

   The AIs everyone talks about are neural networks, and all of them are built from the same small brick, repeated millions of times. - Reading handwriting. In 1989, Yann LeCun and his colleagues at Bell Labs…

2. The logic gate

   You have a very simple machine. It takes two binary inputs ($x_1$ and $x_2$, each 0 or 1) and produces a binary output (0 or 1). The machine works as follows: - It computes $z = w_1 \cdot x_1 + w_2 \cdot x_2 + b$ - If…

3. The invisible boundary

   You are given points in 2D, each colored red (class 0) or blue (class 1):

4. Automatic learning

   In the previous problem, you found $(w_1, w_2, b)$ by hand. But if the data changes, you have to start all over again... Write a function train(X, y, n_iter) that finds the right weights automatically, starting…

5. The neuron that doubts

   The perceptron gives a blunt yes/no (hard threshold) and stalls as soon as the two groups overlap: no line separates them perfectly, so its rule goes around in circles. Let's give it some nuance. Instead of…

6. Two layers are better than one

   Remember XOR, impossible with a single machine. What if we chained two of them? Two-stage architecture: Stage 1: two machines in parallel: - $h_1 = \text{seuil}(w_1 \cdot x_1 + w_2 \cdot x_2 + b_1)$ -…

7. The XOR rematch

   In the previous problem, you defeated XOR by chaining two stages of threshold machines, tuned by hand: three neurons, eight scattered numbers, one point processed at a time. A real network never works that way. It lines…

8. The universal chain

   Your XOR network from the previous exercise has a fixed architecture: two stages, hard-coded. But nothing forces a network to have two layers, or layers of the same size. We want to be able to describe any depth and…

9. The secret ingredient

   So far, your network has used the step function between layers: 0 or 1, nothing in between. Problem: the derivative of this function is zero everywhere (and does not exist at 0). The gradient therefore says nothing…