Vision
From pixels to CNNs: edges, convolutions, augmentation, classification
In this chapter, you will learn to
- Handle an image as an array of pixels
- Understand convolution and build a CNN
- Augment your data without cheating on evaluation
Tools: PyTorch, torchvision
16 exercises
Each exercise is graded automatically, in your browser. The full statements are available once you have an account.
What this is for in real life
To a computer, a photo is just an array of numbers: teaching it to see something in there is one of AI's great successes. - Your phone. Filters, portrait mode, face unlock: each time, a program transforms or…
Photo negative
The user of your app wants a "negative" filter, like the old negatives of photographic film: bright areas become dark and dark areas become bright. Write a function
negative(image)that takes a grayscale image (2D…Farewell, colors
Your "negative" filter works on grayscale images. But a user uploads a color photo (a 3D numpy array of shape height × width × 3, with Red, Green, Blue channels). The filter doesn't work on 3D. You first need to…
Edge detector
The user of your app wants to detect the edges in an image. An edge is a place where the brightness changes abruptly between neighboring pixels: the border of an object, a line, a boundary between two regions. Write…
The pattern scanner
You have just joined the imaging team of an observatory. You are handed a grayscale photograph, a huge one, and a question that sounds simple: the coastline the geologists are after: that sharp transition between dark…
The eye with many filters
Your scanner works, but one thumbnail recognizes only one thing. The geologists are back: they no longer want just coastlines, but also the vertical fractures of the terrain, the diagonals of glacial landforms, and…
The Disguise
Your convolution layers feel like a thing apart, very different from the dense layers
W·x + byou know. Here is a challenge: prove they are not. Take a convolution: a kernelkernel(small matrixkh × kw) slides…Too Deep to Learn
A shallow network learns your task without trouble. You want to do better: you stack 40 layers for a far more powerful network. Surprise: it learns much worse, stuck predicting the average, as if it refused to…
Seeing wider without growing
You are continuing your work at the imaging observatory. Your scanners detect sharp transitions well, the ones that fit within three pixels. But the geologists come back with a structure of an entirely different kind: a…
Perfect framing
You are building a photographer robot for a portrait studio. The robot receives photos of various sizes (50x80, 200x150, etc.) and must send them to a facial-recognition AI that only accepts 32x32 pixel images. A…
Harmonization
The portrait studio uses three different cameras. Problem: each camera produces images with very different brightness and contrast. Camera A produces dark images (pixels between 10 and 50), camera B bright images…
Image multiplier
The studio has only 100 portrait photos to train its AI, but at least 500 would be needed. Buying new photos costs too much. The problem: with so little data, the AI learns the 100 photos by heart instead of…
Lesson: data augmentation, on train, never on test
Making random variants of the training images is called data augmentation. In practice you do not store the 500 images: the transformations are drawn on the fly, at every epoch, so the model never sees exactly…
Lesson: a CNN in PyTorch
Your pattern scanner slid a filter over the image with two
forloops, and you were the one choosing the filter. A convolutional network (CNN) does the same computation, but it learns its filters, just as…Photo sorter
You work in a post office. Thousands of letters arrive each day and the handwritten postcode must be read to sort them. Each digit is a real handwritten-digit image (the MNIST set), grayscale 14x14 pixels, flattened…
The fair's cash box
On Saturday, the high school held its fair. At the stalls, people did not pay in euros but in plastic tokens, bought at the entrance. In the evening, each stall put its tray of tokens on a table and photographed it from…