Advanced vision

Detection, segmentation, self-supervised, ViT, CLIP: beyond classification

In this chapter, you will learn to

• Go beyond classification: locate and segment objects
• Use models that learn without labels
• Connect images and text (CLIP)

Tools: PyTorch, torchvision, Hugging Face Transformers

10 exercises

Each exercise is graded automatically, in your browser. The full statements are available once you have an account.

1. What this is for in real life

   Knowing "there is a pedestrian" is not enough for a car: it needs to know where they are, and which pixels are road. Advanced vision moves from one label per image to understanding the whole scene. ![The same street…

2. Object counter

   Your CNN classifier can recognize an object when it fills the whole image. But now you are given a large image that contains several objects at unknown positions. The CNN alone is no longer enough: you also need to…

3. Too many boxes!

   Your sliding window works, but it produces dozens of detections for a single object. That's because neighboring windows overlap and all detect the same object. You have 30 boxes around the same cat! The problem: how…

4. The pros' detector

   You built detection by hand: a sliding window to scan the image, then NMS to eliminate the redundant boxes. You now know what happens inside a detector. Time for the tools the pros actually use. YOLOv8…

5. The cutter

   You are working on autonomous driving. The system must understand every pixel of the image: is it road, a pedestrian, a vehicle, the sky? This is no longer classification (one label per image) but per-pixel…

6. The splice

   Your segmentation works, but the results are "blurry" -- the edges of the objects are imprecise. The encoder destroys the fine details (sharp edges, small structures) to capture the global context, and the decoder…

7. Cutting up for real

   You cut up an image by hand, pixel by pixel. Here is the tool the pros use: DeepLabV3, a pre-trained segmentation network that assigns a class to every pixel of a real image. Open the Colab notebook…

8. The visual puzzle

   You have a self-attention mechanism that works very well on sequences of words. Could it be applied to images? Problem: a 32x32 image contains 1024 pixels. Self-attention has a quadratic cost ($O(n^2)$), so 1024 tokens…

9. The matchmaker

   You manage a photo library with millions of images. A user types "a dog running on the beach" and wants to find the matching images. Problem: images and text live in totally different spaces -- a vector of pixels and a…

10. The automatic caption

    The matching works: you can find which caption goes with which image. But now you are asked to generate a new caption for a never-before-seen image -- not pick it from a list, but invent it word by word. You have: -…