Transformers
Attention Q·K·V, self-attention: the architecture that dominates text and vision
In this chapter, you will learn to
- Understand the attention mechanism
- Assemble a Transformer and see why it dominates text and images
- Tell apart models that read the whole text from those that generate it
Tools: PyTorch, Hugging Face Transformers
7 exercises
Each exercise is graded automatically, in your browser. The full statements are available once you have an account.
What this is for in real life
Nearly every AI people talk about today rests on a single architecture: the Transformer. This chapter has you build it piece by piece. - Chat assistants. ChatGPT, Claude and Gemini are Transformers. The "T" in…
The simultaneous interpreter
You are working on a machine translation system. Your RNN encoder reads a source sentence word by word and produces a hidden vector at each time step. So far, you pass only the last hidden vector to the decoder, as…
Everyone looks at everyone
Your attention mechanism works: the decoder can consult the encoder states. But you still depend on an RNN to encode the sentence, which is slow (sequential) and still loses information on long sequences. New…
Predicting without cheating
Your self-attention now serves a new purpose: generating text, word after word. You train the model to predict, at each position, the next word in the sentence; during training, you give it whole sentences and each…
Lesson: causal or bidirectional, two families of Transformers
The constraint "never consult the future" is called causal attention. It splits Transformers into two large families, according to what each position is allowed to see: | | Decoder (GPT) | Encoder (BERT) |…
From several angles
Your self-attention works, but it suffers from a limitation: for each position, it produces only one set of attention weights, hence only one way of relating words to each other. Yet a word maintains several types…
Piecing the thread back together
You are refining your translation system. So far, at each step, the decoder produces a word taken from a fixed dictionary. A new task comes up, though: you are handed texts whose fragments (bits of sentence) arrived out…