Sequences & encoder-decoder

RNN, hidden state, encoder-decoder, translation: the missing memory

In this chapter, you will learn to

• Process a sequence with a memory (RNN, hidden state)
• Understand why a network's memory fades and how to fix it
• Turn one sequence into another (encoder-decoder)

Tools: PyTorch

11 exercises

Each exercise is graded automatically, in your browser. The full statements are available once you have an account.

1. What this is for in real life

   The meaning of a sentence, a melody or a curve depends on what came before, sometimes long before. - Your phone keyboard. Suggesting the next word means reading everything you have typed so far. In 2018, Google…

2. The missing memory

   You need to predict the next character in a text sequence. Your approach: look at the last $k$ characters and guess the next one. Characters are one-hot encoded: for an alphabet of $V$ characters, character $i$ is a…

3. The memory box

   To predict the next character of a text, a fixed window that only looks at the last $k$ characters forgets everything older. You need a fundamentally different approach. The idea: instead of looking at the last $k$…

4. The text generator

   Your memory machine can predict the next character. But can it write? The problem: generate text in a loop. Predict a character, append it to the sequence, predict the next one, and so on. But beware: always…

5. Lesson: sentences in PyTorch

   Your memory box read one character at a time, encoded as a vector of 0s and 1s, with given weights. To train this kind of network on real sentences, PyTorch provides three pieces. From numbers to vectors…

6. The dictionary

   You are a volunteer translator for an NGO. You are given a French-English dictionary and sentences to translate. Your first idea: replace each French word with its English translation, one by one. The dictionary is in…

7. The Summary That Forgets

   Word-by-word translation failed: you must first understand the whole sentence, then restate it. That is the idea of the encoder-decoder architecture: the encoder reads the entire sequence and compresses it into…

8. Looking Back

   The single summary of the previous exercise keeps the counts but loses the order. To restate a sequence in the right order, the decoder must be able to consult any position of the input when it needs to. That is…

9. Lesson: training an encoder-decoder

   An encoder squeezes the whole source sentence into a vector; a decoder draws the output sentence from it, word by word. To train both together, you write your own module: a class that inherits from nn.Module, declares…

10. Lesson: Hugging Face in four moves

    Training a real translator takes millions of sentence pairs and days of computation. Others have done it and share their weights on Hugging Face, the big public repository of models. The transformers library uses…

11. The real translation

    You built the encoder-decoder architecture and attention by hand. Here is a real production translator: MarianMT, a pre-trained Encoder-Decoder model. Open the Colab notebook, implement the translate(text)…