Training a language model

Predict the next token, temperature, decoding: what an LLM really is

In this chapter, you will learn to

Each exercise is graded automatically, in your browser. The full statements are available once you have an account.

What this is for in real life
Behind ChatGPT, Claude or Gemini there is a single task, repeated billions of times: guessing what comes next in a text. - Chat assistants. ChatGPT, Claude, Gemini and Mistral AI's Le Chat were all first trained…
The Vocabulary That Learns Itself
A language model reads neither letters nor words: it reads subwords, chunks whose list, the vocabulary, is not fixed in advance but learned from a corpus. Splitting into letters would give endless sequences…
The completion machine
You have a model that, for each token in a sequence, gives the probability distribution over the next token. How do you go from "predicting the next word" to "generating an entire text"? You are given: - vocab: a list…
The thermostat
Your text generator works, but you have two problems: - The deterministic strategy always produces the same boring, repetitive text - The random strategy sometimes veers into gibberish by picking improbable…
Your Own GPT
This is the capstone: in the Colab notebook you build your own GPT from scratch at the character level (token + position embeddings, Transformer blocks with causal self-attention, output head), you train it on…
On the Checkout Belt
You are developing the recommendation system for a supermarket chain. Each customer checks out with a basket: a set of items, each already represented by a feature vector. Your model must summarise this basket into a…