Inference & ML systems

Why generating is expensive, and how large models fit in memory

In this chapter, you will learn to

• Understand what is expensive when a model generates text
• Estimate the memory and compute of a large model
• Split a model that is too large for one machine

6 exercises

Each exercise is graded automatically, in your browser. The full statements are available once you have an account.

1. What this is for in real life

   When an assistant answers you, every token it writes forces it to read through tens, sometimes hundreds, of gigabytes of weights. That memory traffic, far more than the arithmetic, sets the speed and the price of…

2. Why Generating One Word Is Expensive

   A mystery of inference: generating a single token with a big model does relatively little compute, and yet it is slow. The reason is not the compute, it is the memory: the GPU spends its time reading the…

3. The out-of-reach budget

   You are planning the training of a large language model. The full run would cost months of compute and a fortune in GPU hours: launching it blind is out of the question. Before committing such a budget, you need to know…

4. Billed by the bit

   You oversee a fleet of sensors in orbit. Each probe sends back long streams of readings, but the radio link is billed by the bit: every transmitted bit costs money, and the invoice climbs fast over millions of…

5. Too big for a single card

   You are deploying a neural network that has grown too large to fit on a single graphics card. The chosen workaround: split it into S successive slices of layers, the stages, and assign each stage to a different…

6. Sharing to Fit in Memory

   A real Transformer has many attention heads (32, 64…). At inference, to generate fast, we keep in memory the keys K and values V of every head, at every position already seen (the cache). Problem: this cache…