Post-training

After pre-training: aligning and accelerating a language model

In this chapter, you will learn to

• Understand what is done to a language model after pretraining
• Teach a model from preferences
• Speed up generation

3 exercises

Each exercise is graded automatically, in your browser. The full statements are available once you have an account.

1. What this is for in real life

   A model fresh out of pretraining can continue a text, but cannot yet answer you. What turns it into a useful, reliable and fast assistant happens afterwards. ![The life of an assistant: pretraining, sample…

2. Two responses, one preference

   You are fine-tuning a conversational assistant that has already been pre-trained. Rather than numerical scores, your annotators give you pairs: for a single query, one response judged better (the preferred one) and one…

3. Several moves ahead

   You are optimizing the inference service of a large language model. On each pass through the network, the shared trunk produces one hidden vector per position, and an output head turns this vector into a distribution…