Loss functions & information theory

Entropy, cross-entropy, KL divergence, MLE, full pipeline

In this chapter, you will learn to

• Measure information and surprise (entropy)
• Understand where the usual loss functions come from (maximum likelihood)
• Compare two distributions (cross-entropy, KL divergence)

Tools: NumPy

11 exercises

Each exercise is graded automatically, in your browser. The full statements are available once you have an account.

1. What this is for in real life

   For an AI to learn, it needs a number that says how wrong it is, and that number comes from a surprising idea: information can be measured. - Morse code and compression. As early as the 19th century, Morse code…

2. The 20 questions game

   You are playing 20 questions with a friend. They are thinking of an object from a list. You can only ask yes/no questions. Case 1: There are 8 objects, all equally likely (12.5% each). How many questions do you need…

3. The classifier's surprise

   You have built a classifier that predicts 4 categories. For each test example, it gives you probabilities: for instance [0.7, 0.1, 0.15, 0.05] means “a 70% chance that it is category 0”. The true labels and the…

4. Lesson: softmax and cross-entropy

   The score you just invented has a name: cross-entropy. For an example whose true class is $k$, it is $-\ln p_k$, the model's surprise at the right answer; we average it over the examples. It is the loss of almost…

5. The gap between two worlds

   You have two probability distributions over 5 categories: - p: the true distribution (observed in the data) - q: the distribution predicted by your model You want to measure “how far q is from p”. This gap is…

6. Estimating a biased coin

   You find a coin on the ground. You flip it 200 times and record each result in the array flips (1 = heads, 0 = tails). Is the coin biased? If so, what is the probability $p$ of landing on heads? Your mission: 1.…

7. Why the mean?

   In the previous exercise, you found that the MLE of $p$ for Bernoulli flips is $\hat{p} = \bar{x}$ (the mean of the observations). Prove it rigorously. Let $x_1, x_2, \ldots, x_N$ be i.i.d. (independent and…

8. MLE = BCE

   In ML, a binary classification model is often trained by minimizing the Binary Cross-Entropy (BCE): $\text{BCE}(\theta) = -\frac{1}{N}\sum_{i=1}^{N} \Big[ y_i \ln(\hat{p}_i) + (1 - y_i) \ln(1 - \hat{p}_i) \Big]$…

9. MLE = least squares

   The classification loss, BCE, is obtained as a maximum likelihood for Bernoulli labels (this is the subject of the exercise "MLE = BCE"). Let's do the same for regression. We model a continuous target as…

10. The full pipeline

    You are an engineer at a medical start-up. You are given 10 blood samples, each described by 2 biological measurements. Some are healthy (label 0), others pathological (label 1). Your mission: build an automatic…

11. Diagnosis at scale

    The medical start-up is growing. Your first diagnostic system only juggled two clean-cut measurements; the lab now hands you its full records: 100 patients, each described by 5 raw biological measurements…