IOAI Training: Learn AI by solving problems

The problem in brief

TensorFlow Playground is a web page where you train a small neural network on 2D toy datasets and watch it learn. A friend has saved five broken setups as Playground links, and for each one you must explain what goes wrong and then fix it, changing as little as possible and giving the link to your fixed version. All five setups use the hardest dataset of the Playground, the two interleaved spirals, for classification. This is the theory part of the 2024 Hungarian AI Olympiad national final, worth 20 points.

There is no official answer key. To check every claim and every fix, we wrote a small NumPy replica of the Playground, copying its source code (github.com/tensorflow/playground): same spiral generator (500 points, labels $\pm 1$ ), same input features, same initialisation (weights uniform in $[-0.5, 0.5]$ , biases $0.1$ ), $\tanh$ output, squared loss, plain SGD with gradients averaged over each batch. The only thing we cannot reproduce is the browser's random seed, so every setting was run with 5 to 8 different seeds. Numbers below are medians over seeds; they will differ slightly from what you see on screen, but the conclusions do not.

How you are scored

Question 1, 2 and 4 are worth 4 points (2 for the explanation, 2 for the fix), question 3 is worth 2 points (explanation only) and question 5 is worth 6 points (a fix with every change explained). The jury grades the written answers.

One number helps reading all the losses. The Playground's loss is the mean of $\frac{1}{2}(\text{output} - \text{label})^2$ with labels $\pm 1$ . So:

a network that outputs 0 everywhere has loss $\frac{1}{2}$ ;
a network that outputs +1 everywhere (or −1 everywhere) is wrong by 2 on half of the points, so its loss is $\frac{1}{2}\cdot\frac{1}{2}\cdot 2^2 = 1$ ;
a loss of 0.1 means the network is right with some confidence on almost every point.

Our answers, each verified in the replica:

question	diagnosis	fix
1	learning rate 3 is far too large: the neurons saturate	learning rate 0.03
2	network 2-2 on raw $x_1, x_2$ is too small for a spiral	8 and 8 neurons
3	linear activations: the network is one linear model whatever its size	(non-linear activation)
4	linear activations, and a learning rate too small	tanh, rate 0.03, more inputs, 90 % training data
5	linear, single feature, tiny rate, 10 % of data	tanh, rate 0.01, 4 more inputs, 90 % training data

Question 1: a learning rate of 3

Setup. tanh, network 4-3, inputs $x_1, x_2, x_1^2, x_2^2$ , batch 10, learning rate 3, 70 % training data. The test loss never decreases.

Explanation. The learning rate is about 100 times too large. Gradient descent updates every weight by $-\eta \, \partial L / \partial w$ . With $\eta = 3$ , the very first updates are huge: the weights jump to large values, the inputs of the tanh neurons become large (the features $x_1^2$ and $x_2^2$ reach 25), and the neurons saturate at $\pm 1$ . A saturated tanh has derivative $1 - \tanh^2(z) \approx 0$ , so almost no gradient flows back any more and the network is stuck. In our replica the loss sits at about 1.0 (left panel below): that is exactly the loss of a network answering the same class for every point. From time to time a lucky update unsticks it, the loss jumps, and the next oversized step throws it back.

Fix: change only the learning rate to 0.03 (the Playground default): link. Both losses now go down steadily: median test loss after 1000 epochs 0.115 (0.96 with rate 3). A rate of 0.1 works as well (0.106) and 0.01 is slower (0.20 after 1000 epochs); 0.3 and 1 are still too large (0.28 and 0.41).

Training and test loss with learning rates 3 and 0.03

Question 2: a quick drop, then a plateau

Setup. tanh, network 2-2, inputs $x_1, x_2$ only, learning rate 0.03, 50 % training data. The training loss falls quickly, then stagnates.

Explanation. The network is too small for the spiral (underfitting). Each first-layer neuron computes $\tanh(w_1 x_1 + w_2 x_2 + b)$ : a soft step across one straight line. With two such neurons, everything the network does afterwards is a function of two projections of the plane, which can at best cut it into a few wedges. A spiral that turns almost twice around the centre needs many more pieces. The quick drop is the network finding the best boundary it can draw (from 0.5 to about 0.44), and the plateau is the best a 2-2 network can do: the problem is capacity, not optimisation. Waiting longer or tuning the learning rate does not help (rate 0.1: 0.40; ReLU: 0.46).

Fix: more neurons, network 8-8: link. The training loss keeps decreasing to a median of 0.015 (test 0.11). Smaller enlargements help in proportion: 4-4 reaches 0.34, 6-6 0.18, 8-6 0.035. Adding a third layer of 2 neurons does nothing (0.44): two neurons wide is the bottleneck. An alternative within the same idea is to give the 2-2 network richer inputs, but adding $\sin x_1, \sin x_2$ alone is not enough (0.34).

Question 3: more neurons, no improvement

Setup. Linear activation, network 4-4, inputs $x_1, x_2, x_1^2, x_2^2$ .

Explanation. With a linear activation, each layer computes $h = W h_{\text{prev}} + b$ , and a composition of linear maps is linear:

W_3 (W_2 (W_1 \phi + b_1) + b_2) + b_3 = W' \phi + b' .

So however many layers and neurons you add, the whole network is equivalent to one neuron: $\text{output} = \tanh(w \cdot \phi + b)$ with $\phi = (x_1, x_2, x_1^2, x_2^2)$ . Its decision boundary $w \cdot \phi + b = 0$ is a conic (ellipse, parabola or hyperbola). No conic separates two spirals. Extra neurons only add redundant parameters for the same function family. Our replica confirms it: the training loss is 0.476 whatever the width (1-1, 2-2, 4-4 or 8-8), while with tanh it falls from 0.48 to 0.003 as the width grows. This is why neural networks need a non-linear activation.

Training loss against width, linear versus tanh activation

Question 4: stuck above 0.1

Setup. Linear activation, network 4-3, inputs $x_1, x_2, x_1x_2, x_1^2$ , learning rate 0.001, 70 % training data. The test loss never goes below 0.1 in 1000 epochs.

Explanation. Two problems stack up.

As in question 3, linear activations collapse the network into one linear model on $(x_1, x_2, x_1 x_2, x_1^2)$ : a single conic boundary, which cannot follow a spiral. The loss plateaus near 0.48.
Even with a good model, a learning rate of 0.001 is too small to train in 1000 epochs: with tanh and rate 0.001 the test loss is still 0.50 at the end.

Fix (network unchanged). Fixing only the activation and the rate is not enough: tanh with rate 0.03 reaches 0.33, because a 4-3 network is small for a spiral if it must build every curve itself. So we also give it richer inputs and more data:

activation tanh (non-linearity, question 3);
learning rate 0.03 (fast enough for 1000 epochs, but stable, question 1);
add the inputs $x_2^2$ , $\sin x_1$ , $\sin x_2$ . The sines are periodic, the radius of the spiral grows steadily, and $\sin x_1$ , $\sin x_2$ give the network ready-made "stripes" that the small network combines into the curved arms, instead of building them from scratch;
training data 90 %: 450 points along the arms instead of 350, so the gaps between training points are smaller and the test points are better covered.

link. Median final test loss 0.052, below 0.1 for 6 of our 8 seeds (7 of 8 at some point during training). A 4-3 network on a spiral remains sensitive to its initialisation: if one run fails, regenerate and run it again. ReLU instead of tanh works as well (0.055, 7 of 8 seeds).

Question 5: the hardest repair

Setup. Linear activation, one hidden layer of 7 neurons, only the input $x_1 x_2$ , batch 2, learning rate 0.00001, 10 % training data. Target: test loss at most 0.1 in 1000 epochs, without changing the hidden layer, without regularisation, keeping $x_1 x_2$ .

Every parameter of this setup is wrong in its own way, and each change answers one problem:

Activation linear → tanh. Without a non-linearity, the 7 neurons collapse into one (question 3), and with the single input $x_1 x_2$ the boundary is $x_1 x_2 = c$ , a hyperbola: the network can only say "same sign or opposite sign quadrant" (left panel below). A non-linear activation lets the 7 neurons each contribute a different bend.
Add the inputs $x_1$ , $x_2$ , $\sin x_1$ , $\sin x_2$ (keeping $x_1 x_2$ ). Even with tanh, $x_1 x_2$ alone carries no information about which arm a point is on: the spiral turns through all four quadrants. One hidden layer of 7 neurons is small, so ready-made oscillating features (the sines, as in question 4) do much of the work.
Learning rate 0.00001 → 0.01. At $10^{-5}$ the weights barely move in 1000 epochs. 0.01 is fast but stable with a batch of 2. (0.03 also works.)
Training data 10 % → 90 %. With 10 % the network sees only 50 points: in our replica it then memorises them (training loss 0.002) while the test loss stays near 0.46. That is overfitting from too little data, and since regularisation is forbidden, the remedy is more data. With 90 %, the arms are densely sampled.

The batch size of 2 can stay as it is.

link. Median test loss 0.011, below 0.1 for 8 out of 8 seeds. We checked that the changes are needed (these checks used rate 0.03): with only 10 % or 50 % training data the network reaches 0.47 or 0.06 (below 0.1 for 3 of 5 seeds), and without the sines ( $x_1 x_2, x_1, x_2$ only) it stalls at 0.31.

Decision surface of the question 5 network before and after the fix

Results

question	setting	median test loss (replica)
1	as given (rate 3)	0.96
1	rate 0.03	0.115
2	as given (2-2)	train 0.44
2	8-8	train 0.015
3	linear 1-1, 2-2, 4-4, 8-8	train 0.476 for all
4	as given	0.49
4	tanh, rate 0.03 only	0.33
4	tanh, rate 0.03, 7 inputs, 90 % data	0.052 (6/8 seeds below 0.1)
5	as given	0.51
5	tanh, rate 0.01, + $x_1, x_2, \sin x_1, \sin x_2$ , 90 % data	0.011 (8/8 seeds below 0.1)

What to remember

Read the loss value, not only its trend. In this Playground a loss of 0.5 means "outputs 0", and a loss of 1 means "always the same class": question 1 is diagnosed from the number alone.
Too large a learning rate saturates, too small a rate does not move. Both look like "nothing happens"; the fix goes in opposite directions.
Without a non-linear activation, depth and width are useless: stacked linear layers are one linear layer.
Plateau on the training loss means too little capacity (underfit); low training loss with high test loss means too little data or too much freedom (overfit). Questions 2 and 5 show one of each.
Good input features can replace capacity. A small network with $\sin x_1, \sin x_2$ solves a spiral that it cannot solve from raw coordinates.

"""
A faithful NumPy replica of TensorFlow Playground (github.com/tensorflow/playground, src/nn.ts, dataset.ts,
playground.ts) for the classification "spiral" dataset, used to check every answer of HAIO 2024 Neural Playground.

Replicated details: 500 points, spiral generator of dataset.ts, shuffle then split by percTrainData;
inputs x1, x2, x1^2, x2^2, x1x2, sin x1, sin x2 (no normalisation); weights U(-0.5, 0.5), biases 0.1;
hidden activation = setting, output activation tanh; loss = mean of 0.5 (out - label)^2, labels +-1;
plain SGD, gradients averaged over each batch, one epoch = one pass over the training points.
(The browser's random seed cannot be reproduced, so every setting is run with several seeds.)
"""
import numpy as np

FEATURES = {"x": lambda x, y: x, "y": lambda x, y: y, "xSquared": lambda x, y: x * x,
            "ySquared": lambda x, y: y * y, "xTimesY": lambda x, y: x * y,
            "sinX": lambda x, y: np.sin(x), "sinY": lambda x, y: np.sin(y)}
ACT = {"tanh": (np.tanh, lambda z, a: 1 - a * a),
       "relu": (lambda z: np.maximum(z, 0), lambda z, a: (z > 0).astype(float)),
       "sigmoid": (lambda z: 1 / (1 + np.exp(-z)), lambda z, a: a * (1 - a)),
       "linear": (lambda z: z, lambda z, a: np.ones_like(z))}

def spiral(rng, n=500, noise=0.0):
    pts = []
    for dt, lab in ((0, 1), (np.pi, -1)):
        i = np.arange(n // 2); r = i / (n // 2) * 5; t = 1.75 * i / (n // 2) * 2 * np.pi + dt
        pts.append(np.c_[r * np.sin(t) + rng.uniform(-1, 1, n // 2) * noise,
                         r * np.cos(t) + rng.uniform(-1, 1, n // 2) * noise, np.full(n // 2, lab)])
    d = np.vstack(pts); rng.shuffle(d); return d

def run(features, shape, activation="tanh", lr=0.03, batch=10, train_pct=50, epochs=1000, seed=0,
        noise=0.0, return_curve=False, return_model=False):
    rng = np.random.default_rng(seed)
    d = spiral(rng, noise=noise); k = int(len(d) * train_pct / 100)
    X = np.c_[[FEATURES[f](d[:, 0], d[:, 1]) for f in features]].T; Y = d[:, 2]
    Xtr, Ytr, Xte, Yte = X[:k], Y[:k], X[k:], Y[k:]
    sizes = [len(features)] + list(shape) + [1]
    W = [rng.uniform(-0.5, 0.5, (a, b)) for a, b in zip(sizes[:-1], sizes[1:])]
    B = [np.full(b, 0.1) for b in sizes[1:]]
    f, df = ACT[activation]
    def forward(x):
        A, Z = [x], []
        for l in range(len(W)):
            z = A[-1] @ W[l] + B[l]; Z.append(z)
            A.append(np.tanh(z) if l == len(W) - 1 else f(z))
        return A, Z
    loss = lambda x, y: float(np.mean(0.5 * (forward(x)[0][-1][:, 0] - y) ** 2))
    curve = []
    with np.errstate(over="ignore", invalid="ignore"):
        for ep in range(epochs):
            for s in range(0, k - batch + 1, batch):           # leftover points (k % batch) are skipped
                A, Z = forward(Xtr[s:s + batch]); y = Ytr[s:s + batch]
                delta = (A[-1][:, 0] - y)[:, None] * (1 - A[-1] ** 2)      # dE/dz at the tanh output
                for l in range(len(W) - 1, -1, -1):
                    gW, gB = A[l].T @ delta / batch, delta.mean(0)
                    if l > 0: delta = (delta @ W[l].T) * df(Z[l - 1], A[l])
                    W[l] -= lr * gW; B[l] -= lr * gB
            if return_curve or ep == epochs - 1:
                curve.append((loss(Xtr, Ytr), loss(Xte, Yte)))
    tr, te = curve[-1]
    if return_model:
        predict = lambda x, y: forward(np.c_[[FEATURES[f](x, y) for f in features]].T)[0][-1][:, 0]
        return tr, te, predict, d, k
    return (tr, te, np.array(curve)) if return_curve else (tr, te)

def summary(name, seeds=range(5), **kw):
    r = np.array([run(seed=s, **kw) for s in seeds])
    r = np.nan_to_num(r, nan=9.99)
    print(f"{name:55s} train {np.median(r[:,0]):.3f}  test median {np.median(r[:,1]):.3f} "
          f"[min {r[:,1].min():.3f}, max {r[:,1].max():.3f}]  test<=0.1 in {(r[:,1]<=0.1).sum()}/{len(r)}", flush=True)
    return r


if __name__ == "__main__":   # reproduces every number quoted in the correction (about 3 minutes)
    F1, F4 = ["x", "y", "xSquared", "ySquared"], ["x", "y", "xTimesY", "xSquared"]
    print("Q1"); summary("as given, rate 3", features=F1, shape=(4, 3), lr=3, train_pct=70)
    summary("fix: rate 0.03", features=F1, shape=(4, 3), lr=0.03, train_pct=70)
    print("Q2"); summary("as given, 2-2", features=["x", "y"], shape=(2, 2), train_pct=50)
    summary("fix: 8-8", features=["x", "y"], shape=(8, 8), train_pct=50)
    print("Q3")
    for w in (1, 2, 4, 8): summary(f"linear {w}-{w}", features=F1, shape=(w, w), activation="linear", train_pct=70)
    print("Q4"); summary("as given", features=F4, shape=(4, 3), activation="linear", lr=0.001, train_pct=70)
    summary("fix", seeds=range(8), features=F4 + ["ySquared", "sinX", "sinY"], shape=(4, 3), activation="tanh",
            lr=0.03, train_pct=90)
    print("Q5"); summary("as given", features=["xTimesY"], shape=(7,), activation="linear", lr=1e-5, batch=2, train_pct=10)
    summary("fix", seeds=range(8), features=["xTimesY", "x", "y", "sinX", "sinY"], shape=(7,), activation="tanh",
            lr=0.01, batch=2, train_pct=90)

Scientific Task – Neural Playground