IOAI Training: Learn AI by solving problems

Scribble

This page gives the full approach. Read it once you have searched.

Metric: objects found, min(100, S/0.65*100)
Evaluated on: self-generated tests (official tests are hidden): 4000 composites from held-out drawings
Our score: 64.5 % objects (99.2 points)
Expected level: 100 points at S = 65 %; 3 full solves on Kattis

Our solution does not reach the expected level: there is still room to beat it

We solved the problem ourselves: downloaded the data, ran the experiments and measured the score below. The statement and data belong to their organisers.

The problem in brief

Harry sketches 50 kinds of objects (an arm, a beach, a cello, a kangaroo...), and to save paper he draws three objects on top of one another on the same page. You have his clean single drawings, 2000 per class, with labels: 100 000 black-on-white images of $128\times128$ pixels. The task is to name the 3 objects in each of 16 339 overlaid composites. External training data is forbidden. The organisers even show how composites are made: the pixel-wise minimum of three drawings, so ink from any drawing stays black. The task is from the Swedish AI Olympiad qualifier 2026 (by Theodor Beskow). The data zip only links to a public GitHub repository, which holds the images.

The test labels are hidden. We validate on 4 000 composites that we build ourselves, exactly like the organisers' image_merge.py and at full resolution, from 200 drawings per class that are never used in training. We also checked that the test composites are made of new drawings: no training drawing fits inside any test composite. So the held-out-drawing protocol is the faithful one.

How you are scored

Each composite earns one point per correctly named object (order does not matter), and $S$ is the share of the $3N$ possible points obtained:

\text{Points} = \max\Big(0, \min\Big(100, \frac{S}{0.65}\times 100\Big)\Big).

Full marks need 65 % of objects named correctly. On the Kattis statistics page, three contestants reached 100, and the next scores are 75.5 and 56.5.

First look at the data

The drawings look like the "Quick, Draw!" style: thin strokes, centred, filling most of the frame. Three of them on top of each other make a dense tangle (see the figure below), where even a human struggles.
The minimum operation means nothing is ever erased. Every stroke of each object is still present, just mixed with 2 others.
There are $\binom{50}{3} = 19\,600$ possible triples, so you cannot treat the problem as one class per triple. It is multi-label classification: 50 yes/no questions per image, answered by taking the 3 most likely.

A first baseline

The obvious first try is to train a 50-way classifier on the clean single drawings, then take its top 3 classes on each composite. Composites are far from the training distribution (three times more ink), so this transfers badly. We skipped it in favour of the key idea below, which costs nothing extra.

The key idea

We know the generator, so we can manufacture unlimited labelled training data that looks exactly like the test. At each training step, pick three different classes, one random drawing of each, and take their pixel-wise minimum. The label is the 3-hot vector of those classes. The network then learns on precisely the task it will be tested on, and it never sees the same composite twice: with 1 800 drawings per class there are about $19\,600 \times 1\,800^3 \approx 10^{14}$ combinations. That is a built-in regulariser.

The loss is a sigmoid per class (binary cross-entropy), not a softmax. A softmax forces the 50 scores to compete for a single winner, whereas here three classes are present together:

\mathcal{L} = -\sum_{c=1}^{50} \big[y_c \log \sigma(z_c) + (1-y_c)\log(1-\sigma(z_c))\big].

At test time, output the 3 classes with the highest $\sigma(z_c)$ , since we know there are exactly 3.

The full solution

Clone the data repository and convert every image to grey levels at $64\times 64$ (enough for these strokes, and it keeps all 100 000 drawings in 400 MB). Keep drawings 1800 to 1999 of each class aside, to build the validation composites at full resolution.
Generate composites on the fly on the GPU: sample 3 distinct classes with argsort(random)[:, :3] and one drawing per class, apply random horizontal flips to individual drawings, and take the minimum. Batches have 128 composites.
CNN: four blocks of [conv 3x3, batch norm, ReLU] x 2 then max-pooling (32, 64, 128 and 256 channels, $64 \to 4$ ), global average pooling, dropout 0.3, and a linear layer with 50 outputs. AdamW with a one-cycle learning rate peaking at 0.002, for 13 000 steps (1.7 M synthetic composites), about 23 minutes on Apple MPS.
Test-time augmentation: average the sigmoids of the image and of its mirror image, then output the top 3.

Held-out composites and predictions

The figure shows typical behaviour. When objects have distinctive parts (a guitar's neck, a pig's snout), all three are found. Big, plain outlines (table, helmet) and objects made of a few simple strokes (map, toothpaste, passport) are the usual misses.

Results

Object accuracy $S$ on 4 000 held-out composites (built from unseen drawings; test labels are private).

approach	object accuracy $S$	points
CNN on synthetic composites, 1 000 steps	40.8 %	62.7
same, 7 000 steps (smaller budget)	63.3 %	97.3
same, 13 000 steps (final)	64.5 %	99.2

Details for the final model: all 3 objects are right in 21.3 % of the composites, 2 in 53.2 %, 1 in 23.1 % and none in 2.4 %. The right objects are in the top 5 for 76.9 % of them and in the top 10 for 89.3 %. Per class, recall ranges from 32 % (map) to 88 % (campfire). Accuracy was still rising at the end, so a longer run or a wider network would most likely cross the 65 % bar. We stopped at the compute cap.

What to remember

When the organisers tell you how test inputs are generated, simulate it: synthetic data from the true generator beats any amount of clever modelling on the wrong distribution.
Validate on synthetic examples built from held-out originals, at the test's exact resolution and process. Check that the test does not reuse training originals.
Several objects per image means multi-label: one sigmoid per class with binary cross-entropy, then pick the known number of labels.
On-the-fly generation gives a practically infinite dataset. Then more training steps keep helping, and compute becomes the limit rather than overfitting.

Full code (solution.py)

"""SAIO Scribble: name the 3 sketches drawn on top of each other in a composite image.

Only clean single drawings are labelled (2000 per class, 50 classes), but the organisers tell us
how composites are made: pixel-wise minimum of 3 drawings (dark ink on white). So we generate an
unlimited stream of labelled composites ourselves and train a multi-label CNN (50 sigmoids);
the answer is the 3 highest-scoring classes.

Data prep (from github.com/ArvidKristoffersson/scribble-data, linked in the problem's info.txt):
every image converted to grey and resized to 64x64 -> work/Xtr.npy, work/ytr.npy, work/Xte.npy;
validation composites built at full 128 px resolution from the 200 held-out drawings per class
-> work/Xval.npy, work/Yval.npy  (drawings 1800-1999 of each class are never used for training).
  python3 solution.py            # trains, reports validation accuracy, writes work/answers.txt
"""
import os, numpy as np, pandas as pd, torch, torch.nn as nn, torch.nn.functional as F, time
from PIL import Image
torch.set_num_threads(2); torch.manual_seed(0)
DEV = 'mps' if torch.backends.mps.is_available() else 'cpu'


def prepare(repo='work/repo'):
    """git clone --depth 1 https://github.com/ArvidKristoffersson/scribble-data work/repo, then run once."""
    if os.path.exists('work/Xval.npy'): return
    ld = lambda p: np.array(Image.open(f'{repo}/{p}').convert('L').resize((64, 64), Image.BILINEAR))
    tr, te = pd.read_csv(f'{repo}/train.csv'), pd.read_csv(f'{repo}/test.csv')
    np.save('work/Xtr.npy', np.stack([ld(p) for p in tr.image_path]))
    np.save('work/ytr.npy', tr.label.values.astype('U30'))
    np.save('work/Xte.npy', np.stack([ld(p) for p in te.image_path]))
    cls = sorted(tr.label.unique())
    held = np.where(tr.groupby('label').cumcount().values >= 1800)[0]; np.save('work/held_idx.npy', held)
    rng = np.random.default_rng(123); by = {c: held[tr.label.values[held] == c] for c in cls}; V, Y = [], []
    for _ in range(4000):                  # validation composites made exactly like image_merge.py, at 128 px
        cs = rng.choice(cls, 3, replace=False)
        ims = [np.array(Image.open(f'{repo}/' + tr.image_path[rng.choice(by[c])]).convert('L')) for c in cs]
        V.append(np.array(Image.fromarray(np.minimum.reduce(ims)).resize((64, 64), Image.BILINEAR)))
        Y.append([cls.index(c) for c in cs])
    np.save('work/Xval.npy', np.stack(V)); np.save('work/Yval.npy', np.array(Y))


prepare()

Xtr = np.load('work/Xtr.npy'); ytr = np.load('work/ytr.npy'); Xte = np.load('work/Xte.npy')
Xval = np.load('work/Xval.npy'); Yval = np.load('work/Yval.npy')
classes = sorted(set(ytr)); C = len(classes)
yid = np.array([classes.index(c) for c in ytr])
held = set(np.load('work/held_idx.npy'))
by_class = np.stack([np.array([i for i in np.where(yid == c)[0] if i not in held]) for c in range(C)])  # (50, 1800)
X_gpu = torch.tensor(Xtr, dtype=torch.uint8, device=DEV)


def make_batch(n, rng, k=3):
    """n synthetic composites: pixel-wise minimum of k drawings from k distinct classes."""
    cls = np.argsort(rng.random((n, C)), 1)[:, :k]                 # k distinct classes per row
    idx = by_class[cls, rng.integers(0, by_class.shape[1], (n, k))]
    imgs = X_gpu[torch.tensor(idx, device=DEV)]                     # (n, k, 64, 64)
    if rng.random() < 0.5:                                          # random horizontal flips
        imgs = torch.where(torch.rand(n, k, 1, 1, device=DEV) < 0.5, imgs.flip(-1), imgs)
    comp = imgs.min(1).values
    target = torch.zeros(n, C, device=DEV)
    target[torch.arange(n)[:, None], torch.tensor(cls, device=DEV)] = 1
    return prep(comp), target


def prep(x):  # ink = 1, paper = 0
    return (1 - x.float() / 255.0)[:, None]


def block(cin, cout):
    return nn.Sequential(nn.Conv2d(cin, cout, 3, padding=1), nn.BatchNorm2d(cout), nn.ReLU(),
                         nn.Conv2d(cout, cout, 3, padding=1), nn.BatchNorm2d(cout), nn.ReLU(),
                         nn.MaxPool2d(2))


net = nn.Sequential(block(1, 32), block(32, 64), block(64, 128), block(128, 256),   # 64 -> 4
                    nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Dropout(0.3), nn.Linear(256, C)).to(DEV)


@torch.no_grad()
def scores(X):
    net.eval(); out = []
    for i in range(0, len(X), 1000):
        xb = prep(torch.tensor(X[i:i + 1000], device=DEV))
        out.append((torch.sigmoid(net(xb)) + torch.sigmoid(net(xb.flip(-1)))).cpu() / 2)  # flip TTA
    net.train(); return torch.cat(out).numpy()


def object_accuracy(S, Y):
    top3 = np.argsort(-S, 1)[:, :3]
    return np.mean([len(set(a) & set(b)) / 3 for a, b in zip(top3, Y)])


if __name__ == '__main__':
    steps, bs = 13000, 128
    opt = torch.optim.AdamW(net.parameters(), 2e-3, weight_decay=5e-4)
    sched = torch.optim.lr_scheduler.OneCycleLR(opt, 2e-3, total_steps=steps)
    rng = np.random.default_rng(0); t = time.time()
    for step in range(1, steps + 1):
        xb, yb = make_batch(bs, rng)
        loss = F.binary_cross_entropy_with_logits(net(xb), yb)
        opt.zero_grad(); loss.backward(); opt.step(); sched.step()
        if step % 2000 == 0 or step == steps:
            acc = object_accuracy(scores(Xval), Yval)
            print(f'step {step}: loss {loss.item():.4f}, validation object accuracy {acc:.4f} '
                  f'-> points {min(100, acc / 0.65 * 100):.1f}  ({time.time() - t:.0f}s)', flush=True)
    St = scores(Xte)
    top3 = np.argsort(-St, 1)[:, :3]
    open('work/answers.txt', 'w').write('\n'.join(' '.join(classes[j] for j in r) for r in top3) + '\n')
    np.save('work/val_scores.npy', scores(Xval))
    torch.save(net.state_dict(), 'work/net.pt')