IOAI Training: Learn AI by solving problems

The problem in brief

A post office reads handwritten digits with scanners, and some scanners are broken. In subtask 1 you get 500 labelled 28 x 28 images from each of three scanners: a normal one (white digit on black), one that inverts the image (black on white), and one that inverts it on half of the image. In subtask 2 you get 500 images from each of eight scanners, the three above plus five strange ones: digits hidden in a colour-blindness (Ishihara) plate, barcodes, braille dots, Roman numerals photographed on paper, and Eastern Arabic numerals. For each subtask you submit one small TorchScript model, and at test time the model does not know which scanner an image came from. The twist is in the score: every parameter of the model costs points.

We scored our final models with the organisers' own pre-judging script on the practice round's test file (2 000 images per scanner). The test labels are not public, so we only report accuracies, never labels. Every design choice was made on a 20 % validation split of the training images.

How you are scored

For each subtask, with $A$ the accuracy averaged over scanners (each scanner weighs the same) and $P$ the number of parameters:

S = 100\,A - \frac{P}{7\,500}.

\text{subtask 1} = \operatorname{clip}\!\Big(\tfrac{S - 90}{10}\cdot 30,\ 0,\ 30\Big), \qquad \text{subtask 2} = \operatorname{clip}\!\Big(\tfrac{S - 65}{25}\cdot 70,\ 0,\ 70\Big).

So 1 point of accuracy is worth exactly 7 500 parameters. In subtask 1 every point of $S$ above 90 gives 3 points, up to 30 at $S = 100$ ; in subtask 2 every point of $S$ above 65 gives 2.8 points, up to 70 at $S = 90$ . The starter notebook's baseline scores 1/30 (accuracy 0.91) and 0/70 (accuracy 0.65), according to the statement. There is no public leaderboard to compare with, so our target is simply the 100-point maximum.

First look at the data

One training image per digit and per scanner

Scanners 2 and 3 are pure geometry. An inverted digit is $1 - x$ , and a half-inverted one is $1 - x$ on one half. The information is all there; only the colour convention changes.
Scanners 4 to 8 are different writing systems. A barcode or a braille cell encodes the digit by a pattern that has nothing to do with its shape; the network has to learn a separate "alphabet" for each, from only 500 examples (50 per digit).
Tiny training sets. 1 500 images for subtask 1 and 4 000 for subtask 2. A digit classifier trained on 1 500 MNIST images typically tops out around 96 to 97 %.
The scanner identity is not given at test time, so a single model must handle all styles.

A first baseline

A small 3-layer CNN (8, 16 and 24 channels, about 5 000 parameters) with augmentation reproduces roughly the starter's level: on the official test it gets 0.893 accuracy on subtask 1 ( $S = 88.6$ , 0 points) and 0.599 on subtask 2 (0 points), with very poor accuracy on the Ishihara (14 %), braille (41 %) and Roman (27 %) scanners. It is too small to learn eight alphabets.

The key idea

Treat parameters as a currency, and spend them where they buy accuracy.

The score turns model design into an exchange rate: 7 500 parameters for 1 point of accuracy. Three consequences guided the solution.

Depth is cheap, width is expensive. A 3 x 3 convolution from $c_1$ to $c_2$ channels costs $9\,c_1 c_2$ parameters: doubling the width quadruples the cost. Adding a thin extra layer, on the other hand, enlarges the receptive field (the network sees the whole digit) for a few thousand parameters. Our validation sweep (figure below) shows it clearly: 3-layer networks need 40 000 parameters to reach 98 %, while 4-layer networks reach 99 % with 12 000.
Fixed code is free. Only learnable weights are counted, so anything written as a fixed operation inside forward() costs nothing. For subtask 1 we add an "ink" channel: the absolute difference between each pixel and the average colour of its 9 x 9 neighbourhood, which makes a digit look the same whatever the paper colour. Global average pooling (instead of a large dense layer) is another free saving: the classifier head is just $24 \times 10$ weights.
Augmentation manufactures the broken scanners. For subtask 1 we randomly invert 30 % of the training images and half-invert another 30 % (random split position and direction), so the model sees every digit under every broken-scanner convention; plus small rotations, shifts and scalings for all images.

Validation accuracy against parameters for subtask 1; grey lines have equal S

On the plot, a model is better when it sits on a higher grey line. The best point, 4 layers of 10, 20, 24 and 24 channels with the ink channel (11 890 parameters), reached 0.99 on validation, $S \approx 97.4$ .

The full solution

Scale the pixels to $[0, 1]$ ; keep 100 images per scanner for validation.
Network: $L$ blocks of Conv 3x3 (no bias) + BatchNorm + ReLU, with 2 x 2 max-pooling between blocks, then global average pooling and one linear layer to 10 logits. Subtask 1: widths 10, 20, 24, 24 plus the free ink channel (11 890 parameters). Subtask 2: widths 16, 32, 32, 32 on the raw 3 channels (24 026 parameters); the ink channel and a free colour-contrast channel did not help there.
Augmentation: random affine transform (rotation up to 10 degrees, scale 0.9 to 1.1, shift up to 2 pixels, padding with the paper colour), and for subtask 1 the random inversions and half inversions.
Training: 60 epochs of AdamW (learning rate $4 \times 10^{-3}$ with a one-cycle schedule), batch 64, cross-entropy with label smoothing 0.05; then retrain on all training images and export with torch.jit.script. Each subtask takes 1 to 2 minutes on a CPU.

def normalise(self, x):           # fixed code: costs 0 parameters
    g = x.mean(1, keepdim=True)
    bg = F.avg_pool2d(F.pad(g, (4, 4, 4, 4), mode="replicate"), 9, stride=1)
    return torch.cat([x, (g - bg).abs() * 2.0], 1)   # "ink" = distance to the local paper colour

Validation was too optimistic. With only 100 images per scanner, the subtask 1 validation accuracy (0.99) overestimated the test accuracy (0.959) by 3 points, i.e. 9 subtask points. A tiny validation set cannot separate models that differ by 1 %, which is exactly the resolution this score needs; repeated or cross-validated splits would have been a better guide.

Results

Official pre-judging script on the practice test file (2 000 images per scanner); models chosen on the validation split.

model	params	macro accuracy	S	points
subtask 1, 3 layers 8-16-24 (baseline level)	5 098	0.893	88.6	0 / 30
subtask 1, 4 layers 10-20-24-24 + ink (final)	11 890	0.959	94.4	13 / 30
subtask 2, 3 layers 8-16-24 + ink (baseline level)	5 242	0.599	59.2	0 / 70
subtask 2, 4 layers 16-32-32-32 (final)	24 026	0.927	89.5	68 / 70

Per scanner, the final subtask 2 model gets between 0.80 (Roman numerals, the hardest: real photos with varied backgrounds and strokes) and 0.99 (barcodes); the Ishihara plates reach 0.88. In total the two models score 81 / 100. Subtask 2 is nearly saturated; the 17 missing points are in subtask 1, where each extra point of accuracy on ordinary and inverted MNIST is worth 3 points, and where more training data per digit (only 150 images per digit across the three scanners) is the real bottleneck.

What to remember

Read the score as an exchange rate. Here 7 500 parameters = 1 point of accuracy; compute the price of every design decision before making it.
Depth is cheaper than width. Convolution cost grows with the product of channel counts, so a thin, deeper network gives more accuracy per parameter.
Fixed preprocessing inside the model is free. Hand-written normalisations, extra input channels and global pooling add knowledge without adding weights.
Augment towards the test distribution. Generating inverted and half-inverted digits from every image teaches the broken-scanner conventions far better than the 500 examples of each.
Small validation sets lie at fine resolution. 100 images per scanner cannot tell 96 % from 99 %; when 1 % is worth 3 points, use several splits.

"""
CEOAI (EUROAI) 2026 practice round 2, "Panda MNIST": two small TorchScript digit classifiers for broken scanners.
Score per subtask: S = 100 * (macro accuracy over scanners) - (number of parameters) / 7500,
  subtask 1 (scanners 1-3, 1 channel): points = clip((S - 90) / 10 * 30, 0, 30)
  subtask 2 (scanners 1-8, 3 channels): points = clip((S - 65) / 25 * 70, 0, 70)

Data: train_data.zip (500 images per scanner) and test_data.npz (2000 per scanner, with labels) from the Nitro task;
the official scorer pre_judging_script.py is used for the final numbers. Put them in DATA_DIR.
Choices are made on a 20 % validation split of the training images; the test set is scored once at the end.

Key ideas: (1) parameters are expensive (7 500 parameters = 1 accuracy point), so depth beats width (4 thin
conv layers instead of 3 wide ones) and fixed code in forward() is free ("ink vs paper" channel); (2) global
average pooling instead of a big dense head; (3) augmentation that manufactures more of the broken scanners
(inversions, half inversions) plus small affine jitter.
Run: python solution.py --task 1   and   python solution.py --task 2     (1-2 minutes each on CPU)
"""
import argparse, os, sys, zipfile, io, time
import numpy as np, torch, torch.nn as nn, torch.nn.functional as F

p = argparse.ArgumentParser()
p.add_argument("--task", type=int, default=1); p.add_argument("--prep", default=None, choices=["none", "ink", "chroma"])
p.add_argument("--aug", type=int, default=1); p.add_argument("--width", type=str, default=None)
p.add_argument("--epochs", type=int, default=60); p.add_argument("--final", type=int, default=1)
a = p.parse_args()
# final choices (made on the validation split): subtask 1 = 4 conv layers + ink channel, subtask 2 = 4 wider layers
if a.width is None: a.width = "10,20,24,24" if a.task == 1 else "16,32,32,32"
if a.prep is None: a.prep = "ink" if a.task == 1 else "none"
D = os.environ.get("DATA_DIR", "work"); sys.path.insert(0, D)
torch.set_num_threads(2); torch.manual_seed(0); rng = np.random.default_rng(0)
import pre_judging_script as judge                       # official scorer (load_test_images, compute_score, ...)

NS = 3 if a.task == 1 else 8
CIN = 1 if a.task == 1 else 3
with zipfile.ZipFile(f"{D}/train_data.zip") as z:
    ld = lambda n: np.load(io.BytesIO(z.read(n)))
    X = [ld(f"subtask{a.task}/scanner{s}_X.npy").astype(np.float32) / 255 for s in range(1, NS + 1)]
    Y = [ld(f"subtask{a.task}/scanner{s}_y.npy").astype(np.int64) for s in range(1, NS + 1)]

class Net(nn.Module):
    """Small CNN. Everything in `normalise` is fixed code: it costs no parameters."""
    def __init__(self, cin: int, widths, ink: bool):
        super().__init__()
        self.mode = a.prep
        layers, c = [], cin + {"none": 0, "ink": 1, "chroma": cin}[a.prep]
        for i, w in enumerate(widths):
            layers += [nn.Conv2d(c, w, 3, padding=1, bias=False), nn.BatchNorm2d(w), nn.ReLU()]
            if i < len(widths) - 1: layers.append(nn.MaxPool2d(2))
            c = w
        self.features = nn.Sequential(*layers)
        self.head = nn.Linear(c, 10)

    def normalise(self, x):
        # "ink" channel: distance of each pixel from its local paper colour (mean of a 9x9 window).
        # White-on-black, black-on-white and half-inverted digits all become bright strokes on dark.
        g = x.mean(1, keepdim=True)
        if self.mode == "chroma":          # colour contrast (x - grey) x 4: makes Ishihara digits stand out
            return torch.cat([x, (x - g) * 4.0], 1)
        bg = F.avg_pool2d(F.pad(g, (4, 4, 4, 4), mode="replicate"), 9, stride=1)
        return torch.cat([x, (g - bg).abs() * 2.0], 1)

    def forward(self, x):
        if self.mode != "none": x = self.normalise(x)
        return self.head(self.features(x).mean((2, 3)))      # global average pooling

def augment(x):
    """random small affine (shift +-2 px, rotation +-10 deg, scale 0.9-1.1) + random paper inversions"""
    n = x.shape[0]; t = torch.from_numpy(rng.uniform(-1, 1, (n, 4)).astype(np.float32))
    ang, sc = t[:, 0] * 0.17, 1 + 0.1 * t[:, 1]
    th = torch.zeros(n, 2, 3); th[:, 0, 0] = sc * torch.cos(ang); th[:, 0, 1] = -sc * torch.sin(ang)
    th[:, 1, 0] = sc * torch.sin(ang); th[:, 1, 1] = sc * torch.cos(ang); th[:, :, 2] = t[:, 2:] * 2 / 28
    grid = F.affine_grid(th, x.shape, align_corners=False)
    fill = x[:, :, :2, :2].mean((2, 3), keepdim=True)          # pad with the paper colour of the corner
    x = F.grid_sample(x - fill, grid, align_corners=False, padding_mode="zeros") + fill
    if a.task == 1:   # scanners 2 and 3 are inversions: create more of them from every image
        inv = torch.from_numpy(rng.random(n) < 0.3); x[inv] = 1 - x[inv]
        half = torch.from_numpy(rng.random(n) < 0.3); k = int(rng.integers(8, 21))
        m = torch.zeros(1, 1, 28, 28); m[..., :, :k] = 1 if rng.random() < 0.5 else 0
        if rng.random() < 0.5: m = m.transpose(2, 3)
        x[half] = m * (1 - x[half]) + (1 - m) * x[half]
    return x

def train(Xs, Ys, epochs):
    model = Net(CIN, [int(w) for w in a.width.split(",")], a.prep != "none")
    Xt = torch.from_numpy(np.concatenate(Xs)); Yt = torch.from_numpy(np.concatenate(Ys))
    opt = torch.optim.AdamW(model.parameters(), lr=4e-3, weight_decay=1e-4)
    steps = epochs * ((len(Xt) + 63) // 64); sch = torch.optim.lr_scheduler.OneCycleLR(opt, 4e-3, total_steps=steps)
    for ep in range(epochs):
        perm = torch.randperm(len(Xt))
        for i in range(0, len(Xt), 64):
            b = perm[i:i + 64]; x = Xt[b].clone()
            if a.aug: x = augment(x)
            loss = F.cross_entropy(model(x), Yt[b], label_smoothing=0.05)
            opt.zero_grad(); loss.backward(); opt.step(); sch.step()
    return model.eval()

@torch.no_grad()
def accs(model, Xs, Ys):
    return [float((model(torch.from_numpy(x)).argmax(1).numpy() == y).mean()) for x, y in zip(Xs, Ys)]
def points(acc, P):
    return judge.compute_score(acc, P, judge.SUBTASK_MAX[a.task], subtask_id=a.task)

# ---------------------------------------------------------------- validation (20 % of each scanner)
t0 = time.time()
idx = [rng.permutation(len(y)) for y in Y]
tr = [(x[i[100:]], y[i[100:]]) for x, y, i in zip(X, Y, idx)]; va = [(x[i[:100]], y[i[:100]]) for x, y, i in zip(X, Y, idx)]
m = train([t[0] for t in tr], [t[1] for t in tr], a.epochs)
P = sum(q.numel() for q in m.parameters()); v = accs(m, [t[0] for t in va], [t[1] for t in va])
print(f"task {a.task} prep={a.prep} aug={a.aug} width={a.width}: params {P}, val per scanner "
      f"{np.round(v, 2).tolist()}, macro {np.mean(v):.4f}, S {100 * np.mean(v) - P / 7500:.2f}, "
      f"points {points(np.mean(v), P)}  ({(time.time() - t0) / 60:.1f} min)", flush=True)

# ---------------------------------------------------------------- final: retrain on all training images, official test
if a.final:
    m = train(X, Y, a.epochs)
    imgs, labs = judge.load_test_images(f"{D}/test_data.npz"), judge.load_test_labels(f"{D}/test_data.npz")
    keys = [f"scanner{s}" for s in range(1, NS + 1)]
    tx = [np.repeat(imgs[k], 3, 1) if (a.task == 2 and imgs[k].shape[1] == 1) else imgs[k] for k in keys]
    t = accs(m, tx, [labs[k] for k in keys])
    print(f"TEST task {a.task}: per scanner {np.round(t, 3).tolist()} macro {np.mean(t):.4f} "
          f"S {100 * np.mean(t) - P / 7500:.2f} -> {points(np.mean(t), P)} / {judge.SUBTASK_MAX[a.task]} points")
    torch.jit.save(torch.jit.script(m), f"{D}/model_sub{a.task}.pt")