Indonesia IOAI selection 2025 · Correction by IOAI Training

Face or Flag?

This page gives the full approach. Read it once you have searched.

Metric: accuracy
Evaluated on: official test
Our score: 99.35 % accuracy (308/310)
Expected level: baseline 50 %; 100 pts = best contestant (unpublished)

We solved the problem ourselves: downloaded the data, ran the experiments and measured the score below. The statement and data belong to their organisers.

The problem in brief

You are handed small 64 by 64 colour pictures of emoji, each pasted on a plain background, and must write a function that says whether a picture shows a face (label 1) or a flag (label 2). The training set is tiny: 160 labelled images, 80 of each. The test set (310 images) was only released in the last hour of the contest, so there is no way to adapt to it: whatever rule you write must already be right. The notebook nudges you towards hand-crafted features (colour histograms, pixel loops) rather than deep learning, and external emoji data is forbidden.

The organisers later published the test labels, so everything below is scored on the official test set with the official metric.

How you are scored

The metric is plain accuracy on the 310 test images,

\text{acc} = \frac{\#\{\text{correct predictions}\}}{310},

turned into points relative to the best contestant:

\text{points} = 100 \cdot \frac{\text{acc} - 0.5}{\text{acc}_{\text{best}} - 0.5},

with 0 points at or below the 50 % baseline. The best contestant's accuracy is not published, so there is no fixed target: you get 100 points only if nobody does better than you.

First look at the data

Plot a grid of the training images and the job looks almost insultingly easy. Every training face is a yellow Apple smiley, every flag a waving Apple flag. A rule like "is the picture mostly yellow?" separates the 160 training images perfectly.

Now look at the test set (we only did this after choosing our rule, but it is the lesson of the problem):

The test faces come from other emoji designs (thick black outlines, flat colours) and are often not yellow at all: a hamster, a fox, a purple devil, a skull, a red face, a person with grey hair. Meanwhile many test flags are full of yellow (Spain, Romania, Bhutan). This is a textbook distribution shift: the training set is not a fair sample of the test set. Any feature that only happens to separate the training classes will collapse.

Two small practical details matter for feature extraction:

the background is a flat green (71, 112, 76) on most images and white on a few, so we take it as the most frequent colour on the image border;
about a third of the training images carry one or two stray pixels near the edges, so when measuring the silhouette we ignore rows and columns with fewer than 3 object pixels.

A first baseline

The "obvious" feature: the fraction of yellow pixels ( $R>200$ , $G>150$ , $B<120$ ) inside the silhouette. On train, faces have between 29 % and 79 % yellow, flags at most 21 %, so a threshold at the midpoint (about 0.25) is 100 % correct on train.

On test it gets 64.2 %. Barely better than the 50 % baseline. Learned models on raw pixels or colour histograms fail in exactly the same way, because they too latch on to colour:

model (fitted on the 160 training images)	train acc.	test acc.
yellow fraction > 0.25	100 %	64.2 %
1-nearest neighbour on raw pixels	100 %	68.4 %
logistic regression on raw pixels	100 %	66.8 %
1-nearest neighbour on a 64-bin colour histogram	100 %	71.0 %

Perfect training accuracy tells you nothing here: 160 images of one design can be separated by dozens of features, and most of them are accidents.

The key idea

Ask what makes a face a face and a flag a flag, whatever the artist. It is not the colour, which depends on the design and the subject. It is the shape of the object: an emoji face is drawn as a disc (or a head) that fills the whole square canvas, while a flag is a rectangle with a 3:2 or 2:1 ratio, so it only fills about two thirds of the canvas height. Every emoji set obeys this, because it is how the symbols are defined.

Once you see it, one number is enough: the height $h$ of the silhouette's bounding box. On the training set,

h_{\text{faces}} \in [57, 63], \qquad h_{\text{flags}} \in [37, 43],

a gap of 14 pixels. We put the threshold in the middle of the gap, $h \ge 50 \Rightarrow$ face, using the training data only. Putting a threshold at the midpoint of the largest gap is the one-feature version of a maximum-margin classifier (a 1-D SVM): it leaves as much room as possible on both sides for new images that look a little different, which is exactly what a distribution shift will throw at you.

$Left: yellow fraction for train and test. Right: silhouette height against aspect ratio, with the threshold$

The left panel shows the colour feature falling apart: test faces (orange outline) spread across the whole range, test flags pile up where train faces used to be. The right panel shows the shape feature holding firm: test faces still sit high (55 to 64 px), test flags still sit low (40 to 44 px). The colour feature captured how the training images were drawn; the height captures what they are.

Why not use the aspect ratio $w/h$ instead? It works almost as well (98.7 %), but it is less robust: faces with ears, hair or a hat are wider than tall, and the square flags of Switzerland and the Vatican have $w/h \approx 0.93$ , like a face. Their height (41 px) still says "flag".

The full solution

Background: the most frequent colour on the border. Mask: pixels whose colour differs from it by more than 40 (sum of absolute RGB differences).
Bounding box: first and last row (column) containing at least 3 mask pixels, so stray pixels are ignored.
Decision: face if the box is at least 50 px tall, flag otherwise.

def prediksi_gambar_img(img):
    pred_label = 1 if shape_features(img)["h"] >= 50 else 2
    return int(pred_label)

There is nothing to train beyond reading the two height ranges off the training set, which is the point: with 160 examples of a single design, a rule with one parameter chosen for a physical reason generalises far better than a model with thousands of parameters.

Results

All numbers are on the official 310-image test set (140 faces, 170 flags); every rule was fixed on the training set first.

approach	train acc.	test acc.	points if best = ours
yellow fraction	100 %	64.2 %	29
1-NN on raw pixels	100 %	68.4 %	37
1-NN on colour histogram	100 %	71.0 %	42
mirror symmetry of the mask (waving flags are asymmetric)	96.3 %	92.6 %	86
fill ratio of the bounding box	98.8 %	90.3 %	82
aspect ratio $w/h < 1.19$	99.4 %	98.7 %	99
silhouette height $h \ge 50$	100 %	99.35 %	100

The final rule makes 2 mistakes out of 310:

Both are flags drawn with a flagpole (the pirate flag and the white flag), and the pole sticks out below the cloth, raising the height to 51 px. The training set contains no flag on a pole, so no rule chosen on train could anticipate it. You could fix it (the bottom rows of these images contain only the thin pole), but we found that by looking at the test errors, so it would be tuning on the test set: we do not count it. Note also that one test face sits exactly at $h = 50$ : the margin is real but not huge, which is a fair warning about how far a single feature can carry you.

What to remember

Perfect training accuracy is not evidence of anything when the training set is small and homogeneous. Ask how the test data could differ, and check that your features would survive it.
Prefer features tied to the definition of the classes (a flag is a rectangle, a face is round) over features tied to how the examples happen to be drawn (Apple's yellow).
Put thresholds in the middle of the gap, not at the edge of one class: the maximum-margin choice is the one that tolerates shift best.
Robust measurement matters: take the background from the data, ignore stray pixels. Two lines of care turn a fragile feature into a reliable one.
Read your test errors, but do not fix them with the test set: note them as the next thing to improve and keep the reported score honest.

Full code (solution.py)

"""Face or Flag? (OSN AI 2025, Indonesia, exhibition final) -- worked solution.

Data: the two public Google Drive zips used by the official notebook
  OSNAI2025_public.zip    id 1YC1i-DN3DOnV8MGqorKl6yTgSERah0wa  (emoji_train.py, emoji_test.py)
  OSNAI2025_privateGT.zip id 1sgh5ofZ1Eg9J2TchSzdf1WOjB9PIeoO4  (emoji_test_gt.py, labels)
  curl -L -o pub.zip 'https://drive.usercontent.google.com/download?id=<ID>&export=download'
Unzip both into DATA_DIR. The modules only hold a base64 .npz; we decode it ourselves
(allow_pickle=False) instead of importing them.

Idea: colour and texture are a trap (train faces are all yellow Apple smileys, test faces come
from other emoji sets and include a dog, a robot, a pile of poo...). The SHAPE of the
silhouette does not lie: a face is a disc as tall as the canvas, a flag is a wide rectangle
about 43 px tall. One threshold on the silhouette height, chosen on train only, gives 99.35 %.

Run: python solution.py [DATA_DIR] [--figures]
"""
import base64, io, re, sys
import numpy as np

DATA_DIR = sys.argv[1] if len(sys.argv) > 1 and not sys.argv[1].startswith("--") else "work"


def load_module(path):
    """Read the X/y arrays embedded (base64 npz) in the organisers' .py modules."""
    s = open(path).read()
    b64 = re.search(r"_B64 = '''(.*?)'''", s, re.S).group(1)
    return dict(np.load(io.BytesIO(base64.b64decode(b64)), allow_pickle=False))


# ----------------------------------------------------------------------------------------------
# Feature extraction
# ----------------------------------------------------------------------------------------------
def silhouette(img):
    """Boolean mask of the emoji: pixels that differ from the background colour.

    The background is the most frequent colour on the image border (green 71,112,76 for
    most images, white for a few). A few images carry 1-2 stray pixels near the edges,
    so we later ignore rows/columns with fewer than 3 mask pixels."""
    img = np.asarray(img, dtype=int)
    border = np.concatenate([img[0], img[-1], img[:, 0], img[:, -1]])
    colours, counts = np.unique(border, axis=0, return_counts=True)
    bg = colours[counts.argmax()]
    return np.abs(img - bg).sum(-1) > 40


def shape_features(img):
    m = silhouette(img)
    rows = np.where(m.sum(1) >= 3)[0]
    cols = np.where(m.sum(0) >= 3)[0]
    h = rows[-1] - rows[0] + 1          # height of the bounding box (pixels)
    w = cols[-1] - cols[0] + 1          # width of the bounding box
    return {"h": h, "w": w, "aspect": w / h, "fill": m.sum() / (h * w)}


def yellow_fraction(img):
    """The tempting colour feature (used only as the failed baseline)."""
    img = np.asarray(img, dtype=int)
    m = silhouette(img)
    R, G, B = img[..., 0], img[..., 1], img[..., 2]
    return ((R > 200) & (G > 150) & (B < 120) & m).sum() / max(m.sum(), 1)


# ----------------------------------------------------------------------------------------------
# The function the notebook asks for: 1 = Faces, 2 = Flags
# ----------------------------------------------------------------------------------------------
HEIGHT_THRESHOLD = 50   # midpoint of the train gap: faces >= 57 px, flags <= 43 px


def prediksi_gambar_img(img):
    pred_label = 1 if shape_features(img)["h"] >= HEIGHT_THRESHOLD else 2
    return int(pred_label)


def accuracy(y, p):
    return float(np.mean(np.asarray(y) == np.asarray(p)))


def main():
    tr = load_module(f"{DATA_DIR}/emoji_train.py")
    gt = load_module(f"{DATA_DIR}/emoji_test_gt.py")
    X, y, Xt, yt = tr["X_train"], tr["y_train"], gt["X_test"], gt["y_test"]
    print(f"train {X.shape}, classes {np.bincount(y)[1:]} | test {Xt.shape}, classes {np.bincount(yt)[1:]}")

    # Where does the threshold come from? Look at the TRAIN heights only.
    H = np.array([shape_features(i)["h"] for i in X])
    print(f"train face heights {H[y == 1].min()}..{H[y == 1].max()}, "
          f"flag heights {H[y == 2].min()}..{H[y == 2].max()} -> threshold {HEIGHT_THRESHOLD}")

    # Baseline: yellow fraction with the best train threshold
    Y, Yt = np.array([yellow_fraction(i) for i in X]), np.array([yellow_fraction(i) for i in Xt])
    t = (Y[y == 1].min() + Y[y == 2].max()) / 2
    print(f"baseline yellow>{t:.3f}: train {accuracy(y, np.where(Y > t, 1, 2)):.4f}, "
          f"test {accuracy(yt, np.where(Yt > t, 1, 2)):.4f}")

    p_tr = [prediksi_gambar_img(i.tolist()) for i in X]
    p_te = [prediksi_gambar_img(i.tolist()) for i in Xt]
    acc = accuracy(yt, p_te)
    print(f"final rule: train {accuracy(y, p_tr):.4f}, test {acc:.4f} "
          f"({int(round((1 - acc) * len(yt)))} errors, indices {list(np.where(np.array(p_te) != yt)[0])})")
    print(f"normalised score = 100*(acc-0.5)/(best-0.5): 100 if no contestant beat {acc:.4f}")

    if "--figures" in sys.argv:
        make_figures(X, y, Xt, yt)


def make_figures(X, y, Xt, yt):
    import matplotlib
    matplotlib.use("Agg")
    import matplotlib.pyplot as plt
    plt.rcParams.update({"font.size": 11})

    # 1. The distribution shift: train faces vs test faces
    fig, axes = plt.subplots(3, 10, figsize=(12, 4.2))
    rng = np.random.RandomState(1)
    sets = [(X[y == 1], "train faces"), (Xt[yt == 1], "test faces"), (Xt[yt == 2], "test flags")]
    for r, (imgs, name) in enumerate(sets):
        idx = rng.choice(len(imgs), 10, replace=False)
        for c, i in enumerate(idx):
            axes[r, c].imshow(imgs[i]); axes[r, c].set_xticks([]); axes[r, c].set_yticks([])
            for s in axes[r, c].spines.values():
                s.set_visible(False)
        axes[r, 0].set_ylabel(name, rotation=0, ha="right", va="center", fontsize=12)
    fig.suptitle("Train faces are all yellow Apple smileys; test faces are not", fontsize=13)
    fig.tight_layout()
    fig.savefig("figures/shift.png", dpi=150, facecolor="white")
    plt.close(fig)

    # 2. Colour feature vs shape feature, train and test
    def feat(arr):
        return (np.array([yellow_fraction(i) for i in arr]),
                np.array([shape_features(i)["h"] for i in arr]),
                np.array([shape_features(i)["aspect"] for i in arr]))
    Ytr, Htr, Atr = feat(X)
    Yte, Hte, Ate = feat(Xt)
    fig, axes = plt.subplots(1, 2, figsize=(12, 4.6))
    bins = np.linspace(0, 0.8, 33)
    ax = axes[0]
    for lab, col, name in [(1, "#d4a017", "faces"), (2, "#3b6ea8", "flags")]:
        ax.hist(Ytr[y == lab], bins, color=col, alpha=0.35, label=f"train {name}")
        ax.hist(Yte[yt == lab], bins, color=col, histtype="step", lw=2, label=f"test {name}")
    ax.set_xlabel("fraction of yellow pixels in the silhouette"); ax.set_ylabel("number of images")
    ax.set_title("Colour: perfect on train, useless on test"); ax.legend(fontsize=9)
    ax = axes[1]
    jit = np.random.RandomState(0).uniform(-0.3, 0.3, len(Hte))
    for lab, col, name in [(1, "#d4a017", "faces"), (2, "#3b6ea8", "flags")]:
        ax.scatter(Atr[y == lab], Htr[y == lab], s=60, facecolors="none", edgecolors=col, lw=1.2,
                   label=f"train {name}")
        ax.scatter(Ate[yt == lab], Hte[yt == lab] + jit[yt == lab], s=14, color=col, alpha=0.7,
                   label=f"test {name}")
    ax.axhline(HEIGHT_THRESHOLD, color="crimson", ls="--", lw=1.5)
    ax.text(0.73, HEIGHT_THRESHOLD + 0.6, "h = 50 px (chosen on train)", color="crimson", fontsize=10)
    ax.set_xlabel("width / height of the bounding box"); ax.set_ylabel("height of the silhouette (px)")
    ax.set_title("Shape: the same gap on train and test"); ax.legend(fontsize=9, loc="center right")
    fig.tight_layout()
    fig.savefig("figures/features.png", dpi=150, facecolor="white")
    plt.close(fig)

    # 3. The two remaining errors
    p = np.array([prediksi_gambar_img(i) for i in Xt])
    bad = np.where(p != yt)[0]
    fig, axes = plt.subplots(1, len(bad), figsize=(2.6 * len(bad), 2.8))
    for ax, i in zip(np.atleast_1d(axes), bad):
        ax.imshow(Xt[i]); ax.axis("off")
        ax.set_title(f"flag, h = {shape_features(Xt[i])['h']} px", fontsize=10)
    fig.tight_layout()
    fig.savefig("figures/errors.png", dpi=150, facecolor="white")
    plt.close(fig)


if __name__ == "__main__":
    main()