IOAI Training: Learn AI by solving problems

The problem in brief

A patrol drone photographs places and must tell coast from desert. By day it takes colour photos, and 49 of them were labelled by experts. By night it takes only grayscale photos: 149 are available for training but unlabelled. At patrol time a new image may be either colour or grayscale. You must write a training function using both sets and a classify_image function returning a score close to 1 for Coast and close to 0 for Desert. The model is scored on a public colour test set (30 images), a public grayscale test set (70 images) and a secret set. All images are small photos of 128x128 pixels. This is a small domain adaptation problem: labels in one domain (colour), deployment in another (grayscale).

How you are scored

The ROC AUC on each public test set separately, Coast being the positive class:

\mathrm{AUC} = P\big(s(x_{\text{coast}}) > s(x_{\text{desert}})\big),

the probability that a random coast image gets a higher score than a random desert image. Points were given by (unpublished) performance ranges, so there is no target. Two remarks:

AUC only depends on the ranking of scores within one test set. A model whose grayscale scores are all shifted down still gets full AUC if the order is right; accuracy at a 0.5 threshold would suffer, AUC would not.
The provided evaluator would not run as written: classify_image is documented as returning a label and a probability, but the evaluator uses its output directly as a score; roc_curve receives the strings "Coast"/"Desert" without pos_label; and evaluate_classifier returns an undefined roc_auc. We return P(Coast) and compute the AUC with Coast as the positive class.

Data note. The released grayscale_images_train.csv also contains a label column, although the statement says these images are unlabelled. Using it would break the spirit of the problem, so our code reads only the paths.

First look at the data

Classes are balanced (25/24 colour training images, 75/74 grayscale). Coasts show water, sky, vegetation, horizontal horizons and often blue tones; deserts show warm-coloured sand, dune ripples and smooth gradients. Without colour, a white beach and pale sand become very similar, so grayscale is genuinely harder. With 49 labelled images, training a CNN from scratch is hopeless: we need pretrained features (no other data is used, which is what the rules restrict).

A first baseline

Hand-made features on the grayscale version of each image (16-bin intensity histogram, mean and standard deviation, gradient energy, ratio of horizontal to vertical gradients, top-minus-bottom brightness) with a logistic regression reach only AUC 0.72 (colour test) and 0.77 (grayscale test).

Frozen ImageNet ResNet-18 features change the picture: the 512-d global-average-pooled vector (L2-normalised, averaged with its horizontal flip) and a logistic regression trained on the 49 colour images give AUC 0.991 on colour and already 0.949 on grayscale (a grayscale image is fed as three identical channels). The regularisation $C = 10$ was chosen by leave-one-out AUC on the 49 training images (0.931, against 0.914 for $C = 1$ and 0.917 for $C = 100$ ).

The key idea

First, an intuitive idea that failed: make the model colour-blind. Convert the colour training images to grayscale too, so that training and deployment see the same kind of input. Leave-one-out AUC drops from 0.93 to 0.81, and on the tests the result is 0.76 on colour (we threw away the colour) and 0.92 to 0.95 on grayscale (no gain). With only 49 labels, colour is part of the signal, and pretrained features already transfer well to grayscale.

What the domain shift really does. The right panel of the figure shows the day model's scores on grayscale test images: their ranking is good (AUC 0.949), but all coast scores slide towards 0.5, because the "blue water, blue sky" evidence disappears at night. At a 0.5 threshold, accuracy falls to 0.77. The model is mis-calibrated on the new domain, not wrong about the order.

Use the unlabelled night images for self-training. Score the 149 night images with the day model, take the 30 % most confidently Coast and the 30 % most confidently Desert as pseudo-labels (by rank, so the shift does not matter), and refit the logistic regression on colour plus pseudo-labelled night images. The decision boundary now sits where the night images actually are. Accuracy on the grayscale test rises from 0.77 to 0.89 and AUC from 0.949 to 0.953 (within noise on 70 images, but it does not hurt). Finally a router sends colour images to the day model and grayscale images to the night model, so each domain uses the expert trained for it.

Left: ROC curves of the final models on the two public test sets. Right: P(Coast) on the grayscale test images for the day model (filled) and the self-trained night model (outlined); the night model moves the coast images back above 0.5.

The full solution

Features: ResNet-18 (ImageNet weights, frozen), input resized to 224 and normalised, grayscale replicated to 3 channels, flip averaging, global-average-pooled and L2-normalised.
Day model: logistic regression ( $C = 10$ , chosen by leave-one-out AUC) on the 49 labelled colour images.
Night model: pseudo-label the 30 % most confident images of each side among the 149 unlabelled grayscale ones, refit on colour plus pseudo-labelled night images.
classify_image: detect grayscale (mode "L" or equal channels), route to the night or day model, return P(Coast).

p = day.predict_proba(X_night)[:, 1]
lo, hi = np.quantile(p, [.3, .7]); keep = (p <= lo) | (p >= hi)      # rank-based, shift-proof
night = LogisticRegression(C=10).fit(np.vstack([X_day, X_night[keep]]),
                                     np.r_[y_day, (p[keep] >= hi).astype(int)])

Results

Public test sets with their official labels (30 colour, 70 grayscale images).

approach	colour AUC	grayscale AUC	grayscale accuracy at 0.5
hand-made grayscale features + logistic regression	0.720	0.765
ResNet-18 on grayscale-converted images, $C = 1$ ("colour-blind", failed)	0.756	0.918	0.79
ResNet-18 on grayscale-converted images, $C = 10$		0.947	0.84
ResNet-18 colour features, day model only, $C = 10$	0.991	0.949	0.77
+ mean-centring each domain (feature alignment)		0.949	0.86
router: day model for colour, self-trained night model for grayscale (final)	0.991	0.953	0.89

With 30 and 70 test images, differences below about 0.02 AUC are within noise. The robust conclusions are: pretrained features are essential (0.72 to 0.99), colour should be kept when it is available, and the unlabelled night images mainly fix the calibration of the grayscale scores.

What to remember

With 49 labels, borrow representations. Frozen pretrained features plus a linear model beat any hand-made features by a wide margin.
Domain shift often hurts calibration before it hurts ranking. Check both AUC and accuracy at your threshold: here AUC stayed at 0.95 while accuracy fell to 0.77.
Unlabelled target-domain data is precious. Rank-based self-training (keep the most confident on each side) moves the decision boundary to where the new domain lives, without any label.
Do not throw information away to "match" domains. Converting training images to grayscale sounded principled and cost a lot on colour images; route each domain to its own expert instead.
Read the evaluator and the data release critically. The scoring code would not run as written, and the "unlabelled" file actually had labels; a careful contestant notices both and does not use the leaked labels.

"""INAIO 2025 Stage 2, Problem 6: Drone Patrol (Coast vs Desert, colour by day, grayscale by night).

Features: frozen ImageNet ResNet-18 (global average pool, 512-d, L2-normalised, horizontal-flip TTA).
A grayscale image is fed as three identical channels. Classifier: logistic regression, C = 10 chosen
by leave-one-out AUC on the 49 labelled colour images.
Two experts behind a router:
  * colour image   -> model trained on the labelled colour images;
  * grayscale image -> same model, then self-trained on the 149 UNLABELLED night images
                       (pseudo-labels for the 30 % most confident on each side, one round).
The grayscale training CSV of the release also carries labels; they are NEVER read.
Usage: python solution.py <data_dir> <resnet18.safetensors>
"""
import os, sys, numpy as np, pandas as pd, torch
from PIL import Image
from torchvision import models
from safetensors.torch import load_file
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
torch.set_num_threads(2)
MEAN = torch.tensor([0.485, 0.456, 0.406]).view(1, 3, 1, 1); STD = torch.tensor([0.229, 0.224, 0.225]).view(1, 3, 1, 1)
C = 10.0


class Featuriser:
    def __init__(self, weights):
        m = models.resnet18(weights=None); m.load_state_dict(load_file(weights)); m.fc = torch.nn.Identity()
        self.m = m.eval()

    @torch.no_grad()
    def __call__(self, paths, gray=False):
        X = []
        for p in paths:
            im = Image.open(p)
            im = im.convert("L").convert("RGB") if gray else im.convert("RGB")
            x = torch.tensor(np.array(im.resize((224, 224), Image.BILINEAR)), dtype=torch.float32).permute(2, 0, 1)[None] / 255
            x = (x - MEAN) / STD
            X.append(((self.m(x) + self.m(torch.flip(x, dims=[3]))) / 2)[0].numpy())
        X = np.array(X)
        return X / np.linalg.norm(X, axis=1, keepdims=True)


def is_gray(path):
    a = np.array(Image.open(path).convert("RGB")).astype(int)
    return Image.open(path).mode == "L" or (np.abs(a[..., 0] - a[..., 1]).max() < 8 and np.abs(a[..., 1] - a[..., 2]).max() < 8)


def learn_location_type_classifier(color_csv, gray_csv, feat):
    col = pd.read_csv(color_csv, encoding="utf-8-sig")
    unl = pd.read_csv(gray_csv, encoding="utf-8-sig").iloc[:, :1]          # paths only: no labels used
    Xc = feat(col.iloc[:, 0]); yc = (col.label == "Coast").astype(int).values
    day = LogisticRegression(C=C, max_iter=5000).fit(Xc, yc)
    Xu = feat(unl.iloc[:, 0], gray=True)
    p = day.predict_proba(Xu)[:, 1]; lo, hi = np.quantile(p, [.3, .7]); keep = (p <= lo) | (p >= hi)
    night = LogisticRegression(C=C, max_iter=5000).fit(np.vstack([Xc, Xu[keep]]), np.r_[yc, (p[keep] >= hi).astype(int)])
    return dict(day=day, night=night, feat=feat)


def classify_image(image_path, model):
    """P(Coast): close to 1 = Coast, close to 0 = Desert (used directly as the ROC score)."""
    g = is_gray(image_path); clf = model["night"] if g else model["day"]
    return float(clf.predict_proba(model["feat"]([image_path], gray=g))[0, 1])


if __name__ == "__main__":
    os.chdir(sys.argv[1]); model = learn_location_type_classifier("color_images_train.csv", "grayscale_images_train.csv", Featuriser(sys.argv[2]))
    for csv in ("color_test_public.csv", "grayscale_test_public.csv"):
        df = pd.read_csv(csv, encoding="utf-8-sig"); s = np.array([classify_image(p, model) for p in df.iloc[:, 0]])
        y = (df.label == "Coast").values
        print(f"{csv:28s} AUC {roc_auc_score(y, s):.3f}  accuracy {((s > .5) == y).mean():.3f}")

Drone Patrol: From Vivid Landscapes to Monochrome Missions