IOAI Training: Learn AI by solving problems

The problem in brief

Two ImageNet classifiers, a convolutional ResNet-18 ("R") and a transformer ViT-Tiny ("V"), both classify every image of the task correctly. For each image you must craft two tiny additive perturbations at the image's original resolution. Type A leaves R correct but makes V wrong; type B does the opposite. The perturbed image goes through the fixed pipeline (clip to $[0,1]$ , resize the short side to 256, center-crop 224, normalise) before both models see it. You earn a point for each successful perturbation, and the total is multiplied by a penalty that punishes perturbation size. Leaderboard B (100 hidden images, labels given) decided the final ranking. After the contest the organisers published those images and labels, so everything below is scored on the real final split with the official grader code.

How you are scored

With $M$ images, $S_A$ and $S_B$ the numbers of successful type A and type B perturbations:

S = \frac{S_A + S_B}{2M}\cdot PF, \qquad PF = 1.5 - \sigma(50000\,\bar m),

where $\sigma$ is the logistic sigmoid and $\bar m$ is the mean over images of $(\lVert\delta_A\rVert_2 + \lVert\delta_B\rVert_2)/(3HW)$ , the L2 norms divided by the number of raw pixel values. Near zero, $\sigma(x) \approx \tfrac12 + \tfrac x4$ , so

PF \approx 1 - 12500\,\bar m .

$PF$ is 1 only for zero perturbations and falls to 0.5 for large ones. For a typical 500x375 image, two perturbations with L2 norm 0.3 each give $\bar m \approx 1.1\times10^{-6}$ and $PF \approx 0.987$ .

Reference points: the official baseline (a uniform offset of $10^{-4}$ times the mean pixel) scores 0, and the statement gives a Scientific Committee score of 96.99. In the official results table (ioai-official.org, Results 2026), 29 of the 440 contestants scored at least 90 on this task, and the median was 9.5. The best entry in that table is exactly 100, which is impossible for a raw score (any nonzero perturbation gives $PF < 1$ ), so the table seems to be rescaled to the best contestant. We therefore compare with the committee's 96.99.

First look at the data

The images are ordinary ImageNet photos of different sizes (500x375, 600x600, 800x600...). Three facts about the pipeline shape the whole solution.

Everything outside the central crop is invisible. After resizing, only a central square of side $224/256$ of the short side reaches the models. Any perturbation outside it is wasted norm. If you optimise through the real pipeline, autograd gives those pixels a zero gradient and they stay at 0 on their own.
The perturbation is a float tensor, not an 8-bit image. Nothing rounds it to multiples of $1/255$ , so changes far smaller than one gray level are allowed. Our final perturbations change any pixel by at most 0.02, about 5 gray levels, and most pixels by much less than one.
Both models are confident on clean images, but not robust. Standard (not adversarially trained) networks are famously fragile: a perturbation of L2 norm well below 1 is enough to flip them. The real difficulty is to flip one model without flipping the other, and to do it with the smallest possible norm.

A first baseline

The natural first attempt (and the one the GAITE hints suggest) treats the perturbation $\delta$ as a trainable tensor and runs Adam on

\mathcal L = \mathrm{CE}(\text{keep model}) - \mathrm{CE}(\text{fool model}),

clipping $\delta$ to $[-0.005, 0.005]$ after each step and stopping as soon as the keep model is right and the fool model is wrong. On 20 training images this succeeds 39 times out of 40, with a mean $\bar m = 3.4\times10^{-6}$ , so $PF = 0.958$ and a score of 93.4. That is already a strong score, but it has two flaws.

It does not look for the smallest perturbation. Adam with an L-infinity box spreads changes over all pixels. Early stopping keeps the first success, not the cheapest one.
It stops exactly on the decision boundary. At the moment of success the fool model's wrong class wins by a hair. The contest grader ran on a CUDA GPU, where convolutions may use TF32 arithmetic, and logits can differ from your laptop's by a few hundredths. A perturbation that wins by 0.001 logit locally can fail at grading time.

The key idea

Treat the task as what it really is: a minimum-norm adversarial example problem with a two-sided constraint. For type A, writing $m_R$ and $m_V$ for the margins (true-class logit minus the best other logit, positive when the model is right):

\min_\delta \lVert\delta\rVert_2 \quad\text{subject to}\quad m_R(x+\delta) > \kappa,\;\; m_V(x+\delta) < -\kappa .

The safety margin $\kappa$ (we use 0.2 logit) protects against numerical differences on the grading machine. Type B swaps the roles of R and V.

Fixing a norm budget in advance (as the L-infinity box does) is the wrong way to solve this, because the right budget differs a lot from image to image. The DDN attack (Decoupled Direction and Norm, Rony et al., 2019) separates the two questions.

Direction. Take a step of the hinge loss

\ell(\delta) = \operatorname{relu}(m_{\text{fool}} + \kappa) + \operatorname{relu}(\kappa - m_{\text{keep}})

along its normalised gradient. The loss is zero once both constraints hold with margin, so only the violated constraint pushes.

Norm. Keep a radius $\varepsilon$ and rescale $\delta$ onto the sphere $\lVert\delta\rVert_2 = \varepsilon$ after each step. If the current $\delta$ already succeeds, shrink it, $\varepsilon \leftarrow 0.95\,\varepsilon$ ; if not, grow it, $\varepsilon \leftarrow 1.05\,\varepsilon$ .

The radius thus oscillates around the smallest norm at which the current direction still works, while the direction keeps improving. Over 100 steps we remember the smallest successful $\delta$ . Because the whole pipeline (clip, antialiased bilinear resize, crop, normalise) is differentiable, we optimise $\delta$ directly at the raw resolution, and the gradient automatically spreads the change over the raw pixels in the cheapest way for the L2 norm.

Leaderboard B image 7. Middle: the type A perturbation (fool the ViT) is organised in a grid of 16x16 squares, the ViT's patches. Right: the type B perturbation (fool the ResNet) is a fine texture spread over the object. The dashed square is the part of the image the models see; outside it the perturbation is exactly zero.

The figure shows why this task is called a dilemma between two "brains". To fool the ViT (type A), the attack writes inside the 16x16 patches that the transformer tokenises, with visible patch edges, on a few patches over the animal. To fool the ResNet (type B), it writes a high-frequency texture of tiny oriented strokes: exactly what convolutional filters respond to. Each model is attacked through its own inductive bias, which is also why the other model barely notices.

The full solution

Load the two allowed models (torchvision ResNet-18 IMAGENET1K_V1 and timm vit_tiny_patch16_224) in eval mode with frozen weights, and reproduce the grader's _preprocess exactly.
For each image, attack both types at once (a batch of 2 perturbations): 100 DDN steps, start radius 0.5, radius factor $\gamma = 0.05$ , step size $\alpha\varepsilon$ with $\alpha$ decaying from 0.5 to 0.05 (cosine), margin $\kappa = 0.2$ .
After each step, project onto the radius, then onto the valid pixel box ((img + delta).clamp(0, 1) - img), so that no norm is wasted on clipped pixels.
Write {i}_a.pt and {i}_b.pt (float tensors of shape 3xHxW) into a flat submission.zip.

The heart of the attack:

loss = (torch.relu(fool + KAPPA) + torch.relu(KAPPA - keep)).sum()
g, = torch.autograd.grad(loss, delta)
with torch.no_grad():
    for i in range(2):
        delta[i] -= alpha * eps[i] * g[i] / g[i].norm()      # move the direction
        eps[i] *= (1 - GAMMA) if ok[i] else (1 + GAMMA)      # adapt the norm
        delta[i] *= eps[i] / delta[i].norm()                 # back onto the eps-sphere
        delta[i].copy_((img + delta[i]).clamp(0, 1) - img)   # stay inside [0, 1]

Cost: about 0.15 s per step for both models on two CPU threads, so 15 to 20 s per image on a laptop. On the contest GPU the 100 images fit easily in the 12 minutes. The antialiased resize has no backward pass on Apple MPS, so we ran everything on the CPU.

Checking robustness. We scored the zip with the official evaluate.py (build_preds + compute_score), then again with every ResNet convolution's inputs and weights rounded to TF32 precision (10-bit mantissa), to imitate a CUDA grader with cudnn.allow_tf32 = True. All 200 perturbations still succeed: the 0.2-logit margin is worth its small price in norm.

Left: the penalty factor as a function of the mean L2 per pixel, with the baseline and the final attack. Right: L2 norms of the 200 perturbations on leaderboard B: fooling the ResNet costs more than fooling the ViT.

Results

approach	split	success	mean L2/pixel	PF	score
official baseline: uniform offset	statement				0
Adam on CE(keep) - CE(fool), L-inf 0.005, early stop	train, 20 images	39/40	$3.36\times10^{-6}$	0.958	93.4
DDN, 30 steps, start radius 1, $\kappa = 0.3$	train, 3 images	6/6	$1.19\times10^{-6}$	0.985	98.5
DDN, 100 steps, $\kappa = 0.05$	train, 20 images	40/40	$1.02\times10^{-6}$	0.987	98.7
DDN, 100 steps, $\kappa = 0.2$	train, 20 images	40/40	$1.06\times10^{-6}$	0.987	98.7
DDN, 100 steps, $\kappa = 1.0$	train, 20 images	40/40	$1.31\times10^{-6}$	0.984	98.4
DDN, 100 steps, $\kappa = 0.2$ (final, `solution.py`)	leaderboard B	200/200	$1.26\times10^{-6}$	0.984	98.42
same zip, ResNet with TF32-rounded convolutions	leaderboard B	200/200	$1.26\times10^{-6}$	0.984	98.42

The final score of 98.42 on the official final split is above the Scientific Committee's 96.99. Leaderboard A was not run (half an hour of CPU per split), but the train numbers suggest it would be very close. Every perturbation succeeds, so the remaining 1.6 points are entirely the price of the norm, and they are hard to win back: the margin study shows that even $\kappa = 0.05$ gains less than 0.1 point, and 100 steps against 30 gained only about 10 % in norm. The median norms are 0.27 for type A and 0.39 for type B: in this pair the ViT is the easier model to fool.

What to remember

Read the scoring function before choosing the method. $PF \approx 1 - 12500\,\bar m$ told us that success is cheap and that the whole game is the size of the perturbation, so the right tool is a minimum-norm attack, not a fixed-budget one.
Decouple direction and magnitude. DDN's "shrink when it works, grow when it doesn't" finds a per-image budget automatically. The same idea appears in step-size control and in bisection on a threshold.
Optimise through the real pipeline. Differentiating through clip, resize and crop gives zero perturbation outside the crop and the cheapest raw-pixel change for free. Approximating the pipeline costs norm or success.
Leave a margin for the grader's hardware. Stopping exactly on the decision boundary is fragile across GPU and CPU arithmetic; a small logit margin, plus a stress test with lower-precision arithmetic, makes the result trustworthy.
Architectures have their own blind spots. ViTs are attacked through their patch grid, CNNs through fine textures. Differential attacks like this one are a direct way to see what each inductive bias actually looks at.

"""IOAI 2026 Day 2, Double Agent Dilemma: minimum-norm *differential* adversarial perturbations.

For every image we look for two tiny perturbations:
  type A: ResNet18 still right, ViT-Tiny wrong;   type B: ViT-Tiny still right, ResNet18 wrong.
Method: a DDN-style attack (Decoupled Direction and Norm, Rony et al. 2019) on a two-model margin loss.
  * the DIRECTION of delta follows the normalised gradient of a hinge loss on both models' margins;
  * the NORM eps of delta is adapted multiplicatively: shrink when the current delta already works,
    grow when it does not. We keep the smallest successful delta, with a safety margin KAPPA on logits.
The perturbation lives at the raw resolution and goes through the exact grading pipeline
(clip -> Resize(256, antialiased bilinear) -> CenterCrop(224) -> Normalize), so autograd finds the
cheapest raw-pixel change (pixels outside the crop never move).

Usage:  python solution.py <split_dir> <out.zip>      (split_dir contains images/ and labels.json)
Weights: models/resnet18.safetensors (torchvision IMAGENET1K_V1) and models/model.safetensors
(timm vit_tiny_patch16_224.augreg_in21k_ft_in1k), as fetched by the official download-models.py.
"""
import os, sys, json, math, io, zipfile, time
os.environ.setdefault("OMP_NUM_THREADS", "2")
import torch, timm
import torchvision.transforms as T
import torchvision.transforms.functional as TF
from torchvision import models
from safetensors.torch import load_file
from PIL import Image

torch.set_num_threads(2)
torch.manual_seed(0)
STEPS, KAPPA, GAMMA, EPS0 = 100, 0.2, 0.05, 0.5
MODELS = os.environ.get("MODELS_DIR", "models")
MEAN = torch.tensor([0.485, 0.456, 0.406]).view(3, 1, 1)
STD = torch.tensor([0.229, 0.224, 0.225]).view(3, 1, 1)


def load_models():
    r = models.resnet18(weights=None)
    r.load_state_dict(load_file(f"{MODELS}/resnet18.safetensors"))
    v = timm.create_model("vit_tiny_patch16_224", pretrained=True,
                          pretrained_cfg_overlay={"file": f"{MODELS}/model.safetensors", "custom_load": False})
    for p in (*r.parameters(), *v.parameters()):
        p.requires_grad_(False)
    return r.eval(), v.eval()


def preprocess(x):  # identical to the grader's _preprocess
    x = TF.resize(x, size=256, interpolation=T.InterpolationMode.BILINEAR)
    x = TF.center_crop(x, output_size=224)
    return (x - MEAN) / STD


def margins(R, V, x, y):
    """Margin = true-class logit minus best other logit (>0 means correct), for both models."""
    inp = preprocess(x)
    out = []
    for m in (R, V):
        z = m(inp)
        other = z.clone()
        other[:, y] = -1e9
        out.append(z[:, y] - other.max(1).values)
    return out


def attack(R, V, img, y):
    """Returns (delta_A, delta_B). Row 0 of the batch = type A, row 1 = type B."""
    delta = torch.zeros(2, *img.shape, requires_grad=True)
    eps = [EPS0, EPS0]
    best = [torch.zeros_like(img), torch.zeros_like(img)]
    best_norm = [math.inf, math.inf]
    for t in range(STEPS):
        mR, mV = margins(R, V, (img + delta).clamp(0, 1), y)
        keep = torch.stack([mR[0], mV[1]])   # the model that must stay right
        fool = torch.stack([mV[0], mR[1]])   # the model that must become wrong
        ok = (keep > KAPPA) & (fool < -KAPPA)
        for i in range(2):                   # remember the smallest working delta
            n = delta[i].detach().norm().item()
            if ok[i] and n < best_norm[i]:
                best_norm[i], best[i] = n, delta[i].detach().clone()
        loss = (torch.relu(fool + KAPPA) + torch.relu(KAPPA - keep)).sum()
        g, = torch.autograd.grad(loss, delta)
        alpha = 0.05 + 0.225 * (1 + math.cos(math.pi * t / STEPS))  # step, relative to eps
        with torch.no_grad():
            for i in range(2):
                gn = g[i].norm()
                if gn > 0:
                    delta[i] -= alpha * eps[i] * g[i] / gn          # move the direction
                eps[i] *= (1 - GAMMA) if ok[i] else (1 + GAMMA)     # adapt the norm
                dn = delta[i].norm()
                if dn > 0:
                    delta[i] *= eps[i] / dn                         # project on the eps-sphere
                delta[i].copy_((img + delta[i]).clamp(0, 1) - img)  # no wasted norm on clipped pixels
    return best[0], best[1]


def main(split_dir, out_zip):
    R, V = load_models()
    labels = json.load(open(f"{split_dir}/labels.json"))
    t0 = time.time()
    with zipfile.ZipFile(out_zip, "w") as zf:
        for k in sorted(labels, key=int):
            img = TF.to_tensor(Image.open(f"{split_dir}/images/{int(k):04d}.png").convert("RGB"))
            da, db = attack(R, V, img, labels[k])
            for tag, d in (("a", da), ("b", db)):
                buf = io.BytesIO()
                torch.save(d.contiguous(), buf)
                zf.writestr(f"{k}_{tag}.pt", buf.getvalue())
            print(k, f"|dA|={da.norm():.3f} |dB|={db.norm():.3f}  {time.time() - t0:.0f}s", flush=True)


if __name__ == "__main__":
    main(sys.argv[1], sys.argv[2])