IOAI Training: Learn AI by solving problems

The problem in brief

Spaceships race across alien biomes. Each ship has five engineering stats (thermal shield, cryo insulation, wide treads, hull hardness, compressor), and each race takes place in one biome described only by a short name such as "icy tundra" or "volcanic field". For 420 training races you know every ship's finishing time; for 120 test races you must output the finishing order inside each race. The twist: the test biomes carry names that never appear in training ("frozen thin air", "golden sand banks", "magma plains"), so you cannot simply look up how a biome behaves. You have to understand what a name means.

The data are the official Kaggle files of the competition. The test labels are private and we did not submit anything: all scores below come from our own validation on the training races, organised so that each validated biome name is unseen, as in the test.

How you are scored

For each race with $n$ ships, Kendall's $\tau$ compares your order with the true one over all $\binom{n}{2}$ pairs of ships:

\tau = \frac{C - D}{\binom{n}{2}},

where $C$ counts the pairs you put in the right order and $D$ the pairs you put in the wrong order. The leaderboard score is the mean of $\tau$ over races, clipped to $[0, 1]$ . A random order scores about 0, a perfect one 1. Only the order inside a race matters, never the times themselves.

What to aim for. The Evaluation page gives two reference points: the organisers' baseline scores 0.55 and their reference solution 0.86. On the public leaderboard the best entry today has 0.832.

First look at the data

Each race has 6 to 16 ships, the stats lie between 0.05 and 0.98, and the 30 training biomes each cover 8 to 22 races. Finishing times are skewed (median 5.6, max 126), which suggests looking at $\log(\text{time})$ .

The first useful experiment is to fit, inside each race separately, a linear regression of $\log(\text{time})$ on the five stats. It fits almost perfectly: the median $R^2$ is 0.98. So within a race,

\log t_i \approx c_{\text{race}} + w \cdot x_i,

and the order of the ships is just the order of $w \cdot x_i$ . The constant $c_{\text{race}}$ does not matter, and neither does the length of $w$ : the whole problem is to guess the direction of $w$ for a biome you have never seen. If you knew each race's own $w$ , you would score 0.937 (the remaining gap is noise in the times).

Averaging these fitted $w$ per biome reveals a very clean structure (left panel below). The 30 names fall into 6 families of 5 synonyms each, and K-means on the weight vectors finds them with no ambiguity (inertia drops from 1.23 at $k=5$ to 0.20 at $k=6$ , then barely moves):

Weights per biome and the clipped-value leak

family	stats that matter (largest first)
volcanic (lava field, ash volcanic field...)	thermal shield, hull hardness
tundra (cold tundra, icy tundra...)	cryo insulation, compressor
desert (arid desert, hot desert...)	thermal shield, wide treads
thin air (mesosphere, high vacuum...)	compressor
ice spikes (frozen spikes, spiked ice...)	cryo insulation, hull hardness
dunes (sand dunes, rolling dunes...)	wide treads, thermal shield

The test names are different. Some are new synonyms ("magma plains", "sunbaked sands"), but many mix two families: "frozen thin air", "volcanic dunes", "icy sand desert", "tundra vacuum", "hot spike wastes". Nothing in training shows how such a mixture behaves, which is what makes the task hard.

A first baseline

approach	mean $\tau$
all stats count the same ( $w = -\mathbf{1}$ )	0.498
one global $w$ averaged over all races	0.512
per-biome mean $w$ , biome already seen (not possible on test)	0.903
pooled $w$ of the true family, family known (not possible on test)	0.928

The equal-weights baseline lands near the organisers' 0.55. Knowing the biome is worth about 0.4 of $\tau$ : everything hinges on reading the name.

The key idea

There are two complementary sources of information about $w$ , and a good solution uses both.

1. Read the name word by word, and let the words add up. A biome name is a small bag of hazard words. "Frozen" means cold, "desert" means heat and sand, "vacuum" means thin air, "spike" means something sharp. So we give each training word its own weight vector, the average $w$ of the races whose name contains it:

v_{\text{word}} = \operatorname{mean}\{ w_r : \text{word} \in \text{name}(r) \}.

A new name gets the weighted average of its words' vectors. For a word never seen in training ("scorched", "glacial", "magma"), a sentence-embedding model (all-MiniLM-L6-v2) finds the training words with the closest meaning, and we take a softmax-weighted average of their vectors (temperature 0.05). Generic words such as "fields", "plains" or "basin" appear in several families, so their vector sits close to the global mean; we weight each word by $\lVert v_{\text{word}} - \bar v \rVert^2$ so that they barely count. The result for "frozen desert" is a mix of cold and sand, which a nearest-name lookup cannot produce: the closest training name is "desert", which ignores the cold completely.

2. Look at the fleet: it betrays the biome. The organisers' generator leaves a trace in the ships themselves. In a race, the stats that matter are shifted upwards (mean 0.58 instead of 0.51), and the value 0.98, the upper clip of the stats, appears only in stats that matter for that biome (right panel above). In thin-air races, 56 % contain a ship with compressor = 0.98, and no thin-air race ever has a 0.98 in any other stat. This is not hand-labelling: it is a property of the unlabelled test rows, available to any model. It is exactly what helps on the mixed test names. A "frozen vacuum" race contains several ships with cryo insulation at 0.98 and none with compressor at 0.98, which tells you the cold hazard dominates there.

Why should the clip carry information? A plausible story is that the generator draws ships for a biome with a bonus on the useful stats and then clips at 0.98. Only boosted stats ever reach the clip. You never need to know the mechanism, but you do need to look at the raw values: value_counts() on each column shows the spike at 0.98 immediately.

The full solution

Per-race targets. For each training race, fit $\log t$ on the stats by least squares and keep $w_r$ (5 numbers).
Text prior. For the race's biome, compute the word-level prior $p$ described above, using only the other biomes during validation.
Fleet features, one row per (race, stat). Number of ships at exactly 0.98, share of ships above 0.9, share below 0.15, and the stat's mean minus the race's overall mean.
Blend. A linear regression predicts each component $w_{r,j}$ from the prior $p_j$ (centred) and the four fleet features of stat $j$ . It is fitted on the 2 100 (race, stat) rows. We tried gradient boosting and interaction terms here; the plain linear blend was as good or better.
Rank. In each test race, predicted $\log t_i = \hat w \cdot x_i$ ; sort ascending, the smallest is rank 1.

The heart of the text step:

for w in name.split():
    a = np.exp(Ew @ emb([w])[0] / T)       # similarity to every training word
    v = (a / a.sum()) @ V                  # soft nearest training words
    wt = np.linalg.norm(v - gm) ** 2       # informative words weigh more
    tot, z = tot + wt * v, z + wt
return tot / z

How we validated. Leave-one-biome-out (LOBO): for each of the 30 training biomes, everything about that name (its races, its words' statistics from its own races, the blend's training rows) is removed, and its races are ranked from what remains. This mimics "a name you have never seen". It is still kinder than the test, because a held-out synonym like "icy tundra" shares words with "frozen tundra". So we also ran a stricter version in which every word of the held-out name is deleted from the training vocabulary, forcing the embedding model to do all the work.

Results

All scores are mean Kendall $\tau$ on training races, validated leave-one-biome-out. The official test labels are private.

approach	LOBO	LOBO, words also removed
equal weights (baseline)	0.498	0.498
fleet features only, no text	0.631	0.631
word-level family vote (6 families, no stat-level vectors)	0.807	0.720
word-level stat vectors, GloVe embeddings	0.822	0.682
word-level stat vectors, MiniLM embeddings (text only)	0.845	0.739
word-level stat vectors + fleet, linear blend (final)	0.845	0.747
same, gradient-boosted blend	0.807	0.755
nearest whole name (MiniLM), for comparison	0.925	not applicable

On names whose words are known, the fleet features add almost nothing on top of a good text prior (0.845 either way); they help when the words are new (0.739 to 0.747), which is the situation of the test, and they are the only evidence that can arbitrate a mixed name.

The last line deserves a comment. Matching the whole name to the closest training name wins easily on LOBO (0.925), because every training family has 5 near-synonyms. But on the test it would send "frozen desert" to the desert family, "volcanic dunes" to dunes and "frozen vacuum" to thin air, while the fleets of those races show 0.98 values in cryo insulation, thermal shield and cryo insulation respectively. Our validation simply contains no mixed names, so it cannot reward a method for handling them. We chose the word-level model because it is the only one that can represent a mixture, and because its stricter score (0.747) is the more realistic picture of truly new words.

Our final pipeline reaches $\tau = 0.845$ in leave-one-biome-out validation, above the best public leaderboard entry (0.832) and below the organisers' reference (0.86). These numbers are not directly comparable: our validation has no mixed names but keeps shared words, so the true test score could fall on either side.

What to remember

When the metric only looks at order, find the quantity whose order you need. Here $\log t$ is linear in the stats inside a race, so the whole task reduces to estimating the direction of one 5-dimensional vector per biome.
Fit a model per group first, then study the fitted parameters. Thirty weight vectors in a heatmap showed 6 families of synonyms at a glance. That is far more readable than one big model on all rows.
Compose meaning word by word when test inputs are combinations. A nearest-neighbour on whole names is great for synonyms and blind to mixtures. A sum of word vectors can represent "frozen" plus "desert".
Validate the way the test differs from training, and be honest about what your validation cannot see. Leave-one-group-out is the standard tool. Removing shared words gives a harsher, more realistic variant.
Look at raw values for generator artefacts. A clip at 0.98 that only ever appears in useful stats is free, legitimate information about unlabelled test rows.

"""AICC Round 11, "Cosmos Cosmic Run": worked solution.

Idea
1. Inside one race, log(finish_time) is almost exactly linear in the 5 ship stats (R^2 ~ 0.98).
   Only the DIRECTION of the weight vector w matters for the ranking, and w depends on the biome.
2. Biome names are built from words that each carry a hazard ("frozen" -> cold, "desert" -> heat + sand,
   "vacuum" -> thin air...). We learn one weight vector per training WORD, map unseen words to known
   ones with a sentence-embedding model (all-MiniLM-L6-v2), and add up the words of a name.
3. The fleet itself leaks the biome: stats that matter are shifted up, and the value 0.98 (a clip)
   only ever appears in stats that matter. A small linear model blends the text prior with these
   fleet features.

Evaluation: leave-one-biome-out over the 30 training biomes (the test biomes are new names and the
official test labels are private), with mean Kendall tau per race, as on the leaderboard.

Usage: python solution.py [data_dir]   (needs train.csv, test.csv; writes submission.csv)
"""
import os
import sys

os.environ.setdefault("OMP_NUM_THREADS", "2")
import numpy as np
import pandas as pd
import torch
from scipy.stats import kendalltau
from sentence_transformers import SentenceTransformer
from sklearn.linear_model import LinearRegression

torch.set_num_threads(2)
DATA = sys.argv[1] if len(sys.argv) > 1 else "work"
F = ["thermal_shield", "cryo_insulation", "wide_treads", "hull_hardness", "compressor"]
STOP = {"of", "the", "and"}
T = 0.05          # softmax temperature when matching an unseen word to known words
WPOW = 2          # a word counts in proportion to ||v_word - mean||^WPOW (generic words count little)

tr = pd.read_csv(f"{DATA}/train.csv")
te = pd.read_csv(f"{DATA}/test.csv")
tr["ly"] = np.log(tr.finish_time)


# ---------------------------------------------------------------- 1. one weight vector per race
def fit_w(g):
    """Least squares of log(time) on the 5 stats inside one race (intercept dropped)."""
    A = np.c_[np.ones(len(g)), g[F].values]
    return np.linalg.lstsq(A, g.ly.values, rcond=None)[0][1:]


W = {r: fit_w(g) for r, g in tr.groupby("race_id")}
Wr = pd.DataFrame(W, index=F).T
Wr["biome"] = tr.groupby("race_id").biome.first()
names = sorted(tr.biome.unique())
races = {r: g for r, g in tr.groupby("race_id")}

# ---------------------------------------------------------------- 2. text prior from the words
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
_cache = {}


def emb(words):
    missing = [w for w in words if w not in _cache]
    if missing:
        for w, e in zip(missing, model.encode(missing, normalize_embeddings=True)):
            _cache[w] = e
    return np.array([_cache[w] for w in words])


def text_w(name, train_names, banned=()):
    """Weight vector for a biome name: embedding-weighted average of training-word vectors."""
    d = Wr[Wr.biome.isin(train_names)]
    gm = d[F].mean().values
    words = sorted({w for n in train_names for w in n.split()} - set(banned) - STOP)
    V = np.array([d[d.biome.apply(lambda n: w in n.split())][F].mean().values for w in words])
    Ew = emb(words)
    tot, z = np.zeros(5), 0.0
    for w in name.split():
        if w in STOP:
            continue
        a = np.exp(Ew @ emb([w])[0] / T)
        v = (a / a.sum()) @ V                      # soft nearest training words
        wt = np.linalg.norm(v - gm) ** WPOW       # informative words weigh more
        tot, z = tot + wt * v, z + wt
    return tot / z if z > 0 else gm


# ---------------------------------------------------------------- 3. fleet features
def fleet(g):
    X = g[F].values
    return np.c_[(X >= 0.98).sum(0),               # clipped values: only in stats that matter
                 (X >= 0.9).mean(0), (X <= 0.15).mean(0),
                 X.mean(0) - X.mean()]


def rows_for(race_id, g, prior, y=None):
    ff = fleet(g)
    out = []
    for j in range(5):
        out.append(dict(race=race_id, stat=j, prior=prior[j] - prior.mean(), n98=ff[j, 0],
                        hi=ff[j, 1], lo=ff[j, 2], mu=ff[j, 3], y=None if y is None else y[j]))
    return out


COLS = ["prior", "n98", "hi", "lo", "mu"]


def score_race(g, w):
    return kendalltau(g[F].values @ w, g.ly.values).statistic


def build(train_names, target_names, banned_words=False):
    rows = []
    for b in target_names:
        tn = [n for n in train_names if n != b]
        p = text_w(b, tn, banned=b.split() if banned_words else ())
        for r, g in tr[tr.biome == b].groupby("race_id"):
            for d in rows_for(r, g, p, W[r]):
                d["biome"] = b
                rows.append(d)
    return pd.DataFrame(rows)


# ---------------------------------------------------------------- 4. leave-one-biome-out
def lobo(D, cols):
    taus = []
    for b in names:
        a, v = D[D.biome != b], D[D.biome == b]
        m = LinearRegression().fit(a[cols], a.y)
        v = v.assign(p=m.predict(v[cols]))
        taus += [score_race(races[r], q.sort_values("stat").p.values) for r, q in v.groupby("race")]
    return np.mean(taus)


if __name__ == "__main__":
    print("baseline (all stats count the same):",
          round(np.mean([score_race(g, -np.ones(5)) for g in races.values()]), 4))
    print("oracle (weights fitted on the race itself):",
          round(np.mean([score_race(g, W[r]) for r, g in races.items()]), 4))
    D = build(names, names)
    print("LOBO text prior only:", round(lobo(D, ["prior"]), 4))
    print("LOBO text prior + fleet:", round(lobo(D, COLS), 4))
    Dd = build(names, names, banned_words=True)
    print("LOBO, words of the held-out name also removed:", round(lobo(Dd, COLS), 4))

    # ------------------------------------------------------------ 5. test predictions
    final = LinearRegression().fit(D[COLS], D.y)
    sub = []
    for r, g in te.groupby("race_id"):
        p = text_w(g.biome.iloc[0], names)
        X = pd.DataFrame(rows_for(r, g, p))[COLS]
        w = final.predict(X)
        pred_logtime = g[F].values @ w
        ranks = pred_logtime.argsort().argsort() + 1          # 1 = fastest
        sub.append(pd.DataFrame({"id": g.id.values, "predicted_rank": ranks}))
    pd.concat(sub).sort_values("id").to_csv("submission.csv", index=False)
    print("wrote submission.csv (not submitted)")