IOAI Training: Learn AI by solving problems

The problem in brief

Comic-book heroes from many universes have been merged into a single table: one row per character, with descriptive traits (publisher, gender, race, eye and hair colour, height, weight, alignment), about 80 yes/no abilities (Flight, Agility, Telepathy...) and numerical "power stats" (intelligence, strength, speed, durability, power, combat). There are 505 heroes in train.csv and 100 in test.csv. Two warm-up questions ask for simple statistics on the training table, then two modelling tasks: predict each test hero's combat stat (regression) and whether they have Super Strength (binary classification).

How you are scored

100 points in four parts:

task	what	scoring
1	number of distinct `Publisher` values in train	exact answer, 10 points
2	publisher with the most `Alignment = good` heroes (ties: alphabetical)	exact answer, 10 points
3	`powerstats__combat` for each test hero	MAE, 40 points
4	`Super Strength` (0 or 1) for each test hero	accuracy, 40 points

For task 3, with $\text{MAE} = \frac{1}{n}\sum_i |y_i - \hat y_i|$ :

\text{score}_3 = 40 \cdot \frac{30 - \text{MAE}}{30 - 12}, \quad \text{clipped to } [0, 40],

so MAE $\le 12$ earns the full 40 points. For task 4, $\text{score}_4 = 40 \cdot \frac{\text{acc} - 0.55}{0.90 - 0.55}$ clipped to $[0, 40]$ , so accuracy $\ge 0.90$ is the full-score line (statement, "Evaluation"). Test labels are private: the scores below are 5-fold cross-validation on the training set.

Tasks 1 and 2: the warm-up

Both are one line of pandas, as long as you read the question literally.

tr["Publisher"].nunique() gives 23. No value is missing in this column, so there is no "should NaN count?" ambiguity. Marvel (276) and DC (141) dominate; eight publishers appear only once.
Among the 323 heroes with Alignment == "good", value_counts() gives Marvel Comics 172, DC Comics 87, NBC - Heroes 12. No tie, so the answer is Marvel Comics. Careful: 25 heroes have a missing alignment and 5 have the value "-": they are simply not "good".

First look at the data

The interesting discovery is in the power stats. Plot a histogram of any of them, say intelligence:

Left: histogram of the intelligence stat, with a tall spike of about 70 heroes between -5 and 5, far from the bulk of values between 35 and 100. Right: combat (the target) against the number of near-zero visible stats: heroes with 0 or 1 near-zero stats have combat values spread from 5 to 100, heroes with 2 to 5 near-zero stats all have combat exactly -1.

Real values live between about 5 and 100 (with a few noisy values above 100, up to 250 for strength), but about 15 % of each stat sits in a narrow cloud around 0, with negative values like $-0.78$ or $-2.7$ . A stat of $-2.7$ makes no sense. The explanation: in the public superhero database this table comes from, an unknown stat is written as $-1$ . The organisers added a little random noise to the five visible stats, which turned the clean marker $-1$ into a fuzzy cloud in $(-5, 5)$ . The target column powerstats__combat was left clean: it contains exactly $-1$ for 72 heroes (14 %).

And the stats of a hero tend to be unknown together. Counting, for each hero, how many of the five visible stats fall in $(-5, 5)$ :

near-zero visible stats	0	1	2	3	4	5
heroes with combat $= -1$	0	0	1	4	31	36
heroes with a real combat	425	8	0	0	0	0

The rule "2 or more unknown stats $\Rightarrow$ combat $= -1$ " is exact on all 505 training heroes. In the test set, 17 heroes satisfy it.

Two other observations: the extra columns meta_score, efficiency_delta and stability_index look synthetic (correlation with combat $\approx 0$ , flat averages across the A to D categories), and Super Strength is strongly tied to the strength stat (84 % to 100 % of heroes with strength above 20 have it, only 14 % of those between 5 and 20).

A first baseline

Task 3: predicting the median combat (60) for everybody gives MAE 24.9, which is worth only 11 of 40 points. Simply applying the rule above (predict $-1$ for the 72 "unknown" heroes, the median of the others for the rest) already drops the MAE to 16.0.

Task 4: the majority class (Super Strength $= 1$ , 55 %) gives accuracy 0.550, i.e. 0 points. A random forest on all columns gives 0.873.

The key idea

A missing-value marker is information, not a number. The $-1$ convention, made fuzzy by noise, is the single most important structure in this dataset. It matters in two ways.

For the target. Absolute error is unforgiving for this group: if a model predicts 50 for a hero whose combat is $-1$ , it pays 51 points of error on that hero. With 14 % of such heroes, getting them wrong costs about $0.14 \times 51 \approx 7$ points of MAE, more than the whole gap between a mediocre and a perfect model on the others. Since the rule is exact, we hard-code it: those heroes receive exactly $-1$ and cost zero.
For the features. A stat of $0.3$ that really means "unknown" must not be read as "very weak". We replace every visible stat in $(-5, 5)$ by NaN (later filled with the median of known values), clip the remaining values to $[0, 100]$ , and add the count of unknown stats as a feature. A model then learns from real stats only. We also train the combat regressors only on the 433 heroes whose combat is known: mixing the $-1$ heroes into the training set would drag the regression towards $-1$ for anybody with weak-looking stats.

Why does a tree model not discover all this alone? A tree could in principle split on "intelligence $< 5$ ", but it must learn the same pattern separately from noisy columns with only 505 rows, and a squared-error regressor still averages $-1$ heroes with real ones in mixed leaves. Encoding the domain knowledge explicitly is cheaper and exact.

The full solution

Features (shared by tasks 3 and 4): the five visible stats cleaned as above, the number of unknown stats, the ~80 binary abilities, the number of abilities, height, weight, indicators for the most frequent races (Human, Mutant, God / Eternal, Human / Radiation, Android, Alien, Cyborg, "-").

Task 3: for heroes with 2 or more unknown stats, predict $-1$ . For the others, average three regressors trained on the heroes with known combat: a random forest (500 trees, min_samples_leaf=3, max_features=0.3), extra trees, and a Huber regression on standardised features (robust to outliers, which suits an MAE metric). The trees capture interactions (combat is high for "Peak Human Condition", "Weapons Master" or "Marksmanship" heroes: +25 to +30 on average), the linear model the smooth link with intelligence (correlation 0.71).

Task 4: a random forest (800 trees, max_features=0.3) on the same features, without the synthetic columns and the publisher, gender and alignment indicators, which only added noise.

unknown = d[STATS].abs() < 5                 # the noisy "-1 = unknown" marker
X["n_unknown_stats"] = unknown.sum(axis=1)
for s in STATS:
    X[s] = d[s].where(~unknown[s]).clip(0, 100)
...
pred = np.where(n_unknown >= 2, -1, np.mean([rf, et, huber], axis=0))

Results

5-fold cross-validation on the 505 training heroes (repeated 3 to 4 times with different splits for the final lines). Points use the official formulas.

task 3: combat	MAE	points / 40
constant median	24.94	11.2
rule ( $-1$ ) + median of the others	16.02	31.1
HistGradientBoosting (absolute loss), no rule	13.15	37.4
rule + HistGradientBoosting (absolute loss)	13.05	37.7
rule + random forest	12.63 to 12.81	38.2 to 38.6
rule + Huber regression	13.02	37.7
rule + random forest using the median of the trees	12.75	38.3
rule + mean of random forest, extra trees, Huber (final)	12.59	38.7

task 4: Super Strength	accuracy	points / 40
majority class	0.550	0
logistic regression	0.826	31.5
extra trees	0.857	35.1
random forest, all columns	0.873 to 0.885	36.9 to 38.3
HistGradientBoosting	0.882	37.9
random forest without synthetic/meta columns (final)	0.890	38.9

With tasks 1 and 2 correct, the expected total is about $10 + 10 + 38.7 + 38.9 \approx 97.6$ out of 100. With 505 rows the standard error is about 0.4 MAE and 0.014 accuracy, so differences of a few tenths between models in the tables are not significant.

What did not work: adding race indicators and the ability count did not change the combat MAE; blending the classifiers (random forest + logistic regression) made task 4 worse (0.855), because the logistic regression is clearly weaker and drags the average; tuning max_features moved accuracy by less than 0.01. Errors in task 4 are concentrated on heroes with unknown stats (22 % error rate versus 10 % for the others): without a strength value, the model only has the ability list to go on.

What to remember

Always histogram your columns. A spike in an impossible place (negative stats here) is usually a sentinel value for "missing", possibly disguised by noise.
Sentinels must be decoded, not modelled. Turn them into NaN plus a "was missing" indicator, and when the target itself has a sentinel, look for a deterministic rule that predicts it before training anything.
Know what your metric punishes. Under MAE, a handful of grossly wrong predictions (50 instead of $-1$ ) costs more than small errors everywhere; under accuracy, a constant prediction is worth zero points here.
Train on the population you will predict. The combat regressors learn only from heroes with a real combat value, because the others are handled by the rule.
Remove noise columns when the dataset is small: dropping the synthetic columns gained about 0.01 accuracy.

"""Juditis Mission (ONIA 2026, county round, grades 11-12).

Task 1: number of distinct publishers.  Task 2: publisher with the most 'good' heroes.
Task 3: regress powerstats__combat (MAE).  Task 4: classify Super Strength (accuracy).

Key observation: the original superhero database writes -1 for "unknown stat". In this version the
five visible power stats were perturbed with small noise, so an unknown stat now looks like a value
in (-5, 5), while known stats live in [5, 100+]. A hero with >= 2 such near-zero stats has an
unknown combat too (combat = -1 exactly, 72/72 in train, 0 exceptions). We predict -1 for them
and train the models only on heroes with real stats.

Run from the folder holding train.csv and test.csv; writes submission.csv. CPU, < 1 minute.
"""
import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestRegressor, ExtraTreesRegressor, RandomForestClassifier
from sklearn.linear_model import HuberRegressor
from sklearn.metrics import mean_absolute_error, accuracy_score
from sklearn.model_selection import KFold, StratifiedKFold
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

tr = pd.read_csv("train.csv"); te = pd.read_csv("test.csv")
STATS = ["powerstats__intelligence", "powerstats__strength", "powerstats__speed",
         "powerstats__durability", "powerstats__power"]
POWERS = [c for c in tr.columns if tr[c].dtype == bool]          # 80+ binary abilities
POWERS_IN_X = [c for c in POWERS if c in te.columns]             # Super Strength is not in test

# ---------------- Tasks 1 and 2: exact answers ----------------
task1 = tr["Publisher"].nunique()
good = tr[tr["Alignment"] == "good"]["Publisher"].value_counts()
task2 = sorted(good[good == good.max()].index)[0]                # alphabetical tie-break
print("Task 1:", task1, "| Task 2:", task2)


# ---------------- features ----------------
def features(d, with_meta=True):
    X = {}
    unknown = d[STATS].abs() < 5                                  # noisy version of the -1 marker
    X["n_unknown_stats"] = unknown.sum(axis=1)
    for s in STATS:
        X[s] = d[s].where(~unknown[s]).clip(0, 100)               # unknown -> NaN, clip noisy outliers
    X.update({c: d[c].astype(int) for c in POWERS_IN_X})
    X["n_powers"] = d[POWERS_IN_X].sum(axis=1)
    X["Height"], X["Weight"] = d["Height"], d["Weight"]
    for v in ["-", "Human", "Mutant", "God / Eternal", "Human / Radiation", "Android", "Alien", "Cyborg"]:
        X[f"Race={v}"] = (d["Race"] == v).astype(int)
    if with_meta:                                                 # columns that turned out to be noise for task 4
        X["meta_score"], X["efficiency_delta"] = d["meta_score"], d["efficiency_delta"]
        for c in ["Publisher", "Gender", "Alignment"]:
            for v in tr[c].value_counts().index[:6]:
                X[f"{c}={v}"] = (d[c] == v).astype(int)
    return pd.DataFrame(X, index=d.index)


def fill(X, ref):
    return X.fillna(ref.median())


# ---------------- Task 3: combat ----------------
y = tr["powerstats__combat"].values
X3, X3t = features(tr), features(te)
unknown_tr = X3["n_unknown_stats"].values >= 2
unknown_te = X3t["n_unknown_stats"].values >= 2
print(f"combat = -1 for {np.mean(y[unknown_tr] == -1):.0%} of the {unknown_tr.sum()} train heroes with unknown stats,"
      f" {np.mean(y[~unknown_tr] == -1):.0%} of the others")


def combat_models():
    return [RandomForestRegressor(500, min_samples_leaf=3, max_features=0.3, n_jobs=2, random_state=0),
            ExtraTreesRegressor(500, min_samples_leaf=3, max_features=0.5, n_jobs=2, random_state=0),
            make_pipeline(StandardScaler(), HuberRegressor(alpha=10, max_iter=2000))]


def fit_predict_combat(Xa, ya, Xb, unknown_b):
    ref = Xa
    preds = [m.fit(fill(Xa, ref), ya).predict(fill(Xb, ref)) for m in combat_models()]
    return np.where(unknown_b, -1, np.mean(preds, axis=0))


scores = []
for rep in range(3):
    p = np.zeros(len(y))
    for a, b in KFold(5, shuffle=True, random_state=rep).split(X3):
        a = a[~unknown_tr[a]]                                     # learn only from heroes with real stats
        p[b] = fit_predict_combat(X3.iloc[a], y[a], X3.iloc[b], unknown_tr[b])
    scores.append(mean_absolute_error(y, p))
print(f"Task 3 CV MAE: {np.mean(scores):.2f}  (constant median: {mean_absolute_error(y, np.full(len(y), np.median(y))):.2f})")
known = ~unknown_tr
combat_test = fit_predict_combat(X3[known], y[known], X3t, unknown_te)

# ---------------- Task 4: Super Strength ----------------
ys = tr["Super Strength"].values
X4, X4t = features(tr, with_meta=False), features(te, with_meta=False)
make_clf = lambda: RandomForestClassifier(800, max_features=0.3, n_jobs=2, random_state=0)
accs = []
for rep in range(4):
    for a, b in StratifiedKFold(5, shuffle=True, random_state=rep).split(X4, ys):
        m = make_clf().fit(fill(X4.iloc[a], X4.iloc[a]), ys[a])
        accs.append(accuracy_score(ys[b], m.predict(fill(X4.iloc[b], X4.iloc[a]))))
print(f"Task 4 CV accuracy: {np.mean(accs):.3f}  (majority class: {max(ys.mean(), 1 - ys.mean()):.3f})")
ss_test = make_clf().fit(fill(X4, X4), ys).predict(fill(X4t, X4)).astype(float)

# ---------------- submission ----------------
rows = [("GLOBAL", "task1", task1), ("GLOBAL", "task2", task2)]
rows += [(i, "task3", round(float(v), 3)) for i, v in zip(te["id"], combat_test)]
rows += [(i, "task4", v) for i, v in zip(te["id"], ss_test)]
pd.DataFrame(rows, columns=["id", "subtaskID", "answer"]).to_csv("submission.csv", index=False)
print("wrote submission.csv,", len(rows), "rows")