IOAI Training: Learn AI by solving problems

The problem in brief

You must tell dwarf galaxies from ordinary ones using 13 numerical features. The 1 000 labelled galaxies come from a telescope survey, and only 13 % of them are dwarfs, because dwarfs are faint and hard to observe. The statement tells you, in the story, that the real Universe (the 998 000 galaxies of the test set) "confirms the theoretical distribution", which is a balance of dwarfs and ordinary galaxies. So the classifier is trained on one class balance and judged on another. Some feature values are missing (up to 10 % for feature6).

The data are the official Kaggle files. The test labels are private and we did not submit anything. Our scores are cross-validated on the 1 000 training galaxies, reweighted to the sky's 50/50 balance as explained below.

How you are scored

Plain accuracy on the test galaxies, turned into points:

\text{points} = \min\big(100,\ 294\,(\text{accuracy} - 0.55)\big).

Accuracy 0.55 gives 0 points, and accuracy 0.89 gives the full 100 points (statement). On the public leaderboard the best entry has accuracy 0.899, and the top seven lie between 0.88 and 0.90.

First look at the data

Train: 868 ordinary galaxies, 132 dwarfs. Test: 998 000 galaxies, no labels.
The test features look strikingly regular: their quantiles fall on round values (feature2 has deciles at exactly 1, 5 and 9 on a 0 to 10 scale). They look like uniform draws on a grid, a sign of synthetic data.
The test class balance can be read off the features. For the most informative features, the test mean sits halfway between the training means of the two classes. Feature4: dwarfs 3.89, ordinary 2.80, test 3.50. Feature8: 16.8, 13.3, 15.0. That is what a 50/50 sky would give.

A first baseline

A gradient-boosted tree model (scikit-learn's HistGradientBoostingClassifier, which handles missing values natively) separates the classes well: cross-validated AUC 0.93. But with the usual rule "predict ordinary if $\hat p > 0.5$ ", it misses most dwarfs. On a 50/50 sky its accuracy would be only 0.750 (59 points). The model is not bad. It is answering the wrong question: "how likely is ordinary in the survey", where 87 % of galaxies are ordinary.

The key idea

This is label shift: the class proportions change between training and test, but what each class looks like does not. In symbols, $p(x \mid y)$ is the same in the survey and in the sky, while the prior $\pi = P(y = \text{ordinary})$ goes from $\pi_{\text{tr}} = 0.868$ to $\pi_{\text{te}} = 0.5$ .

Checking the assumption. Train a classifier to tell training galaxies from test galaxies. If it cannot, the two sets come from the same distribution. As observed, it reaches AUC 0.67: the survey is visibly different from the sky. Reweight the training galaxies so that each class counts for half, and the AUC drops to 0.52, close to chance (right panel). So re-balancing the classes explains almost all of the difference, which is exactly the label-shift assumption.

Correcting for it. By Bayes' rule, the posterior odds are the likelihood ratio times the prior odds:

\frac{P(\text{ord} \mid x)}{P(\text{dwarf} \mid x)} = \frac{p(x \mid \text{ord})}{p(x \mid \text{dwarf})} \cdot \frac{\pi}{1-\pi}.

The likelihood ratio is the same in the survey and in the sky. So a model trained on the survey only needs its prior odds swapped. On a 50/50 sky we should predict "ordinary" when the likelihood ratio exceeds 1, which in terms of the survey-trained probability means

\hat p_{\text{tr}}(x) > \pi_{\text{tr}} = 0.868 .

This single change lifts the accuracy on a 50/50 sky from 0.750 to 0.831.

Two refinements. The threshold $\pi_{\text{tr}}$ is only right if $\hat p$ is well calibrated, and boosted trees on 1 000 rows are not quite. But we know something more robust: half of the sky is dwarfs. So we simply score the test galaxies and cut at the median of their scores, which predicts exactly 50 % dwarfs whatever the calibration (left panel: the median falls at 0.91, a little above 0.868). Then, removing four features that are pure noise helps a lot on such a small dataset. Each of feature2, feature6, feature9 and feature13 raises the cross-validated AUC when dropped, and dropping all four takes it from 0.93 to 0.94.

Thresholds and the label-shift check

How we measure accuracy on a 50/50 sky without test labels. Under label shift, the accuracy on a sky with half of each class is the average of the two per-class recalls:

\text{acc}_{50/50} = \tfrac12 \Pr(\text{correct} \mid \text{dwarf}) + \tfrac12 \Pr(\text{correct} \mid \text{ordinary}),

which is the balanced accuracy, and the per-class recalls can be measured on the training galaxies by cross-validation. With only 132 dwarfs, each estimate has a standard error of about 0.015, so we repeat 5-fold CV with 3 different seeds and average.

The full solution

Keep 9 features (drop feature2, feature6, feature9, feature13). Missing values are left to the model.
Fit gradient-boosted trees (300 iterations, learning rate 0.05, 15 leaves, L2 penalty 1).
Score a random sample of 30 000 test galaxies and set the threshold at their median.
Self-training, one round: take the 60 % of those sky galaxies with the most confident predictions, add them to the training set with their predicted labels (weight 0.3 each, and the labelled galaxies reweighted to 50/50), refit, and take the new median as the threshold. The pseudo-labelled galaxies come from the sky's own distribution, so they teach the model what a balanced population looks like.
Predict all 998 000 test galaxies. The final model calls 50.4 % of them ordinary.

Results

Accuracy on a 50/50 sky (balanced accuracy, 5-fold CV × 3 seeds on the training galaxies). The official test labels are private.

approach	accuracy on 50/50 sky	points
all 13 features, threshold 0.5	0.750	58.9
all features, class-weighted training, threshold 0.5	0.786	69.4
all features, prior-corrected threshold 0.868	0.831	82.5
9 features, prior-corrected threshold	0.839	85.0
9 features, threshold = median score on the sky	0.867	93.0
9 features, median threshold + self-training (final)	0.873	94.9

The final pipeline reaches an estimated 0.873, worth about 95 points. That is just short of the 0.89 needed for the full 100 points and of the best public score (0.899). Two caveats. The estimate is noisy (about $\pm 0.015$ from the 132 dwarfs). And the four noise features were chosen with the same cross-validation, which flatters the 9-feature rows slightly.

What to remember

When train and test class balances differ, the classifier is fine but the threshold is wrong. Under label shift, predict class 1 when $\hat p_{\text{tr}} > \pi_{\text{tr}}$ (for a 50/50 test set), or more robustly cut the test scores at the known test prior.
Test the assumption with a train-versus-test classifier. If reweighting the classes makes train and test indistinguishable (AUC near 0.5), label shift is the whole story.
Balanced accuracy is accuracy on a balanced population. It lets you validate for a 50/50 test set using an unbalanced training set.
On small data, removing noise features is a real gain. A drop-one-feature loop showed four features whose removal raised the AUC.
Unlabelled test data are information. Here they gave the class balance, the threshold and, through self-training, a few extra points of accuracy.

"""Cyprus AI camp 2026, "Binary Classification in Space": worked solution.

The 1000 training galaxies are 13 % dwarfs, but the sky (the test set, 998 000 galaxies) is
50 % dwarfs. A classifier trained as is learns the training prior and calls almost everything
"ordinary". The fix is a LABEL-SHIFT correction: keep the classifier, move the decision threshold.
Because we know the test prior is 1/2, the cleanest threshold is the MEDIAN of the scores on the
test rows themselves (half of the sky is predicted dwarf). We also drop 4 pure-noise features
and add one round of self-training on confident test pseudo-labels.

Evaluation: the official test labels are private. Under label shift, accuracy on a 50/50 sky equals
the BALANCED accuracy (mean of the two recalls), which we measure by 5-fold CV on the training
galaxies, repeated with 3 seeds. Score = min(100, (accuracy - 0.55) * 294).

Usage: python solution.py [folder with train.csv and test.csv]
"""
import os
import sys
os.environ.setdefault("OMP_NUM_THREADS", "2")
import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingClassifier as HGB
from sklearn.metrics import balanced_accuracy_score
from sklearn.model_selection import StratifiedKFold

D = sys.argv[1] if len(sys.argv) > 1 else "work"
tr, te = pd.read_csv(f"{D}/train.csv"), pd.read_csv(f"{D}/test.csv")
F = [c for c in tr.columns if c.startswith("feature")]
NOISE = ["feature2", "feature6", "feature9", "feature13"]     # dropping each one RAISES the CV AUC
COLS = [c for c in F if c not in NOISE]
y = tr.target.values                                           # 1 = ordinary, 0 = dwarf
PARAMS = dict(max_iter=300, learning_rate=0.05, max_leaf_nodes=15, l2_regularization=1.0)
U = te[COLS].sample(30_000, random_state=2).reset_index(drop=True)   # unlabelled sky sample


def class_weights(labels):
    """Weights that make both classes count half: the training set re-balanced to the sky's prior."""
    return np.where(labels == 1, 0.5 / labels.mean(), 0.5 / (1 - labels.mean()))


def fit_predict(Xa, ya, seed, self_train=True):
    """Returns a model and its threshold (median score on the sky sample)."""
    m = HGB(random_state=seed, **PARAMS).fit(Xa, ya)
    pu = m.predict_proba(U)[:, 1]
    t = np.median(pu)
    if not self_train:
        return m, t
    margin = np.abs(pu - t)
    conf = margin > np.quantile(margin, 0.4)                   # 60 % most confident sky galaxies
    Z = pd.concat([Xa, U[conf]])
    yz = np.r_[ya, (pu[conf] > t).astype(int)]
    wz = np.r_[class_weights(ya), np.full(conf.sum(), 0.3)]
    m2 = HGB(random_state=seed, **PARAMS).fit(Z, yz, sample_weight=wz)
    return m2, np.median(m2.predict_proba(U)[:, 1])


def cv_balanced_accuracy(variant):
    scores = []
    for seed in range(3):
        pred = np.zeros(len(y))
        for a, v in StratifiedKFold(5, shuffle=True, random_state=seed).split(tr, y):
            Xa, ya, Xv = tr[variant["cols"]].iloc[a], y[a], tr[variant["cols"]].iloc[v]
            if variant["rule"] == "0.5":
                pred[v] = HGB(random_state=seed, **PARAMS).fit(Xa, ya).predict_proba(Xv)[:, 1] > 0.5
            elif variant["rule"] == "prior":                   # p > pi_train  <=>  odds ratio > 1 on a 50/50 sky
                pred[v] = HGB(random_state=seed, **PARAMS).fit(Xa, ya).predict_proba(Xv)[:, 1] > ya.mean()
            else:
                global U
                U = te[variant["cols"]].sample(30_000, random_state=2).reset_index(drop=True)
                m, t = fit_predict(Xa, ya, seed, self_train=variant["rule"] == "self")
                pred[v] = m.predict_proba(Xv)[:, 1] > t
        scores.append(balanced_accuracy_score(y, pred))
    return np.mean(scores)


VARIANTS = {
    "all features, threshold 0.5": dict(cols=F, rule="0.5"),
    "all features, prior-corrected threshold": dict(cols=F, rule="prior"),
    "9 features, prior-corrected threshold": dict(cols=COLS, rule="prior"),
    "9 features, threshold = median on the sky": dict(cols=COLS, rule="median"),
    "9 features, median threshold + self-training": dict(cols=COLS, rule="self"),
}
for name, v in VARIANTS.items():
    acc = cv_balanced_accuracy(v)
    print(f"{name:48s} accuracy on a 50/50 sky {acc:.4f}  points {min(100, (acc - 0.55) * 294):.1f}")

# ---------------------------------------------------------------- final model on all 1000 galaxies
U = te[COLS].sample(30_000, random_state=2).reset_index(drop=True)
model, thr = fit_predict(tr[COLS], y, seed=0)
p = model.predict_proba(te[COLS])[:, 1]
pred = (p > thr).astype(int)
print(f"share of the sky predicted ordinary: {pred.mean():.3f}")
pd.DataFrame({"Id": te.Id, "target": pred}).to_csv("submission.csv", index=False)
print("wrote submission.csv (not submitted)")