IOAI Training: Learn AI by solving problems

The problem in brief

The data is the history of Kaggle "Featured" competitions from 2010 to 2024: 230 580 rows, one per user and competition, with the final public and private ranks, the number of teams, the number of submissions and the medal. The test lists the users of 5 later competitions (1 900 rows, each with a team id) and asks for a score: how likely is each user to finish in the top 3 % of that competition, that is with $\text{private rank} / \text{number of teams} \le 0.03$ ? Only users with at least 4 past competitions and 10 past submissions are in the test, and the history contains only competitions that started before the first test competition.

The data comes from the Kaggle competition archive. The test results are private and we never submitted anything, so all scores below come from our own time-based validation inside the history.

How you are scored

ROC AUC between the scores and the true 0/1 outcome, over all test rows:

\text{AUC} = P\big(\text{score}(u^+) > \text{score}(u^-)\big),

for a random user $u^+$ who made the top 3 % and a random user $u^-$ who did not. Only the order of the scores matters.

What to aim for. The organisers' baseline ranks users by their number of past competitions. On the public leaderboard the best entry scores 0.916, and most of the top 10 are between 0.895 and 0.916. Our validation uses different competitions, so the numbers are comparable only roughly.

First look at the data

About 6 % of the rows are top-3 % finishes, more than 3 % because every member of a winning team gets the same rank. Members of one team appear as several rows with exactly the same private rank in the same competition, which lets us rebuild the teams of the history: $(\text{competition}, \text{rank})$ identifies a team.

The key question is how to validate. The test is a "future" situation: features may only use competitions that started before. A random split of rows would leak the future (a user's 2023 result helps predict their 2021 result). So we copy the test exactly:

Validation: the last 5 competitions of the history play the role of the test. Features are computed from all competitions that started before the first of them, users are filtered with the same rule (at least 4 competitions and 10 submissions before), and the label is their result.
Training: we repeat the same construction for every earlier competition $C$ since 2014: users of $C$ with their features computed from what happened before $C$ , labelled with their result in $C$ . This gives 105 768 training rows, each one a small replay of the test situation.

A first baseline

score used to rank users	AUC (last 5 competitions)
number of past competitions (organisers' baseline)	0.542
past top-3 % rate	0.772
recency-weighted mean of $\log(\text{rank}/\text{teams})$	0.744

Experience alone says almost nothing: many users with dozens of competitions never reach the top. Past success says a lot: the share of past competitions finished in the top 3 % already gives 0.77.

The key idea

Two kinds of signal: what the user did before, and who the user is teaming with now.

Success rates by history and by team, and validation scores

History (left panel). The probability of a top-3 % finish climbs steadily with the number of past top-3 % finishes: 3 % for users who never did it, 9 % after one, over 30 % after nine. Skill in competitions is persistent. We describe each user by several views of the past (mean, median and best percentile, recent form over the last 5 competitions, a 2-year exponentially weighted log-percentile, counts of top 3 % and top 10 %, medals, submissions per competition, days since the last competition) and let gradient boosting combine them.

Team (middle panel). The test gives a team id, so we know which test users compete together. In the history, a user who is in a team with other experienced users (users who also pass the filter) ends in the top 3 % in 28 % of cases for a team of two and 64 % for teams of four or more, against 4.5 % for solo users. Strong Kagglers merge with other strong Kagglers when they are near the top of the leaderboard, so a team made of experienced members is itself evidence of a good position. We add the team size (counted among filtered users, exactly as in the test), the best past log-percentile in the team and the total number of past top-3 % finishes in the team. These features are worth 4 points of AUC (0.784 without them, 0.826 with them) and are the most used by the model.

We also add the user's rank within the field of the competition (how strong they are compared with the other filtered users of the same competition), since the top 3 % is a fixed number of places: being strong matters less than being stronger than the others.

The full solution

Rebuild teams in the history from equal private ranks.
For each competition $C$ since 2014 (training) and for the test, compute user features from competitions that started before $C$ , keep users with at least 4 competitions and 10 submissions, and add team and field features inside $C$ :

d['team_size'] = tg.UserId.transform('size')            # filtered users sharing the team
d['team_best_logpct'] = tg.logpct_mean.transform('min')  # best teammate's history
d['team_sum_top3'] = tg.n_top3.transform('sum')          # teammates' past top-3 % finishes
d['rank_in_comp'] = d.logpct_ewm.rank(pct=True)          # strength relative to this field

Train LightGBM (400 trees, learning rate 0.02, 15 leaves, at least 100 rows per leaf, row and column subsampling, L2 = 5), average 3 seeds.
For the test, the history is everything in train.parquet, the team is the given TeamId, and the reference date is the last start date in the history.

Results

ROC AUC on the users of the last 5 (and 10) competitions of the history, built exactly like the test. The official test labels are private.

approach	last 5 competitions	last 10 competitions
number of past competitions (organisers' baseline)	0.542	0.553
past top-3 % rate	0.772	0.776
Elo rating computed over all competitions (Codeforces-style)	0.707	0.723
logistic regression, all features	0.808	0.823
LightGBM without team features	0.784	0.796
LightGBM + Elo rating features	0.826	0.844
LightGBM, 63 leaves / 800 trees / training since 2018 only	0.827 / 0.826 / 0.828	0.842 / 0.842 / 0.843
LightGBM, all features (final)	0.826	0.842

The final model gets 0.826 on the last 5 competitions and 0.842 on the last 10. The best public-leaderboard entry is 0.916, but on different competitions: our two validation windows already differ by 1.6 points, so the competitions themselves change the difficulty a lot. We cannot tell how far we are from that score without the test labels.

Two ideas failed and are worth knowing. An Elo rating, the natural choice for a competition named after Codeforces, is worse than the simple past top-3 % rate (0.707): with ratings updated by rank among hundreds of players, one lucky or unlucky competition moves a rating a lot, while "how often did you reach the top" is exactly the quantity asked. Added to the other features it brings nothing. Tuning the trees changed the score by less than 0.2 points: the information is in the features, not in the model settings.

One more signal exists in the test but cannot be validated: the team id grows with the date a team was created, and in 4 of the 5 test competitions our scores are mildly higher for smaller team ids (Spearman correlation down to $-0.29$ ). Strong users seem to join early. The history has no team ids, so we could not measure this effect and left it out.

What to remember

Validate the way you will be tested. For a "predict the future" task, rebuild the exact test situation in the past (same cut-off, same filter) for both training and validation. A random split would leak future results into the features.
Replay history to multiply training data. Each past competition is a separate snapshot with its own features and labels; together they form a large training set that matches the test distribution.
The best predictor of a rare success is past success, not experience. Rate features ("how often in the top") beat count features ("how many times played").
Look for relational signals. Who a user teams up with tells as much as their own history. Features about the group (team size, best teammate) gave the largest single gain.
A famous method is not automatically the right one. Elo was elegant and weaker than a well-chosen ratio; always compare against a simple baseline.

"""Kaggleforces (NEOAI 2026, day 1): will a Kaggler finish in the top 3 % of a future competition?

Idea: rebuild the test situation many times inside the history.
  For every past competition C (2014+), take the users of C that pass the test filter
  (>= 4 earlier competitions, >= 10 earlier submissions), compute their features from the
  competitions that STARTED BEFORE C only, and label them with their result in C.
  Features: past percentiles (mean, best, recent, recency-weighted), past top-3 % counts,
  medals, submissions, activity, plus TEAM features inside C (team size, best teammate,
  teammates' past top-3 % count) and the user's strength relative to the field of C.
  A LightGBM classifier trained on these snapshots scores the test rows.

Data: kaggle competition "neoai-2026-day-1-kaggleforces" (train.parquet, test.csv).
Usage: python solution.py val    -> time split: last 5 competitions of train.parquet as validation
       python solution.py        -> submission.csv (not submitted)
Runtime: ~1 min on 2 CPU threads, < 1 GB RAM.
"""
import sys, numpy as np, pandas as pd, lightgbm as lgb
from sklearn.metrics import roc_auc_score

PARAMS = dict(n_estimators=400, learning_rate=0.02, num_leaves=15, min_child_samples=100, subsample=0.8,
              subsample_freq=1, colsample_bytree=0.7, reg_lambda=5, n_jobs=2, verbose=-1)


def load():
    t = pd.read_parquet('train.parquet')
    t['pct'] = t.PrivateLeaderboardRank / t.TotalTeams                 # 0 = winner, 1 = last
    t['top3'] = (t.pct <= 0.03).astype(int)
    t['shake'] = (t.PrivateLeaderboardRank - t.PublicLeaderboardRank) / t.TotalTeams
    # team members share their private rank: (competition, rank) identifies a team
    t['team'] = t.CompetitionId.astype(str) + '_' + t.PrivateLeaderboardRank.astype(int).astype(str)
    t['tsize'] = t.groupby('team').UserId.transform('size')
    return t


def user_features(hist, users, start):
    h = hist[hist.UserId.isin(users)].sort_values('EnabledDate')
    h = h.assign(logpct=np.log(h.pct), top10=(h.pct <= 0.10).astype(int),
                 gold=(h.Medal == 1).astype(int), silver=(h.Medal == 2).astype(int), bronze=(h.Medal == 3).astype(int),
                 w=np.exp(-(start - h.EnabledDate).dt.days / 730.0))         # 2-year exponential decay
    g = h.groupby('UserId')
    f = pd.DataFrame({
        'n_comps': g.size(), 'subs': g.n_submissions.sum(), 'subs_mean': g.n_submissions.mean(),
        'pct_mean': g.pct.mean(), 'pct_median': g.pct.median(), 'pct_best': g.pct.min(),
        'logpct_mean': g.logpct.mean(), 'n_top3': g.top3.sum(), 'rate_top3': g.top3.mean(), 'n_top10': g.top10.sum(),
        'n_gold': g.gold.sum(), 'n_silver': g.silver.sum(), 'n_bronze': g.bronze.sum(),
        'shake_mean': g.shake.mean(), 'team_mean': g.tsize.mean(),
        'days_since_last': (start - g.EnabledDate.max()).dt.days, 'days_since_first': (start - g.EnabledDate.min()).dt.days})
    last = h.groupby('UserId').tail(5).groupby('UserId')                    # recent form
    f['pct_last5'], f['logpct_last5'] = last.pct.mean(), last.logpct.mean()
    f['top3_last5'], f['subs_last5'] = last.top3.sum(), last.n_submissions.mean()
    f['logpct_ewm'] = (h.w * h.logpct).groupby(h.UserId).sum() / g.w.sum()
    f['n_comps_2y'] = h[(start - h.EnabledDate).dt.days < 730].groupby('UserId').size()
    f['n_comps_2y'] = f.n_comps_2y.fillna(0)
    return f


def snapshot(hist, rows, start, team_col):
    """Feature rows for the users of one competition, from history only, with the test filter."""
    f = user_features(hist, rows.UserId.unique(), start)
    f = f[(f.n_comps >= 4) & (f.subs >= 10)]
    d = rows[rows.UserId.isin(f.index)].join(f, on='UserId')
    tg = d.groupby(team_col)
    d['team_size'] = tg.UserId.transform('size')                           # counted among filtered users, as in test
    d['team_best_logpct'] = tg.logpct_mean.transform('min')
    d['team_sum_top3'] = tg.n_top3.transform('sum')
    d['comp_n'] = len(d)
    d['rank_in_comp'] = d.logpct_ewm.rank(pct=True)                        # strength relative to this field
    return d


FEATS = ['n_comps', 'subs', 'subs_mean', 'pct_mean', 'pct_median', 'pct_best', 'logpct_mean', 'n_top3', 'rate_top3',
         'n_top10', 'n_gold', 'n_silver', 'n_bronze', 'shake_mean', 'team_mean', 'days_since_last', 'days_since_first',
         'pct_last5', 'logpct_last5', 'top3_last5', 'subs_last5', 'logpct_ewm', 'n_comps_2y', 'team_size',
         'comp_n', 'team_best_logpct', 'team_sum_top3', 'rank_in_comp']


def training_snapshots(t, before):
    starts = t.groupby('CompetitionId').EnabledDate.min()
    ends = t.groupby('CompetitionId').DeadlineDate.max()
    out = []
    for c, s in starts.sort_values().items():
        if s >= pd.Timestamp('2014-01-01') and ends[c] < before:        # label must be known at `before`
            out.append(snapshot(t[t.EnabledDate < s], t[t.CompetitionId == c], s, 'team'))
    return pd.concat(out)


def fit_predict(T, X):
    return np.mean([lgb.LGBMClassifier(**PARAMS, random_state=s).fit(T[FEATS], T.top3).predict_proba(X[FEATS])[:, 1]
                    for s in range(3)], axis=0)                       # 3 seeds averaged


if __name__ == '__main__':
    t = load()
    starts = t.groupby('CompetitionId').EnabledDate.min().sort_values()
    if sys.argv[1:] == ['val']:
        K = 5
        cut = starts.iloc[-K]                                         # history ends where validation begins
        V = pd.concat([snapshot(t[t.EnabledDate < cut], t[t.CompetitionId == c], cut, 'team') for c in starts.index[-K:]])
        p = fit_predict(training_snapshots(t, cut), V)
        print('baseline n_comps AUC', round(roc_auc_score(V.top3, V.n_comps), 4))
        print('LightGBM AUC        ', round(roc_auc_score(V.top3, p), 4))
    else:
        te = pd.read_csv('test.csv')
        start = t.EnabledDate.max()                                   # test competitions start after all history
        X = pd.concat([snapshot(t, g, start, 'TeamId') for _, g in te.groupby('CompetitionId')])
        X['pred_score'] = fit_predict(training_snapshots(t, pd.Timestamp('2100-01-01')), X)
        te = te.merge(X[['CompetitionId', 'UserId', 'pred_score']], on=['CompetitionId', 'UserId'], how='left')
        sub = pd.DataFrame({'Id': te.CompetitionId.astype(str) + '_' + te.UserId.astype(str),
                            'pred_score': te.pred_score.fillna(0)})
        sub.to_csv('submission.csv', index=False)
        print('wrote submission.csv (not submitted)', sub.shape)