IOAI Training: Learn AI by solving problems

The problem in brief

The Kaggle page cyprus-summer-ai-camp-2026-tasks is the hub of the Cyprus summer selection camp (July 2026). Its overview lists the competitions of the three rounds, each hosted on its own Kaggle page, and the hub itself hosts only the warm-up task, "What's the Buzz", which is what this correction solves.

Short messages arrive (a savings tip, a startup rumour, a travel note, a recipe, a stargazing remark, a gadget review) and each must be routed to one of 6 topics: Personal Finance, Startups & Business, Travel & Tourism, Food & Dining, Space & Astronomy, Gadgets & Consumer Tech. This is extreme few-shot learning: for each topic you get exactly one example message and a one-sentence description, nothing else. There are 36 messages to classify. For this warm-up only, the rules allow embeddings from a pretrained language model.

How we evaluate. The official test labels are hidden and we never submitted anything. To measure our method we labelled the 36 test messages ourselves, by reading them (our labels give exactly 6 messages per topic, which suggests that they match the organisers' design). These labels are used only for scoring, never to build or tune the classifier, and the final method was fixed before looking at the scores. Two messages are borderline and could be argued either way: "a night market, sampling street snacks between temple visits" (we chose Travel) and "cutting back on eating out to save each month" (we chose Personal Finance).

How you are scored

Accuracy: the fraction of the 36 messages routed to the right topic. The camp converts it into points relative to the organisers' baseline and the best result:

\text{points} = 100 \cdot \frac{\text{acc} - \text{acc}_{\text{baseline}}}{\text{acc}_{\max} - \text{acc}_{\text{baseline}}},

where $\text{acc}_{\max}$ is the larger of 0.9 times the committee's solution and the best participant's score. What to aim for: on the public leaderboard (18 teams, scores in steps of $1/18$ , so the public part holds 18 of the 36 messages) one team has 1.000, nine teams have 0.944 (17 out of 18), and the lowest score is 0.333.

First look at the data

The six "training" rows look like this: Space & Astronomy has the example "The telescope captured a distant galaxy, revealing thousands of stars billions of light-years away" and the description "Stars, planets, telescopes, space missions and the wider universe." A test message for that topic reads "The rover sent back new images of the crater it has been exploring on the red planet."

Notice that they share no word at all. The test message talks about a rover, a crater and the red planet; the prototype about a telescope, a galaxy and stars. A human sees at once that both are about space, because we know what the words mean. That is the whole difficulty: almost every test message uses vocabulary that never appears in its topic's prototype.

A first baseline

The organisers' baseline turns each topic's example plus description into a TF-IDF vector (word counts weighted by rarity, English stop-words removed) and gives each message the topic with the highest cosine similarity. On our labels it scores 0.306 (11 out of 36), barely above the $1/6 = 0.167$ of random guessing. The left panel below shows why: 25 of the 36 messages share not a single word with their correct prototype, and 19 share no word with any prototype, so all their similarities are 0 and the tie is broken in favour of the first topic, Personal Finance.

TF-IDF word overlap and embedding geometry

The key idea

Compare meanings, not words: map every sentence into a sentence-embedding space, where "rover on the red planet" and "telescope sees a galaxy" are close, and classify by the nearest prototype in that space.

A sentence-embedding model (here BAAI/bge-small-en-v1.5, a 33-million-parameter BERT-style encoder, or all-MiniLM-L6-v2) was trained on hundreds of millions of sentence pairs so that sentences with related meanings get vectors with a high cosine similarity. Each message becomes a unit vector $e \in \mathbb R^{384}$ . For each topic $c$ we build a prototype from its two texts,

p_c = \frac{e(\text{example}_c) + e(\text{description}_c)}{\lVert e(\text{example}_c) + e(\text{description}_c) \rVert},

and predict $\hat c = \arg\max_c \, e^\top p_c$ . This is exactly the "prototypical network" recipe of few-shot learning: with one or two examples per class, the best you can do is to average them and measure distances, and everything rests on the quality of the space. The right panel above shows the 36 messages projected on the first two principal components: they already gather around their topic's star.

We also tried a transductive refinement: assign all test messages, recompute each prototype as the normalised sum of the original prototype and its assigned messages, and repeat three times (a k-means started at the prototypes). It uses the structure of the unlabelled test set, which is legitimate here because all 36 messages are given at once. It helped MiniLM with example-only prototypes (0.944 to 0.972) and changed nothing elsewhere: when the clusters are already clean, there is little left to fix.

The full solution

Embed the 6 examples, the 6 descriptions and the 36 messages with bge-small-en-v1.5 (CPU, under 10 seconds), and normalise the vectors to unit length.
Prototype of each topic = normalised sum of its example and description embeddings.
Three passes of transductive refinement.
Each message goes to the topic of maximal cosine similarity.

P0 = normalise(E_text + E_desc)                      # one prototype per topic
P = P0.copy()
for _ in range(3):                                   # k-means started at the prototypes
    a = np.argmax(T @ P.T, 1)
    P = normalise(np.stack([P0[c] + T[a == c].sum(0) for c in range(6)]))
pred = np.argmax(T @ P.T, 1)

Results

approach (36 messages, scored on our own labels)	accuracy
random guess	0.167
TF-IDF nearest prototype (organisers' baseline)	0.306
MiniLM, description-only prototypes	0.917
MiniLM, example-only prototypes	0.944
MiniLM, example + description	0.944
MiniLM, example only, + refinement	0.972
bge-small, description-only prototypes	0.917
bge-small, example-only prototypes	0.972
bge-small, example + description, + refinement (final)	0.972

The final method classifies 35 of the 36 messages correctly (0.972) on our labels. The public leaderboard has one perfect score and nine teams at 17 out of 18, so this is at the level of the best entries, although on different label sets the comparison is only indicative. The single mistake is "Scaling the sales team too quickly hurt margins in the last fiscal quarter", labelled Startups & Business but sent to Personal Finance (cosine 0.698 against 0.650): margins and fiscal quarters are money words, and nothing in the sentence says "company" explicitly. The descriptions alone are weaker prototypes than the examples (0.917): a real message looks more like another real message than like a definition.

What to remember

Bag-of-words needs shared words. With one example per class, test texts almost never reuse its vocabulary, and TF-IDF collapses to near chance. Check the overlap before trusting a lexical method.
Pretrained sentence embeddings carry meaning across different words. Nearest-prototype classification in such a space is a strong, training-free few-shot classifier.
Prototypes are averages. Combining several views of a class (an example and a description) gives a sturdier centre; an example usually matches real messages better than a definition does.
Unlabelled test data can refine the prototypes (transductive learning, k-means from the prototypes) when the whole test set is available at once.
When labels are hidden, say how you scored. Hand labels are fine for a sanity check, but flag the ambiguous cases and fix the method before looking at the score.

"""What's the Buzz (Cyprus Summer AI Camp 2026, warm-up task): route 36 short messages to 6 topics,
given ONE example message and a one-line description per topic. Metric: accuracy.

Files train.csv / test.csv from the Kaggle competition `cyprus-summer-ai-camp-2026-tasks`; sentence
embedding models (safetensors) in ./models: BAAI/bge-small-en-v1.5 and sentence-transformers/all-MiniLM-L6-v2.
The official test labels are hidden. For evaluation only, MY_LABELS holds our own hand labels of
the 36 test messages (6 per topic); they are never used to build the classifier.

Method: nearest prototype in a sentence-embedding space.
  1. prototype of a topic = normalised mean of the embeddings of its example and of its description;
  2. transductive refinement: assign the test messages, recompute each prototype as the mean of the
     original prototype and its assigned messages, repeat (a k-means started at the prototypes).
"""
import os
os.environ.setdefault("OMP_NUM_THREADS", "2")
import numpy as np, pandas as pd
from sentence_transformers import SentenceTransformer
from sklearn.feature_extraction.text import TfidfVectorizer

train = pd.read_csv("train.csv").sort_values("class").reset_index(drop=True)
test = pd.read_csv("test.csv")
# our own hand labels of msg_000..msg_035, for scoring only (official labels are hidden)
MY_LABELS = [4, 1, 4, 0, 4, 5, 3, 2, 3, 0, 5, 1, 1, 1, 4, 2, 5, 0, 0, 3, 3, 2, 3, 2, 0, 2, 2, 3, 5, 5, 4, 1, 1, 4, 5, 0]
truth = np.array(MY_LABELS) if len(test) == len(MY_LABELS) else None


def normalise(X):
    return X / np.linalg.norm(X, axis=1, keepdims=True)


def accuracy(pred):
    return None if truth is None else float((pred == truth).mean())


def tfidf_baseline():
    """the organisers' baseline: TF-IDF of (example + description), nearest prototype by cosine"""
    protos = (train.text + " " + train.description).tolist()
    M = TfidfVectorizer(stop_words="english").fit_transform(protos + test.text.tolist())
    P, T = M[:6].toarray(), M[6:].toarray()
    return np.argmax(T @ P.T, 1)


def embed(model, texts, query_prefix=""):
    return normalise(model.encode([query_prefix + t for t in texts], batch_size=64, convert_to_numpy=True))


def nearest_prototype(model, use=("text", "description"), refine=0, prefix=""):
    E_text, E_desc = embed(model, train.text, prefix), embed(model, train.description, prefix)
    parts = {"text": E_text, "description": E_desc}
    P0 = normalise(sum(parts[u] for u in use))            # one prototype per topic
    T = embed(model, test.text, prefix)
    P = P0.copy()
    for _ in range(refine):                               # transductive refinement (k-means from P0)
        a = np.argmax(T @ P.T, 1)
        P = normalise(np.stack([P0[c] + T[a == c].sum(0) for c in range(6)]))
    return np.argmax(T @ P.T, 1), T @ P.T


if __name__ == "__main__":
    print("TF-IDF nearest prototype:", accuracy(tfidf_baseline()))
    rows = []
    for name in ["all-MiniLM-L6-v2", "bge-small-en-v1.5"]:
        model = SentenceTransformer(f"models/{name}", device="cpu")
        for use in [("text",), ("description",), ("text", "description")]:
            for refine in [0, 3]:
                pred, _ = nearest_prototype(model, use, refine)
                rows.append((name, "+".join(use), refine, accuracy(pred)))
    print(pd.DataFrame(rows, columns=["model", "prototype", "refine iters", "accuracy"]).to_string(index=False))
    # final choice, fixed in advance: bge-small, example + description, 3 refinement passes
    model = SentenceTransformer("models/bge-small-en-v1.5", device="cpu")
    pred, S = nearest_prototype(model, ("text", "description"), 3)
    print("final accuracy on our hand labels:", accuracy(pred))
    pd.DataFrame({"id": test.id, "class": pred}).to_csv("submission.csv", index=False)

Cyprus Summer AI Camp 2026 Tasks