IOAI Training: Learn AI by solving problems

The problem in brief

Malay is written in two very different registers: bahasa baku, the standard language of newspapers and official texts, and bahasa pasar ("market language"), the relaxed, abbreviated, code-switched language of social media and product reviews. You get 820 labelled texts (class 0 = baku, classes 1 to 5 = various kinds of pasar) and 93 unlabelled test texts, and must label each test text 0 or 1. Predictions were posted to the Malaysian olympiad's API, so the official test labels are private: we estimate our score with repeated cross-validation on the training set, and use the data from the public mirror jaredliw/ioai-tsp-2025.

How you are scored

The server computes the F1 score of the pasar class (label 1), and the task is worth $10 \times F_1$ points:

F_1 = \frac{2\,\text{precision}\cdot\text{recall}}{\text{precision}+\text{recall}} = \frac{2\,TP}{2\,TP + FP + FN}.

The statement hints at a trap: predicting 0 everywhere gives $TP = 0$ , hence $F_1 = 0$ , even though it would be right about half the time. F1 ignores true negatives, so it only rewards finding the positive class.

There is no published threshold. The mirror notebook is a contestant's solution and reports its server scores: 8.94 (counts + logistic regression), 9.57 (TF-IDF + logistic regression) and 9.78, i.e. $F_1 \approx 0.978$ (TF-IDF + linear SGD classifier). We use that last number as the reference level.

First look at the data

The classes are balanced once merged: 415 baku against 405 pasar (45, 90, 45, 90 and 135 texts in sub-classes 1 to 5).
Every baku text looks like a news sentence: "Pasukan Polis Marin (PPM) Wilayah Satu Langkawi membongkar sindiket ... semalam." Capital letter, full sentence, final period, words like semalam (yesterday), lapor (reported), polis.
Pasar texts are comments and reviews: abbreviations (yg for yang, tk for tak, dh, utk), particles (la, je, ni, tu), emojis, stretched letters ("jaaa", "BAGUSSS"), English mixed in ("Thank you seller"), and also Indonesian slang (ngga, bisa, banget).

So the task is easier than "judge the grammar": in this dataset the register is tied to the source (news vs social media), and both leave strong lexical fingerprints.

A first baseline

Two rules with no learning at all:

rule	F1 (pasar)
predict everything pasar	0.661
baku if the text starts with a capital and ends with a period	0.764

The punctuation rule already captures something real, but many casual comments are punctuated properly and some news snippets end with a quote. We need the words.

The key idea

A linear model on bag-of-words features is the right tool when the classes differ in vocabulary. Represent each text as a sparse vector $x \in \mathbb{R}^{|V|}$ of TF-IDF weights (term frequency times inverse document frequency, so that frequent-everywhere words count less), and learn a weight per feature:

\hat y = \mathbb{1}\left[\, w^\top x + b > 0 \,\right].

Each weight says how much a token pushes toward pasar ( $w_j > 0$ ) or baku ( $w_j < 0$ ). With 820 texts and thousands of features, a regularised linear model (logistic regression, linear SVM, Naive Bayes) is hard to beat: it has one parameter per word, which is exactly the amount of structure this data supports.

The learned weights (left panel) are a lesson in themselves:

On the pasar side: yg (abbreviation of yang), tak, dah, nak, ni, tu, aku (informal "I"). These are exactly the markers a linguist would list.
On the baku side: formal function words (di, pada, apabila, seorang, sebuah) but also news topics (polis, negeri, Malaysia, lapor) and semalam. with its period. The model has partly learned "is this a news article?", not only "is this standard Malay?". That works on this test set because it comes from the same sources, but a formal text about another topic (a sermon, a textbook) could fool it.

Character n-grams add robustness to spelling variation, which is the essence of pasar: "bagus", "bagusss" and "bagussss" are three different words but share the n-grams bag, agu, gus. With char_wb (n-grams inside word boundaries, $n = 1..5$ ) the model also sees prefixes and suffixes such as meng-, -kan (frequent in formal derivation) or doubled letters.

The full solution

Merge classes 1 to 5 into label 1.
Features: the union of character 1-5-grams (case kept, sublinear TF-IDF) and whitespace tokens (case kept, so that ALL-CAPS shouting and punctuation glued to words like semalam. stay visible).
Classifier: LinearSVC(C=1).
Evaluate with 5-fold stratified cross-validation repeated 5 times (25 fits): with only 820 texts, one split has a standard deviation of about 0.01 F1, as large as the differences between models.
Refit on all 820 texts and predict the 93 test texts.

chars = TfidfVectorizer(analyzer="char_wb", ngram_range=(1, 5), lowercase=False, sublinear_tf=True)
words = TfidfVectorizer(token_pattern=r"\S+", lowercase=False, sublinear_tf=True)
model = make_pipeline(make_union(chars, words), LinearSVC(C=1.0)).fit(X, y)

Error analysis. Out of 820 texts the out-of-fold predictions make 9 mistakes, and they are instructive. Most are pasar-labelled comments written in perfectly standard Malay ("Asalnya datang daripada penggunaan perkataan 'sarekat'...", a history comment about American colonies): they come from social media, but nothing in the text is colloquial. The baku errors are news sentences on everyday topics ("Hukumnya harus jika bermain..."). These are labelling ambiguities more than model failures, and no bag-of-words model will fix them.

Comparison with the contestant's submission. Our 93 test predictions agree with the predictions printed in the mirror notebook (the contestant's SGD model, for which they report a server score of 9.78, assuming the printed predictions are the ones scored) on 92 texts. On the remaining one (test id 47, an Indonesian question to an archaeologist that starts with "Pak" and contains "sperti") we predict baku and they predicted pasar. We can even pin our test score down to two values. Their score $9.7826 = 10 \times 45/46$ means $2TP/(2TP+FP+FN) = 90/92$ : with 46 predicted positives, that is $TP = 45$ , one false positive and one false negative. If text 47 really is pasar, we have $TP = 44$ , $FP = 1$ , $FN = 2$ , so $F_1 = 88/91 \approx 0.967$ ; if it is baku, $F_1 = 90/91 \approx 0.989$ . Reading it, pasar looks more likely, so our honest estimate on the official test is about 0.967, against $0.988 \pm 0.008$ in cross-validation (a 93-text test set moves by 0.01 per error). We did not adjust the model to that one text: fixing a test prediction by hand would be tuning on the test set.

Results

approach	F1 (pasar), 5 x 5 CV on train
predict everything pasar	0.661
rule: capital start + final period = baku	0.764
word TF-IDF without stopwords + SGD (the contestant's approach)	0.980 $\pm$ 0.012
word counts + multinomial Naive Bayes	0.983 $\pm$ 0.012
word TF-IDF + logistic regression	0.985 $\pm$ 0.011
char 1-5 TF-IDF (lowercased) + linear SVM	0.987 $\pm$ 0.007
char 1-5 + cased tokens + linear SVM (final)	0.988 $\pm$ 0.008

All linear models are within one standard deviation of each other: on a dataset this small, the feature representation matters much more than the choice of linear classifier, and the remaining errors are ambiguous labels. Removing stopwords slightly hurts, because the most informative pasar markers (tak, ni, je, la) are themselves stopwords.

What to remember

Read the metric before modelling: F1 of the positive class gives 0 to an all-negative submission and ignores true negatives.
Sparse text features + a linear model is the strong, fast baseline for short-text classification; on 820 examples it reaches about 0.985 F1.
Character n-grams handle spelling variation (stretched letters, abbreviations, typos) that word features treat as unrelated words.
Do not blindly remove stopwords: in register or style tasks, function words and particles carry the signal.
Inspect the weights: they reveal shortcuts (here, news topics standing in for "standard language") that could break on data from other sources. With small data, use repeated cross-validation, since single-split differences are mostly noise.

"""MAIO 2025 "Baku or pasar": standard (baku, class 0) vs colloquial (pasar,
classes 1-5 merged into 1) Malay text, scored by F1 on the pasar class.

Data (public mirror): https://github.com/jaredliw/ioai-tsp-2025/tree/main/maio-2025
  train.jsonl (820 labelled texts), test.jsonl (93 unlabelled texts)
The official test labels are private (answers were posted to an API), so we
estimate F1 with 5 x 5 repeated stratified cross-validation on train.

Run:  python solution.py        (expects train.jsonl / test.jsonl in work/)
Writes work/answer.json in the format expected by the competition server.
"""
import json
import re
import numpy as np
import pandas as pd
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import f1_score
from sklearn.model_selection import RepeatedStratifiedKFold
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import make_pipeline, make_union
from sklearn.svm import LinearSVC

W = "work/"
train = pd.read_json(W + "train.jsonl", lines=True)
test = pd.read_json(W + "test.jsonl", lines=True)
X = np.array(train.text.tolist(), dtype=object)
y = (train["class"] > 0).astype(int).values           # 1 = pasar (any sub-class)
print("train:", len(X), "texts, pasar share", y.mean().round(3), "| test:", len(test))


def char_word_svm():
    """Final model: character n-grams (inside words, case kept) + raw tokens."""
    chars = TfidfVectorizer(analyzer="char_wb", ngram_range=(1, 5), lowercase=False, sublinear_tf=True)
    words = TfidfVectorizer(token_pattern=r"\S+", lowercase=False, sublinear_tf=True)
    return make_pipeline(make_union(chars, words), LinearSVC(C=1.0))


MODELS = {
    "word counts + Naive Bayes": lambda: make_pipeline(CountVectorizer(), MultinomialNB()),
    "word TF-IDF + logistic regression": lambda: make_pipeline(
        TfidfVectorizer(), LogisticRegression(C=10, max_iter=2000)),
    "char 1-5 TF-IDF + linear SVM": lambda: make_pipeline(
        TfidfVectorizer(analyzer="char_wb", ngram_range=(1, 5), sublinear_tf=True), LinearSVC()),
    "char 1-5 + tokens (cased) + linear SVM": char_word_svm,
}


def cv_f1(make, n_rep=5):
    scores = []
    for tr, te in RepeatedStratifiedKFold(n_splits=5, n_repeats=n_rep, random_state=0).split(X, y):
        m = make().fit(X[tr], y[tr])
        scores.append(f1_score(y[te], m.predict(X[te])))
    return np.mean(scores), np.std(scores)


# ------------------------------------------------------------- baselines
results = {"predict everything pasar": (f1_score(y, np.ones_like(y)), 0.0)}
formal = np.array([t.strip()[:1].isupper() and bool(re.search(r"\.\s*$", t)) for t in X])
results["rule: capital start + final period = baku"] = (f1_score(y, (~formal).astype(int)), 0.0)

# --------------------------------------------------------------- models
for name, make in MODELS.items():
    results[name] = cv_f1(make)
for name, (m, s) in results.items():
    print(f"{name:42s} F1 = {m:.4f} +- {s:.4f}")

# ------------------------------------------- what the linear model learns
words = make_pipeline(TfidfVectorizer(token_pattern=r"\S+"), LogisticRegression(C=10, max_iter=3000)).fit(X, y)
vocab = words[0].get_feature_names_out()
coef = words[1].coef_[0]
order = np.argsort(coef)
top_baku, top_pasar = order[:12], order[-12:]
print("most baku tokens :", list(vocab[top_baku]))
print("most pasar tokens:", list(vocab[top_pasar]))

# ------------------------------------------------- final fit + submission
final = char_word_svm().fit(X, y)
pred = final.predict(np.array(test.text.tolist(), dtype=object))
answer = [{"id": int(i), "class": int(p)} for i, p in zip(test.id, pred)]
json.dump(answer, open(W + "answer.json", "w"))
print("test predictions: pasar share", pred.mean().round(3), "-> work/answer.json")

# ---------------------------------------------------------------- figure
fig, axes = plt.subplots(1, 2, figsize=(10.5, 4.6), dpi=150, gridspec_kw={"width_ratios": [1, 1.25]})
ax = axes[0]
idx = np.r_[top_baku, top_pasar]
cols = ["#4c72b0"] * 12 + ["#dd8452"] * 12
ax.barh(range(24), coef[idx], color=cols)
ax.set_yticks(range(24)); ax.set_yticklabels(vocab[idx], fontsize=8)
ax.axvline(0, color="k", lw=0.6)
ax.set_xlabel("logistic-regression weight (< 0 baku, > 0 pasar)")
ax.set_title("tokens the model relies on")
ax = axes[1]
names = list(results)[::-1]
means = [results[n][0] for n in names]
errs = [results[n][1] for n in names]
ax.barh(range(len(names)), means, xerr=errs, color="#8c8c8c", capsize=3)
ax.set_yticks(range(len(names))); ax.set_yticklabels(names, fontsize=8)
ax.set_xlim(0.6, 1.0)
for i, (v, e) in enumerate(zip(means, errs)):
    ax.text(0.607, i, f"{v:.3f}", va="center", fontsize=9, color="white", weight="bold")
ax.set_xlabel("F1 (pasar), 5 x 5 cross-validation")
ax.set_title("baselines vs linear text models")
for a in axes:
    a.spines[["top", "right"]].set_visible(False)
fig.tight_layout()
fig.savefig("figures/features_scores.png", facecolor="white")
print("saved figures/features_scores.png")