IOAI Training: Learn AI by solving problems

The problem in brief

You get 6,617 English tweets labelled with their overall affect (1 positive, 0 negative), 872 more for validation, 3,357 unlabelled Spanish tweets and 2,687 Spanish test tweets to label. The only model allowed is xlm-roberta-base, a multilingual encoder pretrained on 100 languages; translation tools and other language models are forbidden. This is zero-shot cross-lingual transfer: learn the task in English, apply it in Spanish, and use the unlabelled Spanish text to adapt. Assessment is on "innovation, performance and efficiency". The notebook comes from a third-party repository (github.com/teodora-dimitrova/Singapore-Team-Selection) and the data from github.com/duyngtr16061999/Cross_lingual_sentiment_classification.

About the evaluation. The Spanish test labels were not published with the task. We found that every tweet comes from the public SemEval-2018 Task 1 "E-c" dataset (multi-label emotions, Mohammad et al.), and that the English labels follow an exact rule: label 1 if the tweet is annotated with joy, love, optimism, anticipation or trust, else 0 (this reproduces all 7,489 English labels). Applying the same rule to the Spanish E-c gold file rebuilds the Spanish test labels (48.4 % positive). We used them only to score the final predictions, never to choose anything: every design decision below was made on the English validation set or fixed in advance.

How you are scored

The organisers score the predicted Spanish labels (the notebook computes accuracy on the English validation set, so we use accuracy):

\text{accuracy} = \frac{1}{n}\sum_{i=1}^{n}\mathbf 1[\hat y_i = y_i],

plus a judgement of innovation and efficiency. There is no published threshold. Since the classes are balanced (48.4 % positive in the Spanish test), always predicting one class gives about 50 %.

First look at the data

Tweets are short: a median of 100 characters in English and 74 in Spanish, often with hashtags and emoji. The notebook truncates to 32 tokens, which cuts the end of many English tweets; 64 tokens keeps almost all of them.
The two languages share only the label space. The Spanish tweets come from different users and topics, and their emotional vocabulary ("enfadado", "QEPD", "felicidad") never appears in the training data.
The classes are balanced in all four files, so accuracy is a fair metric and a degenerate model is easy to spot.

A first baseline

The notebook fine-tunes XLM-R with learning rate $10^{-4}$ , 2 epochs, batch 64, 32 tokens. We ran it with one change forced by our memory budget: the word-embedding matrix (250,002 tokens x 768, 192 million of the 278 million parameters) is frozen, which divides the optimizer memory by three. Result: the training loss stays at 0.69 (that is $\ln 2$ , the loss of a coin flip), the model predicts "positive" for every tweet, English validation accuracy 56.1 % and Spanish 48.4 %.

This is a classic failure. A learning rate of $10^{-4}$ is too large for a pretrained transformer: the first updates destroy the pretrained representations faster than the new classifier head can use them, and the model falls back to predicting the majority class. Values between $10^{-5}$ and $5\times10^{-5}$ , with a warm-up, are the standard range for BERT-like models. With $2\times10^{-5}$ , a linear warm-up over 6 % of the steps and linear decay, the same 2 epochs and 32 tokens give 81.0 % on English validation and 76.6 % in Spanish; 3 epochs and 64 tokens give 84.5 to 86.6 % in English and 77.8 to 79.2 % in Spanish (three runs).

The key idea

Why zero-shot transfer works at all. XLM-R was pretrained with masked-language modelling on text in 100 languages with one shared vocabulary. Nothing told it that "happy" and "feliz" mean the same thing, yet its hidden representations end up roughly aligned across languages, because similar contexts produce similar representations. A classifier trained on English representations therefore partly works on Spanish ones. The gap between English validation (about 85 %) and Spanish test (about 78 %) is the price of the imperfect alignment.

Freeze the word embeddings. Fine-tuning on English moves the embeddings of English tokens only; the Spanish tokens never get a gradient. Unfrozen, the English part of the embedding space drifts away from the Spanish part it was aligned with. Freezing the embedding matrix keeps the shared input space intact, saves two thirds of the optimizer memory, and the 12 transformer layers still adapt to the task. (Full fine-tuning did not fit our 3 GB memory budget, so we could not measure the unfrozen variant.)

Self-training: let the model teach itself Spanish. The unlabelled Spanish corpus is the resource that must be used. Self-training (pseudo-labelling) is the simplest efficient way:

train on English;
predict the 3,357 unlabelled Spanish tweets;
keep the most confident 60 % of each predicted class (about 2,000 tweets, balanced so that the model does not amplify a bias towards one class);
retrain from the pretrained checkpoint on English plus these pseudo-labelled Spanish tweets, and repeat.

Why does a model improve by learning from its own predictions? Its confident predictions are mostly right (for a reasonably calibrated model, high-confidence predictions are more accurate than the average one), and they are on Spanish text. Retraining on them teaches the model Spanish cues that it could not see in English: when a tweet with "QEPD" (rest in peace) is confidently negative because of the crying emoji, the next model learns that "QEPD" alone is negative. In other words, the decision boundary is pushed into the low-density regions of the Spanish data, the classic cluster assumption of semi-supervised learning. The balanced selection and the restart from the pretrained checkpoint (rather than continuing to train) limit confirmation bias, where a model reinforces its own mistakes.

Spanish test accuracy (red) and English validation accuracy (blue) after English-only training and after one and two rounds of self-training, for two random seeds. The grey line is the notebook's recipe, which collapses to a single class.

The figure contains the main methodological lesson of the problem. Self-training improves Spanish by 1.2 to 2.4 points, consistently over two seeds, but the English validation accuracy does not follow (it goes down for seed 0, up for seed 1). A source-language validation set cannot select target-language methods. In a real zero-shot setting you would fix the procedure in advance on principled grounds, or annotate a small Spanish development set. We fixed two rounds and 60 % before looking at any Spanish score.

The full solution

Tokenise with the XLM-R tokenizer, 64 tokens, padding to the maximum length.
xlm-roberta-base with a 2-class head; word embeddings frozen.
AdamW, learning rate $2\times10^{-5}$ , weight decay 0.01, batch 32, 6 % linear warm-up then linear decay, 3 epochs.
Two rounds of self-training on the unlabelled Spanish corpus (top 60 % most confident per predicted class), each round retraining from the pretrained weights on English plus pseudo-labelled Spanish.
Predict the Spanish test set and write DT_test_predictions.csv.

p = proba(model, *X_spanish_unlabeled); conf, lab = p.max(1)
for c in (0, 1):                                   # balanced: top 60 % of each predicted class
    idx = torch.where(lab == c)[0]
    keep[idx[conf[idx].argsort(descending=True)[: int(0.6 * len(idx))]]] = True
model = fit(new_model(), English + Spanish[keep], labels + lab[keep])

Efficiency. One English-only training takes about 2.5 minutes on an Apple laptop GPU; the whole pipeline with two rounds of self-training takes 11.5 minutes, with no extra model and no extra pretraining. Domain-adaptive masked-language-model pretraining on the Spanish corpus would be the next idea, but at a much higher cost for 3,357 short tweets.

Results

Spanish scores use the rebuilt SemEval-2018 labels; they were not used for any choice.

approach	English validation	Spanish test
always "positive"	56.1 %	48.4 %
notebook recipe (lr $10^{-4}$ , 2 epochs, 32 tokens)	56.1 %	48.4 % (collapsed)
lr $2\times10^{-5}$ + warm-up, 2 epochs, 32 tokens	81.0 %	76.6 %
same, 3 epochs, 64 tokens (three runs)	84.5 / 86.6 / 84.9 %	77.8 / 78.4 / 79.2 %
+ 1 round of self-training (seeds 0 / 1)	85.8 / 85.9 %	79.8 / 80.1 %
+ 2 rounds of self-training (seeds 0 / 1), final	84.9 / 86.8 %	80.8 / 80.4 %

Our submitted predictions (seed 0) reach 80.8 % on the Spanish test set. Runs on Apple's GPU are not bit-for-bit reproducible even with a fixed seed: identical settings gave 84.5 % and 86.6 % in English, so differences below about 1.5 points are noise.

What to remember

Fine-tuning a pretrained transformer needs a small learning rate and a warm-up. A loss stuck at $\ln 2 = 0.693$ and a model predicting one class are the signature of a learning rate that destroyed the pretrained weights.
Multilingual pretraining gives cross-lingual transfer for free, and freezing the shared word embeddings protects the alignment between languages (and saves memory).
Unlabelled target data is the key resource. Balanced, confidence-filtered self-training, restarted from the pretrained checkpoint, adds about 2 points at almost no cost.
Validate where you deploy. English validation accuracy did not predict the Spanish gains; without target labels, fix the method in advance or label a small target sample.
Look for the source of a dataset. Recognising SemEval-2018 E-c and the exact label rule turned an unscorable exercise into an honest evaluation, used only after all choices were made.

"""Singapore IOAI 2025 camp test (NLP): zero-shot English -> Spanish affect classification with xlm-roberta-base.

Data: github.com/duyngtr16061999/Cross_lingual_sentiment_classification (cloned into ./data);
model: FacebookAI/xlm-roberta-base (config, tokenizer, model.safetensors in ./xlmr).
usage: python solution.py <name> <lr> <epochs> <max_len> <freeze_emb 0|1> <self_train_rounds>
  e.g. python solution.py notebook 1e-4 2 32 0 0     (the notebook's recipe)
       python solution.py final    2e-5 3 64 1 1     (our solution)
Model selection uses ONLY the English validation set. If es_test_labels.npy exists (labels rebuilt
after the fact from the public SemEval-2018 E-c gold files), the Spanish test accuracy is printed too.
"""
import json, os, sys, time
import numpy as np, pandas as pd, torch, torch.nn.functional as F
from transformers import AutoTokenizer, AutoModelForSequenceClassification
torch.set_num_threads(2)
DEV = torch.device("mps" if torch.backends.mps.is_available() else "cpu")
name, LR, EPOCHS, MAXLEN, FREEZE, ROUNDS = sys.argv[1], float(sys.argv[2]), int(sys.argv[3]), int(sys.argv[4]), sys.argv[5] == "1", int(sys.argv[6])

read = lambda f: pd.read_json(f"data/{f}.json", lines=True)
en_tr, en_va, es_un, es_te = read("DE_train"), read("DE_valid"), read("DT_unlabeled"), read("DT_test_unlabeled")
tok = AutoTokenizer.from_pretrained("xlmr")

def encode(texts):
    e = tok(list(texts), truncation=True, padding="max_length", max_length=MAXLEN, return_tensors="pt")
    return e["input_ids"], e["attention_mask"]

def new_model():
    torch.manual_seed(int(os.environ.get("SEED", 0)))
    m = AutoModelForSequenceClassification.from_pretrained("xlmr", num_labels=2)
    if FREEZE:  # keep the shared multilingual word embeddings fixed: English updates cannot drift them away from Spanish
        m.roberta.embeddings.word_embeddings.weight.requires_grad_(False)
    return m.to(DEV)

def fit(m, ids, mask, y, epochs, bs=32):
    params = [p for p in m.parameters() if p.requires_grad]
    opt = torch.optim.AdamW(params, lr=LR, weight_decay=0.01)
    steps = epochs * ((len(y) + bs - 1) // bs)
    sch = torch.optim.lr_scheduler.LambdaLR(opt, lambda s: min(1.0, s / (0.06 * steps)) * max(0.0, (steps - s) / steps))
    for ep in range(epochs):
        m.train(); perm = torch.randperm(len(y))
        for k in range(0, len(y), bs):
            b = perm[k:k + bs]
            out = m(input_ids=ids[b].to(DEV), attention_mask=mask[b].to(DEV))
            loss = F.cross_entropy(out.logits, y[b].to(DEV))
            opt.zero_grad(); loss.backward(); opt.step(); sch.step()
        print(f"  epoch {ep + 1}: loss {loss.item():.3f}", flush=True)
    return m

@torch.no_grad()
def proba(m, ids, mask, bs=128):
    m.eval()
    return torch.cat([m(input_ids=ids[k:k + bs].to(DEV), attention_mask=mask[k:k + bs].to(DEV)).logits.softmax(1).cpu()
                      for k in range(0, len(ids), bs)])

def report(tag, m):
    r = {"en_valid": float((proba(m, *X_va).argmax(1) == torch.tensor(en_va.label.values)).float().mean())}
    p = proba(m, *X_te); r["es_pred_positive"] = float(p.argmax(1).float().mean())
    if os.path.exists("es_test_labels.npy"):
        r["es_test"] = float((p.argmax(1).numpy() == np.load("es_test_labels.npy")).mean())
    print(tag, r, flush=True)
    return r, p

t0 = time.time()
X_tr, X_va, X_un, X_te = encode(en_tr.text), encode(en_va.text), encode(es_un.text), encode(es_te.text)
y_tr = torch.tensor(en_tr.label.values)
m = fit(new_model(), *X_tr, y_tr, EPOCHS)
results = {"round0": report("English only", m)[0]}

# Self-training on the unlabeled Spanish corpus: pseudo-label it with the current model, keep the most
# confident 60 % of each predicted class (balanced), retrain from scratch on English + pseudo-labelled Spanish.
for r in range(1, ROUNDS + 1):
    p = proba(m, *X_un); conf, lab = p.max(1)
    keep = torch.zeros(len(lab), dtype=torch.bool)
    for c in (0, 1):
        idx = torch.where(lab == c)[0]
        keep[idx[conf[idx].argsort(descending=True)[: int(0.6 * len(idx))]]] = True
    print(f"round {r}: {int(keep.sum())} pseudo-labelled Spanish tweets, {float(lab[keep].float().mean()):.2f} positive")
    ids = torch.cat([X_tr[0], X_un[0][keep]]); mask = torch.cat([X_tr[1], X_un[1][keep]]); y = torch.cat([y_tr, lab[keep]])
    del m
    m = fit(new_model(), ids, mask, y, EPOCHS)
    results[f"round{r}"] = report(f"self-training round {r}", m)[0]

_, p = report("final", m)
pd.DataFrame({"text": es_te.text, "predicted_label": p.argmax(1).numpy()}).to_csv(f"DT_test_predictions_{name}.csv", index=False)
results["minutes"] = (time.time() - t0) / 60
json.dump(results, open(f"res_{name}.json", "w"))
print(name, results)

Cross-Lingual Affect in Tweets