IOAI Training: Learn AI by solving problems

The problem in brief

Build a grammatical error correction (GEC) system for Romanian: it receives one possibly wrong sentence and must return the corrected sentence. The errors the organisers have in mind are those of real online comments: missing diacritics ("si" for "și"), one or two wrong letters in a word ("duar" for "doar"), wrong prepositions, gender, number, case or person agreement errors, cacophonies and punctuation mistakes. The catch is in the rules: the model must be an encoder-decoder ("base" size, such as mT5-base or mBART), you may not use any existing GEC data or GEC-finetuned model, and the only training data is a set of clean Romanian Wikipedia pages (about 1,560 pages, 5 MB). You hand in the model on the Hugging Face Hub and a two-page report. In other words, the real task is: invent the training data.

This correction is marked partial: fine-tuning a base-size encoder-decoder (580 M parameters for mT5-base) was impossible on the shared machine we had (2 CPU threads, under 2 GB of RAM per process). We explain the full intended pipeline, and we implement and evaluate the parts that do not need a GPU: the error generator and a strong training-free statistical corrector built from the same Wikipedia text.

How you are scored

The organisers evaluate the hub model on a hidden, diverse set of sentences, one sentence at a time. The official notebook computes three text-similarity metrics between the model output and the reference correction: BLEU, METEOR and ROUGE-L. BLEU, the one we report, is the geometric mean of the modified $n$ -gram precisions $p_n$ ( $n = 1..4$ ) times a brevity penalty:

\text{BLEU} = \min\left(1, e^{1 - r/c}\right)\cdot \exp\Big(\tfrac{1}{4}\textstyle\sum_{n=1}^{4}\log p_n\Big),

with $c$ the total length of the outputs and $r$ that of the references. The statement gives no threshold or leaderboard, so there is no "expected" score to compare with. Our numbers are measured on our own held-out set (described below), not on the organisers' private sentences.

One property of this metric matters a lot for GEC: copying the input already scores high, because most words of a wrong sentence are correct. Any system must be judged against the "do nothing" baseline.

First look at the data

The data are two JSON files of Wikipedia pages (wiki_valid.json, wiki_test.json), each page a few paragraphs. We cut them into sentences (20 to 200 characters, starting with a capital letter): 23,600 sentences. Two cleaning details:

Romanian has two encodings of ș and ț: the correct comma-below (ș, ț) and the legacy cedilla (ş, ţ). Wikipedia mixes them. We normalise everything to comma-below, otherwise a model is penalised for "errors" that are invisible on screen.
We split by page, not by sentence: 10 % of the pages (and their 1,500 sentences we evaluate on) never contribute to training. Sentences of one page share names and phrasing, so a random sentence split would leak.

The examples in the statement are informative: they are YouTube-style comments, fully without diacritics, with missing commas and capitals, with letters dropped ("melodi" for "melodii"). Wikipedia is the opposite style: long, formal, perfect. The whole difficulty of the problem lives in that gap.

The key idea: corrupt clean text to manufacture training pairs

If clean sentences are all we have, we create the wrong sentences ourselves and learn the inverse mapping, wrong $\to$ clean. The quality of the final model depends mostly on how well the synthetic errors imitate real ones. Our generator applies, at random:

error	how	probability
no diacritics at all	ă, â $\to$ a, î $\to$ i, ș $\to$ s, ț $\to$ t on the whole sentence	50 %
some diacritics missing	the same, word by word with probability 0.5	30 %
letter typo	deletion, swap, substitution or doubling of one inner letter, per word	8 % per word
commas	removed (or detached: "uriaș ,")	50 %
final punctuation	removed	40 %
first letter	lower-cased	30 %

Example of a generated pair: "Dupa incheierea carierei de jucator, el a devenit antrneor al echiei de tineret..." $\to$ "După încheierea carierei de jucător, el a devenit antrenor al echipei de tineret...".

The intended full solution then fine-tunes mT5-base on millions of such pairs (each clean sentence can be corrupted many times, with different random errors at every epoch), exactly like the official baseline notebook does with its placeholder corruption ("prepend the letter A"). Good generators also add the harder error types: swapping prepositions (în/la/pe), changing noun or adjective endings to break agreement ("cea mai bună" $\to$ "cea mai bun"), confusing "sa/să", "ca/că", "ia/i-a" and joining words after punctuation ("special,vocea"). A model can only learn to fix what the generator produces.

A first baseline

Copying the input on our 1,500 corrupted held-out sentences: BLEU 0.561, and 5 % of sentences are already exactly right (the generator sometimes leaves a sentence untouched).

The full solution we could run: a statistical corrector

Without a GPU we can still learn a lot from the 21,700 training sentences, because the most frequent errors are local and reversible:

Candidates. For each word of the input, list the spellings seen in Wikipedia that reduce to the same ASCII form ("si" $\to$ {si, și}; "incheierea" $\to$ {încheierea}). For a lowercase word unknown in this form (likely a typo, 4 letters or more), also add the known words at edit distance 1 ("antrneor" $\to$ "antrenor"). Capitalised words keep their capitalisation, so names are not "corrected".
Language model. A bigram model with "stupid backoff": $\log P(w_i \mid w_{i-1}) = \log\frac{\#(w_{i-1}w_i)}{\#(w_{i-1})}$ when the pair was seen, otherwise $\log\big(0.4\,\frac{\#(w_i)+0.1}{N+V}\big)$ .
Decoding. The Viterbi algorithm picks the sequence of candidates with the highest total score, where changing letters (and not only diacritics) costs a penalty of 3 nats, so the system only "fixes" a typo when the context clearly prefers it.
Punctuation and case. Capitalise the first letter, add a final period when the sentence has none.

for c in candidates(w):                          # e.g. "si" -> ["și", "si"]
    cost = 0 if fold(c) == fold(w) else -3.0     # changed letters must be worth it
    score, path = max((s + logp(prev, c) + cost, p) for prev, (s, p) in beams.items())
    new[c] = (score, path + [c])

Why it works: Romanian diacritics are rarely ambiguous in context. "si" alone could be "si" (the musical note) or "și" ("and"), but after a noun and before a verb the bigram counts make "și" overwhelmingly likely. The same goes for "ca/că" or "sa/să", which differ by part of speech and are decided by their neighbours.

Results

1,500 corrupted sentences from held-out Wikipedia pages (our own synthetic test set; the organisers' test sentences are private):

system	BLEU	exactly corrected
copy the input	0.561	5.0 %
restore diacritics (lattice + bigram LM)	0.752	9.3 %
+ capital first letter and final period	0.780	16.6 %
+ typo candidates at edit distance 1 (final)	0.813	21.5 %
+ re-inserting commas where the LM prefers them (600-sentence subset)	+0.001	$-0.7$ point
first version, without case preservation and with a cruder tokeniser (600-sentence subset)	0.730	8.8 %

Bar charts of corpus BLEU and exact-match rate for the four steps: copy 0.561 and 5 percent, diacritics 0.752 and 9.3 percent, capital and period 0.780 and 16.6 percent, typo fixing 0.813 and 21.5 percent.

Lessons from the failed variants: the first version let the language model change capitalisation freely, and it "corrected" proper names ("Bert van Marwijk" became "Beit Van Marwijk", "Guvernul Catalan" became "guvernul catalan"); keeping the writer's case (together with a tokeniser that keeps "s-a" in one piece) raised BLEU by about +0.08. LM-based comma insertion gave almost nothing: commas depend on the syntax of the whole clause, not on two neighbouring words.

On the statement's examples, the limits show clearly: "Am mai auzit melodi dar asta este cea mai bun" becomes "Am mai auzit melodii dar asta este cea mai bun.": the typo is fixed, but the agreement error "bun $\to$ bună" is not, because "bun" is a correct word and a bigram cannot see that it must agree with "cea". "Biatul acesta..." keeps "Biatul" because "băiatul", an everyday word, is rare in Wikipedia. These are exactly the cases where a fine-tuned encoder-decoder, which reads the whole sentence and has seen many corrupted agreements in training, is needed.

What we expect from the full solution: on synthetic data of this kind, fine-tuned mT5 models typically restore diacritics almost perfectly and handle agreement when the generator includes it. The main risk is the style gap: a model trained only on corrupted Wikipedia sentences meets comments with slang, missing words and unusual punctuation. Generating the errors in proportions that match the statement's examples (fully missing diacritics, missing commas, lower case) is the most effective lever.

What to remember

When labelled data is forbidden, synthesise it: corrupt clean text with a noise model of the errors you expect, and learn the inverse. Your model will only fix the error types your generator produces.
Always compare with "copy the input" in correction tasks: it already scores BLEU 0.56 here.
Split by document, not by sentence, to avoid leakage between related sentences.
Normalise invisible variants (ș with cedilla vs comma below) before training and before scoring.
A candidate lattice + language model + Viterbi is a strong, cheap baseline for local errors (diacritics, typos) and a good way to understand what a neural model must add: long-range agreement and real-world style.

"""Grammar Is Everything: Romanian grammatical error correction (ROAI 2024, national round).

The official deliverable is an encoder-decoder (mT5-base / mBART) fine-tuned on errors synthesised
from clean Romanian Wikipedia text and pushed to the Hugging Face Hub. Fine-tuning a "base" model was
out of reach on our shared CPU budget (2 threads, < 2 GB RAM per process), so this script contains:

  1. the part every solution needs: sentence extraction and a synthetic error generator
     (diacritics removed, letter typos, commas dropped or misplaced, final punctuation, case);
  2. a training-free statistical corrector learned from the same clean text: candidate lattice
     (all spellings that fold to the same ASCII form + edit-distance-1 spellings for unknown words),
     bigram language model with stupid backoff, Viterbi decoding, then capitalisation and final period;
  3. an honest evaluation on held-out Wikipedia pages (BLEU with nltk, exact-match rate).

The pairs it generates are exactly what an mT5 fine-tuning run would consume (see the write-up).
Run from the folder holding wiki_valid.json and wiki_test.json (huggingface.co/datasets/Olimpiada-AI/ro_wiki).
"""
import collections, json, math, random, re
import numpy as np
from nltk.translate.bleu_score import corpus_bleu

random.seed(0)

# ---------------- data: sentences, page-level split ----------------
pages = [p["page"] for f in ["wiki_valid.json", "wiki_test.json"] for p in json.load(open(f))]
norm = lambda s: s.replace("ş", "ș").replace("ţ", "ț").replace("Ş", "Ș").replace("Ţ", "Ț")  # cedilla -> comma below
sents = []
for i, p in enumerate(pages):
    for s in re.split(r"(?<=[.!?])\s+|\n", norm(p)):
        s = s.strip()
        if 20 <= len(s) <= 200 and s[0].isupper():
            sents.append((i, s))
random.shuffle(sents)
test_pages = set(random.sample(range(len(pages)), len(pages) // 10))   # whole pages held out: no leakage
train = [s for i, s in sents if i not in test_pages]
test = [s for i, s in sents if i in test_pages][:1500]
print(f"{len(pages)} pages, {len(train)} training sentences, {len(test)} test sentences")

# ---------------- synthetic errors ----------------
DIA = {"ă": "a", "â": "a", "î": "i", "ș": "s", "ț": "t", "Ă": "A", "Â": "A", "Î": "I", "Ș": "S", "Ț": "T"}
fold = lambda w: "".join(DIA.get(ch, ch) for ch in w)
LETTERS = "abcdefghijklmnoprstuvzăâîșț"


def typo(w):
    if len(w) < 4: return w
    i = random.randrange(1, len(w) - 1); op = random.random()
    if op < 0.3: return w[:i] + w[i + 1:]                          # deletion
    if op < 0.55: return w[:i] + w[i + 1] + w[i] + w[i + 2:]       # swap
    if op < 0.8: return w[:i] + random.choice(LETTERS) + w[i + 1:]  # substitution
    return w[:i] + w[i] + w[i:]                                    # duplication


def corrupt(s):
    r = random.random()
    if r < 0.5: s = fold(s)                                                        # typed without diacritics
    elif r < 0.8: s = " ".join(fold(w) if random.random() < 0.5 else w for w in s.split(" "))
    s = " ".join(typo(w) if random.random() < 0.08 else w for w in s.split(" "))
    if random.random() < 0.5: s = re.sub(r",(?=\s)", "" if random.random() < 0.7 else " ,", s)
    if random.random() < 0.4: s = s.rstrip(".!?")
    if random.random() < 0.3: s = s[0].lower() + s[1:]
    return s


random.seed(1)
pairs = [(corrupt(s), s) for s in test]
src, ref = [a for a, _ in pairs], [b for _, b in pairs]

# ---------------- metrics ----------------
tok = lambda s: re.findall(r"\w+(?:[-']\w+)*|[^\w\s]", s)
bleu = lambda hyp, ref: corpus_bleu([[tok(r)] for r in ref], [tok(h) for h in hyp])
exact = lambda hyp, ref: float(np.mean([h == r for h, r in zip(hyp, ref)]))

# ---------------- statistical corrector ----------------
uni, bi = collections.Counter(), collections.Counter()
for s in train:
    t = ["<s>"] + tok(s); uni.update(t[1:]); bi.update(zip(t[:-1], t[1:]))
by_fold = collections.defaultdict(set)
for w in uni: by_fold[fold(w).lower()].add(w)
N, V = sum(uni.values()), len(uni)


def logp(prev, w):
    """Bigram log-probability with 'stupid backoff' to a smoothed unigram."""
    b = bi.get((prev, w), 0)
    if b and uni.get(prev): return math.log(b / uni[prev])
    return math.log(0.4 * (uni.get(w, 0) + 0.1) / (N + V))


def edits1(w):
    sp = [(w[:i], w[i:]) for i in range(len(w) + 1)]
    return set([a + b[1:] for a, b in sp if b] + [a + b[1] + b[0] + b[2:] for a, b in sp if len(b) > 1]
               + [a + c + b[1:] for a, b in sp if b for c in LETTERS] + [a + c + b for a, b in sp for c in LETTERS])


def candidates(w, use_typo, first):
    if not re.match(r"\w", w): return [w]
    cands = set(by_fold.get(fold(w).lower(), ()))           # same word up to diacritics and case
    if use_typo and not cands and len(w) >= 4 and w.islower():
        for e in edits1(fold(w).lower()): cands |= by_fold.get(e, set())
    if first: cands = {c[0].upper() + c[1:] for c in cands}  # sentence start: capitalise
    else: cands = {c for c in cands if c[0].isupper() == w[0].isupper()}  # keep the writer's case
    cands.add(w[0].upper() + w[1:] if first else w)
    return sorted(cands, key=lambda c: -(uni.get(c, 0) + uni.get(c.lower(), 0)))[:8]


def detok(toks):
    s = ""
    for i, t in enumerate(toks):
        if i and not (t in ".,;:!?)»”" or s.endswith(("(", "„", "«", "“"))): s += " "
        s += t
    return s


def correct(s, use_typo=True, use_punct=True, typo_penalty=3.0):
    toks = tok(s)
    beams = {"<s>": (0.0, [])}                                 # Viterbi: state = last chosen word
    for i, w in enumerate(toks):
        new = {}
        for c in candidates(w, use_typo, first=(i == 0 and use_punct)):
            cost = 0.0 if fold(c).lower() == fold(w).lower() else -typo_penalty   # letters changed: pay
            sc, path = max(((sc + logp(p, c) + cost, path) for p, (sc, path) in beams.items()), key=lambda x: x[0])
            new[c] = (sc, path + [c])
        beams = new
    out = detok(max(beams.values(), key=lambda x: x[0])[1])
    if use_punct:
        out = out[0].upper() + out[1:]
        if not re.search(r"[.!?…\"»)]$", out): out += "."
    return out


print(f"identity (no correction)        BLEU {bleu(src, ref):.4f}  exact {exact(src, ref):.3f}")
for name, kw in [("diacritics restoration only", dict(use_typo=False, use_punct=False)),
                 ("+ capital letter, final period", dict(use_typo=False)),
                 ("+ typo candidates (final)", dict())]:
    hyp = [correct(a, **kw) for a in src]
    print(f"{name:31s} BLEU {bleu(hyp, ref):.4f}  exact {exact(hyp, ref):.3f}")

for x in ["Biatul acesta este special,vocea lui,cred ca toi am avut acest gand",
          "Am mai auzit melodi dar asta este cea mai bun"]:
    print("IN :", x, "\nOUT:", correct(x))

Grammar Is Everything: Grammatical Error Correction