IOAI Training: Learn AI by solving problems

The problem in brief

You receive 22,458 Philippine online news articles (2016 to 2018) in English, Filipino, Taglish and some regional languages, and no labels at all. For each article your pipeline must output six fields: veracity (Real or Fake), one of six manipulation types (Authentic, Satire-as-fact, Opinion-as-fact, Politically-motivated fabrication, Misleading-headline, Fabricated), a target group, a harm level, a fact-check flag, and a confidence score. The intended tool is a free-tier LLM API (Gemini, Groq, OpenRouter) driven by few-shot prompts, with strict JSON output and retry logic for rate limits. One rule shapes everything: the hidden labels come from the source website of each article, but mapping sources or URLs to labels is forbidden. You must judge content, tone and rhetoric.

What we could reproduce. Free-tier APIs need accounts, which we may not create, so the LLM part of the intended pipeline cannot be run here. The Kaggle leaderboard labels are private, but the statement says they are derived from the public Philippine Fake News Corpus. A previous step rebuilt per-article labels from that corpus, used only for scoring, never for fitting, and drew a balanced evaluation sample of 360 articles (60 per manipulation type). All numbers below are on that sample, also reweighted to the real class mix.

How you are scored

The Kaggle part is the macro-F1 of veracity and of manipulation type, averaged over classes:

F_1^{\text{macro}} = \frac{1}{C}\sum_{c=1}^{C} \frac{2\,P_c R_c}{P_c + R_c},

with $P_c$ and $R_c$ the precision and recall of class $c$ . It is converted into points relative to the best candidate, $10 \times F_1/F_1^{\text{top}}$ , so there is no absolute target. Calibration of confidence_score is graded by the expected calibration error

\mathrm{ECE} = \sum_{b} \frac{n_b}{n}\,\big|\mathrm{acc}_b - \mathrm{conf}_b\big|

over 10 confidence bins: ECE below 0.1 is "Elite", 0.1 to 0.2 "Strong". Prompts, ethics and engineering are reviewed by hand. There is a 2-point penalty if more than 90 % of the confidences are exactly 1.0, and 5 points for any source lookup.

Two facts about the labels matter. The class mix is very unbalanced: 14,802 Authentic and only 532 Fabricated. And labels are per source: every Adobo Chronicles article is "Satire-as-fact", whatever it says. Content alone therefore has a ceiling, especially for the six-way type.

First look at the data

Credible broadsheet articles look like wire reports: long, many attributions ("said", "according to", "ayon kay", "sinabi"), few exclamation marks, no direct address to the reader. The not-credible sites mix rants in the first and second person, calls to "share" or "watch the video", and much shorter texts. Satire (Adobo Chronicles) is the hard case: it deliberately imitates the broadsheet style.

A first baseline

Naive benchmark (the organisers' file): everything Real / Authentic. On the real class mix this gives veracity macro-F1 0.397 (it gets the 66 % majority right and has F1 0 on Fake) and type macro-F1 0.132.
A small local LLM as a stand-in for the API. Within our 3 GB memory budget the largest instruction model we could run is Qwen2.5-0.5B-Instruct. We gave it a careful system prompt (code-switching, regional languages, "judge by content, never by the website"), category definitions and four few-shot examples, then read the probabilities of the letters A to F, about 1 s per article on the GPU. The result is sobering: the probability of "Authentic" separates Real from Fake with AUC 0.51, chance level, and the macro-F1 drops below the naive baseline (0.25 veracity, 0.07 type on the real mix). Contextual calibration, i.e. dividing by the probabilities given to an empty input, does not help, because there is no signal to recalibrate. A 0.5B model cannot do this task. The intended free-tier models are 10 to 100 times larger.

The key idea

Two things can be done with no labels at all, and both are rewarded by the rubric.

1. Rhetoric as a measurable signal. Encode a few cues as rates per word, fixing the direction of each before looking at any label: exclamation marks, all-caps words, first-person and second-person pronouns, calls to share or watch, length (negative direction), and attribution verbs (negative direction: reporting sounds like reporting). Each cue has its English, Filipino, Cebuano, Ilocano and Waray forms ("ako, kami, tayo, siak", "ayon kay, sinabi, matod pa, kuna ni"), which is also how the pipeline answers the linguistic-equity requirement. Each cue is standardised robustly on the unlabelled test set, $z = (g - \mathrm{median})/\mathrm{IQR}$ , clipped to $[-5, 5]$ , and the score is the plain sum of the $z$ 's. No weight is learned.

2. Use the prior the statement gives you. The statement says 7,656 of the 22,458 articles are not credible. Instead of guessing a threshold, label as Fake exactly the 34 % highest scores:

\text{Fake} \iff s(x) > q_{1 - 7656/22458}(s),

where $q$ is a quantile over the unlabelled test set. Then turn the distance to the threshold into a monotone probability, $p_{\text{fake}} = \sigma\big((s - q)/(0.5\,\mathrm{IQR}_s)\big)$ , and set confidence_score $= \max(p, 1 - p)$ . The confidence is high far from the threshold and close to 0.5 near it, which is exactly what ECE asks for.

Left: the rhetorical score for the six types on the balanced sample, with the prior-matched threshold. Middle: each cue alone, scored by AUC. Right: reliability of the confidence, reweighted to the real class mix.

The middle panel shows why the sum works: every cue alone is weak (AUC 0.54 to 0.76; the capital-letters cue is even slightly backwards), but summing roughly independent weak cues gives AUC 0.80. Politically-motivated fabrications and misleading headlines score very high, while satire overlaps the authentic articles: content cannot separate a good parody from the original style.

The full solution

Features per article: 7 rhetorical rates (plus a "politics" density used only for the type).
Score and veracity: robust $z$ on the unlabelled test set, sum, threshold at the published prior.
Confidence: logistic of the distance to the threshold. No confidence is exactly 1.0 (2 % are above 0.99, median 0.78), so the overconfidence penalty is avoided by construction.
Manipulation type (rules, for articles predicted Fake): heavy politics plus first or second person gives Opinion-as-fact; heavy politics alone, Politically-motivated fabrication; many share/viral cues, Misleading-headline; more attributions than the median (deadpan news style), Satire-as-fact; otherwise Fabricated.
Other fields: target group Political-figure when politics is dense, otherwise None; harm High for political or viral fakes, Medium for other fakes, None for real articles; needs_fact_check = Yes if Fake or confidence below 0.7.
Output the 134,748 rows [index]_[field].

In the intended contest pipeline, the LLM would replace steps 4 and 5 (and refine step 2), with strict JSON parsing and exponential backoff around each API call. The prior-matched threshold and the score-to-confidence mapping still apply on top of the LLM's probabilities: an LLM's raw verbal confidence is notoriously overconfident.

Results

360-article balanced sample, labels rebuilt from the public corpus (scoring only). "Real mix" reweights each type to its true frequency in the 22,458 articles, which is what a leaderboard on the full test set would see.

approach	veracity F1 (balanced)	type F1 (balanced)	veracity F1 (real mix)	type F1 (real mix)	ECE (real mix)
naive benchmark: all Real / Authentic	0.143	0.048	0.397	0.132	0.33
Qwen2.5-0.5B few-shot (letter probabilities)	0.455	0.126	0.254	0.072	0.49
same + contextual calibration	0.455	0.102	0.254	0.036	0.59
rhetorical score + prior threshold + rules (final, `solution.py`)	0.610	0.178	0.653	0.184	0.081

The final pipeline separates Real from Fake with AUC 0.80. Its ECE of 0.081 is in the "Elite" calibration band, and it is far above the naive benchmark on veracity. The six-way type stays weak (0.18): the confusion matrix shows satire is mostly called Authentic (27 of 60), and the four other fake types are mixed among themselves. That is the expected ceiling of rhetoric-only rules against source-level labels.

Honest limits. We could not run the free-tier LLMs the task is built around, so this is not a reproduction of a contestant's leaderboard score. No published leaderboard value exists to compare with (points are relative to the best candidate). The cue lists were written before scoring and evaluated once; the 360-article sample is small (one article is about 0.3 points of F1).

What to remember

With no labels, use every number the organisers publish. The 34 % prior turned a score into a decision rule without any training data.
Calibration is a separate, cheap win. A monotone map from score to probability, centred on the decision threshold, gave ECE 0.08 for free. Never output a constant or verbal "100 % sure" confidence.
Sum many weak, interpretable cues. Seven cues with AUC 0.54 to 0.76 add up to 0.80, and every decision can be explained, which the manual ethics review rewards.
Measure your model before trusting it. A 0.5B LLM with a polished prompt was at chance level. Check AUC against a trivial baseline before building a pipeline around a model.
Know what the labels mean. Source-level labels cap what content can achieve. Satire that perfectly imitates journalism is indistinguishable by style, so do not over-tune on it.

"""IOAI Philippines 2026 selection, Task 3 (Katotohanan): a LABEL-FREE, content-only pipeline.

The contest expects free-tier LLM APIs (accounts: not available to us). This script is the part that
needs no account: interpretable rhetorical features (no source names, no URLs), a veracity score
standardised on the unlabelled test set, a decision threshold set from the class prior PUBLISHED in
the statement (7,656 of 22,458 not credible), a monotone confidence, and simple rules for the other
fields. It writes the 134,748-row submission and, if the ground truth rebuilt from the public corpus is
available (scoring only, never used for fitting), reports the metrics on the 360-article balanced sample.
Usage: python solution.py test.csv submission.csv [truth.csv sample.csv]
"""
import re, sys
import numpy as np, pandas as pd
from sklearn.metrics import f1_score, roc_auc_score

TYPES = ["Authentic", "Satire-as-fact", "Opinion-as-fact", "Politically-motivated fabrication",
         "Misleading-headline", "Fabricated"]
# English + Filipino/Taglish + a few Cebuano/Ilocano/Waray function words for each rhetorical cue
CUES = {
    "first": r"\b(I|we|my|our|ako|kami|tayo|natin|akong|ako'y|ko|ak|siak)\b",
    "you": r"\b(you|your|kayo|ninyo|mo|ikaw|ka|kamo|kayo'y)\b",
    "said": r"\b(said|told|according to|stated|ayon kay|ayon sa|sinabi|matod pa|kuna ni|siring ni)\b",
    "share": r"\b(share|watch|video|comment|netizens|viral|panoorin|i-share)\b",
    "politics": r"\b(senator|president|presidente|mayor|congress|administration|opposition|dilawan|"
                r"liberal party|duterte|aquino|robredo|marcos|trillanes|de lima)\b",
}


def features(t):
    t = str(t); w = t.split(); n = max(len(w), 1)
    f = {k: len(re.findall(p, t, re.I)) / n for k, p in CUES.items()}
    f["excl"] = t.count("!") / n
    f["caps"] = sum(x.isupper() and len(x) > 2 for x in w) / n
    f["log_len"] = np.log(n)
    return f


def predict(test):
    F = pd.DataFrame([features(t) for t in test.content])
    # a-priori direction of each cue: + means "looks less like a sourced broadsheet report"
    sign = {"excl": 1, "caps": 1, "first": 1, "you": 1, "said": -1, "share": 1, "log_len": -1}
    G = F[list(sign)] * pd.Series(sign)
    Z = ((G - G.median()) / (G.quantile(.75) - G.quantile(.25) + 1e-9)).clip(-5, 5)  # robust z on unlabelled data
    score = Z.sum(axis=1).values
    prior_fake = 7656 / 22458                                  # published in the statement
    thr = np.quantile(score, 1 - prior_fake)
    spread = np.quantile(score, .75) - np.quantile(score, .25)
    p_fake = 1 / (1 + np.exp(-(score - thr) / (0.5 * spread)))  # monotone, 0.5 at the threshold
    fake = score > thr
    pol = F.politics > F.politics.quantile(.75)
    opin = (F["first"] + F["you"]) > (F["first"] + F["you"]).quantile(.75)
    manip = np.where(~fake, "Authentic",
             np.where(pol & opin, "Opinion-as-fact",
             np.where(pol, "Politically-motivated fabrication",
             np.where(F.share > F.share.quantile(.9), "Misleading-headline",
             np.where(F.said > F.said.median(), "Satire-as-fact", "Fabricated")))))
    target = np.where(pol, "Political-figure", "None")
    harm = np.where(~fake, "None", np.where(pol | (F.share > F.share.quantile(.9)), "High", "Medium"))
    conf = np.maximum(p_fake, 1 - p_fake)
    check = np.where(fake | (conf < 0.7), "Yes", "No")
    return pd.DataFrame({"record_idx": test.record_idx, "veracity": np.where(fake, "Fake", "Real"),
                         "manipulation_type": manip, "target_group": target, "confidence_score": conf.round(4),
                         "harm_level": harm, "needs_fact_check": check, "_p_fake": p_fake, "_score": score})


def ece(conf, correct, wt, b=10):
    e, bins = 0.0, np.minimum((conf * b).astype(int), b - 1)
    for k in range(b):
        m = bins == k
        if m.any():
            e += wt[m].sum() / wt.sum() * abs(np.average(correct[m], weights=wt[m]) - np.average(conf[m], weights=wt[m]))
    return e


if __name__ == "__main__":
    test = pd.read_csv(sys.argv[1])
    pred = predict(test)
    rows = [(f"{r.record_idx}_{c}", getattr(r, c)) for r in pred.itertuples() for c in
            ("veracity", "manipulation_type", "target_group", "confidence_score", "harm_level", "needs_fact_check")]
    pd.DataFrame(rows, columns=["RowId", "Prediction"]).to_csv(sys.argv[2], index=False)
    print("rows written:", len(rows))
    if len(sys.argv) > 4:   # scoring only: labels rebuilt from the public corpus, never used above
        truth = pd.read_csv(sys.argv[3]).set_index("record_idx")
        s = pd.read_csv(sys.argv[4]); T = truth.loc[s.record_idx]; P = pred.set_index("record_idx").loc[s.record_idx]
        w = T.manip.map(truth.manip.value_counts() / T.manip.value_counts()).values  # back to the real class mix
        y = (T.veracity == "Fake").values
        print("veracity AUC", round(roc_auc_score(y, P._score), 3))
        for name, wt in (("balanced", None), ("population", w)):
            fv = f1_score(T.veracity, P.veracity, average="macro", sample_weight=wt)
            fm = f1_score(T.manip, P.manipulation_type, average="macro", labels=TYPES, sample_weight=wt)
            print(f"{name}: veracity F1 {fv:.3f}  manipulation F1 {fm:.3f}  mean {(fv + fm) / 2:.3f}")
        ok = (P.veracity.values == T.veracity.values).astype(float)
        print("ECE (population)", round(ece(P.confidence_score.values, ok, w), 3))

Katotohanan: Filipino Fake News Classifier