The Archive of Contradictions

The problem in brief

A fact-checking agency wants to automate its analysts. Each input is a short English claim ("The Battle of Shiloh took place in 1864.") and your system must say whether an encyclopedic archive supports it, refutes it, or does not contain enough information. You get 14,998 labelled training claims, 3,000 test claims and the archive itself: wiki_pages.csv, about 487,000 Wikipedia introductions split into numbered sentences. The task is a version of the FEVER benchmark (Thorne et al., 2018); using the original FEVER labels or gold evidence is forbidden, any NLP or ML technique is allowed, and the organisers point to a pretrained natural-language-inference (NLI) model in the documentation.

The expected pipeline is the classic one: retrieve relevant pages, select evidence sentences, decide with an NLI model. The interesting twist, which a careful contestant discovers by looking at the archive, is that this pipeline can only run on a small fraction of the claims.

How you are scored

The metric is macro-F1 over the three classes:

$$\text{F1}_c = \frac{2\,\text{P}_c\,\text{R}_c}{\text{P}_c+\text{R}_c},\qquad \text{Macro F1} = \frac{\text{F1}_{\text{SUPPORTS}} + \text{F1}_{\text{REFUTES}} + \text{F1}_{\text{NEI}}}{3}.$$

Each class weighs the same, so neglecting the rarer REFUTES (20 % of training claims) or NOT ENOUGH INFO (25 %) is costly. Points are

$$\text{points} = \begin{cases} 100 & \text{if Macro F1} \ge 0.6\\ 0 & \text{if Macro F1} \le 0.3\\ \left\lfloor \dfrac{\text{Macro F1}-0.3}{0.3}\times 100\right\rfloor & \text{otherwise,}\end{cases}$$

so 0.6 macro-F1 is the full-score target (statement, "Evaluare"). The test labels are private: every score below is measured on a held-out 20 % of the training claims (fold 0 of a stratified 5-fold split, 3,000 claims), never on the test.

First look at the data

The archive is mostly missing. Count the first characters of the 487k page titles: 255k start with "A", about 230k with a digit, and almost nothing else. The alphabetically last "A" page is Arthur_Sweeney, and even within that range many famous pages (Angelina Jolie, Earth, The Avengers) are absent. In the original FEVER the evidence pages are spread over all 5.4 million Wikipedia pages, so a claim about "Nikolaj Coster-Waldau" simply has no evidence here.

To measure it, we look up the claim's leading entity (the longest prefix of the claim, optionally after "The", that is a page title, ignoring the "(film)"-style disambiguation suffix). Only 6.1 % of the training claims and 6.8 % of the test claims have their subject page in the archive. When it is there, retrieval works nicely ("Anne Hathaway played the character Mia Thermopolis." finds the sentence "...before landing the lead role of Mia Thermopolis in the Disney comedy film The Princess Diaries"), but for 93 % of the claims the decision has to come from the claim itself.

The claims carry strong clues of their own. FEVER claims were written by annotators who were asked to mutate true sentences into false ones, and the easiest way to make a claim false is to negate or restrict it. The result is a famous annotation artefact (Schuster et al., 2019):

NOT ENOUGH INFO claims have softer cues: vague or speculative vocabulary such as "attended", "popular", "successful", "will". A model that reads the claim alone can therefore get far above chance; the question is how far.

A first baseline

Claim-only, bag of words: TF-IDF on word unigrams and bigrams (plus character 2 to 5-grams), class-balanced logistic regression. 5-fold cross-validated macro-F1: 0.517 (per class: NEI 0.38, REFUTES 0.53, SUPPORTS 0.64). Adding a frozen sentence embedding of the claim (all-MiniLM-L6-v2, 384 dimensions) to the TF-IDF features gives 0.549. That is 82 points: decent, but the full score needs 0.6.

The key idea

Since retrieval can only help 6 % of the claims, the gains must come from reading claims better, and the best reader is a pretrained transformer fine-tuned end to end on the 12,000 training claims.

Why is fine-tuning so much better than "frozen embedding + linear model"? A frozen sentence embedding was trained to make paraphrases close to each other. For that objective, "X was in the film Y" and "X was not in the film Y" are near-identical, so the negation, which is the single most useful signal here, is almost erased. Fine-tuning moves the whole network so that the directions that separate the labels (negation, restriction, hedging, specificity of the claimed fact, plausibility of entity pairs memorised during pretraining) become large in the pooled vector. With 3 classes and only 22 M parameters (MiniLM-L6), three epochs on CPU are enough.

Two details matter for macro-F1:

• Class-weighted cross-entropy, with weight $w_c = \frac{N}{3 N_c}$, so that a REFUTES or NEI error costs about twice a SUPPORTS error. Without it the model drifts to the majority class and the NEI F1 collapses.
• Blending the fine-tuned model with the bag-of-words model, $P = 0.3\,P_{\text{LR}} + 0.7\,P_{\text{MiniLM}}$: the two make different mistakes (exact n-gram cues versus contextual meaning), so averaging probabilities helps a little.

And what about the evidence pipeline? We built it: subject-title retrieval, TF-IDF ranking of the page's sentences against the claim (keeping the top 5 plus the page's first sentence), then the cross-encoder nli-deberta-v3-base (trained on SNLI and MultiNLI, not on FEVER) scores each (sentence, claim) pair. On the 6 % of claims with evidence, stacking the NLI scores on top of the bag-of-words model raised macro-F1 from 0.51 to 0.62 on that subset. But the fine-tuned claim model already reaches 0.64 on the same subset, and adding the NLI scores to it made things slightly worse (0.633 for the lightest weight we tried). Our NLI scores are noisy: often the retrieved sentences are about the right entity but not about the claimed fact, and "neutral" then pushes towards NOT ENOUGH INFO for claims that are in fact supported by a sentence the page does not contain. A better sentence selector could fix it, but at most it can move 6 % of the claims.

The full solution

1. Split: stratified 5-fold on train.csv, fold 0 (3,000 claims) is our validation set, the models are trained on the remaining 12,000.
2. Model A: TF-IDF (1-2 grams, sublinear tf, min_df 2) concatenated with the frozen mean-pooled MiniLM embedding, logistic regression ($C = 2$, class_weight="balanced").
3. Model B: MiniLM-L6 + mean pooling + linear head, AdamW (lr $5\cdot10^{-5}$, one-cycle schedule, weight decay 0.01), batch 32, max 48 tokens, 3 epochs, class-weighted cross-entropy. About 13 minutes on 2 CPU threads.
4. Blend $0.3\,P_A + 0.7\,P_B$, predict the argmax, write id,subtaskID,label.

The heart of model B:

h = self.enc(**batch).last_hidden_state            # (B, T, 384)
m = batch["attention_mask"].unsqueeze(-1).float()
logits = self.head((h * m).sum(1) / m.sum(1))       # mean pooling over real tokens
loss = F.cross_entropy(logits, labels, weight=class_weights)

Results

Macro-F1 on our validation fold (official test labels are private). The first line is 5-fold cross-validation.

approach	macro-F1	points (formula)
TF-IDF words + chars, logistic regression (5-fold CV)	0.517	72
frozen MiniLM embedding, logistic regression (5-fold CV)	0.515	71
TF-IDF + frozen MiniLM, logistic regression (5-fold CV)	0.549	82
same + retrieval/NLI features stacked for all claims (5-fold CV)	0.543	81
fine-tuned MiniLM, claim only	0.595	98
fine-tuned MiniLM + NLI evidence fusion (weight 0.5 / 1 / 2)	0.595 / 0.592 / 0.590	98 to 96
0.3 x logistic regression + 0.7 x fine-tuned MiniLM (final)	0.600 (exploration run), 0.596 (clean solution.py run)	100 / 98
final blend + per-class decision biases tuned for macro-F1 (cross-fitted)	0.600	100 (no gain)

Per-class F1 of the final blend (exploration run): NEI 0.46, REFUTES 0.63, SUPPORTS 0.71. NOT ENOUGH INFO stays the hardest class, which makes sense: whether the archive could verify a claim is not visible in the claim's wording, only in how specific and checkable it sounds.

Reading the table honestly: the same pipeline gave 0.600 and 0.596 in two runs that differ only by the random seed of fine-tuning, so we sit right at the 0.6 line on 3,000 held-out claims, and the standard error of a macro-F1 on 3,000 examples is about 0.01. On the private test, expect roughly 0.58 to 0.61, that is 93 to 100 points. Training the final models on all 15,000 claims instead of 12,000 (we did not, to keep a clean validation) would most likely add a little.

Ideas we tried that did not help: tuning per-class decision biases for macro-F1 (no gain once cross-fitted), stacking NLI scores into the fine-tuned model. An idea we could not finish: probing a small instruction-tuned LLM (Qwen2.5-1.5B) for its world knowledge of each claim; the shared machine's GPU was saturated. It is the natural next step, because it attacks the SUPPORTS versus REFUTES confusion with actual knowledge rather than wording.

What to remember

• Inspect the knowledge source before building on it. Five minutes of counting page titles showed that 93 % of the claims have no evidence in the archive, which completely changes where the effort should go.
• Annotation artefacts are real signal. Crowd-written datasets leak their labels through wording (negations for REFUTES, vague words for NEI). Exploit them in a competition, and remember them when you read a paper's claim that a model "verifies facts".
• Fine-tune rather than freeze when the label depends on small words. General-purpose sentence embeddings flatten negation; end-to-end fine-tuning of even a 22 M-parameter encoder brought +0.05 macro-F1.
• Macro-F1 punishes ignored classes: use class weights (or balanced sampling) during training, and check per-class F1, not only the average.
• Validate the add-on, not the idea. Retrieval + NLI is the textbook pipeline and it did help a weak model, but it added nothing on top of a strong one: only a held-out comparison could tell.