← Back to the problem

Hungary HAIO 2024 · Correction by IOAI Training

Scientific Task – ChatGPT

This page gives the full approach. Read it once you have searched.

Metric
jury points (30 max)
Evaluated on
written answers
Our score
23 of 30 pts answerable (screens 404)
Expected level
30 points

We solved the problem ourselves: downloaded the data, ran the experiments and measured the score below. The statement and data belong to their organisers.

The problem in brief

This 30-point "scientific task" of the 2024 Hungarian AI Olympiad final asks you to diagnose the weaknesses of a chatbot from five real exchanges about Formula 1 (2024 season). A fan asks ChatGPT for the last race winner, for a summary of a Wikipedia page given by link, for the earliest race at which Max Verstappen could clinch the title, for five Python programs computing championship totals, and finally to find a sentence hidden in a long pasted text. Each time the answer is wrong in an instructive way, and you must explain why, using what you know about how large language models are built.

A caveat before reading. The five ChatGPT answers were shared as screenshots on Google Drive, and these links now return "404 not found". The questions that only ask for the mechanism (most of the paper) are fully answered below; the parts that require reading the exact screenshot (the details of the calculation in Q3 and the list of correct programs in Q4) are answered with the method you would apply, not with the final list.

How you are scored

Points are attached to each sub-question, 30 in total: Q1 8 points (1 + 2 + 5), Q2 5 points (3 + 2), Q3 5 points (3 + 2), Q4 6 points (2 + 4), Q5 6 points (2 + 4). Grading is by the jury, so what earns the points is a precise mechanism ("training data stops at a cutoff date"), not a vague one ("ChatGPT is not perfect").

The key idea

Almost every failure in this task comes from one sentence: a language model is a function that predicts the next token from the text in its context window, using parameters frozen at the end of training. Three consequences follow, and each question tests one of them.

  1. Frozen knowledge. Training data stops at a knowledge cutoff. Anything after it (the latest race) simply does not exist for the model, unless a tool such as web search injects it into the context.
  2. Plausibility, not truth. The training objective rewards text that looks like a likely continuation. When the right information is missing, the most likely continuation is still a fluent, confident answer: a hallucination. The model has no built-in signal that says "I do not know".
  3. Limited, uneven reading. The model only "sees" its context window, which has a maximum length, and attention over very long inputs is uneven: information in the middle of a long text is retrieved worse than information at the beginning or the end.

Arithmetic and code are a fourth, related weakness: the model writes the steps of a calculation token by token, it does not execute them, so it can produce a formula that looks rigorous but is wrong, or code that looks correct and is not.

Question by question

Q1. The last race winner (8 points)

Why can it not answer? (1 point) The race took place after the model's knowledge cutoff. The model's parameters were fixed when training ended, so it has never seen the result. In 2024 the free ChatGPT also had no live web access, so nothing could bring the result into its context.

How can John get up-to-date information? (2 points) Either by bringing the information to the model or by not using the model as a source:

  • use a version with web search / browsing (a retrieval tool fetches current pages and places them in the context, then the model summarises them: retrieval-augmented generation);
  • paste the text of an official report into the prompt himself;
  • or simply use primary sources: the official Formula 1 website or app, the FIA results, a sports news site.

Five questions it certainly cannot answer (5 points). A good list covers different reasons, which is what makes each item worth its point:

  1. "Who won yesterday's Grand Prix?" (after the cutoff);
  2. "What is the weather in Budapest right now?" (real-time data);
  3. "What did I eat for breakfast this morning?" (private information never in the training data);
  4. "Who will win the 2030 world championship?" (the future is unknowable, any answer is a guess);
  5. "What number am I thinking of?" or "What is written on the paper on my desk?" (information that exists nowhere in text).

Questions like "how many r's are in strawberry" are also classic failures, but they are a different mechanism (tokenisation) and the model can sometimes get them right, so they are weaker examples of "definitely cannot".

Q2. Summarising a page from its link (5 points)

Explanation of the answer (3 points). The model did not read the page; it received only the text of the URL. A Wikipedia URL contains the race's name and year (for example ".../2024_..._Grand_Prix"), which is enough for the model to generate what a summary of such a race usually looks like: drivers, teams, a plausible winner. It picked a real driver (Pierre Gasly) because that name fits the context statistically, not because it is true. This is a textbook hallucination: fluent, specific, confident, and false. Two factors make it worse: the race is after the cutoff, so there is no memorised fact to fall back on, and models are trained to be helpful, which pushes them to produce an answer instead of admitting they cannot open links.

Did it really open the page? (2 points) No. Evidence: (i) the version used had no browsing tool, so it could not make an HTTP request; (ii) the page plainly states that Max Verstappen won, so a model that had read it would not have named Gasly; (iii) the answer contains nothing that is not guessable from the URL alone. A quick experiment confirms it: give the model a URL with a made-up race name and it will "summarise" a race that never existed.

Q3. When can Verstappen clinch the title? (5 points)

What the correct computation looks like (3 points). The question has a clean worst-case answer. Let LL be Verstappen's current lead over his closest rival and let grg_r be the maximum points available in race weekend rr (in 2024: 25 for a win plus 1 for the fastest lap, plus 8 more on a sprint weekend). In the most favourable scenario Verstappen takes the maximum and the rival scores nothing, so after kk more weekends the lead is L+∑r≤kgrL + \sum_{r \le k} g_r, while the rival can still collect at most ∑r>kgr\sum_{r > k} g_r. The title is mathematically decided at the first kk where

L+∑r=1kgr  >  ∑r=k+1Rgr.L + \sum_{r=1}^{k} g_r \; > \; \sum_{r=k+1}^{R} g_r .

With identical weekends of value gg this gives k>gR−L2gk > \frac{gR - L}{2g}: each race won both increases the lead and removes points from the pool, which is why the lead and the remaining pool cross roughly halfway through the season.

Lead of the championship leader versus points still available to the rival, for a hypothetical lead of 48 points with 17 weekends left: the two curves cross after 8 weekends, while a naive formula that ignores the shrinking pool predicts about 15.

Helping John read ChatGPT's answer means checking it against this formula: which lead did it use, which points per race (forgetting the fastest-lap point and the sprints is a frequent mistake), and did it account for the fact that the remaining pool shrinks as races are run? The screenshot is no longer available, so we cannot reproduce its exact numbers; solution.py shows the correct calculation and contrasts it with a typical flawed formula ("(points available minus lead) divided by 25"), which overestimates the answer almost twofold.

Why is the formalism wrong? (2 points) The model imitates the shape of a mathematical argument (variables, an equation, a confident final number) without carrying out logical reasoning. It generates each step as the most plausible next text, so a wrong modelling choice early on (for example dividing the remaining points by the points per win, which ignores that the rival also scores and that the leader's own points also grow) is never checked afterwards. Arithmetic done "in text" adds errors on top. Remedies: ask it to reason step by step and verify each step, or better, ask it to write code and run the code yourself.

Q4. Five Python programs for the standings (6 points)

What you notice when you test them (2 points). Several programs that all look reasonable give different results on the same data, and some do not even run. This is the key observation: plausible-looking code is not correct code, and the only way to know is to execute it on a small example whose answer you computed by hand. Typical slips in this kind of task: the gap is computed to the previous driver instead of to the leader, the leader's total is taken before summing all races, drivers are mixed up after sorting, or a test database is generated with points that the F1 scoring system cannot produce.

Which programs are correct? (4 points) This sub-question can only be answered from the screenshot of the five programs, which is no longer online, so we do not give a list. The method that earns the points is the one implemented in solution.py: write a three-line reference

totals = {d: sum(p) for d, p in points.items()}
best = max(totals.values())
result = {d: (t, best - t) for d, t in totals.items()}

then run each candidate on a few hundred random small databases and keep exactly those that always agree with the reference. On our test harness, a candidate with the "gap to the previous driver" bug fails immediately.

Q5. The hidden sentence (6 points)

Why it failed (2 points). Two mechanisms, and a good answer names both. First, the context window is finite: a long pasted text may be truncated, and if the sentence fell in the cut part the model never saw it. Second, even inside the window, long-context retrieval is uneven: models are best at using information at the start and the end of the input and noticeably worse in the middle (the "lost in the middle" effect, measured on needle-in-a-haystack tests). A third, subtler reason fits the screenshot's question "who is the best driver?": the model's prior from training (Verstappen dominated in 2023 and 2024) competes with a single unsupported sentence in the input, and a lone claim buried in a factual text looks like noise to it.

Where and how to hide the text (4 points). If the goal is that the model finds it:

  • put the sentence at the very beginning or the very end of the text, where retrieval is best;
  • make it stand out from its surroundings: its own line or paragraph, a marker such as "Important note:", or a format different from the rest;
  • keep the whole input well below the context limit (or paste only the relevant part), so nothing is truncated;
  • ask a question that points to it ("According to the text, who is described as the best driver?"), which tells the model to rely on the document rather than on its prior.

If instead John wants the sentence to stay hidden from a human but still be read by the model, the same principles apply in reverse: it must still sit where the model looks, which is exactly how prompt injection attacks place their instructions. It is worth mentioning, because it shows you understand that position and salience control what the model attends to.

Results

questionpointsstatus of our answer
Q1 cutoff, how to get fresh news, 5 impossible questions8complete
Q2 hallucination, the page was not opened5complete
Q3 explain the calculation (3) and why it is wrong (2)5mechanism and correct method complete; exact numbers of the screenshot unavailable
Q4 what you notice (2), which codes are correct (4)6method complete; the list of correct codes needs the deleted screenshot
Q5 why it failed (2), where to place the text (4)6complete

About 23 of the 30 points rest only on the mechanisms and are fully answered here; the remaining 7 depend on screenshots that are no longer online.

What to remember

  • An LLM predicts plausible text from its context with frozen parameters. Cutoff, hallucinations and context limits all follow from that one sentence; name the mechanism, not just the symptom.
  • A URL is not a document. Without a browsing tool the model sees only the characters of the link and will happily invent the page.
  • Never trust generated maths or code without executing it. Write a tiny reference, compare on random tests: it is fast and decisive.
  • Position matters in long contexts. Put what must be used at the start or end, keep inputs short, and ask questions that point to the source.
  • Tools fix knowledge gaps, not reasoning gaps. Search brings fresh facts into the context; checking the logic is still your job.
Full code (solution.py)
"""HAIO 2024 summer final, ChatGPT task: the two computations behind Q3 and Q4.

Q3 asks for the earliest race at which a championship leader can clinch the title. The correct
formalism is a worst-case argument: after k more races the leader is champion if his lead exceeds
the maximum number of points the best rival can still collect. Q4 asks for codes computing totals
and gaps to the leader; we give a reference implementation and a randomised test harness, which is
exactly how one should "test ChatGPT's answers".
"""
import random
import numpy as np
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt

GP_MAX = 26      # 2024 rules: 25 for a win + 1 for fastest lap
SPRINT_MAX = 8   # sprint win

def max_points(n_gp, n_sprint):
    return n_gp * GP_MAX + n_sprint * SPRINT_MAX

def earliest_clinch(lead, remaining):
    """remaining = list of 'gp' / 'sprint+gp' weekends still to race.
    Best case for the leader: he takes the maximum, the best rival scores 0 in each weekend.
    He is champion after weekend k if lead_k > maximum the rival can still score afterwards.
    (Ties are broken by count-back of wins, ignored here: we ask for a strict lead.)"""
    gain = [GP_MAX + (SPRINT_MAX if w == "sprint+gp" else 0) for w in remaining]
    for k in range(len(remaining) + 1):
        lead_k = lead + sum(gain[:k])
        still = sum(gain[k:])
        if lead_k > still:
            return k
    return None

def naive_wrong(lead, n_races):
    """A typical flawed formula: forgets that the leader's own points also grow and that
    the rival can score too, e.g. 'races = (points still available - lead) / 25'."""
    return (n_races * 25 - lead) / 25

# ---- Q3 illustration (hypothetical standings: lead of 48 points, 17 weekends left, 5 with sprints)
remaining = ["gp"] * 12 + ["sprint+gp"] * 5
random.Random(0).shuffle(remaining)
k = earliest_clinch(48, remaining)
print("earliest clinch after", k, "more weekends; naive formula says", round(naive_wrong(48, 17), 2))

# ---- Q4: reference implementation + randomised tests for any candidate code
def reference(points):  # points: dict driver -> list of race points
    totals = {d: sum(p) for d, p in points.items()}
    best = max(totals.values())
    return {d: (t, best - t) for d, t in totals.items()}

def candidate_buggy(points):  # typical LLM slip: gap computed to the previous driver, not the leader
    totals = sorted(((sum(p), d) for d, p in points.items()), reverse=True)
    out, prev = {}, totals[0][0]
    for t, d in totals:
        out[d] = (t, prev - t); prev = t
    return out

def test(candidate, trials=200):
    rng = random.Random(1)
    for _ in range(trials):
        pts = {f"D{i}": [rng.choice([0, 1, 2, 4, 6, 8, 10, 12, 15, 18, 25]) for _ in range(rng.randint(1, 8))]
               for i in range(rng.randint(2, 6))}
        if candidate(pts) != reference(pts):
            return False
    return True

print("reference passes:", test(reference), "| buggy candidate passes:", test(candidate_buggy))

# ---- figure: the clinching argument
lead = [48]; still = [sum(GP_MAX + (SPRINT_MAX if w == "sprint+gp" else 0) for w in remaining)]
for w in remaining:
    g = GP_MAX + (SPRINT_MAX if w == "sprint+gp" else 0)
    lead.append(lead[-1] + g); still.append(still[-1] - g)
x = np.arange(len(lead))
fig, ax = plt.subplots(figsize=(8.5, 4.2))
ax.plot(x, lead, "o-", lw=2, label="leader's lead (best case: he wins, rival scores 0)")
ax.plot(x, still, "s-", lw=2, label="points the rival can still score")
ax.axvline(k, ls="--", c="0.4"); ax.text(k + 0.2, max(still) * 0.9, f"clinched after {k} weekends", color="0.3")
ax.set_xlabel("weekends raced from now"); ax.set_ylabel("points")
ax.set_title("Q3: the title is decided when the lead exceeds what is still available")
ax.legend(loc="center right", fontsize=9); ax.spines[["top", "right"]].set_visible(False)
fig.tight_layout(); fig.savefig("figures/clinch.png", dpi=150)