The Analytical Language of John Wilkins

<a href="https://colab.research.google.com/github/IOAI-official/IOAI-2026/blob/main/At-Home-Round/Home-Task-3.ipynb" target="_parent"></a>

"These ambiguities, redundancies, and deficiencies recall those attributed by Dr. Franz Kuhn to a certain Chinese encyclopedia called the Celestial Emporium of Benevolent Knowledge. On those remote pages it is written that animals are divided into (a) those that belong to the Emperor, (b) embalmed ones, (c) those that are trained, (d) suckling pigs, (e) mermaids, (f) fabulous ones, (g) stray dogs, (h) those that are included in this classification..."

— Jorge Luis Borges, The Analytical Language of John Wilkins

The story

In the seventeenth century a churchman named John Wilkins set out to build a perfect language — one in which the very spelling of a word would declare the nature of the thing it named. Each animal would be filed under a rigorous tree of yes-and-no distinctions: beast or fish, winged or finned, tame or wild, until the creature stood alone at the end of a single branch, named by the path that led to it.

The scheme failed, as all such schemes fail. But somewhere a clerk kept building it anyway. He bound every animal of the world into a great Cabinet of Distinctions — and then, before the index could be written, he died. The drawers remain. Each holds one creature behind a small brass grille, and the creature will not say its name. It will only answer yes or no to questions about its own nature.

The Cabinet has come to you with its labels lost. Two lists survived in the clerk's hand:

• animals_pool.txt — every creature filed in the Cabinet (~1,400 entries).
• questions_pool.txt — every distinction the clerk thought to draw (~500 yes/no questions).

Open a drawer. Ask your distinctions. Find the path that names the beast.

Your task

Each hidden creature sits inside a sealed oracle called an Interactor — a brass grille over a drawer, holding one animal. You cannot see it. You may put to it one of two kinds of question:

Call	Returns	What you are asking
interactor.ask(question)	"yes" or "no"	A yes/no question about the hidden animal. The question must be a line from questions_pool.txt.
interactor.guess(animal)	"correct" or "wrong"	"Is this the hidden animal?" "correct" ends the row. The animal must be a word from animals_pool.txt.

Each question in the pool refers to the creature generically — "is it a mammal?", "does it live in water?", "can it fly?" — and the oracle answers about whichever animal is hidden in that drawer.

If you submit a question or animal not in the relevant pool, the oracle refuses without spending its strength: a ValueError is raised and your budget is unchanged. Typos cost nothing.

Each drawer will entertain at most fifteen questions before the grille falls shut.

Scoring

For each creature:

score = max(0, (1 if you ever guessed correctly else 0) - 0.02 × queries_used)

• Correct guess on question 1 → 0.98
• Correct guess on question 5 → 0.90
• Correct guess on question 15 → 0.70
• Never correct → 0

Your score is the mean across all creatures in a test set. Tune on dev, then run the final cell to get your test1 score — that summary table is what you submit a screenshot of. The organizers keep a second, hidden test set for official grading, so a solution that genuinely deduces (rather than overfits dev/test1) is what scores well.

The oracle

In plain language, the oracle inside each Interactor is a local language model — by default, Qwen/Qwen2.5-3B-Instruct. When you call ask(question) it prompts the model with:

You are answering a question about one specific animal.
The animal is: <hidden animal>.
Answer with a single word, yes or no.
Question: <question>

at temperature 0, parses the first word of the reply, and returns "yes" or "no" to your code.

The model is deterministic (the same (animal, question) pair always gives the same answer) and runs entirely inside the Interactor. You are free to run the same model in your own code to predict what it will say without spending the oracle's strength — that is a large part of what makes a clever solution. Note the oracle answers from the model's beliefs about the animal, which are usually right but not infallible; a good solution is robust to the occasional surprising answer.

Step 1: Setup

The dataset and helper code (interactor.py, evaluate.py, the two pools, the dev/test CSVs) live in the shared IOAI-2026/AnimalDeduction/dataset Drive folder. The cell below just downloads them into Colab — no sign-in, no shortcuts, just run it. Use a GPU runtime: Runtime → Change runtime type → T4 (free tier is enough).

!pip install -q gdown transformers accelerate

import os, sys
from pathlib import Path
import gdown

# Dataset + helper code live in the shared IOAI-2026/AnimalDeduction/dataset folder
# (public link). Download locally and import from there — no sign-in needed.
LOCAL_DIR = Path('/content/animaldeduction')
if not LOCAL_DIR.exists() or not any(LOCAL_DIR.iterdir()):
    gdown.download_folder(id='1YheHvGfQw5YUa7MjdUF0hQC4sdtLZ5UC',
                          output=str(LOCAL_DIR), quiet=True, use_cookies=False)

sys.path.insert(0, str(LOCAL_DIR))
os.chdir(LOCAL_DIR)
print('Working directory:', os.getcwd())
print('Files:', sorted(p.name for p in LOCAL_DIR.iterdir()))

Step 2: Load data and try the oracle

The Interactor owns the hidden gold animal and runs a local LLM (Qwen 2.5 3B Instruct by default) to answer yes/no questions about it. The first Interactor(...) instantiation triggers the LLM download (~6 GB on first run, takes 30-60 s on T4). Every subsequent Interactor reuses the same LLM that's already loaded in memory.

import random
import numpy as np
import pandas as pd
import torch

from interactor import Interactor
from evaluate import evaluate, load_pools

DEVICE = 'cuda' if torch.cuda.is_available() else 'cpu'
print('Device:', DEVICE)

animals_pool, questions_pool = load_pools()
print(f'animals_pool size:   {len(animals_pool):>6}  (e.g. {animals_pool[:5]})')
print(f'questions_pool size: {len(questions_pool):>6}  (e.g. {questions_pool[:3]})')

# Sanity probe: create one Interactor and ask two questions about an octopus.
probe = Interactor(gold_animal='octopus', animals_pool=animals_pool, questions_pool=questions_pool)
print("\nask('is it a mammal?')      ->", probe.ask('is it a mammal?'))
print("ask('does it live in water?') ->", probe.ask('does it live in water?'))
print('Queries used:', probe.queries_used, '/', probe.budget)

# The model loads in bfloat16, which a T4 (Turing) cannot accelerate. Convert to
# float16 (T4 has fp16 tensor cores) -> identical answers, several times faster.
import torch
if torch.cuda.is_available() and next(Interactor._model.parameters()).dtype != torch.float16:
    Interactor._model = Interactor._model.half()
    print('oracle model -> float16 (faster on T4)')

Step 3: Solution interface

Your solution is a class with two methods:

• __init__(self, animals_pool, questions_pool) — runs once. Load models, precompute tables, etc.
• solve(self, interactor) — runs once per test row. Use the oracle to identify the hidden animal.

Inside solve, you have:

interactor.ask(question)      -> 'yes' or 'no'        (question must be in questions_pool)
interactor.guess(animal)      -> 'correct' or 'wrong' (animal must be in animals_pool)
interactor.is_done()          -> True after a correct guess or budget exhausted
interactor.remaining_budget() -> int

Scoring per row: score = max(0, (1 if you ever guess correctly else 0) - 0.02 * total_queries).

Budget is 15 questions per row. Information theory: log₂(1400) ≈ 10.5 bits, each yes/no answer is at most 1 bit — so ~11 well-chosen questions plus 1 final guess fit the budget, if each question splits the remaining candidates in half. Most don't: "does it have a backbone?" sounds decisive but the model calls a great many creatures vertebrates. A good question splits the remaining candidates roughly in half, given everything you have already learned — so the right next question depends on the answers so far.

Baseline: random guessing (the floor)

Ignores ask() entirely. Just guesses random animals until the budget runs out. Expected score: ~0 (15 random guesses out of ~1,400 candidates ≈ 1% solve rate). Any reasonable solution needs to beat this by a lot.

class RandomBaseline:
    def __init__(self, animals_pool, questions_pool, seed=0):
        self.animals_pool = animals_pool
        self.questions_pool = questions_pool
        self.rng = random.Random(seed)

    def solve(self, interactor):
        guessed = set()
        while not interactor.is_done():
            cand = self.rng.choice(self.animals_pool)
            while cand in guessed:
                cand = self.rng.choice(self.animals_pool)
            guessed.add(cand)
            interactor.guess(cand)

baseline_results = evaluate(RandomBaseline(animals_pool, questions_pool), 'dev.csv')

Reference: a non-adaptive 20-questions sketch

This reference shows the shape of a real solution without giving away the points. In __init__ it precomputes, with its own copy of the model, the oracle's yes/no answer to a small fixed list of broad questions for every animal — a bit-vector per animal. In solve it asks those same fixed questions, reads off the oracle's bit-vector, and guesses the animals whose precomputed vector is closest.

It works, but it's deliberately weak: the questions are the same for every row (not chosen adaptively to split the remaining candidates), and it uses only a handful. Beating it is mostly about (1) precomputing the full animal × question table and (2) choosing each next question greedily to most evenly split the animals still consistent with the answers so far. That's your job in Step 4.

Precomputing even this small table calls the model a few thousand times (~5-15 min on T4). Skip this cell if you just want to get to your own solution — it is only a reference.

💡 Speed tip

Building the animal×question table by calling interactor.ask() one at a time is slow. You can make your precompute much faster — without changing any answers — by batching your model calls (many prompts through the model per forward pass). That optimization is up to you.

# Reference solution (optional, slow to init). Demonstrates precompute + match,
# but uses FIXED, non-adaptive questions -> leaves most of the score on the table.
FIXED_QUESTIONS = [
    'is it a mammal?',
    'is it a bird?',
    'is it a fish?',
    'is it an insect?',
    'does it live in water?',
    'can it fly?',
    'is it a carnivore?',
    'is it bigger than a human?',
    'does it have a backbone?',
    'is it commonly kept as a pet?',
    'does it have legs?',
    'does it lay eggs?',
]

class FixedQuestionsReference:
    def __init__(self, animals_pool, questions_pool, max_animals=None):
        self.animals_pool = animals_pool
        self.questions_pool = set(questions_pool)
        self.fixed = [q for q in FIXED_QUESTIONS if q in self.questions_pool]
        from interactor import Interactor
        cand = animals_pool if max_animals is None else animals_pool[:max_animals]
        self.candidates = cand
        print(f'  [reference] precomputing {len(cand)} x {len(self.fixed)} answer table...')
        self.table = {}
        for i, a in enumerate(cand):
            sim = Interactor(gold_animal=a, animals_pool=self.animals_pool,
                             questions_pool=self.questions_pool, budget=10**9)
            self.table[a] = tuple(1 if sim.ask(q) == 'yes' else 0 for q in self.fixed)
            if (i + 1) % 200 == 0:
                print(f'    {i+1}/{len(cand)}')
        print('  [reference] table ready.')

    def solve(self, interactor):
        obs = []
        for q in self.fixed:
            if interactor.remaining_budget() <= 1:
                break
            obs.append(1 if interactor.ask(q) == 'yes' else 0)
        obs = tuple(obs)
        def agree(a):
            vec = self.table[a]
            return sum(1 for x, y in zip(vec, obs) if x == y)
        ranked = sorted(self.candidates, key=agree, reverse=True)
        for a in ranked:
            if interactor.is_done():
                break
            interactor.guess(a)

# Example (commented out by default — uncomment to run; slow to init):
# ref = FixedQuestionsReference(animals_pool, questions_pool)
# ref_results = evaluate(ref, 'dev.csv')

Step 4: Your solution

Replace the body of MySolution.solve (and __init__ if you precompute anything) with your strategy. Iterate on dev.csv until you're happy with the score, then jump to Step 5 to evaluate on test1 + test2.

The intended approach:

1. In __init__, precompute once — with your own copy of the model — the oracle's yes/no answer for every (animal, question) pair you care about. This costs no oracle budget.
2. In solve, keep a set of candidate animals still consistent with the answers so far. At each step pick the question whose answer most evenly splits that set (maximize information gain), ask it, and shrink the set. Guess when one candidate dominates or the budget is nearly gone.
3. Be robust: the oracle occasionally answers in a way your table didn't predict. Don't let one surprising bit eliminate the true animal forever.

class MySolution:
    def __init__(self, animals_pool, questions_pool):
        self.animals_pool = animals_pool
        self.questions_pool = questions_pool
        # TODO: precompute the (animal x question) answer table and anything else.

    def solve(self, interactor):
        # TODO: your strategy. Each ask is ~1 bit; log2(1400) ~ 10.5 bits needed.
        # Budget is 15. Choose each question to split the remaining candidates.
        while not interactor.is_done():
            interactor.guess(self.animals_pool[0])

my_dev = evaluate(MySolution(animals_pool, questions_pool), 'dev.csv')

Step 5: Final scoring

Once you're happy with your dev score, run this cell. It scores dev and test1 (and test2 automatically, if that file is present). The FINAL line — the n-weighted mean over the available test split(s) — is what you submit a screenshot of.

The organizers also score your submitted MySolution on a separate hidden test set that is not included here. Aim for a strategy that deduces the animal from scratch each row, so it transfers to unseen creatures.

import os

solution = MySolution(animals_pool, questions_pool)
dev_results   = evaluate(solution, 'dev.csv')
test1_results = evaluate(solution, 'test1.csv')

splits = [('dev', dev_results), ('test1', test1_results)]
# test2 is a held-out set; included automatically only if present in the folder.
if os.path.exists('test2.csv'):
    splits.append(('test2', evaluate(solution, 'test2.csv')))

rows = [{
    'split': name, 'n': r['n'], 'mean_score': r['mean_score'],
    'solved_rate': r['solved_rate'], 'mean_queries': r['mean_queries'],
} for name, r in splits]

# FINAL = n-weighted mean over every test split available (test1 [+ test2]).
tests = [r for name, r in splits if name.startswith('test')]
n_test = sum(r['n'] for r in tests)
rows.append({
    'split': 'FINAL',
    'n': n_test,
    'mean_score':   sum(r['mean_score']   * r['n'] for r in tests) / n_test,
    'solved_rate':  sum(r['solved_rate']  * r['n'] for r in tests) / n_test,
    'mean_queries': sum(r['mean_queries'] * r['n'] for r in tests) / n_test,
})
pd.DataFrame(rows)