Find the Order

The problem in brief

Each test item is a short spoken English conversation between two people, cut into turns (one audio file per turn, 7 to 20 turns). The files have been shuffled, and you must put them back in chronological order. You are told which two files are the first and second turns, which fixes where the dialogue starts. Only three pretrained models are allowed (wav2vec 2.0, Whisper and the small language model Qwen2.5-0.5B), the machine has one GPU and no internet, and the whole notebook, including any training, must finish within 10 minutes on 100 hidden dialogues. You get 1 288 training dialogues with their true order and 100 public test dialogues to check yourself.

How you are scored

For a dialogue with $n$ turns there are $M = n(n-1)/2$ pairs of turns. A pair is correct when your order puts the two turns in the same relative order as the truth. If $I$ pairs are inverted, the dialogue scores

$$\text{score} = 1 - \frac{I}{M},$$

and the task score is the mean over the 100 dialogues, times 100. A random order scores about 50, so 50 is the floor, not 0. An invalid permutation scores 0 for that dialogue.

The organisers released the graded splits after the contest (Hugging Face dataset IOAI-official/ioai-2026-find-the-order), so we score our final predictions on leaderboard B, the split that decided the final ranking, with the official metric. Every choice below was made on training dialogues only.

What is a good score? The ranking does not use the raw number directly. The IOAI 2026 contest rules (technical appendix, section 2.5) normalise every task:

$$\text{Norm} = 100 \cdot \frac{\text{raw} - \text{Min}}{\text{Max} - \text{Min}},$$

where Min is the official baseline (here the identity permutation, raw 49.0 on leaderboard B) and Max is the larger of $0.9 \times$ the Scientific Committee's own solution and the best raw score of any contestant. The official results table (ioai-official.org, Results 2026) lists 466 contestants: 333 got 0 on this task, a cluster sits around 5 to 15 (raw scores barely above the baseline), and 36 contestants scored above 80. The best normalised score was 93.8. Since nobody reached 100, Max must be $0.9 \times$ the committee score, so Max is at most 90 raw. That lets us convert our raw score into a safe lower bound on the normalised one.

First look at the data

Three facts shape the whole solution.

1. The speakers alternate. Each file contains one speaker turn, so in the true order speaker A is always at even positions (0, 2, 4, ...) and speaker B at odd positions. The two prefix files tell you which voice is A and which is B. If you can tell the two voices apart, you already know the parity of every turn's position.
2. The content is very local. These are everyday textbook dialogues (shopping, a dinner, asking for directions), with a median of 7 words per turn. A turn usually answers the turn just before it: "What is your favorite?" is followed by "I love anything with chocolate the best!". So the key quantity is "how well does turn $j$ follow turn $i$?".
3. The transcripts are good but not perfect. Whisper-base gets most turns right, with a few amusing slips ("I scream please" for "Ice cream, please", "1336 we in" for "1336 yuan"). The ordering method must tolerate noisy text.

A first baseline

The official baseline submits the identity permutation (files in their given order) and scores 49.0 on leaderboard B, the same as a random order (50.9).

The cheapest real improvement uses the prefix: put the two known turns first and leave the rest in file order. That alone gives 69.1, because 2 of the 7 to 20 turns are now perfect and they sit at the start, which fixes all pairs that involve them.

Next, the voices. For each chunk we compute 20 MFCCs (a compact summary of the spectrum, obtained by a discrete cosine transform of Whisper's log-mel bands) and keep their mean and standard deviation over time: a 40-number voice fingerprint. A 2-means clustering seeded by the two prefix chunks, forced to give speaker A exactly $\lceil n/2 \rceil$ turns, assigns 99.5 % of the chunks to the right speaker, and 97 of the 100 dialogues are split perfectly. Placing A's turns at even positions and B's at odd positions (in arbitrary order inside each group) brings the score to 70.2. The speaker split is almost free and almost perfect, but on its own it barely helps: it halves the number of candidates for each position, and the real work is ordering the turns of each speaker.

The key idea

Turn the problem into "which turn comes next?", score that with the language model, and correct the scores for how likely each turn is on its own.

For every ordered pair $(i, j)$ where $i$ and $j$ belong to different speakers, ask Qwen for the log-probability of turn $j$ given turn $i$:

$$s(i \to j) = \log P_{\text{LM}}(\text{"B: } t_j\text{"} \mid \text{"A: } t_i\text{"}).$$

A dialogue order is then a path through all turns that starts with the two prefix turns, and its score is the sum of $s$ along the path. Finding the best path is a travelling-salesman-like search, but with at most 20 turns and the speaker constraint, a beam search is enough.

There is a trap, visible in the left panel below. Raw log-probabilities are dominated by the length and banality of the candidate: a short, generic turn ("Yes, I know.", "Okay.") is likely after almost anything, while a long specific turn is unlikely everywhere. The row-wise best candidate (white star) then becomes the same "magnet" turn for many rows, and the true successor (red box) wins only 24 % of the time over all leaderboard B turns.

The fix comes from information theory. What we want is not "is $j$ probable after $i$?" but "is $j$ more probable after $i$ than in general?". That is the pointwise mutual information

$$\text{PMI}(i, j) = \log P(t_j \mid t_i) - \log P(t_j),$$

where $\log P(t_j)$ is the probability of turn $j$ after a neutral header. We use a softened version $s(i\to j) - \alpha \log P(t_j)$; $\alpha = 0.75$ was the best value on 60 training dialogues (with $\alpha = 1$ the correction starts to over-reward long, unusual turns). With it, the true successor becomes the top candidate 39 % of the time, and the magnet disappears (right panel).

Notice also the checkerboard: since speakers alternate, a turn can only be followed by a turn of the other speaker, so half of the matrix is never computed. The speaker split from the baseline is what makes this possible.

Even with PMI, a single "best next turn" is right less than half the time, so greedy decoding is a poor idea. A global search that adds up the evidence along the whole chain is much better: the sum rewards orders in which every transition is plausible, not just the next one.

The full solution

1. Transcribe and fingerprint. Resample each chunk from 44.1 kHz to 16 kHz, transcribe it with Whisper-base (batched per dialogue), and compute the 40-number MFCC fingerprint.

2. Split the speakers with the seeded, balanced 2-means described above.

3. Pair matrix. For every pair $(i,j)$ of different speakers, compute $s(i \to j)$ with the prompt A conversation between two people.\nA: <t_i>\nB: <t_j>\n and sum the log-probabilities of the tokens of the second line. Also compute $\log P(t_j)$ once per turn. About 50 pairs per dialogue, batched 24 at a time.

4. Beam search with width 200 over chains that start with the prefix, adding $s(i \to j) - 0.75 \log P(t_j)$ at each step. Two partial chains that have used the same set of turns and end on the same turn have the same possible futures, so we keep only the better one (the classic dynamic-programming trick for the travelling salesman, here inside a beam).

   key = (frozenset(seq + (j,)), j)          # (turns used, last turn)
   if key not in cand or cand[key][0] < s + w:
       cand[key] = (s + w, seq + (j,))
   

5. Re-rank with the full context. The pair scores only see one previous turn. We take the 8 best chains from the beam and score each as a complete dialogue: the sum over positions $p \ge 2$ of $\log P(t_{\sigma(p)} \mid t_{\sigma(0)}, \dots, t_{\sigma(p-1)})$, one forward pass of Qwen per chain. The chain with the highest full-context likelihood wins. Here no PMI correction is needed: all 8 chains contain the same turns, so the per-turn priors add up to the same total and cancel.

6. Convert the chain into the rank format ($P[i]$ = position of chunk $i$) and write answers.json.

On our CPU (2 threads), transcription takes about 10 minutes, the pair matrix about 9 minutes and the re-ranking about 2 minutes for 100 dialogues; on the contest GPU the same pipeline would run in a few minutes, within the 10-minute limit. We did not fine-tune Qwen: the zero-shot model with the PMI correction already does most of the work.

Results

All scores are pairwise ordering accuracy $\times 100$ on the official leaderboard B split (100 dialogues). Hyperparameters ($\alpha$, beam width, $K$) were chosen on training dialogues.

approach	leaderboard B
official baseline: identity permutation	49.0
random order	50.9
prefix first, rest in file order	69.1
prefix + speaker parity (MFCC 2-means)	70.2
greedy next-turn choice, PMI scores	70.2
beam search, raw pair scores ($\alpha = 0$)	79.1
beam search, PMI scores ($\alpha = 0.75$)	80.9
beam + full-context re-ranking of the top 8 (final)	82.1
for reference: best of the 8 candidates (oracle)	87.3

On the 60 training dialogues we used for tuning, the same stages give 78.4 (beam, PMI) and 80.1 after re-ranking (oracle 85.5), so the gain from re-ranking is consistent and not an artefact of the test set.

With the official normalisation (Min 49.0, Max at most 90), the raw 82.1 is worth at least 80.7 normalised points, which would have placed it around 37th of 466 contestants or better; the best contestant reached 93.8 normalised, which corresponds to a raw score of at most about 87.4. The exact normalised value depends on the committee's reference score, which is not published. The gap to the top is mostly in the language model: the oracle line shows that a better order is often among our 8 candidates (87.3 if we always picked the best one), but a 0.5B model cannot always tell which one it is. Natural next steps would be a bigger Whisper for cleaner text, fine-tuning Qwen on the 1 288 training dialogues to predict the next turn, and adding acoustic continuity (pauses, intonation at the end of a question).

What to remember

• Use the structure you are given. The prefix and the alternation of speakers are free information: together they take you from 50 to 70 before any language model is used, and alternation halves the search space.
• Reduce ordering to pairwise "what comes next" scores, then search globally. Greedy choice wastes the scores; a beam search with dynamic-programming merging (same set used, same last item) finds much better chains.
• Correct language-model likelihoods for the prior. Raw $\log P(j \mid i)$ favours short, generic candidates; subtracting $\alpha \log P(j)$ (a PMI-style score) measures how much $i$ actually explains $j$. The same trick appears in retrieval, multiple-choice scoring with LMs and collocation mining.
• Generate a few candidates cheaply, then re-rank them with a more expensive model. Local scores to shortlist, full context to decide: a pattern that transfers to translation, speech recognition and search.
• Know the floor of your metric. Pairwise accuracy gives 50 to a random order, so 70 is a modest gain and 82 a big one.