Ghost of the Machine

The problem in brief

Each passage (500 to 800 words) starts as human-written text from a book or encyclopedia and, at one unknown point, continues as text generated by a language model that was given the human part as a prompt. Each side has at least 180 words. You must output the character index of the switch. The only pretrained model allowed is the text encoder bge-base-en-v1.5 (110 M parameters, no generation), and the whole notebook, training included, must run in 10 minutes on one GPU. You get 1,221 labelled training passages. Leaderboard A (380 passages) was shown live and leaderboard B (380 passages) decided the ranking; both answer files are now public, so we score on them directly.

How you are scored

With $p$ the predicted index and $t$ the true one:

$$\text{score} = 100 \cdot \operatorname{mean}\, e^{-|p - t|/\tau}, \qquad \tau = 100 .$$

One sentence is about 120 characters, so being off by one sentence already costs about 70 % of that passage's points: you must find the exact sentence. Reference points from the statement: the official baseline (the average boundary fraction times the length) scores 28.6 on A, and the Scientific Committee, who fine-tuned the encoder, scores 93.41 on A. In the official results, only 6 of 440 contestants scored at least 90 on this task, and the median was 0.

First look at the data

• The switch is always at a sentence start. In all 1,221 training passages the boundary comes right after ". " (1,207 times), "! " or "? ". Splitting on these gives about 27 candidate positions per passage, and the true one is always among them. Snapping the baseline guess to the nearest sentence start already lifts it from 32.7 to 42.9 on B.
• The human side has fingerprints. The human texts carry tokenisation artefacts of their source ("Anselm ," with a space before punctuation, " ."), many more en dashes (1.0 against 0.02 per 1,000 characters) and straight double quotes (2.2 against 0.1). The machine continuation has clean typography. The model, on the other hand, "sounds like an LLM": smooth transitions, evaluative summaries ("despite the profound material and intellectual losses...").
• The boundary sits at $59.5 \% \pm 6.4 \%$ of the length, a useful but weak prior.

A first baseline

Classify each sentence as human or machine, then pick the split that agrees best with the labels. With 13 style counts only (the marks above, sentence length, mean word length, relative position) and a logistic regression, 97.4 % of sentences are classified correctly, yet the change-point score is only 71.2 on B. The few errors are concentrated exactly where it matters: around the switch, where the first machine sentence continues the human topic and imitates its style.

The key idea

Two ideas turn a good sentence classifier into an exact locator.

1. Use the fact that there is exactly one switch. With $p_i = P(\text{machine} \mid \text{sentence } i)$, the log-likelihood that the switch is at candidate $k$ is

$$L(k) = \sum_{i<k} \log(1 - p_i) + \sum_{i \ge k} \log p_i .$$

One noisy sentence in the middle of a human paragraph cannot move the answer, because every other sentence votes against it. Adding the bge CLS embedding to the style counts raises sentence accuracy to 98.7 %; we temper the likelihood ($\alpha L$, since neighbouring sentences are not independent), add the Gaussian prior on the position, and forbid splits leaving fewer than 150 words on a side. That gives 90.0 on B.

2. Model the transition itself. Most of the remaining errors are one sentence off, because the first machine sentence looks human. What gives it away is not the sentence alone but the change relative to the previous sentence. A second logistic regression is trained on pairs of consecutive sentences, with features $[e_k - e_{k-1},\ e_k \odot e_{k-1}]$ on the standardised embeddings and style counts, to predict "the switch is at $k$". Its log-odds are added to the posterior:

$$\log \mathrm{post}(k) = \alpha L(k) + \beta \operatorname{logit} q_k + \log \pi(k) + \text{const}.$$

Decision. Since the score is $e^{-|p - t|/100}$, the best answer under the posterior is

$$p^\star = \arg\max_{c} \sum_k \mathrm{post}(k)\, e^{-|c - c_k|/100},$$

which hedges between two neighbouring sentences when the posterior is split. Here the posteriors are so peaked that it almost always coincides with the most probable sentence (91.43 against 91.43 on B).

The full solution

1. Split each passage into sentences at the candidate positions.
2. Encode every sentence with frozen bge-base-en-v1.5 (CLS vector, L2-normalised, 96 tokens). This takes about 3 minutes for train plus one test split on a laptop GPU, so it fits the 10-minute budget on the contest GPU.
3. Sentence classifier: logistic regression ($C = 0.01$) on [embedding, 13 style counts], standardised.
4. Transition classifier: logistic regression on pair features of consecutive sentences.
5. Posterior over candidates with $\alpha = 2$, $\beta = 1$, the position prior and the 150-word constraint; output the expected-score maximiser.

$\alpha$, $\beta$ and $C$ were chosen by 5-fold cross-validation grouped by passage on the training set (92.5), never on the leaderboards. Grouping matters: sentences of one passage share topic and style, and an ungrouped split would leak them across folds.

Results

approach	CV on train	leaderboard A	leaderboard B
official baseline: mean fraction of the length		28.6	32.7
mean fraction snapped to the nearest sentence start		40.1	42.9
style counts + change point	68.1	68.5	71.2
bge CLS + style + change point ($\alpha = 1$)	91.7	90.1	91.0
bge CLS + style + change point ($\alpha = 2$, prior)	92.2	90.1	90.0
+ transition classifier, $\beta = 1$ (final, solution.py)	92.5	93.0	91.4
same, most probable sentence instead of expected score		93.0	91.4

On leaderboard B the final model finds the exact sentence in 88.7 % of passages; most misses are one sentence early or late (as in the left panel above), and a handful are 3 to 7 sentences off. On A, the split where the committee reports 93.41, we reach 93.0 without any fine-tuning, using only frozen embeddings and two logistic regressions. Fine-tuning bge on the sentence task, as the committee did, is the natural next step to recover the remaining boundary sentences.

What to remember

• Find the discrete structure first. The switch is always at a sentence start: this cuts thousands of positions down to about 27, and a regression on characters into a classification over sentences.
• Aggregate with the constraint. A single change point turns 98.7 %-accurate but noisy sentence votes into an exact location; sum log-probabilities instead of thresholding each sentence.
• Model transitions, not only states. The hardest sentence is the one right after the switch; a classifier on consecutive pairs captures the change that a sentence-level classifier misses.
• Decide with the metric. For $e^{-|p - t|/\tau}$, choose the position maximising the expected score under the posterior; it hedges only when the model is genuinely unsure.
• Cheap features still count. Typography (en dashes, quotes, tokenisation artefacts) gives 97 % sentence accuracy on its own: always look at the raw characters before reaching for embeddings.