Speed-up

The problem in brief

A camp records a nightly storytelling hour read by about thirty regular readers. Someone has secretly sped up some of the clips so that they fit in the time slot, in a way that keeps the pitch of the voice almost unchanged. You get 865 clips of 7 seconds (16 kHz) that are all at normal speed, about 29 per reader, and 280 test clips by the same 30 readers, some of them sped up. For each test clip you output a score, higher meaning "more likely sped up". The only pretrained model allowed is a provided Whisper speech recogniser; we did not need it.

How we evaluate. The test labels are private and we never submitted anything. We therefore build our own labelled test: for each reader we set 10 training clips aside, speed up a random half of them ourselves, and measure how well the detector ranks them (265 clips, 120 sped up). This tells us how well we detect our speed-up; whether the organisers used the same kind of processing is checked separately on the real test clips (see the results).

How you are scored

The ROC AUC of the scores: the probability that a randomly chosen sped-up clip gets a higher score than a randomly chosen normal one,

$$\text{AUC} = \Pr\big(\text{score}(\text{sped up}) > \text{score}(\text{normal})\big).$$

Only the ranking matters, and 0.5 is chance. What to aim for: on the public leaderboard the organisers' solution and three teams reach 1.000; five entries sit at 0.643 to 0.644, which is presumably the Whisper-confidence baseline of the starter notebook.

First look at the data

• Every clip lasts exactly 7.000 s, with no padding at the end, so the length gives nothing away: a sped-up clip was made from a longer stretch of reading.
• "The pitch barely shifts" tells us how the clips were sped up. Playing audio faster by resampling raises the pitch by the same factor (the "chipmunk" effect). Keeping the pitch requires time-stretching, usually with a phase vocoder: cut the sound into overlapping frames, compute their spectra, read the frames faster (interpolating between neighbouring spectra), re-synthesise each frequency with a phase that advances at the right rate, and add the frames back.
• Speakers differ a lot: a fast reader at normal speed talks faster than a slow reader sped up by 20 %. Any feature has to be compared with the same reader's normal clips.

A first look at the test set already shows something odd: compared with the training clips of the same readers, the test clips are on average quieter and have smoother spectra. That is a hint that a processing step left marks on part of them.

A first baseline

The starter notebook runs Whisper on every clip and uses its average token log-probability: sped-up speech is unusual for the recogniser, so it should be less confident. That idea works only a little (the cluster of leaderboard entries at 0.643 to 0.644 is presumably this baseline). A plausible reason: Whisper is trained to be robust to fast speakers and noisy recordings, so its confidence on a clip depends more on the text and the voice than on the speed. We did not run it ourselves (its checkpoint is a 1.6 GB pickle and our method does not need it).

The most natural hand-made baseline is the speaking rate: count energy peaks (roughly syllables) per second and compare with the reader's usual rate. On our self-made test this gives only AUC 0.66: the rate of one reader varies a lot from sentence to sentence, so a 10 to 20 % speed-up hides inside the natural variation.

The key idea

Do not detect "fast speech", detect "speech that went through a phase vocoder". Then learn that detector from examples you make yourself, by time-stretching the normal training clips.

A phase vocoder leaves fingerprints that are much more regular than the speaking rate:

1. Smoother spectra over time. The stretched frames are interpolations between neighbouring original frames, so the spectrum changes less from one 8 ms step to the next. We measure the spectral flux, the root-mean-square change of the log-spectrum between consecutive frames, $\Phi = \sqrt{\overline{(\log P_{t+1}(f) - \log P_t(f))^2}}$, taken as the median over the loud frames.
2. Lower loudness. Re-synthesised phases no longer line up exactly across overlapping frames, so neighbouring frames partly cancel each other out ("phasiness"), and the overall RMS drops.
3. A smeared energy envelope. Syllable onsets become less sharp, so the modulation spectrum of the energy envelope (its centroid between 1 and 16 Hz) moves down, even though the speech is faster. Fewer frames are cleanly periodic, so our pitch tracker finds fewer voiced frames.

To turn these clues into a detector we need labelled examples, and we can make them: for every training clip we create a time-stretched copy with our own phase vocoder (2048-sample frames, hop 512, the usual library defaults) at a random rate between 1.05 and 1.5. Every feature is then turned into a z-score within the reader, $z = (x - \mu_{\text{reader}})/\sigma_{\text{reader}}$, using that reader's normal training clips, and a logistic regression learns "original versus stretched" from 9 features: modulation centroid, energy-peak rate, mean length of loud stretches, median pitch, number of voiced frames, spectral centroid, spectral flux, RMS and crest factor.

On our self-made test the single features already tell the story:

feature (z-score within the reader)	AUC alone
spectral flux (lower when stretched)	0.968
RMS loudness (lower)	0.948
modulation centroid of the envelope (lower)	0.895
number of voiced frames (lower)	0.834
spectral centroid (higher)	0.707
energy-peak rate, i.e. speaking rate (higher)	0.660
median pitch (higher)	0.640

The artefacts of the processing beat the speed itself by a wide margin, and the combination is almost perfect.

The full solution

1. For every clip compute the 9 features (band-pass 300 Hz to 4 kHz energy envelope at 100 Hz for the rate features, autocorrelation pitch on 40 ms frames, STFT with 512-sample frames for centroid and flux).
2. Normalise each feature with the mean and standard deviation of the same reader's normal training clips.
3. Positive examples: every training clip time-stretched at a random rate in $[1.05, 1.5]$; negative examples: the original clips.
4. Logistic regression ($C = 1$) on the 9 z-scores; the test score is its log-odds.

P = table(tr.audio_path, tr.speaker_id, rates)       # the same clips, time-stretched by us
X = np.vstack([speaker_z(A, A, cols), speaker_z(P, A, cols)])
clf = LogisticRegression(C=1).fit(X, np.r_[np.zeros(len(A)), np.ones(len(A))])
score = clf.decision_function(speaker_z(B, A, cols))

Everything runs on the CPU in about 2 minutes, without any pretrained model.

Results

approach (self-made test: 265 held-out clips, 120 stretched by us)	AUC
Whisper confidence baseline (presumed, public leaderboard on real test clips)	about 0.64
speaking rate alone (energy peaks per second, within reader)	0.660
spectral flux alone	0.968
gradient boosting on the 9 within-reader features	0.998
logistic regression on the 9 within-reader features (final)	0.999
same, only clips stretched by 5 to 15 %	0.997

On our self-made test the detector reaches AUC 0.999, and it stays at 0.997 even for speed-ups of only 5 to 15 %.

Does it transfer to the organisers' speed-up? The real test labels are hidden, but the test clips answer the question themselves:

In the left panel, the test clips (black circles) do not form one cloud around the normal clips: about half of them sit on the normal cloud and half on the cloud of clips we stretched ourselves. The scores of the final detector on the test clips (right panel) are strongly bimodal: 49 % of the clips are confidently called sped up, 51 % confidently called normal, with very few in between. A detector that missed the organisers' processing would give one blurred bump. This strongly suggests that the organisers also used a phase vocoder and that our ranking of the real test set is close to the 1.000 reached on the leaderboard, but we cannot verify that number without the labels.

What to remember

• Ask how the data was made. "Faster but same pitch" means time-stretching, and time-stretching leaves its own artefacts. Detecting the tool is often easier than detecting its intended effect.
• When you have no positives, manufacture them. Applying the suspected transformation to clean training data gives a labelled set for free; the method then becomes ordinary supervised learning.
• Normalise per individual. Readers differ more than the effect you look for; z-scores within each reader remove that nuisance.
• Check transfer without labels. A clean two-group structure of the scores on the real test set is good evidence that your synthetic positives resemble the real ones.
• A big pretrained model is not always the answer. Whisper is built to ignore speed and recording quirks, which is exactly the information this task needs.