IOAI Training: Learn AI by solving problems

The problem in brief

You get five podcast recordings (17 to 31 minutes each) and, for each, the number of people who speak in it (2, 3 or 5). For every recording you must say, second by second, who is speaking: one line per speaker listing that speaker's turns as mm:ss-mm:ss intervals. You do not know who the speakers are, only how many; the judge will match your speaker lines to the real people in the most favourable way. The training data are six recordings of a single person speaking each (about 22 minutes in total), with no labels inside them. This task is known as speaker diarisation.

How you are scored

For every second and every speaker line, claiming that this person speaks earns $+1$ if they do and $-1$ if they do not; seconds you leave unclaimed earn nothing. With $S$ the total over all files (after the best matching of your lines to the true speakers),

\text{Score} = \max\Big(0, \min\Big(100,\ 100\sqrt{\frac{S}{4000}}\Big)\Big).

The five test files last 8028 seconds in total, so a perfect diarisation (one speaker per second) would give $S \approx 8000$ : the cap of 100 is reached at about half of that, for example by being right on 75 % of the seconds and wrong on 25 % ( $0.75 - 0.25 = 0.5$ net point per second). There are no labels for the test files at all, so we must build our own validation.

First look at the data

The test files are long and mostly continuous speech (their loudness rarely drops to silence), so almost every second belongs to someone.
Turns are long: in the output of our final model, a typical turn lasts from 10 seconds to several minutes. This means that neighbouring seconds almost always have the same speaker, a strong prior we should use.
The training recordings come from different sessions, so each one has its own "sound" (microphone, room): a classifier can tell them apart using the recording channel as much as the voice. That will matter when we interpret our validation.

Validation design. We cut the six training recordings into turns of 3 to 25 seconds and glue turns of $K$ different people together, giving fake 10-minute podcasts with known truth (15 of them, with $K = 2, 2, 3, 3, 5$ ). We score them with the official rule, matching predicted to true speakers with the Hungarian algorithm (scipy.optimize.linear_sum_assignment), and report the net points per second (1 is perfect, 0.5 reaches the cap). Because channel differences make the fake podcasts too easy, we also build a harder version where each recording's average cepstrum (its channel signature) is subtracted first.

A first baseline

Claim that the first speaker speaks all the time, and leave the others empty. You get $+1$ for that person's seconds and $-1$ for everybody else's, so you only gain when one person talks more than half of the time. On the fake podcasts this gives $-0.14$ net point per second (and 0 points on Kattis). Saying everybody speaks all the time is even worse. We need to separate the voices.

The key idea

Represent each second by a fingerprint of the voice, then cluster. The classic fingerprint is the MFCC: cut the audio into 25 ms frames every 10 ms, take the power spectrum, pool it into 40 bands spaced on the mel scale (which mimics the ear's resolution), take the logarithm and a discrete cosine transform, and keep the first 20 coefficients. They describe the shape of the spectral envelope, that is the shape of the speaker's vocal tract, largely independently of the words being said. We summarise each second by the mean and the standard deviation of its 100 MFCC frames: 40 numbers per second.

Since we know the number of speakers $K$ , we cluster the seconds of each file into $K$ groups with k-means, after standardising and keeping 20 PCA directions. Each cluster is one speaker.

The second idea is temporal smoothing. A single second of speech is a noisy fingerprint (a laugh, a breath, a whispered word), so per-second clustering produces short false switches. Because turns last many seconds, we replace each label by the majority label of a 5-second window centred on it. In probability terms, this is a cheap version of a hidden Markov model in which the speaker rarely changes from one second to the next.

Four timelines coloured by speaker. A 5-minute synthetic podcast with three speakers: the truth, the per-second k-means labels with many one-second glitches, and the smoothed labels that follow the truth closely. Bottom: the first 10 minutes of real test file 1 labelled by the final model, showing long turns of three speakers.

The full solution

Decode each mp3 to 16 kHz mono with ffmpeg.
Compute MFCCs with NumPy (pre-emphasis, Hamming window, 512-point FFT, 40 mel bands, DCT), then the per-second mean and standard deviation.
For each test file: standardise, PCA to 20 dimensions, k-means with the given $K$ (10 restarts).
Smooth the labels by a majority vote over 5 seconds.
Turn consecutive seconds of each label into intervals and print one bracketed list per speaker, with no spaces.

Z = PCA(20).fit_transform(StandardScaler().fit_transform(F))   # F: one row per second
lab = KMeans(K, n_init=10).fit_predict(Z)
oh = np.eye(K)[lab]                                            # one-hot labels
lab = uniform_filter1d(oh, 5, axis=0, mode="nearest").argmax(1) # 5-second majority vote

Results

Net points per second on the 15 synthetic podcasts (1 = perfect, 0.5 = enough for the cap):

approach	raw recordings	channel removed
first speaker speaks all the time	$-0.14$	$-0.14$
k-means per second, no smoothing	0.89	0.15
k-means + 5 s majority vote (final)	0.94	0.24
k-means + 9 s majority vote	0.91	0.24
agglomerative (Ward) + 5 s vote	0.94	0.25
Gaussian mixture + 5 s vote	0.84
failed idea: abstain when the cluster posterior is below 0.6	0.65	0.14

Rerunning the final solution.py on a fresh set of synthetic podcasts gives 0.885 and 0.328 net point per second. Scaled to the 8028 test seconds, that is $S \approx 7100$ (score 100, capped) in the easy setting and $S \approx 2600$ (score about 81) in the hard one. The truth for real podcasts probably lies in between: speakers recorded on separate microphones are easy to separate, speakers sharing one room are not, and real conversations also contain overlaps and laughter that our fake podcasts lack. These are estimates from synthetic data, not test scores.

Two lessons from the table. Smoothing helps a lot when features are weak; a window that is too long starts to erase short turns. And abstaining on "uncertain" seconds hurt, although leaving a second unclaimed costs 0 instead of $-1$ : k-means posteriors are not calibrated probabilities, and the costly mistakes are confident ones (two voices merged into one cluster), not uncertain ones.

What to remember

No labels for the test? Build a labelled proxy from what you have (here, fake podcasts glued from single-speaker recordings), and score it with the official rule.
Check what your proxy makes too easy. Different recording channels made the fake podcasts much easier than real ones; removing the channel signature gave a more honest lower estimate.
MFCC statistics + clustering is the classic diarisation baseline: the mel spectrum captures the voice, clustering finds the speakers when you know how many there are.
Use temporal structure. Labels of neighbouring seconds are strongly correlated; a majority vote (or an HMM) removes most isolated errors.
Abstention only pays with calibrated confidence. A score that rewards silence needs probabilities you can trust.

"""SAIO "Spudcast": who speaks when? (speaker diarisation, one decision per second)

Pipeline:
  1. decode each mp3 to 16 kHz mono with ffmpeg;
  2. MFCC features (25 ms frames, 40 mel bands, 20 coefficients) written with NumPy,
     summarised per second by their mean and standard deviation (40 numbers);
  3. standardise, PCA to 20 dimensions, k-means with K = the given number of speakers;
  4. temporal smoothing: majority vote over a 5-second sliding window;
  5. merge consecutive seconds into "mm:ss-mm:ss" segments, one line per speaker.
Validation: the test files have no labels, so we build fake podcasts from the six
single-speaker training files (turns of 3-25 s) and compute the official +1/-1 score
with the best speaker matching (Hungarian algorithm).

Run: python solution.py data/            -> prints the answer (one line per speaker)
     python solution.py data/ --validate -> scores on synthetic podcasts
"""
import os, sys, subprocess, tempfile
import numpy as np
from scipy.io import wavfile
from scipy.fft import dct
SR=16000
def mel_fb(n_mels=40, n_fft=512, sr=SR):
    mel=lambda f: 2595*np.log10(1+f/700); inv=lambda m: 700*(10**(m/2595)-1)
    pts=inv(np.linspace(mel(50),mel(sr/2-200),n_mels+2)); bins=np.floor((n_fft+1)*pts/sr).astype(int)
    fb=np.zeros((n_mels,n_fft//2+1))
    for i in range(n_mels):
        a,b,c=bins[i],bins[i+1],bins[i+2]
        fb[i,a:b]=(np.arange(a,b)-a)/max(b-a,1); fb[i,b:c]=(c-np.arange(b,c))/max(c-b,1)
    return fb
FB=mel_fb()
def mfcc(x, n=20):
    """25 ms frames every 10 ms -> log-mel (40) -> DCT -> first n coefficients + log energy."""
    x=x.astype(np.float32); x=np.append(x[0],x[1:]-0.97*x[:-1])
    hop,win=160,400; nf=1+(len(x)-win)//hop
    idx=np.arange(win)[None]+hop*np.arange(nf)[:,None]
    fr=x[idx]*np.hamming(win)
    P=np.abs(np.fft.rfft(fr,512))**2/512
    lm=np.log(P@FB.T+1e-6)
    c=dct(lm,type=2,axis=1,norm='ortho')[:,1:n+1]
    e=np.log(P.sum(1)+1e-6)
    return c,e
def seconds(path):
    """One feature vector per second: mean and std of MFCCs, plus loudness."""
    sr,x=wavfile.read(path); c,e=mfcc(x); T=len(c)//100
    c=c[:T*100].reshape(T,100,-1); e=e[:T*100].reshape(T,100)
    F=np.concatenate([c.mean(1),c.std(1)],1)
    loud=np.percentile(e,90,axis=1)
    return F,loud


import numpy as np, itertools
from scipy.optimize import linear_sum_assignment
from scipy.ndimage import median_filter
from sklearn.cluster import KMeans, AgglomerativeClustering
from sklearn.mixture import GaussianMixture
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
def synth(FT, K, T, rng):
    """Fake podcast: K train speakers taking turns of 3-25 s, T seconds in total."""
    spk=rng.choice(np.arange(1,7),K,replace=False); pos={s:0 for s in spk}
    F=[];y=[]; cur=None
    while len(y)<T:
        s=rng.choice([x for x in spk if x!=cur]) if K>1 else spk[0]; cur=s
        L=int(rng.integers(3,26)); a=pos[s]%len(FT[s]); seg=np.take(FT[s],np.arange(a,a+L),axis=0,mode='wrap'); pos[s]+=L
        F.append(seg); y+= [list(spk).index(s)]*L
    return np.vstack(F)[:T], np.array(y[:T])
def S(pred, y, K):
    """Organisers' metric with the best matching of predicted to true speakers. pred=-1 means 'nobody'."""
    M=np.zeros((K,K))
    for p in range(K):
        for t in range(K):
            claim=pred==p; M[p,t]=np.sum(claim&(y==t))-np.sum(claim&(y!=t))
    r,c=linear_sum_assignment(-M); return M[r,c].sum()
def cluster(F, K, method='kmeans', smooth=5, pca=20):
    Z=StandardScaler().fit_transform(F)
    if pca: Z=PCA(pca,random_state=0).fit_transform(Z)
    if method=='kmeans': lab=KMeans(K,n_init=10,random_state=0).fit_predict(Z)
    elif method=='gmm': lab=GaussianMixture(K,covariance_type='diag',n_init=3,random_state=0).fit_predict(Z)
    elif method=='agglo': lab=AgglomerativeClustering(K,linkage='ward').fit_predict(Z)
    elif method=='one': lab=np.zeros(len(F),int)
    if smooth>1 and K>1:   # majority vote in a sliding window = temporal smoothing
        oh=np.eye(K)[lab]; from scipy.ndimage import uniform_filter1d
        lab=uniform_filter1d(oh,smooth,axis=0,mode='nearest').argmax(1)
    return lab

def load(mp3):
    with tempfile.TemporaryDirectory() as tmp:
        wav = os.path.join(tmp, "a.wav")
        subprocess.run(["ffmpeg", "-v", "error", "-y", "-i", mp3, "-ac", "1", "-ar", str(SR), wav], check=True)
        return seconds(wav)[0]


def mmss(s):
    return f"{s // 60:02d}:{s % 60:02d}"


def segments(lab, k):
    """Seconds labelled k -> 'start-end' strings (end = first second after the turn)."""
    out, s = [], None
    for t, v in enumerate(list(lab) + [-2]):
        if v == k and s is None: s = t
        if v != k and s is not None: out.append(f"{mmss(s)}-{mmss(t)}"); s = None
    return "[" + ",".join(out) + "]"


if __name__ == "__main__":
    root = sys.argv[1]
    if "--validate" in sys.argv:
        FT = {k: load(f"{root}/train/{k}.mp3") for k in range(1, 7)}
        for hard in (False, True):
            if hard:   # remove each recording's average cepstrum: no channel cue left
                FT = {k: F - np.r_[F[:, :20].mean(0), np.zeros(20)] for k, F in FT.items()}
            rng = np.random.default_rng(0); res = []
            for K in [2, 2, 3, 3, 5] * 3:
                F, y = synth(FT, K, 600, rng)
                res.append(S(cluster(F, K), y, K) / len(y))
            print(("channel removed" if hard else "raw recordings") + f": net points per second {np.mean(res):.3f}")
    else:
        Ks = [int(l) for l in open(f"{root}/testspeakers.txt").read().split()[-5:]]
        for i, K in enumerate(Ks, 1):
            lab = cluster(load(f"{root}/test/{i}.mp3"), K)
            print("\n".join(segments(lab, k) for k in range(K)))