IOAI Training: Learn AI by solving problems

The Last Frequency

This page gives the full approach. Read it once you have searched.

Metric: accuracy, points = 166.67 (acc - 0.40)
Evaluated on: held-out validation (official test labels are private)
Our score: 95.1 % accuracy (91.8 pts), 15 % hold-out
Expected level: public LB best 0.963; baseline notebook 40-50 %

Our solution does not reach the expected level: there is still room to beat it

We solved the problem ourselves: downloaded the data, ran the experiments and measured the score below. The statement and data belong to their organisers.

The problem in brief

A voice-controlled bunker must understand 35 spoken commands again: the ten digits, directions such as up, down, left, go, stop, actions such as forward, follow, learn, and a few plain words (bed, bird, cat, marvin, sheila, tree, wow...). Each example is one second of raw audio sampled at 16 kHz, so a vector of 16 000 numbers, taken from Google's public Speech Commands v2 dataset. You get 8 479 labelled clips (125 to 325 per word) and must name the word in 2 120 test clips.

The test labels are private on Kaggle and we never submitted anything. All scores below are measured on a stratified held-out 15 % of the training clips (1 272 clips), which no model saw during training.

How you are scored

Plain accuracy, the fraction of clips whose word is right, converted to points:

\text{points} = \max\big(0, \min(100,\ 166.67\,(\text{accuracy} - 0.40))\big),

so 40 % accuracy gives 0 points and 100 % gives 100; 90 % is worth 83 points, 95 % is worth 92. What to aim for: the organisers' baseline notebook announces 40-50 % accuracy, and the best public leaderboard score is 0.963.

First look at the data

Waveforms and log-mel spectrograms of four words

Every clip is scaled so that its loudest sample is at $\pm 1$ , and the word occupies only 0.2 to 0.5 s somewhere inside the second, not always at the same place: "up" above starts at 0.75 s, "three" at 0.4 s. A good model must not care where the word is.
The raw waveform is hard to read; the log-mel spectrogram (energy in 64 frequency bands, every 10 ms, on a log scale) shows the structure at once: the hiss of the "th" or "f" as a burst at high frequencies, the vowels as stacked horizontal lines (harmonics), the "r" of "tree" as a band sliding upwards.
Some pairs differ by a single sound: three / tree, off / on / up, go / two / no. The statement itself warns about them.

A first baseline

The organisers' baseline turns each clip into a 64 × 64 mel spectrogram and trains a three-layer CNN, announcing 40-50 % accuracy. The two big weaknesses are a tiny network and no augmentation: with about 240 examples per word, the model memorises speakers instead of words.

A stronger from-scratch baseline fixes both:

front end: 25 ms windows every 10 ms, 64 mel bands, $\log(\text{energy} + 10^{-6})$ , giving a 64 × 101 image per clip;
network: four blocks of two 3 × 3 convolutions with batch normalisation (32, 64, 128, 256 channels), max-pooling in between, then a global average over time and frequency, so the decision does not depend on where the word sits;
augmentation, chosen to imitate what changes between two people saying the same word: a random shift of ±100 ms, a random volume between 0.5 and 1.5, a little white noise, and SpecAugment (two random frequency bands and two random time spans hidden in the spectrogram);
AdamW with a one-cycle learning rate schedule, label smoothing 0.1, 30 epochs (4.5 minutes on the Apple GPU).

This CNN reaches 94.0 % on the held-out clips, already far above the announced baseline.

The key idea

Do not learn what speech sounds like from 8 000 clips: borrow a model that has already listened to 960 hours of speech, and fine-tune only its upper layers.

HuBERT-base is a 12-layer transformer that was trained on unlabelled audiobooks (LibriSpeech) with a self-supervised task: hide parts of the audio and predict discrete "pseudo-phoneme" units for the hidden frames. It never saw a label, let alone our 35 commands, but to solve that task it had to learn how speech is built: phonemes, their transitions, independence from the speaker's voice. Such a representation is exactly what keyword spotting needs.

Which layer holds the most useful information? We averaged each layer's output over time and trained a linear classifier on top:

Layer-wise probe and results

The left panel tells a classic story. The convolutional encoder output (layer 0) gives only 70 %; accuracy rises through the lower transformer layers as the model builds phoneme-like units, and plateaus around layers 7 to 11 at about 93 %, with a frozen network and a linear classifier. That plateau suggests the plan:

Freeze the bottom. Run the clips once through the convolutional encoder and the first $K$ transformer layers, and store the output (49 frames of 768 numbers per clip, in 16-bit floats). These layers compute generic acoustic features that do not need to change.
Fine-tune the top $12 - K$ layers on the cached frames, with a small learning rate ( $5 \times 10^{-5}$ ), and put an attention-pooling classifier on top: a learned score $s_t = w^\top h_t$ for each frame, weights $a_t = \operatorname{softmax}(s)_t$ , and the clip vector $\sum_t a_t h_t$ . The model learns to look at the frames where the word is and ignore the silence around it, which plain averaging cannot do.
Augment the frames: a random shift of up to 5 frames (100 ms) and two random time masks.

Caching makes this cheap: each epoch only runs the top layers on 49 tokens per clip, 40 seconds for 8 layers. With $K = 8$ (top 4 layers trained) we get 94.6 %, with $K = 4$ (top 8 layers trained) 94.9 %.

Finally, the CNN and HuBERT see the problem very differently (a spectrogram image versus a learned phonetic representation), so their mistakes are partly independent: averaging their probabilities gives 95.1 %.

The full solution

Cache the output of HuBERT layer 4 for all clips (90 s on the GPU, batches of 8 to keep memory under 2 GB).
Model A: the log-mel CNN above, 30 epochs.
Model B: HuBERT layers 5-12 (pretrained weights) + attention pooling + linear layer, AdamW with learning rates $5 \times 10^{-5}$ (transformer) and $10^{-3}$ (new head), one-cycle schedule, 10 epochs, batch 16, label smoothing 0.1.
Average the two probability vectors and take the most probable word.

h = run_layers(frames)                       # HuBERT layers 5..12, fine-tuned
a = F.softmax(att(h).squeeze(-1), dim=1)     # one weight per 20 ms frame
logits = out(dropout((a[..., None] * h).sum(1)))

The HuBERT weights are only published as a PyTorch pickle; we converted them to safetensors once, inside a sandbox without network access, and never unpickled third-party files elsewhere.

Results

approach	held-out accuracy	points
organisers' baseline CNN (as announced in the notebook)	0.40 to 0.50	0 to 17
frozen HuBERT, time-averaged, best single layer + logistic regression	0.929	88.2
frozen HuBERT, learned mix of all 13 layers + MLP	0.927	87.8
CNN from scratch on log-mel, with augmentation	0.940	89.9
HuBERT, layers 9-12 fine-tuned, attention pooling	0.946	91.0
HuBERT, layers 5-12 fine-tuned, attention pooling	0.949	91.5
CNN + HuBERT 5-12, probabilities averaged (final)	0.951	91.8

Our final model gets 95.1 % on the 1 272 held-out clips (91.8 points), against 96.3 % for the best public leaderboard entry, measured on other clips. With 1 272 clips the standard error of our estimate is about 0.6 %, so the gap is small but probably real; the natural next steps would be fine-tuning all of HuBERT on raw audio with waveform augmentation, or averaging several runs, which our memory and time budget did not allow.

The remaining 63 errors are mostly the expected near-homophones: wow heard as one (3), off as on, up as off, one as on, tree as three (2 each). The hardest words are wow, forward and off, all around 85 % correct.

What to remember

For speech, the log-mel spectrogram turns audio into an image, and an image CNN with a global average over time handles words that appear anywhere in the clip.
Augment with what really varies: timing, loudness, noise, and SpecAugment masks. With about 240 examples per word, every source of variation you can simulate is extra data.
Self-supervised pretrained models transfer extremely well. A frozen HuBERT with a linear classifier already matches a carefully trained CNN, without seeing any labelled command during pretraining.
Probe the layers before fine-tuning. The middle-to-upper layers hold the word identity; freezing and caching the lower ones makes fine-tuning fast and light.
Ensembles of different views help: a spectrogram CNN and a transformer on learned features make different mistakes.

Full code (solution.py)

"""The Last Frequency (Cyprus AI Camp 2026): 35-word speech commands from 1 s of raw 16 kHz audio.

Data: train_waveforms.npy, train_labels.npy, public/private_test_waveforms.npy, label_map.json from
the competition; facebook/hubert-base-ls960 config + weights in ./hubert (the .bin was converted to
model.safetensors inside run_sandboxed.sh). Test labels are private: `python solution.py holdout`
scores a stratified 85/15 split, `python solution.py final` trains on all 8 479 clips and writes
submission.csv (public then private test, Id 0..2119).

Two models, averaged:
  A. a small CNN trained from scratch on log-mel spectrograms, with waveform and SpecAugment augmentation;
  B. self-supervised HuBERT-base: layers 1-K frozen (their outputs are cached once), layers K+1..12
     fine-tuned with an attention-pooling classifier, time shift + time masking on the frames.
"""
import os, sys, math, json
os.environ.setdefault("OMP_NUM_THREADS", "2")
import numpy as np, torch, torch.nn as nn, torch.nn.functional as F, pandas as pd
from transformers import HubertModel, HubertConfig
from safetensors.torch import load_file
from sklearn.model_selection import train_test_split

torch.set_num_threads(2); torch.manual_seed(0); np.random.seed(0)
DEV = "mps" if torch.backends.mps.is_available() else "cpu"
if DEV == "mps": torch.mps.set_per_process_memory_fraction(0.1)
K = 4                     # number of frozen HuBERT layers (cached); layers 5-12 are fine-tuned
W = np.load("train_waveforms.npy", mmap_mode="r"); y = np.load("train_labels.npy")
TESTS = [np.load(f"{s}_test_waveforms.npy", mmap_mode="r") for s in ["public", "private"]]
yt = torch.tensor(y)
def empty():
    if DEV == "mps": torch.mps.empty_cache()

# ---------------- A. log-mel CNN from scratch ----------------
def mel_fb(n_mels=64, n_fft=400, sr=16000):
    f = np.linspace(0, sr / 2, n_fft // 2 + 1)
    mel, imel = (lambda h: 2595 * np.log10(1 + h / 700)), (lambda m: 700 * (10 ** (m / 2595) - 1))
    pts = imel(np.linspace(mel(20), mel(sr / 2), n_mels + 2))
    fb = [np.clip(np.minimum((f - pts[i]) / (pts[i + 1] - pts[i]), (pts[i + 2] - f) / (pts[i + 2] - pts[i + 1])), 0, None)
          for i in range(n_mels)]
    return torch.tensor(np.array(fb), dtype=torch.float32)
FB, WIN = mel_fb().to(DEV), torch.hann_window(400).to(DEV)
def logmel(w):            # 25 ms windows, 10 ms hop, 64 mel bands -> (B, 64, 101)
    return torch.log(FB @ torch.stft(w, 400, 160, window=WIN, return_complex=True).abs() ** 2 + 1e-6)

def wave_aug(w):
    w = torch.roll(w, np.random.randint(-1600, 1601), 1)                       # +-100 ms shift
    g = torch.empty(len(w), 1, device=DEV).uniform_(0.5, 1.5)                  # loudness
    return w * g + 0.005 * torch.rand(len(w), 1, device=DEV) * torch.randn_like(w)   # light noise

def spec_aug(S):
    for _ in range(2):    # SpecAugment: hide random frequency bands and time spans
        f0 = np.random.randint(0, 56); S[:, f0:f0 + np.random.randint(0, 8)] = S.mean()
        t0 = np.random.randint(0, 89); S[:, :, t0:t0 + np.random.randint(0, 12)] = S.mean()
    return S

def block(i, o):
    return nn.Sequential(nn.Conv2d(i, o, 3, padding=1, bias=False), nn.BatchNorm2d(o), nn.ReLU(),
                         nn.Conv2d(o, o, 3, padding=1, bias=False), nn.BatchNorm2d(o), nn.ReLU())
class CNN(nn.Module):
    def __init__(s):
        super().__init__()
        s.norm = nn.BatchNorm2d(1)
        s.f = nn.Sequential(block(1, 32), nn.MaxPool2d(2), block(32, 64), nn.MaxPool2d(2),
                            block(64, 128), nn.MaxPool2d(2), block(128, 256))
        s.h = nn.Sequential(nn.Dropout(0.3), nn.Linear(256, 35))
    def forward(s, S):
        return s.h(s.f(s.norm(S[:, None])).mean((2, 3)))      # global average pool over time and frequency

def model_a(tr, tests, epochs=30, bs=64):
    m = CNN().to(DEV)
    opt = torch.optim.AdamW(m.parameters(), lr=3e-3, weight_decay=1e-2)
    sched = torch.optim.lr_scheduler.OneCycleLR(opt, 3e-3, total_steps=epochs * math.ceil(len(tr) / bs), pct_start=0.15)
    for ep in range(epochs):
        m.train(); perm = np.random.permutation(tr)
        for i in range(0, len(perm), bs):
            b = np.sort(perm[i:i + bs])
            S = spec_aug(logmel(wave_aug(torch.tensor(np.array(W[b])).to(DEV))))
            loss = F.cross_entropy(m(S), yt[b].to(DEV), label_smoothing=0.1)
            opt.zero_grad(); loss.backward(); opt.step(); sched.step()
    m.eval(); out = []
    with torch.no_grad():
        for X in tests:
            out.append(np.concatenate([F.softmax(m(logmel(torch.tensor(np.array(X[i:i + 128])).to(DEV))), 1).cpu().numpy()
                                       for i in range(0, len(X), 128)]))
    del m; empty()
    return out

# ---------------- B. HuBERT, top layers fine-tuned on cached frames ----------------
def hubert():
    m = HubertModel(HubertConfig.from_pretrained("hubert"))
    m.load_state_dict(load_file("hubert/model.safetensors"), strict=False)
    return m

def cache_layer(X, name):  # output of the K-th transformer layer, 49 frames x 768 per clip (fp16, on disk)
    m, out = hubert().eval().to(DEV), []
    with torch.no_grad():
        for i in range(0, len(X), 8):
            w = torch.tensor(np.array(X[i:i + 8])).to(DEV)
            w = (w - w.mean(1, keepdim=True)) / (w.std(1, keepdim=True) + 1e-7)
            out.append(m(w, output_hidden_states=True).hidden_states[K].half().cpu().numpy())
    del m; empty()
    np.save(f"cache_L{K}_{name}.npy", np.concatenate(out))
    return np.load(f"cache_L{K}_{name}.npy", mmap_mode="r")

class Top(nn.Module):
    def __init__(s):
        super().__init__()
        s.layers = nn.ModuleList(hubert().encoder.layers[K:])
        s.att, s.drop, s.out = nn.Linear(768, 1), nn.Dropout(0.2), nn.Linear(768, 35)
    def forward(s, h):
        for l in s.layers:
            o = l(h); h = o[0] if isinstance(o, tuple) else o
        a = F.softmax(s.att(h).squeeze(-1), 1)                # attention pooling: where is the word?
        return s.out(s.drop((a[..., None] * h).sum(1)))

def frame_aug(h):
    h = torch.roll(h, np.random.randint(-5, 6), 1)            # one frame = 20 ms
    for _ in range(2):
        t0 = np.random.randint(0, h.shape[1] - 5); h[:, t0:t0 + np.random.randint(0, 5)] = 0
    return h

def model_b(Htr_all, tr, Htests, epochs=10, bs=16):
    m = Top().to(DEV)
    head = list(m.att.parameters()) + list(m.out.parameters())
    opt = torch.optim.AdamW([{"params": m.layers.parameters(), "lr": 5e-5}, {"params": head, "lr": 1e-3}], weight_decay=1e-2)
    sched = torch.optim.lr_scheduler.OneCycleLR(opt, [5e-5, 1e-3], total_steps=epochs * math.ceil(len(tr) / bs), pct_start=0.1)
    for ep in range(epochs):
        m.train(); perm = np.random.permutation(tr)
        for i in range(0, len(perm), bs):
            b = np.sort(perm[i:i + bs])
            loss = F.cross_entropy(m(frame_aug(torch.tensor(np.array(Htr_all[b])).float().to(DEV))), yt[b].to(DEV), label_smoothing=0.1)
            opt.zero_grad(); loss.backward(); opt.step(); sched.step()
    empty(); m.eval(); out = []
    with torch.no_grad():
        for Hx in Htests:
            out.append(np.concatenate([F.softmax(m(torch.tensor(np.array(Hx[i:i + 32])).float().to(DEV)), 1).cpu().numpy()
                                       for i in range(0, len(Hx), 32)]))
    del m; empty()
    return out


if __name__ == "__main__":
    mode = sys.argv[1] if len(sys.argv) > 1 else "holdout"
    Htr = cache_layer(W, "train")
    if mode == "holdout":
        tr, ho = train_test_split(np.arange(len(y)), test_size=0.15, stratify=y, random_state=42)
        ho = np.sort(ho)
        (Pa,), (Pb,) = model_a(tr, [W[ho]]), model_b(Htr, tr, [np.array(Htr[ho])])
        for name, P in [("A CNN", Pa), ("B HuBERT", Pb), ("A+B", (Pa + Pb) / 2)]:
            print(name, "holdout accuracy", round((P.argmax(1) == y[ho]).mean(), 4))
    else:
        tr = np.arange(len(y))
        Pa = model_a(tr, TESTS)
        Pb = model_b(Htr, tr, [cache_layer(X, n) for X, n in zip(TESTS, ["public", "private"])])
        P = np.concatenate([(a + b) / 2 for a, b in zip(Pa, Pb)])
        names = json.load(open("label_map.json"))
        pd.DataFrame({"Id": np.arange(len(P)), "Command": [names[str(i)] for i in P.argmax(1)]}).to_csv("submission.csv", index=False)
        print("wrote submission.csv", len(P))