The problem in brief
You receive a working notebook that tags Dutch newspaper sentences (CoNLL-2002) with named entities: persons, locations, organisations and "miscellaneous" names, in the BIO scheme (B-PER opens a person name, I-PER continues it, O is everything else). The starter model is a one-layer BiLSTM over randomly initialised word embeddings, trained on 10 000 sampled training sentences. You must (1) stop padding tokens from polluting training and evaluation, (2) replace the token-level metric by a proper entity-level one, (3) initialise the embeddings with pretrained vectors, and (4) improve the architecture, explaining each choice.
How you are scored
This is a camp test graded by the coaches on the notebook: each task must be implemented, run, and explained. There is no hidden test server, and no score threshold is published. The natural yardstick is the entity-level micro F1 on the official test split ned.testb (labels public), the metric of the CoNLL shared task:
An entity counts only if both its span and its type are exactly right. For reference, the classic BiLSTM-CRF with character features (Lample et al., 2016) reports 81.7 F1 on this test set, trained on the full 15 806 training sentences; we train on the 10 000-sentence sample imposed by the notebook.
First look at the data
Three facts shape the whole solution.
- The classes are wildly imbalanced. 90.3 % of the training tokens are
O. A tagger that answersOeverywhere already has 90 % token accuracy while finding zero entities. - Entities are often words never seen in training. In the test set, 44 % of entity tokens are outside the training vocabulary, against 12 % of tokens overall. A word-embedding model maps all of them to the same
<unk>vector. - But entities look like entities. 95 % of entity tokens start with a capital letter, against 6 % of the other tokens. Names also have telling suffixes (
-sen,-burg,-dam).
The conclusion writes itself: the model must be able to read the spelling of a word, not just its identity. That is Task 4.
A first baseline
The starter code (rewritten in plain PyTorch since torchtext 0.6 no longer installs) reports a dev "F-score" of 0.906 after 5 epochs. That number is an illusion, for two reasons that are exactly Tasks 1 and 2.
Task 1: padding. Sentences in a batch are padded to the same length. The starter masks <pad> in the loss (ignore_index), but its evaluation flattens the whole padded batch and feeds every position to precision_recall_fscore_support. Padding positions, trivially predicted, inflate the score. The fix is a boolean mask built from the sentence lengths, applied everywhere:
loss = cross_entropy(emissions[mask], gold[mask]) # training
pred = model.predict(...)[j, :length_j] # evaluation: cut each sentence back
Task 2: the metric. Even without padding, average="micro" over all tags is just token accuracy, dominated by the 90 % of O. And token-level scores give partial credit: if the gold entity is B-ORG I-ORG I-ORG ("Raad van State") and the model predicts B-ORG I-ORG O, two tokens out of three are right, yet the organisation was not found. NER is evaluated on whole entities, so we use seqeval, which rebuilds the spans from the BIO tags and applies the formula above.
The same baseline model, measured three ways on the test set:
metric on ned.testb | value |
|---|---|
token accuracy, O included (what the starter effectively measures) | 0.951 |
| token micro F1 on the 8 entity tags, padding removed | 0.581 |
| entity-level F1 (seqeval) | 0.510 |
From 0.95 to 0.51: the honest metric reveals that the baseline misses about half of the entities.
The key idea
Two weaknesses remain once we measure correctly, and each has a classic fix.
1. The model cannot read words (fixed by pretrained vectors and a character CNN). With random embeddings learnt from 10 000 sentences, rare words get poor vectors and unseen words are all <unk>. Task 3 initialises the embedding matrix with Dutch fastText vectors (cc.nl.300, trained on Common Crawl): 95 % of our vocabulary gets a vector that already places "Amsterdam" next to "Rotterdam". Because these vectors are unsupervised, we may also add to the vocabulary the dev and test words that have a fastText vector (no labels are involved). For Task 4 we add a character-level CNN: each word is spelled into character embeddings (dimension 30), a 1D convolution of width 3 with 50 filters slides over them, and a max over positions gives a 50-dimensional "spelling vector" concatenated to the word vector. A filter can learn "capital letter at the start", "ends in -sen", "contains a digit", so even an unseen surname gets a meaningful representation.
2. The model decides each tag independently (fixed by a CRF). A softmax per token can output O I-PER, an I- tag that continues nothing, or B-LOC I-ORG, a span with two types. Each such inconsistency destroys an entity under the exact-match metric. A linear-chain CRF scores the whole tag sequence:
where are the BiLSTM emission scores and is a learnt transition matrix (plus start and end scores). The model learns, for example, a very negative . Training minimises ; the sum over the sequences in the denominator is computed in by the forward algorithm (a running log-sum-exp), and prediction uses Viterbi, the same recursion with a max. This is dynamic programming, the same trick as in HMMs.
The full solution
solution.py runs the three models with the same data protocol (same 10 000 training sentences, random.seed(42)), selecting the epoch with the best dev entity F1 and reporting the test score once.
- Data and masks. Sentences are encoded to word ids, character ids (words truncated to 20 characters) and tag ids; every batch carries a boolean mask of real tokens. Vocabulary: the 20 000 most frequent training words, as in the starter.
- Embeddings (Task 3). Rows of the embedding matrix are copied from fastText when the word (or its lower-cased form) has a vector, random otherwise; they stay trainable. To save disk we streamed the 1.2 GB
cc.nl.300.vec.gzonce and kept only the 36 213 words that occur in the corpus. - Encoder (Task 4).
[word vector ; char-CNN vector]with dropout 0.5, a BiLSTM (256 units per direction), dropout 0.5, a linear layer to 9 emission scores. - CRF (Task 4). The CRF fits in about 30 lines; the heart of the forward algorithm is
alpha = start + E[:, 0] # log-scores of all prefixes, per last tag
for t in range(1, L):
nxt = torch.logsumexp(alpha[:, :, None] + T[None] + E[:, t, None, :], dim=1)
alpha = torch.where(mask[:, t, None], nxt, alpha) # padded positions leave alpha unchanged
log_Z = torch.logsumexp(alpha + end, dim=1)
- Training. Adam (learning rate ), batches of 32, gradient clipping at 5, at most 8 epochs with early stopping on dev entity F1 (patience 3).
Results
All numbers are on the official test set ned.testb (5 195 sentences, 3 941 entities), model chosen on ned.testa.
| approach | dev entity F1 | test entity F1 | test token acc. (with O) |
|---|---|---|---|
starter as given, its own metric (padding + O counted) | "0.906" | not measured | |
| Tasks 1-2: same BiLSTM, random embeddings, honest metric | 0.473 | 0.510 | 0.951 |
| Task 3: + fastText initialisation | 0.640 | 0.685 | 0.969 |
| Task 4: + char-CNN + CRF (+ input dropout) | 0.783 | 0.802 | 0.981 |
Per type for the final model: PER 0.883, LOC 0.855, ORG 0.744, MISC 0.730. MISC is the hardest class (nationalities, events, titles: a fuzzy category), and ORG suffers from long multi-word names where one wrong boundary loses the whole entity.
Observations worth keeping:
- Token accuracy barely moves (0.951 to 0.981) while entity F1 jumps from 0.51 to 0.80: the wrong metric would have hidden almost all of the progress.
- Pretrained vectors bring +17.5 points, character features and the CRF another +11.7. We changed the char-CNN and the CRF together, so their separate contributions are not measured here; Lample et al. find both matter.
- The models overfit quickly (the dev curve peaks at epoch 3 to 5, then drifts down while the training loss keeps falling), which is why we keep the best dev epoch rather than the last one.
- Our 0.802 is close to the published 0.817 for this architecture while using 63 % of the training sentences and 8 epochs on a CPU.
What to remember
- Mask padding everywhere, in the loss and in the metric; a mask built from sequence lengths is the cleanest way.
- Evaluate the unit the task cares about. NER is about entities, so use span-level F1 (
seqeval); token accuracy on a 90 %-Ocorpus is nearly meaningless. - Unseen words are the main enemy in NER. Pretrained embeddings give them semantics, a character CNN gives them shape (capitals, suffixes, digits).
- Structured outputs deserve structured decoders. A CRF learns which tag transitions are legal and decodes the best whole sequence with Viterbi, an dynamic program.
- Select the model on the dev set with the final metric, and look at per-class scores: they tell you where the remaining errors live (here MISC and ORG).