The problem in brief
Drone images of Philippine coastal waters during Holy Week. You must do two things. First, detect swimmers, boats and floating debris with bounding boxes. Second, flag two anomaly classes that never appear in the training data: a soggy plush bunny (class 8) and painted Easter eggs (class 9). There are 500 labelled training images (mostly frames from a handful of drone videos) and 100 unlabelled test images. The organisers explicitly expect generative data or zero-shot vision-language models for the anomalies.
The test labels are private. Our detection score is measured on held-out video sequences from the training set. The anomaly part cannot be scored there, because no training image contains an anomaly. We checked it on the test images against labels we made ourselves by looking at the images. Both numbers are therefore our own estimates, not official scores.
How you are scored
A blended metric, half and half:
is the mean over the standard classes 0-2 of the average precision. A prediction counts as correct when its IoU with an unmatched ground-truth box is at least 0.5:
The anomaly accuracy is the per-image categorical accuracy of the easter_egg_found column (none, 8 or 9). A detected anomaly must also be reported as a 1-pixel "ghost box" at in the box list. The organisers' report gives no per-task threshold, so there is no target to compare with.
First look at the data
- Frames come in sequences. Image ids are consecutive frames of a few drone flights, so neighbouring images are near duplicates. A random train/validation split would leak: the model would be validated on frames almost identical to training frames. We split by sequence, starting a new group whenever the id jumps by more than 3. The result is 385 training images and 115 held-out images from different flights.
- Very small objects. The images are 2-4K pixels wide. A swimmer box has a median area of about 1 500 px², roughly , while a boat is about 5 000 to 14 000 px². Shrinking a whole 4K image to the usual 640 px would turn swimmers into 6-pixel blobs.
- Rare and shifting debris. There are only 81 debris boxes, mostly large ones from one flight in the training part (median 22 500 px²). The held-out debris boxes are small (median 1 400 px²).
- Two undocumented classes. The labels also contain classes 3 (13 boxes) and 4 (74 boxes), which the statement never mentions. We train on them, but the score uses only classes 0-2.
- Some frames contain pasted black rectangles (occlusions), and a few test images contain the pasted anomalies: a pile of rainbow eggs, a giant plush toy in a life jacket.
A first baseline
Detection is a job for a pretrained detector. Fine-tuning YOLOv8s (COCO-pretrained) on images simply resized to 640 px fails on swimmers, because they become too small to see. The real baseline is therefore already the tiling idea below. For the anomalies, the naive baseline "always answer none" would already be right on 95 % of test images. That shows why the per-class choice matters more than overall accuracy.
The key idea
Two different problems, two different tools.
- Detection by tiles. Each training image is resized so its long side is 1920 px and cut into overlapping tiles (stride 512). A box is kept on a tile if at least 40 % of it lies inside, and most empty tiles are dropped. Swimmers stay about 20-40 px tall, so the network sees them at a usable scale. At inference we run the whole 1920-px image at once (
imgsz=1920). The convolutional detector works at any size, and the objects keep the scale they had in training. - Anomalies by zero-shot CLIP. You cannot train a detector for a class you have no examples of, but CLIP has learned from captions what "colourful painted Easter eggs floating on water" looks like. For each image we embed the full view and 9 overlapping half-size crops (the anomaly may be small), and compare them with text prompts for "normal" sea scenes (open sea, boat, swimmers, buoy, jet ski, sun glare, black rectangle), for eggs and for a plush toy. The class score is the softmax mass on that class's prompts, taking the most suspicious crop. The thresholds come from 300 training images, which are guaranteed anomaly-free: a test image is flagged only if its score exceeds everything seen on clean data.
The full solution
- Grouped split by video sequence, then 1 871 training tiles of 640 px built from the 1920-px resized images.
- Fine-tune YOLOv8s (ultralytics) for 4 epochs (a 20-minute budget on a shared Apple GPU) with mosaic augmentation, horizontal and vertical flips (a drone view has no "up") and % scale jitter.
- Inference on full images at 1920 px, confidence threshold 0.01 (low thresholds help AP), NMS IoU 0.6.
- CLIP ViT-B/16 zero-shot scoring of every image (full view plus 3×3 crops). Eggs are flagged when their score exceeds 0.9 (the training maximum is 0.50). The plush toy is flagged when its score exceeds 0.9995 (the training maximum is 0.998, see below).
- Submission: the detected boxes for classes 0-2, plus the ghost box
8 1.0 0 0 1 1or9 1.0 0 0 1 1for flagged images.
lab = '9' if score['9'] > 0.9 else '8' if score['8'] > 0.9995 else 'none'
if lab != 'none': pred = f'{lab} 1.0 0 0 1 1 ' + pred # ghost box protocol
Results
Detection (115 held-out images from unseen flights, VOC all-point AP at IoU 0.5):
| model and inference | swimmer | boat | debris | mAP50 (classes 0-2) |
|---|---|---|---|---|
| YOLOv8s on 640 tiles, 4 epochs, inference at 1280 px | 0.577 | 0.986 | 0.000 | 0.521 |
| same model, inference at 1920 px | 0.685 | 0.984 | 0.000 | 0.556 |
| second fine-tuning run (crashed in an MPS kernel, degraded checkpoint) | 0.193 | 0.020 | 0.000 | 0.071 |
Boats are essentially solved. Swimmers gain 11 points from full-resolution inference, the scale argument above in action. Debris gets AP 0: there are 23 small held-out debris boxes, while the training debris (58 boxes) is mostly large debris from one flight. The model never learned what small debris looks like. This is the clearest gap. A real contestant would fix it with copy-paste augmentation of debris crops at small scales, or more epochs. Our 20-minute compute cap stopped us there.
Anomalies (100 test images, checked against our own visual inspection: eggs in 4899, 5257, 5545 and 6047, the plush toy in 526):
| method | eggs found | plush found | false alarms | accuracy |
|---|---|---|---|---|
| always "none" | 0 / 4 | 0 / 1 | 0 | 95 % |
| CLIP, generic bunny prompts, threshold = training maximum | 4 / 4 | 1 / 1 (score 0.98996 against a training max of 0.98911) | 0 | 100 % |
| CLIP, bunny + plush-toy prompts, thresholds 0.9 / 0.9995 | 4 / 4 | 1 / 1 | 0 | 100 % |
The egg score is an easy call: the four egg images score above 0.98, and no clean training image goes above 0.50. The plush toy is fragile. It scores 0.99992, but a clean training frame reaches 0.998 (CLIP matches some swimmer clusters with life jackets to "plush toy in the water"), so the margin is tiny, and the test object is closer to a teddy bear than to a bunny. Our blended estimate is . Keep in mind that its two halves are measured on different images, and that the second half uses our own labels.
What to remember
- Split by group when samples are correlated. Consecutive video frames must stay on the same side of the split, or validation measures memorisation.
- Keep small objects at a sensible scale. Train on high-resolution tiles and run inference on the full high-resolution image. For swimmers this was worth 11 AP points.
- For a class with no training examples, use a model that already knows it. CLIP zero-shot scoring over several crops finds pasted objects without a single labelled example.
- Calibrate zero-shot thresholds on data that is known to be negative. The clean training set tells you how high a "normal" score can go. When the margin is tiny, as for the plush toy, say so.
- Look at the per-class AP, not just the mean. A zero for debris shows a gap in training coverage (small debris was never seen), which better augmentation can address.