Trash Trail asks volunteers to photograph litter before they clean it up. The model’s job is to say what is in the photo, because “what kind of trash is here” is what turns a pile of photos into something a city can act on — and eventually into a target list for a robot that has to pick the stuff up.
For months we measured that model the way everyone measures models: mAP on a held-out split. It looked mediocre but improving. Then we measured it a different way, and found we had been asking the wrong question.
Grading every box by hand
We took 119 real volunteer photos and had them labelled blind — reviewers saw the image and nothing else, no model output. Then we took all 295 boxes the detector produced on those photos, cropped each one, and had those graded blind too: is this actually litter, and what material is it?
Two numbers came out, and they point in opposite directions.
95% of the boxes were on real litter. The detector fires on leaves, twigs, feathers and signage only about 4% of the time. It is not hallucinating, and its sense of where is good.
36% of those boxes carried the right label. Nearly a third landed in a catch-all class — other.other, other.plastic, food.other.
The model was not blind. It was mute.
The composition was inverted
That mattered more than a low score, because the thing we want out of this model is a composition — what kinds of trash, in what proportion. Here is what the reviewers found versus what the detector reported:
| soft / film / paper | rigid containers | |
|---|---|---|
| ground truth | 76% | 23% |
| detector reported | 45% | 54% |
Reality is roughly three-to-one soft over rigid. The model said the opposite. Per class, it recovered bottles at 62% and paper at 7%, with foam at zero across 53 occurrences.
Anyone designing a gripper from that output would build for bottles and cans, when the actual workload is wrappers, film and paper. The number was not just wrong, it was wrong in the direction that would have cost the most to discover later.
Why: 43 classes, ~7,700 boxes
The old label space had 43 leaf classes — softdrinks.bottle, alcohol.bottle, food.jar, coffee.cup — trained on roughly 7,700 boxes. That is about 180 boxes per class before you account for a long tail, and rigid containers alone were split across eight of them while three separate catch-alls absorbed everything uncertain.
So we rebuilt the label space around material rather than product: rigid container, film, paper/card, small rigid, foam, glass, textile, cigarette butt. Eight classes, chosen because they map to how you’d handle the object — which is what both a gripper and a cleanup-time estimate actually need.
Before spending anything on training, we tested whether the collapse alone would help by re-taking the model’s own argmax within material groups, no retraining. It bought 10 points, from 36% to 46%, and the correct group sat in the model’s top three 78% of the time. That 78% is the ceiling for any relabelling-only fix — worth knowing before assuming new data is the answer.
Borrowing data, and what came back
We pulled two CC BY 4.0 datasets from Roboflow Universe, chosen by looking at the pixels rather than reading class lists. That distinction did real work: the single largest polystyrene source on Universe is aerial drone footage over open water, and two other tempting matches turned out to be harbour CCTV and a conveyor belt. All three list the classes we wanted. None of them would transfer to a phone photo of a roadside.
Merged into the material taxonomy: 3,658 images, 13,248 boxes, with 9,200 of those in the three materials the model could not name. Trained RF-DETR Medium on Roboflow. 73 minutes.
It reported mAP@50 of 73.8%, with six of eight classes between 0.96 and 1.00.
We did not believe it.
The vendor split lies
Those test images come from the same source datasets as the training images — same photographers, same sessions, plausibly near-duplicate frames. So we scored the model on our own 119 blind-labelled photos instead: a different domain, different cameras, labels produced with no model in the loop.
| class | reported test mAP@50 | our external recall |
|---|---|---|
| foam | 1.000 | 3% |
| paper_card | 0.980 | 17% |
| film_flexible | 0.975 | 79% |
| rigid_container | 0.967 | 74% |
| cigarette_butt | 0.000 | 26% |
Foam reads as perfect on the split it was evaluated against, and 3% on real photographs. This is the second time we have caught this exact trap — an earlier evaluation against a public TACO re-split returned 0.636 that turned out to be memorisation, because most of its test images were in our training set.
Read the last row, though, because it is the one that settles the argument. Cigarette butts score a flat zero on the vendor split and 26% on our photographs — the best relative showing of any class we had never trained properly. The split is not biased optimistic. It is uninformative in both directions: it overstates foam by a factor of thirty and understates cigarette butts infinitely. A number that is wrong in only one direction can at least be discounted. This one cannot be corrected for at all.
The lesson is not that the platform is wrong. It is that a test split inherited from a public dataset tells you how well you memorised that dataset, and nothing about your deployment. If you have not built an external set, you do not have a number.
What actually improved
| material | images | before | after |
|---|---|---|---|
| film_flexible | 107 | 19% | 79% |
| rigid_container | 71 | 69% | 74% |
| glass | 4 | 0% | 25% |
| cigarette_butt | 15 | — | 26% |
| paper_card | 102 | 20% | 17% |
| foam | 53 | 0% | 3% |
| small_rigid | 16 | 55% | 12% |
Photos where the model saw nothing at all dropped from 30 to 17. Total detections went from 515 to 897.
Film is the single largest real-world category and it went from useless to genuinely useful. That is a real result on in-the-wild volunteer photography, where TACO-trained detectors typically sit around 19 mAP@50.
The finding worth keeping
Look at the two biggest rows again.
film_flexible got 4,379 training boxes and moved 19% → 79%.
paper_card got 4,438 training boxes — more — and moved 20% → 17%.
Near-identical volume. Opposite outcomes. So this is not a data-quantity story, and it is about as close to a controlled experiment as borrowed data ever gets.
The donor dataset’s “paper” is structured packaging: paper bags, paper cups, cardboard boxes, Tetra Paks. Our corpus’s paper is loose scraps, receipts and tissue — flat, crumpled, low-contrast, half-buried in leaf litter. Same word. Different object. Foam repeats it exactly: their Styrofoam is cups and containers, ours is crumbled EPS fragments on sand.
Matching class names across datasets does not match visual concepts, and name-matching cannot detect the difference. The borrowed data that worked was the data that happened to share the concept. The rest was inert, and small_rigid — 205 boxes — actively regressed.
That is the thing we would want to have known three weeks ago, and it is cheap to check: before importing a dataset because its class list matches, open twenty images and ask whether they show your object.
Where this leaves us
We have a material-taxonomy litter detector that works on volunteer phone photos for the largest category, and an external benchmark — 119 blind-labelled photos, 295 blind-graded boxes — that did not previously exist for this domain. Both are a starting point rather than a finish.
Still open, honestly:
- Paper and foam need in-domain annotation. No public dataset has our version of these objects. Borrowing is exhausted; the next boxes have to be ours.
small_rigidregressed and needs either real data or removal. Shipping six honest classes beats eight where two are noise.- Counting is still wrong. Reviewers found roughly 2,777 items across those 119 photos; the model finds a fraction. Presence is usable, quantity is not.
Some of the tooling came out of this too: a proposal for a public cross-image matching primitive in supervision, a cookbook for the before/after diff, and a Workflow block that does the set difference between two detection sets.