SwitchBench leaderboard
Recall of sheet-switch detectors on naturally occurring sheet switches, scored by the SwitchBench scorer kit.
In plain English
- What is a "sheet switch"?
- A scroll is a spiral of papyrus sheet. When we trace a surface through the CT scan, the trace can accidentally hop from one wind of the spiral onto the wind next to it, like a needle skipping a groove on a record. A detector's job is to notice where a traced surface (a "patch") does that, so the mistake can be fixed before anyone reads text off the wrong sheet.
- What does "recall" mean here?
- Of all the confirmed sheet switches we know about in the test set, recall is the fraction a detector actually flagged (hit) within 1.0 mm of the switch. Higher is better. The 95% interval (Wilson score) shows how much that number could plausibly move with more test data.
- What is "false alarms / 100 mm"?
- How often a detector cries wolf on parts of the surface we've confirmed have no switch, per 100 mm of that confirmed-clean surface. Lower is better.
- What is "coverage"?
- How many of the test patches a detector even produced a verdict for. Recall still counts an event on a patch a detector never looked at as a miss — that's unavoidable if recall is to mean "fraction of all known switches caught." Coverage is what tells you whether a low recall means the detector looked and missed, or mostly didn't look: a detector that only covers a handful of patches will show a low recall for that reason alone, not necessarily because it performs badly where it does run.
- How do I submit a detector?
- Run your detector, score its output with the scorer kit, then open a Detector submission issue with your alarms file and the exact command you ran — see CONTRIBUTING.md. We re-score everyone's alarms ourselves before publishing a row.
Leaderboard
| 95% CI | FA 95% CI | Tool @ commit |
|---|
Coverage matters: a detector with few patches with a verdict is limited in scope here, not wrong — it gets a Limited scope badge (fewer than half the patches got a verdict) and sorts after full-coverage detectors at the same recall, so a low false-alarm rate earned on a sliver of the corpus doesn't out-rank one earned on all of it. The false-alarm rate also gets its own 95% confidence interval, the same patch-block bootstrap used for recall, so its rate is judged with the same rigor. Every row is reproducible with the scorer kit on the same inputs.
Answer-key check (Sep 30, 2026): the answer key was built with automated rules plus a 12-event blind human spot-check (10 of 12 real), not an exhaustive human annotation. Recall only needs the labelled switches to be real, so it stands. The false-alarm columns were then checked on the human-verified spiral-fit patches, which the Vesuvius team treats as switch-free (pre-registered test P7): tifxyz-doctor raised 0.37 false alarms per 100 mm there [0.27, 0.48], consistent with the 0.43 in this table. A fully human-verified subset (SwitchBench v1.1) is next. In plain English: both the "caught" and the "cried wolf" numbers have now been checked against human-verified data.
Score your own detector
Pick an alarms.json file from your computer. Nothing is uploaded — scoring
runs entirely in your browser with the same corpus data this page already loaded.
Format: {"patch_id": [[x, y, z], ...], ...} (level-2 voxel coordinates,
one list of alarm points per patch; a patch you looked at but raised no alarm on should still
be a key, mapped to an empty list).
False alarms per 100 mm need the confirmed-clean patch geometry, which this page does not ship. Run the scorer kit for the full report, including false alarms:
python -m tools.switchbench_kit score --corpus results/switchbench_natural_v1_events.json \ --alarms alarms.json --patches-dir DIR