Evidence
Everything on this page is a measured result from a run that was specified before it was run. Where a test failed, the failure is here too.
The test
Public RFUAV dataset (Apache-2.0), 37 emitter classes recorded off-air by the dataset's authors. From it, a 14-class split: 10 drones the system learns, and 4 it never sees during training or calibration. Those 4 are the real test - the system has to refuse them rather than name them.
Training and test images are separated by recording order, not shuffled, so the system is tested on later recordings than it learned from. Shuffling would have let near-identical frames appear on both sides and inflated every number here.
Results
| Measure | Result | What it means |
|---|---|---|
| Known-drone accuracy (macro) | 84.21% | Averaged equally across the 10 known drones, so the best-represented ones cannot carry the score. |
| Known-drone accuracy (micro) | 79.78% | Pooled across all test images. |
| Unknown catch rate | 77.08% | Share of never-seen drones correctly refused instead of named. |
| Separability (AUROC) | 0.915 | Threshold-free. 0.5 is a coin flip; 1.0 is perfect. |
Refusing a drone the system was trained on counts as a miss in the accuracy figures above, not as a neutral abstention. For a detector, "I don't know" about a known drone is a failure.
The cheating check
A model can score well by memorising where a signal sits on the chart rather than what it looks like. To test that, every test signal was moved and re-scored:
- Shifted in time by 25%: accuracy fell 0.58 points. When a signal occurs should not matter, and it does not.
- Shifted in frequency by 2% (the scale of real tuning drift): accuracy fell 2.22 points. Small real-world drift is tolerated.
Both well inside the 10-point failure line set before the test was run. A large frequency shift does change the answer - correctly, because a signal moved to a different band is a different signal.
What it was compared against
An ordinary confidence-threshold detector, built and run by us on the identical split with identical rules - not a number quoted from someone else's paper on different data. An earlier internal method was beaten by that baseline and was retired rather than reported. Its failure is recorded in Limits.