Evidence
Everything on this page is a measured result from a run that was specified before it was run. Where a test failed, the failure is here too.
The test
Public RFUAV dataset (Apache-2.0), 37 emitter classes recorded off-air by the dataset's authors. From it, four different 14-class splits, each with 10 drones the system learns and 4 it never sees during training or calibration. Those 4 are the real test - the system has to refuse them rather than name them. All four draws were fixed before any of them was run.
Training and test images are separated by recording order, not shuffled, so the system is tested on later recordings than it learned from. Shuffling would have let near-identical frames appear on both sides and inflated every number here.
Results
Each of four different known/unknown splits was run through the same pipeline. Known-drone accuracy is fairly steady across them. Spotting never-seen drones is not: the seed 42 split, the first one run, turned out to be the best of the four at it.
| Measure | Mean of 4 splits (range) | Seed 42 | Seed 43 | Seed 44 | Seed 45 | What it means |
|---|---|---|---|---|---|---|
| Known-drone accuracy (macro) | 82.4% (75.6 to 86.8) | 84.21% | 86.82% | 83.10% | 75.62% | Averaged equally across the 10 known drones, so the best-represented ones cannot carry the score. |
| Known-drone accuracy (micro) | 77.7% (72.2 to 82.7) | 79.78% | 82.65% | 76.03% | 72.22% | Pooled across all test images. |
| Unknown catch rate | 48.1% (16.3 to 77.1) | 77.08% | 47.50% | 51.67% | 16.25% | Share of never-seen drones correctly refused instead of named. Depends heavily on which drones are held back. |
| Separability (AUROC) | 0.73 (0.52 to 0.91) | 0.915 | 0.784 | 0.722 | 0.516 | Threshold-free. 0.5 is a coin flip; 1.0 is perfect. Seed 45 is close to a coin flip. |
Seed 42 was the first split run and, until now, the only one shown here. Before the other three were run, the seed 42 run was repeated with the same code and reproduced its numbers exactly. Four splits is still few: the averages are themselves uncertain.
Refusing a drone the system was trained on counts as a miss in the accuracy figures above, not as a neutral abstention. For a detector, "I don't know" about a known drone is a failure.
The cheating check
A model can score well by memorising where a signal sits on the chart rather than what it looks like. To test that, every test signal was moved and re-scored:
- Shifted in time by 25%: accuracy fell 0.58 points. When a signal occurs should not matter, and it does not.
- Shifted in frequency by 2% (the scale of real tuning drift): accuracy fell 2.22 points. Small real-world drift is tolerated.
Both well inside the 10-point failure line set before the test was run. A large frequency shift does change the answer - correctly, because a signal moved to a different band is a different signal.
This check was run on the first split (seed 42) only. It has not been repeated on the other three.
What it was compared against
An ordinary confidence-threshold detector, built and run by us on the identical splits with identical rules - not a number quoted from someone else's paper on different data. On the first split, an earlier internal method was beaten by that baseline and was retired. Across all four splits the two turned out about equal at spotting unseen drones (the earlier method 49%, the baseline 48%, on average), each doing better on two of the four, and no single way of scoring unknown drones did best on all four. Neither is reliable: see Limits. The baseline stays the reported method because it was chosen before the other three splits were run.
Bandwidth, measured from real radio samples
A separate, early result - not part of the classifier above, and not yet feeding it.
Beyond the picture-based classifier, we tested whether a real physical property - how wide a signal is, in MHz - can be measured directly from raw radio samples rather than from rendered images. This is the first real step toward Angel Eyes' longer-term goal: reasoning about what physical transmitter produced a signal, not just naming it.
One second of real, raw signal was pulled directly from the dataset's own recordings (not the pre-rendered pictures used above), and its occupied bandwidth was measured with a method fixed in advance and checked two ways: against signals we built ourselves with mathematically known widths, and against two real drone recordings whose own files state their true bandwidth.
| Check | Result |
|---|---|
| Known-width signals (2, 5, 10 MHz) | Measured within 12% of the true value in every case |
| DJI MAVIC3 PRO (real recording, true width 10 MHz) | Measured 8.86 MHz |
| DJI MINI3 (real recording, true width 10 MHz) | Measured 8.15 MHz |
Validated working range: roughly 2-10 MHz. Above about 10-15 MHz the method under-reads - a signal built at 20 MHz measured as roughly half that. A correction was tried, and tested properly before being trusted: it fixed the wide-signal case on built-in-house signals, but broke one of the two real recordings that already worked, so it was rejected. The simpler, original method is what is reported here.
What this range does and does not mean. It says when the method is right: if a signal's true width is inside 2-10 MHz, the reading is accurate. It does not work in reverse. A reading of about 8-11 MHz does not, on its own, prove the true width is inside that range - in our tests a 20 MHz signal read about 10.7 MHz and a 40 MHz signal read about 7.9 MHz. A test of telling those cases apart did not pass its own pre-set bar; see Limits.