Limits
What Angel Eyes cannot yet do, and what has already failed. This page exists because a capability claim without its limits is not a claim, it is marketing.
Not yet established
- Four splits is still few. The classifier figures on this site are averages over four different known/unknown draws (seeds 42 to 45). Four draws is not many: the averages are themselves uncertain by several points, and the results vary a lot from one draw to the next (see below). Treat them as an early estimate, not a stable performance level.
- Public recordings, not live air. The dataset was recorded with transmitters at close range under good conditions. Real deployment adds distance, movement, interference, weather, and other emitters in the same bands. None of that is tested.
- The classifier currently reads pre-rendered spectrogram pictures from the dataset, not raw radio samples from a receiver. The bandwidth work described below is a separate, early step toward changing that, and does not yet feed the classifier.
What has already failed
Spotting drones it has never seen: not reliable yet. Angel Eyes was tested four times, each time with a different set of 4 drones held back and never shown to it. The share of those unseen drones it correctly refused ranged from 16% to 77% (48% on average). The separability score (AUROC) was below 0.80 on three of the four draws (seeds 43, 44 and 45: 0.78, 0.72 and 0.52), and the last of those is close to a coin flip (0.5). Which drones happen to be unseen changes the result a great deal, and none of the three ways of scoring "unknown" that we tried was best on every draw. The 77% this site showed earlier came from the best of the four draws. For now, treat Angel Eyes as a recogniser of drones it was trained on (about 82% on average), not as a dependable detector of drones it has never seen.
Training the image model on the radio pictures themselves did not pass our test. On four held-back splits it made unseen drones much easier to spot on two and slightly harder on the other two, and on one split it cost 11 points of known-drone accuracy. It is not used.
Measuring physical traits from pictures did not work. The first attempt at
the "what made this signal" idea tried to measure real bandwidth and hopping
behaviour from the dataset's rendered spectrogram images. Three detection methods
were tried against a bar fixed in advance - measured video bandwidth within 20% of
the dataset paper's own published figures - and all three failed, one of them
reporting signals filling the entire 100 MHz band for drones whose real
bandwidth is under 6 MHz. The cause was traced to the source data: recovering a
physical measurement from a rendered picture of a chart means recovering numbers
that were discarded when the picture was drawn. That route is closed for this
dataset's images.
Telling narrow signals from wide ones: tested, did not pass. We tested whether a
second way of measuring width could tell a genuinely narrow signal from a wide one that
only reads narrow. The pass marks were fixed in writing before the test. It used 18
signals built by us and 2 real drone recordings of known width. It did not pass. In one
case it gave an answer that rested on a reading outside the range we have tested, which
our own rules forbid. As we predicted in advance, it was also fooled by a signal made of
evenly spaced single tones. On the built-in-house signals it answered 14 of 15 correctly
and got both real recordings right. But a much simpler rule did just as well, so nothing
here is claimed. The same test also showed why the caveat below matters: taking the
width reading at face value called 5 of 7 wide test signals narrow.
What has partially worked: bandwidth from raw signal
Moving from pictures to the dataset's actual raw radio recordings, the same kind of measurement was tried again - and this time, with real limits, it works.
- Validated from roughly 2 to 10 MHz. Checked against three signals of mathematically known width and two real drone recordings whose own files state their true bandwidth (both 10 MHz, both DJI models - measured 8.86 and 8.15 MHz).
- Under-reads above roughly 10-15 MHz. A signal built at 20 MHz measured as roughly half that width using the validated method.
- Accuracy depends on the signal's shape. On test signals whose edges fade slowly, the reading came out 22-35% below the true width. On signals with sharp edges it was within 3%.
- A reading inside the range does not prove the true width is inside it. The validated range is right in one direction only: when the true width is 2-10 MHz, the reading is accurate. In tests, a 20 MHz signal read about 10.7 MHz and a 40 MHz signal read about 7.9 MHz - readings that look like a validated narrow signal. A test of whether a second measurement can tell the two cases apart has now been run, and it did not pass (see above).
- A fix was tried and rejected, on the record. A change to how the measurement finds a signal's edges corrected the wide-signal case on built-in-house test signals, but broke one of the two real recordings that had been working correctly. Because it failed on real data, it was not adopted, even though it looked like a clean improvement on synthetic signals alone. The simpler, original method - with its known 2-10 MHz range - is what stands.
- Only tested at one real value. Both real drones checked happen to share the same true bandwidth, 10 MHz. The two drones in the dataset with a different labelled bandwidth (20 MHz) do not carry a readable answer inside their own files, so they could not be used to check whether the method generalises to a different true value using real data - only using signals built by us.
- Hop timing was tested and is not validated on real recordings. The same measurement records how a signal's frequency moves over time. Every drone in the dataset hops, so a bare "does it hop" test would prove nothing; the test measured how long a signal stays on one frequency and included signals that never hop, so it could fail. It passed on clean built-in-house signals. Run on two real DJI recordings it missed the dataset paper's figures (both read 0.37 ms, against 1.7 ms and 0.56 ms). The cause was found: a wide video signal that never hops gets counted as hopping. One fix was allowed. It passed a mixed control-plus-video test but still reported 251 false hops on a wide signal that never hops (3,489 before the fix), so it failed its bar. It was not re-run on real recordings and the work has stopped. Hop timing is not claimed.
- Not yet connected to anything. This measurement does not feed the classifier above, and no reasoning layer sits on top of it yet.
Honesty rules this project runs under
- Pass marks and failure lines are fixed in writing before a test runs, never after seeing the result.
- When a method is corrected, the old number stays on the record beside the new one.
- A measuring instrument that misfires is reported as a broken instrument, not as evidence against the thing being measured.
- A fix is only adopted if it improves the new case without breaking a real result that already worked - checked, not assumed.
- Comparisons are re-run by us on identical data, never quoted from a different dataset.