ANGEL EYESVyroNyx Private Limited

Limits

What Angel Eyes cannot yet do, and what has already failed. This page exists because a capability claim without its limits is not a claim, it is marketing.

Not yet established

What has already failed

Spotting drones it has never seen: not reliable yet. Angel Eyes was tested four times, each time with a different set of 4 drones held back and never shown to it. The share of those unseen drones it correctly refused ranged from 16% to 77% (48% on average). The separability score (AUROC) was below 0.80 on three of the four draws (seeds 43, 44 and 45: 0.78, 0.72 and 0.52), and the last of those is close to a coin flip (0.5). Which drones happen to be unseen changes the result a great deal, and none of the three ways of scoring "unknown" that we tried was best on every draw. The 77% this site showed earlier came from the best of the four draws. For now, treat Angel Eyes as a recogniser of drones it was trained on (about 82% on average), not as a dependable detector of drones it has never seen.
Training the image model on the radio pictures themselves did not pass our test. On four held-back splits it made unseen drones much easier to spot on two and slightly harder on the other two, and on one split it cost 11 points of known-drone accuracy. It is not used.
Measuring physical traits from pictures did not work. The first attempt at the "what made this signal" idea tried to measure real bandwidth and hopping behaviour from the dataset's rendered spectrogram images. Three detection methods were tried against a bar fixed in advance - measured video bandwidth within 20% of the dataset paper's own published figures - and all three failed, one of them reporting signals filling the entire 100 MHz band for drones whose real bandwidth is under 6 MHz. The cause was traced to the source data: recovering a physical measurement from a rendered picture of a chart means recovering numbers that were discarded when the picture was drawn. That route is closed for this dataset's images.
Telling narrow signals from wide ones: tested, did not pass. We tested whether a second way of measuring width could tell a genuinely narrow signal from a wide one that only reads narrow. The pass marks were fixed in writing before the test. It used 18 signals built by us and 2 real drone recordings of known width. It did not pass. In one case it gave an answer that rested on a reading outside the range we have tested, which our own rules forbid. As we predicted in advance, it was also fooled by a signal made of evenly spaced single tones. On the built-in-house signals it answered 14 of 15 correctly and got both real recordings right. But a much simpler rule did just as well, so nothing here is claimed. The same test also showed why the caveat below matters: taking the width reading at face value called 5 of 7 wide test signals narrow.

What has partially worked: bandwidth from raw signal

Moving from pictures to the dataset's actual raw radio recordings, the same kind of measurement was tried again - and this time, with real limits, it works.

Honesty rules this project runs under