ECCV 2024 workshop: combining event and frame data for more robust drone detection

A workshop paper at ECCV 2024 demonstrates that fusing event-camera and RGB frame data for drone detection outperforms either modality alone — and analyzes when each modality dominates performance.

Neuromorphic Drone Detection: An Event-RGB Multimodal Approach (ECCV 2024 Workshops) presents a systematic study of combining event-camera and conventional RGB frame data for drone detection, including an analysis of which conditions favor each modality.

The case for multimodal fusion

A frame camera and an event camera see the same scene differently. The frame camera integrates light over a fixed exposure window and produces a dense image with rich color and texture information. The event camera responds instantaneously to brightness changes and produces a sparse representation of motion.

Against a static background with good illumination, the frame camera has the advantage: its dense imagery supports more accurate classification, and its color channels provide additional discriminative features. Against a dynamic background with rapid motion, the event camera has the advantage: it does not blur, and its motion-selective output rejects the background clutter that degrades frame-based detection.

The multimodal fusion approach attempts to capture the advantages of both, weighting each modality’s contribution by the conditions that favor it.

Fusion architecture

The paper evaluates several fusion architectures: early fusion (concatenate event and frame features at the input), late fusion (run separate detectors and merge their outputs), and a learned attention-based fusion that dynamically weights the two modalities based on input content.

The attention-based fusion consistently outperforms both early and late fusion, and both significantly outperform either single-modality approach. The attention mechanism learned to up-weight the event modality in conditions with fast target motion and dynamic backgrounds, and to up-weight the frame modality in conditions with slow motion and high-texture backgrounds.

When each modality wins

The condition analysis is the most practically useful part of the paper:

At low target velocities (under 5 m/s apparent motion): frame camera dominates, event camera adds marginal benefit
At moderate velocities (5–20 m/s): multimodal fusion provides the largest improvement over either alone
At high velocities (above 20 m/s): event camera dominates, frame camera blurs and becomes unreliable

For C-UAS applications, target velocities in the 20–100 m/s range — the high-velocity regime where the event camera dominates — are common in Group 2–3 UAS engagements. This argues for event-primary sensing with frame fusion for classification enhancement rather than frame-primary sensing with event assistance.

The sensor synchronization requirement

One practical challenge the paper highlights is synchronization: the event camera generates asynchronous data while the frame camera produces synchronous frames. Fusing them requires either forcing synchronization (which wastes events) or building an asynchronous fusion pipeline (which is architecturally complex).

The paper uses a fixed-length event window tied to the frame timestamp, which discards temporal precision but is simpler to implement. An asynchronous fusion architecture would likely perform better at high target velocities.

Neuromorphic Drone Detection: An Event-RGB Multimodal Approach. ECCV 2024 Workshops.