Fusing event cameras with infrared imaging at CVPR 2024 — what the combination unlocks

Cold targets against warm backgrounds. Warm targets against cold backgrounds. A drone with no thermal signature and a matte carbon-fiber body. Small UAS are difficult to detect precisely because they don’t cooperate with any single sensing modality. A small electric motor produces very little heat. A carbon airframe absorbs rather than reflects radar. A dark body against a bright sky defeats a conventional imager. Any single-sensor approach to this problem has a failure mode that an adversary can exploit by choosing the right time or geometry.

Sensor fusion is the obvious response, but fusion is harder than it sounds. If you fuse two sensors naively — average their outputs, concatenate their feature maps — you lose the structure that makes each one useful in the first place. The event camera’s advantage is temporal resolution and dynamic range. The infrared sensor’s advantage is scene-independent contrast at targets that emit differently from the background. A fusion architecture that discards those complementary properties in favor of a blended representation has gained little.

The approach

The CVPR 2024 work on event-visible-infrared fusion treats the two streams as providing different views of the same physical event and trains a network to reason about their relationship rather than simply combining them. The event stream captures edges in motion at high temporal resolution; the infrared channel captures thermal contrast that is stable over time. The fusion model learns to weight each channel according to the detection context — privileging the event stream when a target is fast and maneuvering, privileging infrared when a target is slow or stationary but thermally distinguishable.

The result is detection performance that degrades gracefully when one modality is degraded. A blinded infrared sensor (sunlit scene, no thermal contrast) still leaves the event channel operating. A fast maneuver that overwhelms the infrared’s frame rate is still handled by the event stream’s microsecond-scale updates.

The seeker integration problem

For seeker design, the fusion result raises a packaging question. An event camera and an infrared sensor co-boresighted on an airframe add size, weight, and interface complexity. Whether the detection improvement justifies that cost depends on the target class. For Group 1 UAS — small, cold, fast — the combination may be the only approach that achieves reliable detection across the full engagement envelope. For Group 3 UAS, the larger signature may give a single-modality approach sufficient margin.

The relevant figure from the paper is the detection rate on small cold targets at realistic engagement ranges. Event-only detection on a cold target with low solar contrast is the hard case, and the infrared channel provides measurable lift in exactly that scenario. The complement goes the other way at night: infrared holds contrast, event cameras require some scene texture to generate events.

Calibration and registration

One practical challenge the paper addresses is spatial calibration between the two sensors. Events encode pixel-level change; infrared frames encode pixel-level intensity. Registering these requires knowing the extrinsic transform between the two apertures to sub-pixel accuracy, and maintaining that calibration under airframe vibration. The authors use a calibration board with resistive heating elements to produce features visible in both modalities simultaneously — a method that transfers directly to seeker bring-up procedures.

What we take from it

The demonstration that event cameras and infrared sensors complement each other on precisely the target class that matters most for terminal guidance is the key takeaway. A seeker that integrates both modalities is more complex than one that uses either alone, but the performance envelope it covers — cold targets, fast maneuvers, mixed illumination conditions — maps well onto the actual engagement scenarios C-UAS interceptors face. The work at CVPR 2024 demonstrates that the fusion architecture to achieve this exists; the remaining work is productizing it into a flight-qualified seeker package.