Event cameras guide low-light video enhancement at ECCV 2024 — night operations get more capable

Night operations are a recurring constraint for optical terminal guidance. A conventional imager needs light. It can be supplemented with IR illuminators, but those are active signatures. It can be replaced with a thermal imager, but cold targets remain difficult. Low-light video enhancement is a software approach to stretching the envelope of conventional cameras — and event cameras turn out to be a useful guide signal for doing it well.

An ECCV 2024 paper makes the case for event-guided low-light enhancement. The core insight is that event cameras, because of their per-pixel logarithmic response, maintain useful information at light levels where frame cameras saturate or fall into noise. An event camera pointed at the same scene as a frame camera in low light continues to generate events at scene edges and moving objects, even when the frame camera output is dominated by shot noise.

The guidance mechanism

The paper trains a network to enhance noisy low-light frames using the event stream as a side channel. The events encode the high-contrast structure of the scene — edges, motion boundaries — that the noisy frame has degraded but not eliminated. The network uses the event data as a structural prior to regularize the denoising process: where events indicate a strong edge, the denoised output should also have a strong edge, regardless of what the noisy frame pixel values say.

This is different from image fusion, where you’re combining two independent observations of the same signal. Here the event stream and the frame are measuring different things — temporal contrast change versus spatial intensity — and the network is using one as structural guidance for processing the other.

The results on nighttime video benchmarks are meaningful: the event-guided approach outperforms frame-only denoising methods, especially at motion boundaries where frame-based methods tend to produce ghosting artifacts. The event stream preserves the temporal structure of moving objects even when the frame camera is struggling.

Night terminal guidance implications

The relevance for terminal guidance depends on the engagement scenario. If the terminal phase occurs at night — which is a realistic and increasingly common scenario for C-UAS operations — an optical seeker faces a degraded environment. A thermal seeker avoids the problem if the target has adequate thermal signature, but small commercial drones have very low heat output and may not be reliably detectable at engagement range.

Event-guided enhancement of a low-light visible channel is one approach to extending the optical seeker’s night envelope without requiring a dedicated thermal imager. The event camera’s low-light performance isn’t magical — in true darkness with no scene texture there are no events — but in the ambient light conditions of an inhabited area at night (streetlights, building lights, moonlight), an event camera continues to produce useful data when a frame camera is marginal.

The compute cost of running an event-guided enhancement network in real time is a constraint. The ECCV paper targets offline enhancement for video quality rather than real-time guidance. Bringing the latency into the guidance budget — sub-millisecond from image to centroid estimate — would require significant network compression and likely FPGA acceleration.

The dual-modality seeker concept

What this paper reinforces is the value of a dual-modality seeker that includes both an event camera and a low-light frame camera sharing an optical axis. The event camera provides the high-temporal-resolution tracking channel that drives guidance; the frame camera provides context for target classification and fuze triggering. In low-light conditions the event channel remains useful longer than the frame channel, and the frame channel can be enhanced using the event data as guidance.

This is a more integrated concept than simple sensor fusion — the two sensors are providing different types of information to different stages of the processing pipeline rather than two redundant views of the same target. That integration is worth designing in from the seeker architecture stage rather than bolting together two independent sensor channels.