Frequency-aware event-guided deblurring at CVPR 2024 puts sharp frames back in reach

Motion blur is the technical problem that animates most of our engineering work. A frame camera integrating light over a millisecond shutter produces a streak from a target crossing at 30 meters per second. That streak doesn’t have a centroid you can feed to a guidance law. Event cameras avoid the problem entirely at the sensing stage — but there are scenarios where you have existing frame-camera infrastructure and want to extend its useful range rather than replace it.

A CVPR 2024 paper on frequency-aware event-guided video deblurring provides a method for recovering sharp frames from a frame camera using the event stream as a computational guide. The frequency-aware part is the key insight: motion blur is a low-pass filter in the spatial frequency domain. An image with motion blur has the high-frequency content — edges, fine structure — attenuated or destroyed. The event stream, because it responds to spatial contrast change, preferentially captures high-frequency content. Combining these observations, you can use the event stream to estimate what spatial frequencies the frame is missing and restore them.

How the method works

The paper formulates deblurring as a reconstruction problem in the frequency domain. The blurred frame provides low-frequency content reliably. The event stream, integrated over the same time window as the frame exposure, provides an estimate of the scene motion that caused the blur and implicitly encodes the high-frequency structure the blur destroyed. A network trained to invert this relationship can produce a deblurred frame by combining the two inputs.

The frequency-aware loss function is the innovation: rather than minimizing pixel-wise error between the deblurred output and a ground-truth sharp frame, the authors penalize errors in specific frequency bands according to their perceptual and computational significance. High-frequency edge information is weighted more heavily than low-frequency intensity gradients, which produces sharper edge reconstruction and better target localization from the deblurred output.

Benchmark results on the GoPro and HIDE datasets show state-of-the-art deblurring quality, and the method generalizes to the event camera simulation setup where real blur at high angular rates is the test condition.

The relevance to C-UAS

The application we’re thinking about is the hybrid seeker: a system that uses a conventional frame camera for classification (large format sensor, good signal-to-noise, high spatial resolution) and an event camera for guidance (fast, high dynamic range, blur-free). In this architecture, the frame camera needs sharp images for classification even though the event camera is handling the guidance loop.

At closing speeds above a few hundred meters per second, a frame camera with any reasonable shutter speed will produce blurred images of the target. If event-guided deblurring can recover sharp imagery in real time — fast enough to run within the classification loop — then the hybrid seeker can maintain classification capability through the full engagement, not just at opening ranges.

Real-time deblurring is a demanding compute requirement. The CVPR paper reports results on an offline dataset without reporting latency. A reasonable estimate for a network of the size they use would be 10–50 ms on a GPU-class processor, which is marginal for a classification loop with a 100 ms budget and probably too slow for a guidance loop with a 5 ms budget. Distillation to a smaller network, or moving the deblurring to an FPGA pipeline, would be required for airborne deployment.

The broader point

What this paper demonstrates is that the event camera and the frame camera together provide more information about a fast-moving scene than either provides alone. The event camera captures the temporal structure of the motion; the frame camera captures the spatial context of the scene. Deblurring is one use case for combining them. The same complementarity applies to other processing tasks — detection, tracking, classification — and the research community is actively exploring all of them. The sensor fusion architectures that emerge from this work will shape how multi-modal seekers are designed.