Unsupervised optical flow from event cameras reaches supervised-method accuracy

Unsupervised Joint Learning of Optical Flow and Intensity with Event Cameras (arxiv 2503.17262) proposes a training approach that learns optical flow from event data without ground truth flow labels, using photometric consistency as the only supervisory signal. The result is a flow estimator that matches supervised baselines without requiring the expensive, difficult-to-collect labeled training data that supervised methods need.

Why optical flow matters for event cameras

Optical flow — the per-pixel velocity field of a scene — is a core intermediate representation for many event-camera applications. It connects events to the motion that generated them: if you know the flow field, you can predict which pixels will generate events next and can distinguish target motion from background motion.

For tracking, optical flow provides a motion prior: if the flow at a target’s current position points to the right at 10 pixels per frame, the tracker should look for the target to the right in the next time step. This prior is especially useful for event-camera tracking because the sparse event stream provides ambiguous direct position measurements between event bursts.

For ego-motion compensation — the problem of separating camera motion events from target motion events — optical flow provides the mechanism: camera rotation and translation generate predictable flow fields, and events consistent with the predicted flow are background; events inconsistent with it are targets.

The unsupervised training approach

Supervised optical flow methods require paired event streams and ground truth flow fields collected under known conditions — typically from synthetic data or from special setups where ground truth can be measured with expensive equipment.

The unsupervised approach uses photometric consistency: if the estimated flow field correctly captures the scene motion, then warping the reconstructed intensity image at time t by the flow field should produce the reconstructed intensity image at time t + dt. The training loss is the difference between the predicted and actual reconstructed images.

This requires simultaneously estimating optical flow and reconstructing intensity images from events — hence the “joint learning” in the title. The intensity reconstruction provides the photometric reference that supervises the flow estimation.

Accuracy on standard benchmarks

The paper evaluates on the MVSEC and DSEC benchmarks, which are the standard datasets for event-camera optical flow. Results show that the unsupervised method matches or slightly exceeds the best supervised methods on most test sequences.

The improvement on supervised baselines is largest for sequences with novel motion patterns not well-represented in the training data — precisely the case where supervised methods, trained on a specific distribution, are most likely to fail. The unsupervised method generalizes better because it uses physics-based consistency rather than dataset-specific statistics.

Implications for deployment

An event-camera optical flow estimator that does not require labeled training data can be trained on data collected during actual deployment conditions — in the specific lighting, altitude, and motion regime where the system will operate. This is a significant practical advantage over supervised methods, which require expensive data collection campaigns to match training to deployment.

For defense sensing applications where the operational environment differs from any available public dataset, unsupervised training on operationally-relevant data is a compelling approach.

---

Unsupervised Joint Learning of Optical Flow and Intensity with Event Cameras. arxiv 2503.17262