Egocentric event cameras advance gesture recognition in first-person applications

EgoEvGesture demonstrates gesture recognition from a head-mounted event camera that handles the high ego-motion and variable lighting typical of first-person sensing — a capability that transfers to any platform where the sensor shares rigid body dynamics with the observer.

EgoEvGesture: Gesture Recognition Based on Egocentric Event Camera (arxiv 2503.12419) addresses the specific challenge of event-camera sensing from a first-person perspective: the sensor moves continuously with the user’s head, generating persistent ego-motion events that must be separated from the gesture events of interest.

The egocentric sensing challenge

Most event-camera gesture recognition datasets are collected with fixed cameras observing gesturing subjects. In this configuration, ego-motion events are absent or minimal, and the gesture events are the dominant signal.

In egocentric sensing — wearable cameras, head-mounted displays, cockpit sensors — the camera itself moves constantly. Head movement generates events from every static surface in the field of view at a rate proportional to angular velocity. A user looking around the scene generates more total background events than the gesture they are making.

This is a direct parallel to the seeker problem: in a terminal guidance scenario, the seeker head is moving relative to the target, the target is moving relative to the scene, and separating the two motion sources is the core technical challenge.

The approach

EgoEvGesture separates ego-motion and gesture motion using a learned decomposition: a network trained on egocentric event streams learns to factorize the input into an ego-motion component (which is consistent with rigid-body camera motion) and a residual component (which represents object motion independent of camera motion).

The gesture recognition classifier then operates only on the residual component, which contains the gesture signal with reduced background contamination. The factorization is learned end-to-end, without requiring an explicit IMU or separate ego-motion measurement.

Performance in high-motion conditions

The key evaluation result is that EgoEvGesture significantly outperforms methods that do not separate ego-motion under high-motion conditions. When the user’s head is relatively stationary, both methods perform similarly. As head angular velocity increases above 30 deg/s, the baseline method degrades rapidly while EgoEvGesture maintains reasonable accuracy.

For a seeker application, this threshold is too conservative — seeker angular rates can reach hundreds of deg/s in the terminal phase — but the principle of learned ego-motion factorization is applicable at any rate, and the required training data would need to reflect the relevant dynamics.

Applicability to moving-sensor tracking

The EgoEvGesture approach is not specific to gesture recognition or the human-motion domain. The same learned decomposition can in principle be applied to any scenario where a sensor is moving and the target of interest is moving independently of the sensor.

The modification required for a seeker application would be to replace the gesture classifier with a target detector and tracker, and to train on event data collected from a moving platform observing a moving aerial target — a data collection requirement significantly more expensive than the head-mounted gesture collection in this paper, but technically tractable.

EgoEvGesture: Gesture Recognition Based on Egocentric Event Camera. arxiv 2503.12419